OpenCloudOS-Kernel/drivers/block/rnbd
Guoqing Jiang 5e6e08087a block/rnbd: replace REQ_OP_FLUSH with REQ_OP_WRITE
Since flush bios are implemented as writes with no data and
the preflush flag per Christoph's comment [1].

And we need to change it in rnbd accordingly. Otherwise, I
got splatting when create fs from rnbd client.

[  464.028545] ------------[ cut here ]------------
[  464.028553] WARNING: CPU: 0 PID: 65 at block/blk-core.c:751 submit_bio_noacct+0x32c/0x5d0
[ ... ]
[  464.028668] CPU: 0 PID: 65 Comm: kworker/0:1H Tainted: G           OE      6.4.0-rc1 #9
[  464.028671] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.15.0-0-g2dd4b9b-rebuilt.opensuse.org 04/01/2014
[  464.028673] Workqueue: ib-comp-wq ib_cq_poll_work [ib_core]
[  464.028717] RIP: 0010:submit_bio_noacct+0x32c/0x5d0
[  464.028720] Code: 03 0f 85 51 fe ff ff 48 8b 43 18 8b 88 04 03 00 00 85 c9 0f 85 3f fe ff ff e9 be fd ff ff 0f b6 d0 3c 0d 74 26 83 fa 01 74 21 <0f> 0b b8 0a 00 00 00 e9 56 fd ff ff 4c 89 e7 e8 70 a1 03 00 84 c0
[  464.028722] RSP: 0018:ffffaf3680b57c68 EFLAGS: 00010202
[  464.028724] RAX: 0000000000060802 RBX: ffffa09dcc18bf00 RCX: 0000000000000000
[  464.028726] RDX: 0000000000000002 RSI: 0000000000000000 RDI: ffffa09dde081d00
[  464.028727] RBP: ffffaf3680b57c98 R08: ffffa09dde081d00 R09: ffffa09e38327200
[  464.028729] R10: 0000000000000000 R11: 0000000000000000 R12: ffffa09dde081d00
[  464.028730] R13: ffffa09dcb06e1e8 R14: 0000000000000000 R15: 0000000000200000
[  464.028733] FS:  0000000000000000(0000) GS:ffffa09e3bc00000(0000) knlGS:0000000000000000
[  464.028735] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  464.028736] CR2: 000055a4e8206c40 CR3: 0000000119f06000 CR4: 00000000003506f0
[  464.028738] Call Trace:
[  464.028740]  <TASK>
[  464.028746]  submit_bio+0x1b/0x80
[  464.028748]  rnbd_srv_rdma_ev+0x50d/0x10c0 [rnbd_server]
[  464.028754]  ? percpu_ref_get_many.constprop.0+0x55/0x140 [rtrs_server]
[  464.028760]  ? __this_cpu_preempt_check+0x13/0x20
[  464.028769]  process_io_req+0x1dc/0x450 [rtrs_server]
[  464.028775]  rtrs_srv_inv_rkey_done+0x67/0xb0 [rtrs_server]
[  464.028780]  __ib_process_cq+0xbc/0x1f0 [ib_core]
[  464.028793]  ib_cq_poll_work+0x2b/0xa0 [ib_core]
[  464.028804]  process_one_work+0x2a9/0x580

[1]. https://lore.kernel.org/all/ZFHgefWofVt24tRl@infradead.org/

Signed-off-by: Guoqing Jiang <guoqing.jiang@linux.dev>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Chaitanya Kulkarni <kch@nvidia.com>
Link: https://lore.kernel.org/r/20230512034631.28686-1-guoqing.jiang@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2023-05-12 08:56:42 -06:00
..
Kconfig block/rnbd: Select SG_POOL for RNBD_CLIENT 2021-01-08 08:19:18 -07:00
Makefile rnbd-srv: remove rnbd_dev_{open,close} 2022-09-21 08:35:23 -06:00
README block/rnbd: Adding name to the Contributors List 2021-01-08 08:19:18 -07:00
rnbd-clt-sysfs.c driver core: class: remove module * from class_create() 2023-03-17 15:16:33 +01:00
rnbd-clt.c block/rnbd-clt: fix wrong max ID in ida_alloc_max 2023-01-17 08:33:36 -07:00
rnbd-clt.h rnbd-clt: pass sector_t type for resize capacity 2022-08-02 17:14:45 -06:00
rnbd-common.c
rnbd-log.h
rnbd-proto.h block/rnbd: replace REQ_OP_FLUSH with REQ_OP_WRITE 2023-05-12 08:56:42 -06:00
rnbd-srv-sysfs.c driver core: class: remove module * from class_create() 2023-03-17 15:16:33 +01:00
rnbd-srv-trace.c block/rnbd-srv: Add event tracing support 2022-08-22 07:52:51 -06:00
rnbd-srv-trace.h block/rnbd-srv: Add event tracing support 2022-08-22 07:52:51 -06:00
rnbd-srv.c v6.1 merge window pull request 2022-10-07 12:05:29 -07:00
rnbd-srv.h rnbd-srv: remove struct rnbd_dev 2022-09-21 08:35:23 -06:00

README

********************************
RDMA Network Block Device (RNBD)
********************************

Introduction
------------

RNBD (RDMA Network Block Device) is a pair of kernel modules
(client and server) that allow for remote access of a block device on
the server over RTRS protocol using the RDMA (InfiniBand, RoCE, iWARP)
transport. After being mapped, the remote block devices can be accessed
on the client side as local block devices.

I/O is transferred between client and server by the RTRS transport
modules. The administration of RNBD and RTRS modules is done via
sysfs entries.

Requirements
------------

  RTRS kernel modules

Quick Start
-----------

Server side:
  # modprobe rnbd_server

Client side:
  # modprobe rnbd_client
  # echo "sessname=blya path=ip:10.50.100.66 device_path=/dev/ram0" > \
            /sys/devices/virtual/rnbd-client/ctl/map_device

  Where "sessname=" is a session name, a string to identify the session
  on client and on server sides; "path=" is a destination IP address or
  a pair of a source and a destination IPs, separated by comma.  Multiple
  "path=" options can be specified in order to use multipath  (see RTRS
  description for details); "device_path=" is the block device to be
  mapped from the server side. After the session to the server machine is
  established, the mapped device will appear on the client side under
  /dev/rnbd<N>.


RNBD-Server Module Parameters
=============================

dev_search_path
---------------

When a device is mapped from the client, the server generates the path
to the block device on the server side by concatenating dev_search_path
and the "device_path" that was specified in the map_device operation.

The default dev_search_path is: "/".

dev_search_path option can also contain %SESSNAME% in order to provide
different device namespaces for different sessions.  See "device_path"
option for details.

============================
Protocol (rnbd/rnbd-proto.h)
============================

1. Before mapping first device from a given server, client sends an
RNBD_MSG_SESS_INFO to the server. Server responds with
RNBD_MSG_SESS_INFO_RSP. Currently the messages only contain the protocol
version for backward compatibility.

2. Client requests to open a device by sending RNBD_MSG_OPEN message. This
contains the path to the device and access mode (read-only or writable).
Server responds to the message with RNBD_MSG_OPEN_RSP. This contains
a 32 bit device id to be used for  IOs and device "geometry" related
information: side, max_hw_sectors, etc.

3. Client attaches RNBD_MSG_IO to each IO message send to a device. This
message contains device id, provided by server in his rnbd_msg_open_rsp,
sector to be accessed, read-write flags and bi_size.

4. Client closes a device by sending RNBD_MSG_CLOSE which contains only the
device id provided by the server.

=========================================
Contributors List(in alphabetical order)
=========================================
Danil Kipnis <danil.kipnis@profitbricks.com>
Fabian Holler <mail@fholler.de>
Guoqing Jiang <guoqing.jiang@cloud.ionos.com>
Jack Wang <jinpu.wang@profitbricks.com>
Kleber Souza <kleber.souza@profitbricks.com>
Lutz Pogrell <lutz.pogrell@cloud.ionos.com>
Milind Dumbare <Milind.dumbare@gmail.com>
Roman Penyaev <roman.penyaev@profitbricks.com>
Swapnil Ingle <ingleswapnil@gmail.com>