UbiquitousOS
/
OpenCloudOS-Kernel
mirror of https://gitee.com/OpenCloudOS/OpenCloudOS-Kernel.git
11
0
Fork 0
Code Issues Projects Releases Wiki Activity

xprtrdma: Refresh the documenting comment in frwr_ops.c 

Things have changed since this comment was written. In particular,
the reworking of connection closing, on-demand creation of MRs, and
the removal of fr_state all mean that deferring MR recovery to
frwr_map is no longer needed. The description is obsolete.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
This commit is contained in:
Chuck Lever 2019-08-19 18:37:52 -04:00 committed by Anna Schumaker
parent 95bd8304b3
commit 2fb2a4d529
1 changed files with 17 additions and 47 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

64
net/sunrpc/xprtrdma/frwr_ops.c
View File

@ -7,67 +7,37 @@
/* Lightweight memory registration using Fast Registration Work /* Lightweight memory registration using Fast Registration Work
* Requests (FRWR). * Requests (FRWR).
* *
* FRWR features ordered asynchronous registration and deregistration * FRWR features ordered asynchronous registration and invalidation
* of arbitrarily sized memory regions. This is the fastest and safest * of arbitrarily-sized memory regions. This is the fastest and safest
* but most complex memory registration mode. * but most complex memory registration mode.
*/ */
/* Normal operation /* Normal operation
* *
* A Memory Region is prepared for RDMA READ or WRITE using a FAST_REG * A Memory Region is prepared for RDMA Read or Write using a FAST_REG
* Work Request (frwr_map). When the RDMA operation is finished, this * Work Request (frwr_map). When the RDMA operation is finished, this
* Memory Region is invalidated using a LOCAL_INV Work Request * Memory Region is invalidated using a LOCAL_INV Work Request
* (frwr_unmap_sync). * (frwr_unmap_async and frwr_unmap_sync).
* *
* Typically these Work Requests are not signaled, and neither are RDMA * Typically FAST_REG Work Requests are not signaled, and neither are
* SEND Work Requests (with the exception of signaling occasionally to * RDMA Send Work Requests (with the exception of signaling occasionally
* prevent provider work queue overflows). This greatly reduces HCA * to prevent provider work queue overflows). This greatly reduces HCA
* interrupt workload. * interrupt workload.
*
* As an optimization, frwr_unmap marks MRs INVALID before the
* LOCAL_INV WR is posted. If posting succeeds, the MR is placed on
* rb_mrs immediately so that no work (like managing a linked list
* under a spinlock) is needed in the completion upcall.
*
* But this means that frwr_map() can occasionally encounter an MR
* that is INVALID but the LOCAL_INV WR has not completed. Work Queue
* ordering prevents a subsequent FAST_REG WR from executing against
* that MR while it is still being invalidated.
*/ */
/* Transport recovery /* Transport recovery
* *
* ->op_map and the transport connect worker cannot run at the same * frwr_map and frwr_unmap_* cannot run at the same time the transport
* time, but ->op_unmap can fire while the transport connect worker * connect worker is running. The connect worker holds the transport
* is running. Thus MR recovery is handled in ->op_map, to guarantee * send lock, just as ->send_request does. This prevents frwr_map and
* that recovered MRs are owned by a sending RPC, and not one where * the connect worker from running concurrently. When a connection is
* ->op_unmap could fire at the same time transport reconnect is * closed, the Receive completion queue is drained before the allowing
* being done. * the connect worker to get control. This prevents frwr_unmap and the
* connect worker from running concurrently.
* *
* When the underlying transport disconnects, MRs are left in one of * When the underlying transport disconnects, MRs that are in flight
* four states: * are flushed and are likely unusable. Thus all flushed MRs are
* * destroyed. New MRs are created on demand.
* INVALID: The MR was not in use before the QP entered ERROR state.
*
* VALID: The MR was registered before the QP entered ERROR state.
*
* FLUSHED_FR: The MR was being registered when the QP entered ERROR
* state, and the pending WR was flushed.
*
* FLUSHED_LI: The MR was being invalidated when the QP entered ERROR
* state, and the pending WR was flushed.
*
* When frwr_map encounters FLUSHED and VALID MRs, they are recovered
* with ib_dereg_mr and then are re-initialized. Because MR recovery
* allocates fresh resources, it is deferred to a workqueue, and the
* recovered MRs are placed back on the rb_mrs list when recovery is
* complete. frwr_map allocates another MR for the current RPC while
* the broken MR is reset.
*
* To ensure that frwr_map doesn't encounter an MR that is marked
* INVALID but that is about to be flushed due to a previous transport
* disconnect, the transport connect worker attempts to drain all
* pending send queue WRs before the transport is reconnected.
*/ */
#include <linux/sunrpc/rpc_rdma.h> #include <linux/sunrpc/rpc_rdma.h>