• Chuck Lever's avatar
    xprtrdma: Cap req_cqinit · e7104a2a
    Chuck Lever authored
    Recent work made FRMR registration and invalidation completions
    unsignaled. This greatly reduces the adapter interrupt rate.
    
    Every so often, however, a posted send Work Request is allowed to
    signal. Otherwise, the provider's Work Queue will wrap and the
    workload will hang.
    
    The number of Work Requests that are allowed to remain unsignaled is
    determined by the value of req_cqinit. Currently, this is set to the
    size of the send Work Queue divided by two, minus 1.
    
    For FRMR, the send Work Queue is the maximum number of concurrent
    RPCs (currently 32) times the maximum number of Work Requests an
    RPC might use (currently 7, though some adapters may need more).
    
    For mlx4, this is 224 entries. This leaves completion signaling
    disabled for 111 send Work Requests.
    
    Some providers hold back dispatching Work Requests until a CQE is
    generated.  If completions are disabled, then no CQEs are generated
    for quite some time, and that can stall the Work Queue.
    
    I've seen this occur running xfstests generic/113 over NFSv4, where
    eventually, posting a FAST_REG_MR Work Request fails with -ENOMEM
    because the Work Queue has overflowed. The connection is dropped
    and re-established.
    
    Cap the rep_cqinit setting so completions are not left turned off
    for too long.
    
    BugLink: https://bugzilla.linux-nfs.org/show_bug.cgi?id=269Signed-off-by: default avatarChuck Lever <chuck.lever@oracle.com>
    Signed-off-by: default avatarAnna Schumaker <Anna.Schumaker@Netapp.com>
    e7104a2a
xprt_rdma.h 14.3 KB