• Steve Wise's avatar
    RDMA/cxgb4: Avoid false GTS CIDX_INC overflows · 1973e8b8
    Steve Wise authored
    The T4 IQ hw design assumes CIDX_INC credits will be returned on a
    regular basis and always before the CIDX counter crosses over the PIDX
    counter.  For RDMA CQs, however, returning CIDX_INC credits is only
    needed and desired when and if the CQ is armed for notification.  This
    can lead to a GTS write returning credits that causes the HW to reject
    the credit update because it causes CIDX to pass PIDX.  Once this
    happens, the CIDX/PIDX counters get out of whack and an application
    can miss a notification and get stuck blocked awaiting a notification.
    
    To avoid this, we allocate the HW IQ 2x times the requested size.
    This seems to avoid the false overflow failures.  If we see more
    issues with this, then we'll have to add code in the poll path to
    return credits periodically like when the amount reaches 1/2 the queue
    depth).  I would like to avoid this as it adds a PCI write transaction
    for applications that never arm the CQ (like most MPIs).
    Signed-off-by: default avatarSteve Wise <swise@opengridcomputing.com>
    Signed-off-by: default avatarRoland Dreier <rolandd@cisco.com>
    1973e8b8
cq.c 22.1 KB