• Pradeep Satyanarayana's avatar
    IPoIB/cm: Add connected mode support for devices without SRQs · 68e995a2
    Pradeep Satyanarayana authored
    Some IB adapters (notably IBM's eHCA) do not implement SRQs (shared
    receive queues).  The current IPoIB connected mode support only works
    on devices that support SRQs.
    
    Fix this by adding support for using the receive queue of each
    connected mode receive QP.  The disadvantage of this compared to using
    an SRQ is that it means a full queue of receives must be posted for
    each remote connected mode peer, which means that total memory usage
    is potentially much higher than when using SRQs.  To manage this, add
    a new module parameter "max_nonsrq_conn_qp" that limits the number of
    connections allowed per interface.
    
    The rest of the changes are fairly straightforward: we use a table of
    struct ipoib_cm_rx to hold all the active connections, and put the
    table index of the connection in the high bits of receive WR IDs.
    This is needed because we cannot rely on the struct ib_wc.qp field for
    non-SRQ receive completions.  Most of the rest of the changes just
    test whether or not an SRQ is available, and post receives or find
    received packets in the right place depending on the answer.
    
    Cleaning up dead connections actually becomes simpler, because we do
    not have to do the "last WQE reached" dance that is required to
    destroy QPs attached to an SRQ.  We just move the QP to the error
    state and wait for all pending receives to be flushed.
    Signed-off-by: default avatarPradeep Satyanarayana <pradeeps@linux.vnet.ibm.com>
    
    [ Completely rewritten and split up, based on Pradeep's work.  Several
      bugs fixed and no doubt several bugs introduced.  - Roland ]
    Signed-off-by: default avatarRoland Dreier <rolandd@cisco.com>
    68e995a2
ipoib_main.c 32 KB