• Zach Brown's avatar
    RDS/IB: add refcount tracking to struct rds_ib_device · 3e0249f9
    Zach Brown authored
    The RDS IB client .remove callback used to free the rds_ibdev for the given
    device unconditionally.  This could race other users of the struct.  This patch
    adds refcounting so that we only free the rds_ibdev once all of its users are
    done.
    
    Many rds_ibdev users are tied to connections.  We give the connection a
    reference and change these users to reference the device in the connection
    instead of looking it up in the IB client data.  The only user of the IB client
    data remaining is the first lookup of the device as connections are built up.
    
    Incrementing the reference count of a device found in the IB client data could
    race with final freeing so we use an RCU grace period to make sure that freeing
    won't happen until those lookups are done.
    
    MRs need the rds_ibdev to get at the pool that they're freed in to.  They exist
    outside a connection and many MRs can reference different devices from one
    socket, so it was natural to have each MR hold a reference.  MR refs can be
    dropped from interrupt handlers and final device teardown can block so we push
    it off to a work struct.  Pool teardown had to be fixed to cancel its pending
    work instead of deadlocking waiting for all queued work, including itself, to
    finish.
    
    MRs get their reference from the global device list, which gets a reference.
    It is left unprotected by locks and remains racy.  A simple global lock would
    be a significant bottleneck.  More scalable (complicated) locking should be
    done carefully in a later patch.
    Signed-off-by: default avatarZach Brown <zach.brown@oracle.com>
    3e0249f9
ib_rdma.c 16.6 KB