• Adit Ranadive's avatar
    IB: Add vmw_pvrdma driver · 29c8d9eb
    Adit Ranadive authored
    This patch series adds a driver for a paravirtual RDMA device. The
    device is developed for VMware's Virtual Machines and allows existing RDMA
    applications to continue to use existing Verbs API when deployed in VMs
    on ESXi. We recently did a presentation in the OFA Workshop [1] regarding
    this device.
    
    Description and RDMA Support
    ============================
    The virtual device is exposed as a dual function PCIe device. One part
    is a virtual network device (VMXNet3) which provides networking properties
    like MAC, IP addresses to the RDMA part of the device. The networking
    properties are used to register GIDs required by RDMA applications to
    communicate.
    
    These patches add support and the all required infrastructure for
    letting applications use such a device. We support the mandatory Verbs API as
    well as the base memory management extensions (Local Inv, Send with Inv and
    Fast Register Work Requests). We currently support both Reliable Connected
    and Unreliable Datagram QPs but do not support Shared Receive Queues
    (SRQs).
    
    Also, we support the following types of Work Requests:
     o Send/Receive (with or without Immediate Data)
     o RDMA Write (with or without Immediate Data)
     o RDMA Read
     o Local Invalidate
     o Send with Invalidate
     o Fast Register Work Requests
    
    This version only adds support for version 1 of RoCE. We will add RoCEv2
    support in a future patch. We do support registration of both MAC-based
    and IP-based GIDs. I have also created a git tree for our user-level driver
    [2].
    
    Testing
    =======
    We have tested this internally for various types of Guest OS - Red Hat,
    Centos, Ubuntu 12.04/14.04/16.04, Oracle Enterprise Linux, SLES 12
    using backported versions of this driver. The tests included several
    runs of the performance tests (included with OFED), Intel MPI PingPong
    benchmark on OpenMPI, krping for FRWRs. Mellanox has been kind enough
    to test the backported version of the driver internally on their hardware
    using a VMware provided ESX build. I have also applied and tested this
    with Doug's k.o/for-4.9 branch (commit 5603910b). Note, that this patch
    series should be applied all together. I split out the commits so that
    it may be easier to review.
    
    PVRDMA Resources
    ================
    [1] OFA Workshop Presentation -
    https://openfabrics.org/images/eventpresos/2016presentations/102parardma.pdf
    
    [2] Libpvrdma User-level library -
    http://git.openfabrics.org/?p=~aditr/libpvrdma.git;a=summaryReviewed-by: default avatarJorgen Hansen <jhansen@vmware.com>
    Reviewed-by: default avatarGeorge Zhang <georgezhang@vmware.com>
    Reviewed-by: default avatarAditya Sarwade <asarwade@vmware.com>
    Reviewed-by: default avatarBryan Tan <bryantan@vmware.com>
    Reviewed-by: default avatarLeon Romanovsky <leonro@mellanox.com>
    Signed-off-by: default avatarAdit Ranadive <aditr@vmware.com>
    Signed-off-by: default avatarDoug Ledford <dledford@redhat.com>
    29c8d9eb
pvrdma_verbs.c 16 KB