• Max Gurtovoy's avatar
    RDMA/mlx5: Use PA mapping for PI handover · 2563e2f3
    Max Gurtovoy authored
    If possibe, avoid doing a UMR operation to register data and protection
    buffers (via MTT/KLM mkeys). Instead, use the local DMA key and map the
    SG lists using PA access. This is safe, since the internal key for data
    and protection never exposed to the remote server (only signature key
    might be exposed). If PA mappings are not possible, perform mapping
    using MTT/KLM descriptors.
    
    The setup of the tested benchmark (using iSER ULP):
     - 2 servers with 24 cores (1 initiator and 1 target)
     - ConnectX-4/ConnectX-5 adapters
     - 24 target sessions with 1 LUN each
     - ramdisk backstore
     - PI active
    
    Performance results running fio (24 jobs, 128 iodepth) using
    write_generate=1 and read_verify=1 (w/w.o patch):
    
    bs      IOPS(read)        IOPS(write)
    ----    ----------        ----------
    512   1266.4K/1262.4K    1720.1K/1732.1K
    4k    793139/570902      1129.6K/773982
    32k   72660/72086        97229/96164
    
    Using write_generate=0 and read_verify=0 (w/w.o patch):
    bs      IOPS(read)        IOPS(write)
    ----    ----------        ----------
    512   1590.2K/1600.1K    1828.2K/1830.3K
    4k    1078.1K/937272     1142.1K/815304
    32k   77012/77369        98125/97435
    Signed-off-by: default avatarMax Gurtovoy <maxg@mellanox.com>
    Signed-off-by: default avatarIsrael Rukshin <israelr@mellanox.com>
    Suggested-by: default avatarSagi Grimberg <sagi@grimberg.me>
    Signed-off-by: default avatarJason Gunthorpe <jgg@mellanox.com>
    2563e2f3
mlx5_ib.h 41.3 KB