• Dan Williams's avatar
    libnvdimm/pmem: Bypass CONFIG_HARDENED_USERCOPY overhead · 52f476a3
    Dan Williams authored
    Jeff discovered that performance improves from ~375K iops to ~519K iops
    on a simple psync-write fio workload when moving the location of 'struct
    page' from the default PMEM location to DRAM. This result is surprising
    because the expectation is that 'struct page' for dax is only needed for
    third party references to dax mappings. For example, a dax-mapped buffer
    passed to another system call for direct-I/O requires 'struct page' for
    sending the request down the driver stack and pinning the page. There is
    no usage of 'struct page' for first party access to a file via
    read(2)/write(2) and friends.
    
    However, this "no page needed" expectation is violated by
    CONFIG_HARDENED_USERCOPY and the check_copy_size() performed in
    copy_from_iter_full_nocache() and copy_to_iter_mcsafe(). The
    check_heap_object() helper routine assumes the buffer is backed by a
    slab allocator (DRAM) page and applies some checks.  Those checks are
    invalid, dax pages do not originate from the slab, and redundant,
    dax_iomap_actor() has already validated that the I/O is within bounds.
    Specifically that routine validates that the logical file offset is
    within bounds of the file, then it does a sector-to-pfn translation
    which validates that the physical mapping is within bounds of the block
    device.
    
    Bypass additional hardened usercopy overhead and call the 'no check'
    versions of the copy_{to,from}_iter operations directly.
    
    Fixes: 0aed55af ("x86, uaccess: introduce copy_from_iter_flushcache...")
    Cc: <stable@vger.kernel.org>
    Cc: Jeff Moyer <jmoyer@redhat.com>
    Cc: Ingo Molnar <mingo@redhat.com>
    Cc: Christoph Hellwig <hch@lst.de>
    Cc: Al Viro <viro@zeniv.linux.org.uk>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Cc: Matthew Wilcox <willy@infradead.org>
    Reported-and-tested-by: default avatarJeff Smits <jeff.smits@intel.com>
    Acked-by: default avatarKees Cook <keescook@chromium.org>
    Acked-by: default avatarJan Kara <jack@suse.cz>
    Signed-off-by: default avatarDan Williams <dan.j.williams@intel.com>
    52f476a3
pmem.c 15.8 KB