Commit 3fa6c507 authored by Mikulas Patocka's avatar Mikulas Patocka Committed by Linus Torvalds

mm: optimize copy_page_to/from_iter_iovec

copy_page_to_iter_iovec() and copy_page_from_iter_iovec() copy some data
to userspace or from userspace.  These functions have a fast path where
they map a page using kmap_atomic and a slow path where they use kmap.

kmap is slower than kmap_atomic, so the fast path is preferred.

However, on kernels without highmem support, kmap just calls
page_address, so there is no need to avoid kmap.  On kernels without
highmem support, the fast path just increases code size (and cache
footprint) and it doesn't improve copy performance in any way.

This patch enables the fast path only if CONFIG_HIGHMEM is defined.

Code size reduced by this patch:
  x86 (without highmem)	  928
  x86-64		  960
  sparc64		  848
  alpha			 1136
  pa-risc		 1200

[akpm@linux-foundation.org: use IS_ENABLED(), per Andi]
Link: http://lkml.kernel.org/r/alpine.LRH.2.02.1607221711410.4818@file01.intranet.prod.int.rdu2.redhat.comSigned-off-by: default avatarMikulas Patocka <mpatocka@redhat.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Alexander Viro <viro@zeniv.linux.org.uk>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Andi Kleen <andi@firstfloor.org>
Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
parent 7e4411bf
...@@ -144,7 +144,7 @@ static size_t copy_page_to_iter_iovec(struct page *page, size_t offset, size_t b ...@@ -144,7 +144,7 @@ static size_t copy_page_to_iter_iovec(struct page *page, size_t offset, size_t b
buf = iov->iov_base + skip; buf = iov->iov_base + skip;
copy = min(bytes, iov->iov_len - skip); copy = min(bytes, iov->iov_len - skip);
if (!fault_in_pages_writeable(buf, copy)) { if (IS_ENABLED(CONFIG_HIGHMEM) && !fault_in_pages_writeable(buf, copy)) {
kaddr = kmap_atomic(page); kaddr = kmap_atomic(page);
from = kaddr + offset; from = kaddr + offset;
...@@ -175,6 +175,7 @@ static size_t copy_page_to_iter_iovec(struct page *page, size_t offset, size_t b ...@@ -175,6 +175,7 @@ static size_t copy_page_to_iter_iovec(struct page *page, size_t offset, size_t b
copy = min(bytes, iov->iov_len - skip); copy = min(bytes, iov->iov_len - skip);
} }
/* Too bad - revert to non-atomic kmap */ /* Too bad - revert to non-atomic kmap */
kaddr = kmap(page); kaddr = kmap(page);
from = kaddr + offset; from = kaddr + offset;
left = __copy_to_user(buf, from, copy); left = __copy_to_user(buf, from, copy);
...@@ -193,6 +194,7 @@ static size_t copy_page_to_iter_iovec(struct page *page, size_t offset, size_t b ...@@ -193,6 +194,7 @@ static size_t copy_page_to_iter_iovec(struct page *page, size_t offset, size_t b
bytes -= copy; bytes -= copy;
} }
kunmap(page); kunmap(page);
done: done:
if (skip == iov->iov_len) { if (skip == iov->iov_len) {
iov++; iov++;
...@@ -225,7 +227,7 @@ static size_t copy_page_from_iter_iovec(struct page *page, size_t offset, size_t ...@@ -225,7 +227,7 @@ static size_t copy_page_from_iter_iovec(struct page *page, size_t offset, size_t
buf = iov->iov_base + skip; buf = iov->iov_base + skip;
copy = min(bytes, iov->iov_len - skip); copy = min(bytes, iov->iov_len - skip);
if (!fault_in_pages_readable(buf, copy)) { if (IS_ENABLED(CONFIG_HIGHMEM) && !fault_in_pages_readable(buf, copy)) {
kaddr = kmap_atomic(page); kaddr = kmap_atomic(page);
to = kaddr + offset; to = kaddr + offset;
...@@ -256,6 +258,7 @@ static size_t copy_page_from_iter_iovec(struct page *page, size_t offset, size_t ...@@ -256,6 +258,7 @@ static size_t copy_page_from_iter_iovec(struct page *page, size_t offset, size_t
copy = min(bytes, iov->iov_len - skip); copy = min(bytes, iov->iov_len - skip);
} }
/* Too bad - revert to non-atomic kmap */ /* Too bad - revert to non-atomic kmap */
kaddr = kmap(page); kaddr = kmap(page);
to = kaddr + offset; to = kaddr + offset;
left = __copy_from_user(to, buf, copy); left = __copy_from_user(to, buf, copy);
...@@ -274,6 +277,7 @@ static size_t copy_page_from_iter_iovec(struct page *page, size_t offset, size_t ...@@ -274,6 +277,7 @@ static size_t copy_page_from_iter_iovec(struct page *page, size_t offset, size_t
bytes -= copy; bytes -= copy;
} }
kunmap(page); kunmap(page);
done: done:
if (skip == iov->iov_len) { if (skip == iov->iov_len) {
iov++; iov++;
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment