Commit 0a2c4048 authored by Nadav Amit's avatar Nadav Amit Committed by Linus Torvalds

mm: migrate: fix barriers around tlb_flush_pending

Reading tlb_flush_pending while the page-table lock is taken does not
require a barrier, since the lock/unlock already acts as a barrier.
Removing the barrier in mm_tlb_flush_pending() to address this issue.

However, migrate_misplaced_transhuge_page() calls mm_tlb_flush_pending()
while the page-table lock is already released, which may present a
problem on architectures with weak memory model (PPC).  To deal with
this case, a new parameter is added to mm_tlb_flush_pending() to
indicate if it is read without the page-table lock taken, and calling
smp_mb__after_unlock_lock() in this case.

Link: http://lkml.kernel.org/r/20170802000818.4760-3-namit@vmware.comSigned-off-by: default avatarNadav Amit <namit@vmware.com>
Acked-by: default avatarRik van Riel <riel@redhat.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Mel Gorman <mgorman@suse.de>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jeff Dike <jdike@addtoit.com>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Mel Gorman <mgorman@techsingularity.net>
Cc: Nadav Amit <nadav.amit@gmail.com>
Cc: Russell King <linux@armlinux.org.uk>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Yoshinori Sato <ysato@users.sourceforge.jp>
Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
parent 16af97dc
...@@ -526,12 +526,12 @@ static inline cpumask_t *mm_cpumask(struct mm_struct *mm) ...@@ -526,12 +526,12 @@ static inline cpumask_t *mm_cpumask(struct mm_struct *mm)
/* /*
* Memory barriers to keep this state in sync are graciously provided by * Memory barriers to keep this state in sync are graciously provided by
* the page table locks, outside of which no page table modifications happen. * the page table locks, outside of which no page table modifications happen.
* The barriers below prevent the compiler from re-ordering the instructions * The barriers are used to ensure the order between tlb_flush_pending updates,
* around the memory barriers that are already present in the code. * which happen while the lock is not taken, and the PTE updates, which happen
* while the lock is taken, are serialized.
*/ */
static inline bool mm_tlb_flush_pending(struct mm_struct *mm) static inline bool mm_tlb_flush_pending(struct mm_struct *mm)
{ {
barrier();
return atomic_read(&mm->tlb_flush_pending) > 0; return atomic_read(&mm->tlb_flush_pending) > 0;
} }
...@@ -554,7 +554,13 @@ static inline void inc_tlb_flush_pending(struct mm_struct *mm) ...@@ -554,7 +554,13 @@ static inline void inc_tlb_flush_pending(struct mm_struct *mm)
/* Clearing is done after a TLB flush, which also provides a barrier. */ /* Clearing is done after a TLB flush, which also provides a barrier. */
static inline void dec_tlb_flush_pending(struct mm_struct *mm) static inline void dec_tlb_flush_pending(struct mm_struct *mm)
{ {
barrier(); /*
* Guarantee that the tlb_flush_pending does not not leak into the
* critical section, since we must order the PTE change and changes to
* the pending TLB flush indication. We could have relied on TLB flush
* as a memory barrier, but this behavior is not clearly documented.
*/
smp_mb__before_atomic();
atomic_dec(&mm->tlb_flush_pending); atomic_dec(&mm->tlb_flush_pending);
} }
#else #else
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment