Commit 1a44e149 authored by Andrea Arcangeli's avatar Andrea Arcangeli Committed by Linus Torvalds

[PATCH] .text page fault SMP scalability optimization

We had a problem on ppc64 where with more than 4 threads a large system
wouldn't scale well while faulting in the .text (most of the time was spent
in the kernel despite it was an userland compute intensive app).  The
reason is the useless overwrite of the same pte from all cpu.

I fixed it this way (verified on an older kernel but the forward port is
almost identical).  This will benefit all archs not just ppc64.
Signed-off-by: default avatarAndrea Arcangeli <andrea@suse.de>
Cc: Hugh Dickins <hugh@veritas.com>
Signed-off-by: default avatarAndrew Morton <akpm@osdl.org>
Signed-off-by: default avatarLinus Torvalds <torvalds@osdl.org>
parent 2e9b367c
...@@ -1980,9 +1980,10 @@ static inline int handle_pte_fault(struct mm_struct *mm, ...@@ -1980,9 +1980,10 @@ static inline int handle_pte_fault(struct mm_struct *mm,
pte_t *pte, pmd_t *pmd, int write_access) pte_t *pte, pmd_t *pmd, int write_access)
{ {
pte_t entry; pte_t entry;
pte_t old_entry;
spinlock_t *ptl; spinlock_t *ptl;
entry = *pte; old_entry = entry = *pte;
if (!pte_present(entry)) { if (!pte_present(entry)) {
if (pte_none(entry)) { if (pte_none(entry)) {
if (!vma->vm_ops || !vma->vm_ops->nopage) if (!vma->vm_ops || !vma->vm_ops->nopage)
...@@ -2009,9 +2010,20 @@ static inline int handle_pte_fault(struct mm_struct *mm, ...@@ -2009,9 +2010,20 @@ static inline int handle_pte_fault(struct mm_struct *mm,
entry = pte_mkdirty(entry); entry = pte_mkdirty(entry);
} }
entry = pte_mkyoung(entry); entry = pte_mkyoung(entry);
if (!pte_same(old_entry, entry)) {
ptep_set_access_flags(vma, address, pte, entry, write_access); ptep_set_access_flags(vma, address, pte, entry, write_access);
update_mmu_cache(vma, address, entry); update_mmu_cache(vma, address, entry);
lazy_mmu_prot_update(entry); lazy_mmu_prot_update(entry);
} else {
/*
* This is needed only for protection faults but the arch code
* is not yet telling us if this is a protection fault or not.
* This still avoids useless tlb flushes for .text page faults
* with threads.
*/
if (write_access)
flush_tlb_page(vma, address);
}
unlock: unlock:
pte_unmap_unlock(pte, ptl); pte_unmap_unlock(pte, ptl);
return VM_FAULT_MINOR; return VM_FAULT_MINOR;
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment