• Kentaro Makita's avatar
    fix soft lock up at NFS mount via per-SB LRU-list of unused dentries · da3bbdd4
    Kentaro Makita authored
    [Summary]
    
     Split LRU-list of unused dentries to one per superblock to avoid soft
     lock up during NFS mounts and remounting of any filesystem.
    
     Previously I posted here:
     http://lkml.org/lkml/2008/3/5/590
    
    [Descriptions]
    
    - background
    
      dentry_unused is a list of dentries which are not referenced.
      dentry_unused grows up when references on directories or files are
      released.  This list can be very long if there is huge free memory.
    
    - the problem
    
      When shrink_dcache_sb() is called, it scans all dentry_unused linearly
      under spin_lock(), and if dentry->d_sb is differnt from given
      superblock, scan next dentry.  This scan costs very much if there are
      many entries, and very ineffective if there are many superblocks.
    
      IOW, When we need to shrink unused dentries on one dentry, but scans
      unused dentries on all superblocks in the system.  For example, we scan
      500 dentries to unmount a filesystem, but scans 1,000,000 or more unused
      dentries on other superblocks.
    
      In our case , At mounting NFS*, shrink_dcache_sb() is called to shrink
      unused dentries on NFS, but scans 100,000,000 unused dentries on
      superblocks in the system such as local ext3 filesystems.  I hear NFS
      mounting took 1 min on some system in use.
    
    * : NFS uses virtual filesystem in rpc layer, so NFS is affected by
      this problem.
    
      100,000,000 is possible number on large systems.
    
      Per-superblock LRU of unused dentried can reduce the cost in
      reasonable manner.
    
    - How to fix
    
      I found this problem is solved by David Chinner's "Per-superblock
      unused dentry LRU lists V3"(1), so I rebase it and add some fix to
      reclaim with fairness, which is in Andrew Morton's comments(2).
    
      1) http://lkml.org/lkml/2006/5/25/318
      2) http://lkml.org/lkml/2006/5/25/320
    
      Split LRU-list of unused dentries to each superblocks.  Then, NFS
      mounting will check dentries under a superblock instead of all.  But
      this spliting will break LRU of dentry-unused.  So, I've attempted to
      make reclaim unused dentrins with fairness by calculate number of
      dentries to scan on this sb based on following way
    
      number of dentries to scan on this sb =
      count * (number of dentries on this sb / number of dentries in the machine)
    
    - ToDo
     - I have to measuring performance number and do stress tests.
    
     - When unmount occurs during prune_dcache(), scanning on same
      superblock, It is unable to reach next superblock because it is gone
      away.  We restart scannig superblock from first one, it causes
      unfairness of reclaim unused dentries on first superblock.  But I think
      this happens very rarely.
    
    - Test Results
    
      Result on 6GB boxes with excessive unused dentries.
    
    Without patch:
    
    $ cat /proc/sys/fs/dentry-state
    10181835        10180203        45      0       0       0
    # mount -t nfs 10.124.60.70:/work/kernel-src nfs
    real    0m1.830s
    user    0m0.001s
    sys     0m1.653s
    
     With this patch:
    $ cat /proc/sys/fs/dentry-state
    10236610        10234751        45      0       0       0
    # mount -t nfs 10.124.60.70:/work/kernel-src nfs
    real    0m0.106s
    user    0m0.002s
    sys     0m0.032s
    
    [akpm@linux-foundation.org: fix comments]
    Signed-off-by: default avatarKentaro Makita <k-makita@np.css.fujitsu.com>
    Cc: Neil Brown <neilb@suse.de>
    Cc: Trond Myklebust <trond.myklebust@fys.uio.no>
    Cc: David Chinner <dgc@sgi.com>
    Cc: "J. Bruce Fields" <bfields@fieldses.org>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    da3bbdd4
super.c 22.8 KB