1. 07 Jan, 2011 40 commits
    • Nick Piggin's avatar
      fs: dcache per-inode inode alias locking · 873feea0
      Nick Piggin authored
      dcache_inode_lock can be replaced with per-inode locking. Use existing
      inode->i_lock for this. This is slightly non-trivial because we sometimes
      need to find the inode from the dentry, which requires d_inode to be
      stabilised (either with refcount or d_lock).
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      873feea0
    • Nick Piggin's avatar
      fs: dcache per-bucket dcache hash locking · ceb5bdc2
      Nick Piggin authored
      We can turn the dcache hash locking from a global dcache_hash_lock into
      per-bucket locking.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      ceb5bdc2
    • Nick Piggin's avatar
      bit_spinlock: add required includes · 626d6074
      Nick Piggin authored
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      626d6074
    • Nick Piggin's avatar
      kernel: add bl_list · 4e35e607
      Nick Piggin authored
      Introduce a type of hlist that can support the use of the lowest bit in the
      hlist_head. This will be subsequently used to implement per-bucket bit spinlock
      for inode and dentry hashes, and may be useful in other cases such as network
      hashes.
      Reviewed-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      4e35e607
    • Nick Piggin's avatar
      xfs: provide simple rcu-walk ACL implementation · 880566e1
      Nick Piggin authored
      This simple implementation just checks for no ACLs on the inode, and
      if so, then the rcu-walk may proceed, otherwise fail it.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      880566e1
    • Nick Piggin's avatar
      btrfs: provide simple rcu-walk ACL implementation · 258a5aa8
      Nick Piggin authored
      This simple implementation just checks for no ACLs on the inode, and
      if so, then the rcu-walk may proceed, otherwise fail it.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      258a5aa8
    • Nick Piggin's avatar
      ext2,3,4: provide simple rcu-walk ACL implementation · 73598611
      Nick Piggin authored
      This simple implementation just checks for no ACLs on the inode, and
      if so, then the rcu-walk may proceed, otherwise fail it.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      73598611
    • Nick Piggin's avatar
      fs: provide simple rcu-walk generic_check_acl implementation · 1e1743eb
      Nick Piggin authored
      This simple implementation just checks for no ACLs on the inode, and
      if so, then the rcu-walk may proceed, otherwise fail it.
      
      This could easily be extended to put acls under RCU and check them
      under seqlock, if need be. But this implementation is enough to show
      the rcu-walk aware permissions code for path lookups is working, and
      will handle cases where there are no ACLs or ACLs in just the final
      element.
      
      This patch implicity converts tmpfs to rcu-aware permission check.
      Subsequent patches onvert ext*, xfs, and, btrfs. Each of these uses
      acl/permission code in a different way, so convert them all to provide
      templates and proof of concept.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      1e1743eb
    • Nick Piggin's avatar
      fs: provide rcu-walk aware permission i_ops · b74c79e9
      Nick Piggin authored
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      b74c79e9
    • Nick Piggin's avatar
      fs: rcu-walk aware d_revalidate method · 34286d66
      Nick Piggin authored
      Require filesystems be aware of .d_revalidate being called in rcu-walk
      mode (nd->flags & LOOKUP_RCU). For now do a simple push down, returning
      -ECHILD from all implementations.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      34286d66
    • Nick Piggin's avatar
      fs: cache optimise dentry and inode for rcu-walk · 44a7d7a8
      Nick Piggin authored
      Put dentry and inode fields into top of data structure.  This allows RCU path
      traversal to perform an RCU dentry lookup in a path walk by touching only the
      first 56 bytes of the dentry.
      
      We also fit in 8 bytes of inline name in the first 64 bytes, so for short
      names, only 64 bytes needs to be touched to perform the lookup. We should
      get rid of the hash->prev pointer from the first 64 bytes, and fit 16 bytes
      of name in there, which will take care of 81% rather than 32% of the kernel
      tree.
      
      inode is also rearranged so that RCU lookup will only touch a single cacheline
      in the inode, plus one in the i_ops structure.
      
      This is important for directory component lookups in RCU path walking. In the
      kernel source, directory names average is around 6 chars, so this works.
      
      When we reach the last element of the lookup, we need to lock it and take its
      refcount which requires another cacheline access.
      
      Align dentry and inode operations structs, so members will be at predictable
      offsets and we can group common operations into head of structure.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      44a7d7a8
    • Nick Piggin's avatar
      fs: dcache reduce branches in lookup path · fb045adb
      Nick Piggin authored
      Reduce some branches and memory accesses in dcache lookup by adding dentry
      flags to indicate common d_ops are set, rather than having to check them.
      This saves a pointer memory access (dentry->d_op) in common path lookup
      situations, and saves another pointer load and branch in cases where we
      have d_op but not the particular operation.
      
      Patched with:
      
      git grep -E '[.>]([[:space:]])*d_op([[:space:]])*=' | xargs sed -e 's/\([^\t ]*\)->d_op = \(.*\);/d_set_d_op(\1, \2);/' -e 's/\([^\t ]*\)\.d_op = \(.*\);/d_set_d_op(\&\1, \2);/' -i
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      fb045adb
    • Nick Piggin's avatar
      fs: dcache remove d_mounted · 5f57cbcc
      Nick Piggin authored
      Rather than keep a d_mounted count in the dentry, set a dentry flag instead.
      The flag can be cleared by checking the hash table to see if there are any
      mounts left, which is not time critical because it is performed at detach time.
      
      The mounted state of a dentry is only used to speculatively take a look in the
      mount hash table if it is set -- before following the mount, vfsmount lock is
      taken and mount re-checked without races.
      
      This saves 4 bytes on 32-bit, nothing on 64-bit but it does provide a hole I
      might use later (and some configs have larger than 32-bit spinlocks which might
      make use of the hole).
      
      Autofs4 conversion and changelog by Ian Kent <raven@themaw.net>:
      In autofs4, when expring direct (or offset) mounts we need to ensure that we
      block user path walks into the autofs mount, which is covered by another mount.
      To do this we clear the mounted status so that follows stop before walking into
      the mount and are essentially blocked until the expire is completed. The
      automount daemon still finds the correct dentry for the umount due to the
      follow mount logic in fs/autofs4/root.c:autofs4_follow_link(), which is set as
      an inode operation for direct and offset mounts only and is called following
      the lookup that stopped at the covered mount.
      
      At the end of the expire the covering mount probably has gone away so the
      mounted status need not be restored. But we need to check this and only restore
      the mounted status if the expire failed.
      
      XXX: autofs may not work right if we have other mounts go over the top of it?
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      5f57cbcc
    • Nick Piggin's avatar
      fs: fs_struct use seqlock · c28cc364
      Nick Piggin authored
      Use a seqlock in the fs_struct to enable us to take an atomic copy of the
      complete cwd and root paths. Use this in the RCU lookup path to avoid a
      thread-shared spinlock in RCU lookup operations.
      
      Multi-threaded apps may now perform path lookups with scalability matching
      multi-process apps. Operations such as stat(2) become very scalable for
      multi-threaded workload.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      c28cc364
    • Nick Piggin's avatar
      fs: rcu-walk for path lookup · 31e6b01f
      Nick Piggin authored
      Perform common cases of path lookups without any stores or locking in the
      ancestor dentry elements. This is called rcu-walk, as opposed to the current
      algorithm which is a refcount based walk, or ref-walk.
      
      This results in far fewer atomic operations on every path element,
      significantly improving path lookup performance. It also avoids cacheline
      bouncing on common dentries, significantly improving scalability.
      
      The overall design is like this:
      * LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
      * Take the RCU lock for the entire path walk, starting with the acquiring
        of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
        not required for dentry persistence.
      * synchronize_rcu is called when unregistering a filesystem, so we can
        access d_ops and i_ops during rcu-walk.
      * Similarly take the vfsmount lock for the entire path walk. So now mnt
        refcounts are not required for persistence. Also we are free to perform mount
        lookups, and to assume dentry mount points and mount roots are stable up and
        down the path.
      * Have a per-dentry seqlock to protect the dentry name, parent, and inode,
        so we can load this tuple atomically, and also check whether any of its
        members have changed.
      * Dentry lookups (based on parent, candidate string tuple) recheck the parent
        sequence after the child is found in case anything changed in the parent
        during the path walk.
      * inode is also RCU protected so we can load d_inode and use the inode for
        limited things.
      * i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
      * i_op can be loaded.
      
      When we reach the destination dentry, we lock it, recheck lookup sequence,
      and increment its refcount and mountpoint refcount. RCU and vfsmount locks
      are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
      not match, we can not drop rcu-walk gracefully at the current point in the
      lokup, so instead return -ECHILD (for want of a better errno). This signals the
      path walking code to re-do the entire lookup with a ref-walk.
      
      Aside from the final dentry, there are other situations that may be encounted
      where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
      a reference on the last good dentry) and continue with a ref-walk. Again, if
      we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
      using ref-walk. But it is very important that we can continue with ref-walk
      for most cases, particularly to avoid the overhead of double lookups, and to
      gain the scalability advantages on common path elements (like cwd and root).
      
      The cases where rcu-walk cannot continue are:
      * NULL dentry (ie. any uncached path element)
      * parent with d_inode->i_op->permission or ACLs
      * dentries with d_revalidate
      * Following links
      
      In future patches, permission checks and d_revalidate become rcu-walk aware. It
      may be possible eventually to make following links rcu-walk aware.
      
      Uncached path elements will always require dropping to ref-walk mode, at the
      very least because i_mutex needs to be grabbed, and objects allocated.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      31e6b01f
    • Nick Piggin's avatar
      kernel: optimise seqlock · 3c22cd57
      Nick Piggin authored
      Add branch annotations for seqlock read fastpath, and introduce
      __read_seqcount_begin and __read_seqcount_end functions, that can avoid the
      smp_rmb() if used carefully. These will be used by store-free path walking
      algorithm performance is critical and seqlocks are in use.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      3c22cd57
    • Nick Piggin's avatar
      fs: avoid inode RCU freeing for pseudo fs · ff0c7d15
      Nick Piggin authored
      Pseudo filesystems that don't put inode on RCU list or reachable by
      rcu-walk dentries do not need to RCU free their inodes.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      ff0c7d15
    • Nick Piggin's avatar
      fs: icache RCU free inodes · fa0d7e3d
      Nick Piggin authored
      RCU free the struct inode. This will allow:
      
      - Subsequent store-free path walking patch. The inode must be consulted for
        permissions when walking, so an RCU inode reference is a must.
      - sb_inode_list_lock to be moved inside i_lock because sb list walkers who want
        to take i_lock no longer need to take sb_inode_list_lock to walk the list in
        the first place. This will simplify and optimize locking.
      - Could remove some nested trylock loops in dcache code
      - Could potentially simplify things a bit in VM land. Do not need to take the
        page lock to follow page->mapping.
      
      The downsides of this is the performance cost of using RCU. In a simple
      creat/unlink microbenchmark, performance drops by about 10% due to inability to
      reuse cache-hot slab objects. As iterations increase and RCU freeing starts
      kicking over, this increases to about 20%.
      
      In cases where inode lifetimes are longer (ie. many inodes may be allocated
      during the average life span of a single inode), a lot of this cache reuse is
      not applicable, so the regression caused by this patch is smaller.
      
      The cache-hot regression could largely be avoided by using SLAB_DESTROY_BY_RCU,
      however this adds some complexity to list walking and store-free path walking,
      so I prefer to implement this at a later date, if it is shown to be a win in
      real situations. I haven't found a regression in any non-micro benchmark so I
      doubt it will be a problem.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      fa0d7e3d
    • Nick Piggin's avatar
      fs: consolidate dentry kill sequence · 77812a1e
      Nick Piggin authored
      The tricky locking for disposing of a dentry is duplicated 3 times in the
      dcache (dput, pruning a dentry from the LRU, and pruning its ancestors).
      Consolidate them all into a single function dentry_kill.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      77812a1e
    • Nick Piggin's avatar
      ec33679d
    • Nick Piggin's avatar
      be182bff
    • Nick Piggin's avatar
      fs: dcache reduce prune_one_dentry locking · 89e60548
      Nick Piggin authored
      prune_one_dentry can avoid quite a bit of locking in the common case where
      ancestors have an elevated refcount. Alternatively, we could have gone the
      other way and made fewer trylocks in the case where d_count goes to zero, but
      is probably less common.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      89e60548
    • Nick Piggin's avatar
      fs: dcache reduce d_parent locking · a734eb45
      Nick Piggin authored
      Use RCU to simplify locking in dget_parent.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      a734eb45
    • Nick Piggin's avatar
      fs: dcache rationalise dget variants · dc0474be
      Nick Piggin authored
      dget_locked was a shortcut to avoid the lazy lru manipulation when we already
      held dcache_lock (lru manipulation was relatively cheap at that point).
      However, how that the lru lock is an innermost one, we never hold it at any
      caller, so the lock cost can now be avoided. We already have well working lazy
      dcache LRU, so it should be fine to defer LRU manipulations to scan time.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      dc0474be
    • Nick Piggin's avatar
      fs: dcache reduce dcache_inode_lock · 357f8e65
      Nick Piggin authored
      dcache_inode_lock can be avoided in d_delete() and d_materialise_unique()
      in cases where it is not required.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      357f8e65
    • Nick Piggin's avatar
      fs: dcache reduce locking in d_alloc · 89ad485f
      Nick Piggin authored
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      89ad485f
    • Nick Piggin's avatar
      fs: dcache reduce dput locking · 61f3dee4
      Nick Piggin authored
      It is possible to run dput without taking data structure locks up-front. In
      many cases where we don't kill the dentry anyway, these locks are not required.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      61f3dee4
    • Nick Piggin's avatar
      fs: dcache avoid starvation in dcache multi-step operations · 58db63d0
      Nick Piggin authored
      Long lived dcache "multi-step" operations which retry on rename seq can
      be starved with a lot of rename activity. If they fail after the 1st pass,
      take the rename_lock for writing to avoid further starvation.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      58db63d0
    • Nick Piggin's avatar
      fs: dcache remove dcache_lock · b5c84bf6
      Nick Piggin authored
      dcache_lock no longer protects anything. remove it.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      b5c84bf6
    • Nick Piggin's avatar
      fs: Use rename lock and RCU for multi-step operations · 949854d0
      Nick Piggin authored
      The remaining usages for dcache_lock is to allow atomic, multi-step read-side
      operations over the directory tree by excluding modifications to the tree.
      Also, to walk in the leaf->root direction in the tree where we don't have
      a natural d_lock ordering.
      
      This could be accomplished by taking every d_lock, but this would mean a
      huge number of locks and actually gets very tricky.
      
      Solve this instead by using the rename seqlock for multi-step read-side
      operations, retry in case of a rename so we don't walk up the wrong parent.
      Concurrent dentry insertions are not serialised against.  Concurrent deletes
      are tricky when walking up the directory: our parent might have been deleted
      when dropping locks so also need to check and retry for that.
      
      We can also use the rename lock in cases where livelock is a worry (and it
      is introduced in subsequent patch).
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      949854d0
    • Nick Piggin's avatar
      fs: increase d_name lock coverage · 9abca360
      Nick Piggin authored
      Cover d_name with d_lock in more cases, where there may be concurrent
      modification to it.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      9abca360
    • Nick Piggin's avatar
      fs: scale inode alias list · b23fb0a6
      Nick Piggin authored
      Add a new lock, dcache_inode_lock, to protect the inode's i_dentry list
      from concurrent modification. d_alias is also protected by d_lock.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      b23fb0a6
    • Nick Piggin's avatar
      fs: dcache scale subdirs · 2fd6b7f5
      Nick Piggin authored
      Protect d_subdirs and d_child with d_lock, except in filesystems that aren't
      using dcache_lock for these anyway (eg. using i_mutex).
      
      Note: if we change the locking rule in future so that ->d_child protection is
      provided only with ->d_parent->d_lock, it may allow us to reduce some locking.
      But it would be an exception to an otherwise regular locking scheme, so we'd
      have to see some good results. Probably not worthwhile.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      2fd6b7f5
    • Nick Piggin's avatar
      fs: dcache scale d_unhashed · da502956
      Nick Piggin authored
      Protect d_unhashed(dentry) condition with d_lock. This means keeping
      DCACHE_UNHASHED bit in synch with hash manipulations.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      da502956
    • Nick Piggin's avatar
      fs: dcache scale dentry refcount · b7ab39f6
      Nick Piggin authored
      Make d_count non-atomic and protect it with d_lock. This allows us to ensure a
      0 refcount dentry remains 0 without dcache_lock. It is also fairly natural when
      we start protecting many other dentry members with d_lock.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      b7ab39f6
    • Nick Piggin's avatar
      fs: dcache scale lru · 23044507
      Nick Piggin authored
      Add a new lock, dcache_lru_lock, to protect the dcache LRU list from concurrent
      modification. d_lru is also protected by d_lock, which allows LRU lists to be
      accessed without the lru lock, using RCU in future patches.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      23044507
    • Nick Piggin's avatar
      fs: dcache scale hash · 789680d1
      Nick Piggin authored
      Add a new lock, dcache_hash_lock, to protect the dcache hash table from
      concurrent modification. d_hash is also protected by d_lock.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      789680d1
    • Nick Piggin's avatar
      hostfs: simplify locking · ec2447c2
      Nick Piggin authored
      Remove dcache_lock locking from hostfs filesystem, and move it into dcache
      helpers. All that is required is a coherent path name. Protection from
      concurrent modification of the namespace after path name generation is not
      provided in current code, because dcache_lock is dropped before the path is
      used.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      ec2447c2
    • Nick Piggin's avatar
      fs: change d_hash for rcu-walk · b1e6a015
      Nick Piggin authored
      Change d_hash so it may be called from lock-free RCU lookups. See similar
      patch for d_compare for details.
      
      For in-tree filesystems, this is just a mechanical change.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      b1e6a015
    • Nick Piggin's avatar
      fs: change d_compare for rcu-walk · 621e155a
      Nick Piggin authored
      Change d_compare so it may be called from lock-free RCU lookups. This
      does put significant restrictions on what may be done from the callback,
      however there don't seem to have been any problems with in-tree fses.
      If some strange use case pops up that _really_ cannot cope with the
      rcu-walk rules, we can just add new rcu-unaware callbacks, which would
      cause name lookup to drop out of rcu-walk mode.
      
      For in-tree filesystems, this is just a mechanical change.
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      621e155a