• Vincent Whitchurch's avatar
    cifs: fix potential deadlock in direct reclaim · cc391b69
    Vincent Whitchurch authored
    The srv_mutex is used during writeback so cifs should ensure that
    allocations done when that mutex is held are done with GFP_NOFS, to
    avoid having direct reclaim ending up waiting for the same mutex and
    causing a deadlock.  This is detected by lockdep with the splat below:
    
     ======================================================
     WARNING: possible circular locking dependency detected
     5.18.0 #70 Not tainted
     ------------------------------------------------------
     kswapd0/49 is trying to acquire lock:
     ffff8880195782e0 (&tcp_ses->srv_mutex){+.+.}-{3:3}, at: compound_send_recv
    
     but task is already holding lock:
     ffffffffa98e66c0 (fs_reclaim){+.+.}-{0:0}, at: balance_pgdat
    
     which lock already depends on the new lock.
    
     the existing dependency chain (in reverse order) is:
    
     -> #1 (fs_reclaim){+.+.}-{0:0}:
            fs_reclaim_acquire
            kmem_cache_alloc_trace
            __request_module
            crypto_alg_mod_lookup
            crypto_alloc_tfm_node
            crypto_alloc_shash
            cifs_alloc_hash
            smb311_crypto_shash_allocate
            smb311_update_preauth_hash
            compound_send_recv
            cifs_send_recv
            SMB2_negotiate
            smb2_negotiate
            cifs_negotiate_protocol
            cifs_get_smb_ses
            cifs_mount
            cifs_smb3_do_mount
            smb3_get_tree
            vfs_get_tree
            path_mount
            __x64_sys_mount
            do_syscall_64
            entry_SYSCALL_64_after_hwframe
    
     -> #0 (&tcp_ses->srv_mutex){+.+.}-{3:3}:
            __lock_acquire
            lock_acquire
            __mutex_lock
            mutex_lock_nested
            compound_send_recv
            cifs_send_recv
            SMB2_write
            smb2_sync_write
            cifs_write
            cifs_writepage_locked
            cifs_writepage
            shrink_page_list
            shrink_lruvec
            shrink_node
            balance_pgdat
            kswapd
            kthread
            ret_from_fork
    
     other info that might help us debug this:
    
      Possible unsafe locking scenario:
    
            CPU0                    CPU1
            ----                    ----
       lock(fs_reclaim);
                                    lock(&tcp_ses->srv_mutex);
                                    lock(fs_reclaim);
       lock(&tcp_ses->srv_mutex);
    
      *** DEADLOCK ***
    
     1 lock held by kswapd0/49:
      #0: ffffffffa98e66c0 (fs_reclaim){+.+.}-{0:0}, at: balance_pgdat
    
     stack backtrace:
     CPU: 2 PID: 49 Comm: kswapd0 Not tainted 5.18.0 #70
     Call Trace:
      <TASK>
      dump_stack_lvl
      dump_stack
      print_circular_bug.cold
      check_noncircular
      __lock_acquire
      lock_acquire
      __mutex_lock
      mutex_lock_nested
      compound_send_recv
      cifs_send_recv
      SMB2_write
      smb2_sync_write
      cifs_write
      cifs_writepage_locked
      cifs_writepage
      shrink_page_list
      shrink_lruvec
      shrink_node
      balance_pgdat
      kswapd
      kthread
      ret_from_fork
      </TASK>
    
    Fix this by using the memalloc_nofs_save/restore APIs around the places
    where the srv_mutex is held.  Do this in a wrapper function for the
    lock/unlock of the srv_mutex, and rename the srv_mutex to avoid missing
    call sites in the conversion.
    
    Note that there is another lockdep warning involving internal crypto
    locks, which was masked by this problem and is visible after this fix,
    see the discussion in this thread:
    
     https://lore.kernel.org/all/20220523123755.GA13668@axis.com/
    
    Link: https://lore.kernel.org/r/CANT5p=rqcYfYMVHirqvdnnca4Mo+JQSw5Qu12v=kPfpk5yhhmg@mail.gmail.com/Reported-by: default avatarShyam Prasad N <nspmangalore@gmail.com>
    Suggested-by: default avatarLars Persson <larper@axis.com>
    Reviewed-by: default avatarRonnie Sahlberg <lsahlber@redhat.com>
    Reviewed-by: default avatarEnzo Matsumiya <ematsumiya@suse.de>
    Signed-off-by: default avatarVincent Whitchurch <vincent.whitchurch@axis.com>
    Signed-off-by: default avatarSteve French <stfrench@microsoft.com>
    cc391b69
cifs_swn.c 16.9 KB