1. 04 Mar, 2021 31 commits
    • Jens Axboe's avatar
      io-wq: ensure all pending work is canceled on exit · f0127254
      Jens Axboe authored
      If we race on shutting down the io-wq, then we should ensure that any
      work that was queued after workers shutdown is canceled. Harden the
      add work check a bit too, checking for IO_WQ_BIT_EXIT and cancel if
      it's set.
      
      Add a WARN_ON() for having any work before we kill the io-wq context.
      
      Reported-by: syzbot+91b4b56ead187d35c9d3@syzkaller.appspotmail.com
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      f0127254
    • Jens Axboe's avatar
      io_uring: ensure that threads freeze on suspend · e4b4a13f
      Jens Axboe authored
      Alex reports that his system fails to suspend using 5.12-rc1, with the
      following dump:
      
      [  240.650300] PM: suspend entry (deep)
      [  240.650748] Filesystems sync: 0.000 seconds
      [  240.725605] Freezing user space processes ...
      [  260.739483] Freezing of tasks failed after 20.013 seconds (3 tasks refusing to freeze, wq_busy=0):
      [  260.739497] task:iou-mgr-446     state:S stack:    0 pid:  516 ppid:   439 flags:0x00004224
      [  260.739504] Call Trace:
      [  260.739507]  ? sysvec_apic_timer_interrupt+0xb/0x81
      [  260.739515]  ? pick_next_task_fair+0x197/0x1cde
      [  260.739519]  ? sysvec_reschedule_ipi+0x2f/0x6a
      [  260.739522]  ? asm_sysvec_reschedule_ipi+0x12/0x20
      [  260.739525]  ? __schedule+0x57/0x6d6
      [  260.739529]  ? del_timer_sync+0xb9/0x115
      [  260.739533]  ? schedule+0x63/0xd5
      [  260.739536]  ? schedule_timeout+0x219/0x356
      [  260.739540]  ? __next_timer_interrupt+0xf1/0xf1
      [  260.739544]  ? io_wq_manager+0x73/0xb1
      [  260.739549]  ? io_wq_create+0x262/0x262
      [  260.739553]  ? ret_from_fork+0x22/0x30
      [  260.739557] task:iou-mgr-517     state:S stack:    0 pid:  522 ppid:   439 flags:0x00004224
      [  260.739561] Call Trace:
      [  260.739563]  ? sysvec_apic_timer_interrupt+0xb/0x81
      [  260.739566]  ? pick_next_task_fair+0x16f/0x1cde
      [  260.739569]  ? sysvec_apic_timer_interrupt+0xb/0x81
      [  260.739571]  ? asm_sysvec_apic_timer_interrupt+0x12/0x20
      [  260.739574]  ? __schedule+0x5b7/0x6d6
      [  260.739578]  ? del_timer_sync+0x70/0x115
      [  260.739581]  ? schedule_timeout+0x211/0x356
      [  260.739585]  ? __next_timer_interrupt+0xf1/0xf1
      [  260.739588]  ? io_wq_check_workers+0x15/0x11f
      [  260.739592]  ? io_wq_manager+0x69/0xb1
      [  260.739596]  ? io_wq_create+0x262/0x262
      [  260.739600]  ? ret_from_fork+0x22/0x30
      [  260.739603] task:iou-wrk-517     state:S stack:    0 pid:  523 ppid:   439 flags:0x00004224
      [  260.739607] Call Trace:
      [  260.739609]  ? __schedule+0x5b7/0x6d6
      [  260.739614]  ? schedule+0x63/0xd5
      [  260.739617]  ? schedule_timeout+0x219/0x356
      [  260.739621]  ? __next_timer_interrupt+0xf1/0xf1
      [  260.739624]  ? task_thread.isra.0+0x148/0x3af
      [  260.739628]  ? task_thread_unbound+0xa/0xa
      [  260.739632]  ? task_thread_bound+0x7/0x7
      [  260.739636]  ? ret_from_fork+0x22/0x30
      [  260.739647] OOM killer enabled.
      [  260.739648] Restarting tasks ... done.
      [  260.740077] PM: suspend exit
      
      Play nice and ensure that any thread we create will call try_to_freeze()
      at an opportune time so that memory suspend can proceed. For the io-wq
      worker threads, mark them as PF_NOFREEZE. They could potentially be
      blocked for a long time.
      Reported-by: default avatarAlex Xu (Hello71) <alex_y_xu@yahoo.ca>
      Tested-by: default avatarAlex Xu (Hello71) <alex_y_xu@yahoo.ca>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      e4b4a13f
    • Pavel Begunkov's avatar
      io_uring: remove extra in_idle wake up · b23fcf47
      Pavel Begunkov authored
      io_dismantle_req() is always followed by io_put_task(), which already do
      proper in_idle wake ups, so we can skip waking the owner task in
      io_dismantle_req(). The rules are simpler now, do io_put_task() shortly
      after ending a request, and it will be fine.
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      b23fcf47
    • Pavel Begunkov's avatar
      io_uring: inline __io_queue_async_work() · ebf93667
      Pavel Begunkov authored
      __io_queue_async_work() is only called from io_queue_async_work(),
      inline it.
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      ebf93667
    • Pavel Begunkov's avatar
      io_uring: inline io_req_clean_work() · f85c310a
      Pavel Begunkov authored
      Inline io_req_clean_work(), less code and easier to analyse
      tctx dependencies and refs usage.
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      f85c310a
    • Pavel Begunkov's avatar
      io_uring: choose right tctx->io_wq for try cancel · 64c72123
      Pavel Begunkov authored
      When we cancel SQPOLL, @task in io_uring_try_cancel_requests() will
      differ from current. Use the right tctx from passed in @task, and don't
      forget that it can be NULL when the io_uring ctx exits.
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      64c72123
    • Jens Axboe's avatar
      io_uring: fix -EAGAIN retry with IOPOLL · 3e6a0d3c
      Jens Axboe authored
      We no longer revert the iovec on -EIOCBQUEUED, see commit ab2125df,
      and this started causing issues for IOPOLL on devies that run out of
      request slots. Turns out what outside of needing a revert for those, we
      also had a bug where we didn't properly setup retry inside the submission
      path. That could cause re-import of the iovec, if any, and that could lead
      to spurious results if the application had those allocated on the stack.
      
      Catch -EAGAIN retry and make the iovec stable for IOPOLL, just like we do
      for !IOPOLL retries.
      
      Cc: <stable@vger.kernel.org> # 5.9+
      Reported-by: default avatarAbaci Robot <abaci@linux.alibaba.com>
      Reported-by: default avatarXiaoguang Wang <xiaoguang.wang@linux.alibaba.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      3e6a0d3c
    • Jens Axboe's avatar
      io-wq: fix error path leak of buffered write hash map · dc7bbc9e
      Jens Axboe authored
      The 'err' path should include the hash put, we already grabbed a reference
      once we get that far.
      
      Fixes: e941894e ("io-wq: make buffered file write hashed work map per-ctx")
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      dc7bbc9e
    • Pavel Begunkov's avatar
      io_uring: remove sqo_task · 16270893
      Pavel Begunkov authored
      Now, sqo_task is used only for a warning that is not interesting anymore
      since sqo_dead is gone, remove all of that including ctx->sqo_task.
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      16270893
    • Pavel Begunkov's avatar
      io_uring: kill sqo_dead and sqo submission halting · 70aacfe6
      Pavel Begunkov authored
      As SQPOLL task doesn't poke into ->sqo_task anymore, there is no need to
      kill the sqo when the master task exits. Before it was necessary to
      avoid races accessing sqo_task->files with removing them.
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      [axboe: don't forget to enable SQPOLL before exit, if started disabled]
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      70aacfe6
    • Jens Axboe's avatar
      io_uring: ignore double poll add on the same waitqueue head · 1c3b3e65
      Jens Axboe authored
      syzbot reports a deadlock, attempting to lock the same spinlock twice:
      
      ============================================
      WARNING: possible recursive locking detected
      5.11.0-syzkaller #0 Not tainted
      --------------------------------------------
      swapper/1/0 is trying to acquire lock:
      ffff88801b2b1130 (&runtime->sleep){..-.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline]
      ffff88801b2b1130 (&runtime->sleep){..-.}-{2:2}, at: io_poll_double_wake+0x25f/0x6a0 fs/io_uring.c:4960
      
      but task is already holding lock:
      ffff88801b2b3130 (&runtime->sleep){..-.}-{2:2}, at: __wake_up_common_lock+0xb4/0x130 kernel/sched/wait.c:137
      
      other info that might help us debug this:
       Possible unsafe locking scenario:
      
             CPU0
             ----
        lock(&runtime->sleep);
        lock(&runtime->sleep);
      
       *** DEADLOCK ***
      
       May be due to missing lock nesting notation
      
      2 locks held by swapper/1/0:
       #0: ffff888147474908 (&group->lock){..-.}-{2:2}, at: _snd_pcm_stream_lock_irqsave+0x9f/0xd0 sound/core/pcm_native.c:170
       #1: ffff88801b2b3130 (&runtime->sleep){..-.}-{2:2}, at: __wake_up_common_lock+0xb4/0x130 kernel/sched/wait.c:137
      
      stack backtrace:
      CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.11.0-syzkaller #0
      Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
      Call Trace:
       <IRQ>
       __dump_stack lib/dump_stack.c:79 [inline]
       dump_stack+0xfa/0x151 lib/dump_stack.c:120
       print_deadlock_bug kernel/locking/lockdep.c:2829 [inline]
       check_deadlock kernel/locking/lockdep.c:2872 [inline]
       validate_chain kernel/locking/lockdep.c:3661 [inline]
       __lock_acquire.cold+0x14c/0x3b4 kernel/locking/lockdep.c:4900
       lock_acquire kernel/locking/lockdep.c:5510 [inline]
       lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
       __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
       _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
       spin_lock include/linux/spinlock.h:354 [inline]
       io_poll_double_wake+0x25f/0x6a0 fs/io_uring.c:4960
       __wake_up_common+0x147/0x650 kernel/sched/wait.c:108
       __wake_up_common_lock+0xd0/0x130 kernel/sched/wait.c:138
       snd_pcm_update_state+0x46a/0x540 sound/core/pcm_lib.c:203
       snd_pcm_update_hw_ptr0+0xa75/0x1a50 sound/core/pcm_lib.c:464
       snd_pcm_period_elapsed+0x160/0x250 sound/core/pcm_lib.c:1805
       dummy_hrtimer_callback+0x94/0x1b0 sound/drivers/dummy.c:378
       __run_hrtimer kernel/time/hrtimer.c:1519 [inline]
       __hrtimer_run_queues+0x609/0xe40 kernel/time/hrtimer.c:1583
       hrtimer_run_softirq+0x17b/0x360 kernel/time/hrtimer.c:1600
       __do_softirq+0x29b/0x9f6 kernel/softirq.c:345
       invoke_softirq kernel/softirq.c:221 [inline]
       __irq_exit_rcu kernel/softirq.c:422 [inline]
       irq_exit_rcu+0x134/0x200 kernel/softirq.c:434
       sysvec_apic_timer_interrupt+0x93/0xc0 arch/x86/kernel/apic/apic.c:1100
       </IRQ>
       asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:632
      RIP: 0010:native_save_fl arch/x86/include/asm/irqflags.h:29 [inline]
      RIP: 0010:arch_local_save_flags arch/x86/include/asm/irqflags.h:70 [inline]
      RIP: 0010:arch_irqs_disabled arch/x86/include/asm/irqflags.h:137 [inline]
      RIP: 0010:acpi_safe_halt drivers/acpi/processor_idle.c:111 [inline]
      RIP: 0010:acpi_idle_do_entry+0x1c9/0x250 drivers/acpi/processor_idle.c:516
      Code: dd 38 6e f8 84 db 75 ac e8 54 32 6e f8 e8 0f 1c 74 f8 e9 0c 00 00 00 e8 45 32 6e f8 0f 00 2d 4e 4a c5 00 e8 39 32 6e f8 fb f4 <9c> 5b 81 e3 00 02 00 00 fa 31 ff 48 89 de e8 14 3a 6e f8 48 85 db
      RSP: 0018:ffffc90000d47d18 EFLAGS: 00000293
      RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
      RDX: ffff8880115c3780 RSI: ffffffff89052537 RDI: 0000000000000000
      RBP: ffff888141127064 R08: 0000000000000001 R09: 0000000000000001
      R10: ffffffff81794168 R11: 0000000000000000 R12: 0000000000000001
      R13: ffff888141127000 R14: ffff888141127064 R15: ffff888143331804
       acpi_idle_enter+0x361/0x500 drivers/acpi/processor_idle.c:647
       cpuidle_enter_state+0x1b1/0xc80 drivers/cpuidle/cpuidle.c:237
       cpuidle_enter+0x4a/0xa0 drivers/cpuidle/cpuidle.c:351
       call_cpuidle kernel/sched/idle.c:158 [inline]
       cpuidle_idle_call kernel/sched/idle.c:239 [inline]
       do_idle+0x3e1/0x590 kernel/sched/idle.c:300
       cpu_startup_entry+0x14/0x20 kernel/sched/idle.c:397
       start_secondary+0x274/0x350 arch/x86/kernel/smpboot.c:272
       secondary_startup_64_no_verify+0xb0/0xbb
      
      which is due to the driver doing poll_wait() twice on the same
      wait_queue_head. That is perfectly valid, but from checking the rest
      of the kernel tree, it's the only driver that does this.
      
      We can handle this just fine, we just need to ignore the second addition
      as we'll get woken just fine on the first one.
      
      Cc: stable@vger.kernel.org # 5.8+
      Fixes: 18bceab1 ("io_uring: allow POLL_ADD with double poll_wait() users")
      Reported-by: syzbot+28abd693db9e92c160d8@syzkaller.appspotmail.com
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      1c3b3e65
    • Jens Axboe's avatar
      io_uring: ensure that SQPOLL thread is started for exit · 3ebba796
      Jens Axboe authored
      If we create it in a disabled state because IORING_SETUP_R_DISABLED is
      set on ring creation, we need to ensure that we've kicked the thread if
      we're exiting before it's been explicitly disabled. Otherwise we can run
      into a deadlock where exit is waiting go park the SQPOLL thread, but the
      SQPOLL thread itself is waiting to get a signal to start.
      
      That results in the below trace of both tasks hung, waiting on each other:
      
      INFO: task syz-executor458:8401 blocked for more than 143 seconds.
            Not tainted 5.11.0-next-20210226-syzkaller #0
      "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
      task:syz-executor458 state:D stack:27536 pid: 8401 ppid:  8400 flags:0x00004004
      Call Trace:
       context_switch kernel/sched/core.c:4324 [inline]
       __schedule+0x90c/0x21a0 kernel/sched/core.c:5075
       schedule+0xcf/0x270 kernel/sched/core.c:5154
       schedule_timeout+0x1db/0x250 kernel/time/timer.c:1868
       do_wait_for_common kernel/sched/completion.c:85 [inline]
       __wait_for_common kernel/sched/completion.c:106 [inline]
       wait_for_common kernel/sched/completion.c:117 [inline]
       wait_for_completion+0x168/0x270 kernel/sched/completion.c:138
       io_sq_thread_park fs/io_uring.c:7115 [inline]
       io_sq_thread_park+0xd5/0x130 fs/io_uring.c:7103
       io_uring_cancel_task_requests+0x24c/0xd90 fs/io_uring.c:8745
       __io_uring_files_cancel+0x110/0x230 fs/io_uring.c:8840
       io_uring_files_cancel include/linux/io_uring.h:47 [inline]
       do_exit+0x299/0x2a60 kernel/exit.c:780
       do_group_exit+0x125/0x310 kernel/exit.c:922
       __do_sys_exit_group kernel/exit.c:933 [inline]
       __se_sys_exit_group kernel/exit.c:931 [inline]
       __x64_sys_exit_group+0x3a/0x50 kernel/exit.c:931
       do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
       entry_SYSCALL_64_after_hwframe+0x44/0xae
      RIP: 0033:0x43e899
      RSP: 002b:00007ffe89376d48 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
      RAX: ffffffffffffffda RBX: 00000000004af2f0 RCX: 000000000043e899
      RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000
      RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000010000000
      R10: 0000000000008011 R11: 0000000000000246 R12: 00000000004af2f0
      R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000001
      INFO: task iou-sqp-8401:8402 can't die for more than 143 seconds.
      task:iou-sqp-8401    state:D stack:30272 pid: 8402 ppid:  8400 flags:0x00004004
      Call Trace:
       context_switch kernel/sched/core.c:4324 [inline]
       __schedule+0x90c/0x21a0 kernel/sched/core.c:5075
       schedule+0xcf/0x270 kernel/sched/core.c:5154
       schedule_timeout+0x1db/0x250 kernel/time/timer.c:1868
       do_wait_for_common kernel/sched/completion.c:85 [inline]
       __wait_for_common kernel/sched/completion.c:106 [inline]
       wait_for_common kernel/sched/completion.c:117 [inline]
       wait_for_completion+0x168/0x270 kernel/sched/completion.c:138
       io_sq_thread+0x27d/0x1ae0 fs/io_uring.c:6717
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
      INFO: task iou-sqp-8401:8402 blocked for more than 143 seconds.
      
      Reported-by: syzbot+fb5458330b4442f2090d@syzkaller.appspotmail.com
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      3ebba796
    • Pavel Begunkov's avatar
      io_uring: replace cmpxchg in fallback with xchg · 28c4721b
      Pavel Begunkov authored
      io_run_ctx_fallback() can use xchg() instead of cmpxchg(). It's simpler
      and faster.
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      28c4721b
    • Pavel Begunkov's avatar
      io_uring: fix __tctx_task_work() ctx race · 2c32395d
      Pavel Begunkov authored
      There is an unlikely but possible race using a freed context. That's
      because req->task_work.func() can free a request, but we won't
      necessarily find a completion in submit_state.comp and so all ctx refs
      may be put by the time we do mutex_lock(&ctx->uring_ctx);
      
      There are several reasons why it can miss going through
      submit_state.comp: 1) req->task_work.func() didn't complete it itself,
      but punted to iowq (e.g. reissue) and it got freed later, or a similar
      situation with it overflowing and getting flushed by someone else, or
      being submitted to IRQ completion, 2) As we don't hold the uring_lock,
      someone else can do io_submit_flush_completions() and put our ref.
      3) Bugs and code obscurities, e.g. failing to propagate issue_flags
      properly.
      
      One example is as follows
      
        CPU1                                  |  CPU2
      =======================================================================
      @req->task_work.func()                  |
        -> @req overflwed,                    |
           so submit_state.comp,nr==0         |
                                              | flush overflows, and free @req
                                              | ctx refs == 0, free it
      ctx is dead, but we do                  |
      	lock + flush + unlock           |
      
      So take a ctx reference for each new ctx we see in __tctx_task_work(),
      and do release it until we do all our flushing.
      
      Fixes: 65453d1e ("io_uring: enable req cache for task_work items")
      Reported-by: syzbot+a157ac7c03a56397f553@syzkaller.appspotmail.com
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      [axboe: fold in my one-liner and fix ref mismatch]
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      2c32395d
    • Jens Axboe's avatar
      io_uring: kill io_uring_flush() · 0d30b3e7
      Jens Axboe authored
      This was always a weird work-around or file referencing, and we don't
      need it anymore. Get rid of it.
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      0d30b3e7
    • Jens Axboe's avatar
      io_uring: kill unnecessary io_run_ctx_fallback() in io_ring_exit_work() · 914390bc
      Jens Axboe authored
      We already run the fallback task_work in io_uring_try_cancel_requests(),
      no need to duplicate at ring exit explicitly.
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      914390bc
    • Jens Axboe's avatar
      io_uring: move cred assignment into io_issue_sqe() · 5730b27e
      Jens Axboe authored
      If we move it in there, then we no longer have to care about it in io-wq.
      This means we can drop the cred handling in io-wq, and we can drop the
      REQ_F_WORK_INITIALIZED flag and async init functions as that was the last
      user of it since we moved to the new workers. Then we can also drop
      io_wq_work->creds, and just hold the personality u16 in there instead.
      Suggested-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      5730b27e
    • Jens Axboe's avatar
      io_uring: kill unnecessary REQ_F_WORK_INITIALIZED checks · 1575f21a
      Jens Axboe authored
      We're no longer checking anything that requires the work item to be
      initialized, as we're not carrying any file related state there.
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      1575f21a
    • Jens Axboe's avatar
      io_uring: remove unused argument 'tsk' from io_req_caches_free() · 4010fec4
      Jens Axboe authored
      We prune the full cache regardless, get rid of the dead argument.
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      4010fec4
    • Pavel Begunkov's avatar
      io_uring: destroy io-wq on exec · 8452d4a6
      Pavel Begunkov authored
      Destroy current's io-wq backend and tctx on __io_uring_task_cancel(),
      aka exec(). Looks it's not strictly necessary, because it will be done
      at some point when the task dies and changes of creds/files/etc. are
      handled, but better to do that earlier to free io-wq and not potentially
      lock previous mm and other resources for the time being.
      
      It's safe to do because we wait for all requests of the current task to
      complete, so no request will use tctx afterwards. Note, that
      io_uring_files_cancel() may leave some requests for later reaping, so it
      leaves tctx intact, that's ok as the task is dying anyway.
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      8452d4a6
    • Pavel Begunkov's avatar
      io_uring: warn on not destroyed io-wq · ef8eaa4e
      Pavel Begunkov authored
      Make sure that we killed an io-wq by the time a task is dead.
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      ef8eaa4e
    • Jens Axboe's avatar
      io_uring: fix race condition in task_work add and clear · 1d5f360d
      Jens Axboe authored
      We clear the bit marking the ctx task_work as active after having run
      the queued work, but we really should be clearing it before. Otherwise
      we can hit a tiny race ala:
      
      CPU0					CPU1
      io_task_work_add()			tctx_task_work()
      					run_work
      	add_to_list
      	test_and_set_bit
      					clear_bit
      		already set
      
      and CPU0 will return thinking the task_work is queued, while in reality
      it's already being run. If we hit the condition after __tctx_task_work()
      found no more work, but before we've cleared the bit, then we'll end up
      thinking it's queued and will be run. In reality it is queued, but we
      didn't queue the ctx task_work to ensure that it gets run.
      
      Fixes: 7cbf1722 ("io_uring: provide FIFO ordering for task_work")
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      1d5f360d
    • Jens Axboe's avatar
      io-wq: provide an io_wq_put_and_exit() helper · afcc4015
      Jens Axboe authored
      If we put the io-wq from io_uring, we really want it to exit. Provide
      a helper that does that for us. Couple that with not having the manager
      hold a reference to the 'wq' and the normal SQPOLL exit will tear down
      the io-wq context appropriate.
      
      On the io-wq side, our wq context is per task, so only the task itself
      is manipulating ->manager and hence it's safe to check and clear without
      any extra locking. We just need to ensure that the manager task stays
      around, in case it exits.
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      afcc4015
    • Jens Axboe's avatar
      io_uring: don't use complete_all() on SQPOLL thread exit · 8629397e
      Jens Axboe authored
      We want to reuse this completion, and a single complete should do just
      fine. Ensure that we park ourselves first if requested, as that is what
      lead to the initial deadlock in this area. If we've got someone attempting
      to park us, then we can't proceed without having them finish first.
      
      Fixes: 37d1e2e3 ("io_uring: move SQPOLL thread io-wq forked worker")
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      8629397e
    • Pavel Begunkov's avatar
      io_uring: run fallback on cancellation · ba50a036
      Pavel Begunkov authored
      io_uring_try_cancel_requests() matches not only current's requests, but
      also of other exiting tasks, so we need to actively cancel them and not
      just wait, especially since the function can be called on flush during
      do_exit() -> exit_files().
      Even if it's not a problem for now, it's much nicer to know that the
      function tries to cancel everything it can.
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      ba50a036
    • Jens Axboe's avatar
      io_uring: SQPOLL stop error handling fixes · e54945ae
      Jens Axboe authored
      If we fail to fork an SQPOLL worker, we can hit cancel, and hence
      attempted thread stop, with the thread already being stopped. Ensure
      we check for that.
      
      Also guard thread stop fully by the sqd mutex, just like we do for
      park.
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      e54945ae
    • Jens Axboe's avatar
      io-wq: fix double put of 'wq' in error path · 470ec4ed
      Jens Axboe authored
      We are already freeing the wq struct in both spots, so don't put it and
      get it freed twice.
      
      Reported-by: syzbot+7bf785eedca35ca05501@syzkaller.appspotmail.com
      Fixes: 4fb6ac32 ("io-wq: improve manager/worker handling over exec")
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      470ec4ed
    • Jens Axboe's avatar
      io-wq: wait for manager exit on wq destroy · d364d9e5
      Jens Axboe authored
      The manager waits for the workers, hence the manager is always valid if
      workers are running. Now also have wq destroy wait for the manager on
      exit, so we now everything is gone.
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      d364d9e5
    • Jens Axboe's avatar
      io-wq: rename wq->done completion to wq->started · dbf99620
      Jens Axboe authored
      This is a leftover from a different use cases, it's used to wait for
      the manager to startup. Rename it as such.
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      dbf99620
    • Jens Axboe's avatar
      io-wq: don't ask for a new worker if we're exiting · 613eeb60
      Jens Axboe authored
      If we're in the process of shutting down the async context, then don't
      create new workers if we already have at least the fixed one.
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      613eeb60
    • Jens Axboe's avatar
      io-wq: have manager wait for all workers to exit · fb3a1f6c
      Jens Axboe authored
      Instead of having to wait separately on workers and manager, just have
      the manager wait on the workers. We use an atomic_t for the reference
      here, as we need to start at 0 and allow increment from that. Since the
      number of workers is naturally capped by the allowed nr of processes,
      and that uses an int, there is no risk of overflow.
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      fb3a1f6c
  2. 01 Mar, 2021 2 commits
  3. 28 Feb, 2021 7 commits
    • Linus Torvalds's avatar
      Merge tag 'ide-5.11-2021-02-28' of git://git.kernel.dk/linux-block · d346da4a
      Linus Torvalds authored
      Pull ide fix from Jens Axboe:
       "This is a leftover fix from 5.11, where I forgot to ship it your way"
      
      * tag 'ide-5.11-2021-02-28' of git://git.kernel.dk/linux-block:
        ide/falconide: Fix module unload
      d346da4a
    • Linus Torvalds's avatar
      Merge tag 'kbuild-fixes-v5.12' of... · 06d5d309
      Linus Torvalds authored
      Merge tag 'kbuild-fixes-v5.12' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild
      
      Pull Kbuild fixes from Masahiro Yamada:
      
       - Fix UNUSED_KSYMS_WHITELIST for Clang LTO
      
       - Make -s builds really silent irrespective of V= option
      
       - Fix build error when SUBLEVEL or PATCHLEVEL is empty
      
      * tag 'kbuild-fixes-v5.12' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild:
        kbuild: Fix <linux/version.h> for empty SUBLEVEL or PATCHLEVEL again
        kbuild: make -s option take precedence over V=1
        ia64: remove redundant READELF from arch/ia64/Makefile
        kbuild: do not include include/config/auto.conf from adjust_autoksyms.sh
        kbuild: fix UNUSED_KSYMS_WHITELIST for Clang LTO
        kbuild: lto: add _mcount to list of used symbols
      06d5d309
    • Linus Torvalds's avatar
      Merge tag 'csky-for-linus-5.12-rc1' of git://github.com/c-sky/csky-linux · cd278456
      Linus Torvalds authored
      Pull arch/csky updates from Guo Ren:
       "Features:
         - add new memory layout 2.5G(user):1.5G(kernel)
         - add kmemleak support
         - reconstruct VDSO framework: add VDSO with GENERIC_GETTIMEOFDAY,
           GENERIC_TIME_VSYSCALL, HAVE_GENERIC_VDSO
         - add faulthandler_disabled() check
         - support (fix) swapon
         - add (fix) _PAGE_ACCESSED for default pgprot
         - abort uaccess retries upon fatal signal (from arm)
      
        Fixes and optimizations:
         - fix perf probe failure
         - fix show_regs doesn't contain regs->usp
         - remove custom asm/atomic.h implementation
         - fix barrier design
         - fix futex SMP implementation
         - fix asm/cmpxchg.h with correct ordering barrier
         - cleanup asm/spinlock.h
         - fix PTE global for 2.5:1.5 virtual memory
         - remove prologue of page fault handler in entry.S
         - fix TLB maintenance synchronization problem
         - add show_tlb for CPU_CK860 debug
         - fix FAULT_FLAG_XXX param for handle_mm_fault
         - fix update_mmu_cache called with user io mapping
         - fix do_page_fault parent irq status
         - fix a size determination in gpr_get()
         - pgtable.h: Coding convention
         - kprobe: Fix code in simulate without 'long'
         - fix pfn_valid error with wrong max_mapnr
         - use free_initmem_default() in free_initmem()
         - fix compile error"
      
      * tag 'csky-for-linus-5.12-rc1' of git://github.com/c-sky/csky-linux: (30 commits)
        csky: Fixup compile error
        csky: use free_initmem_default() in free_initmem()
        csky: Fixup pfn_valid error with wrong max_mapnr
        csky: Add VDSO with GENERIC_GETTIMEOFDAY, GENERIC_TIME_VSYSCALL, HAVE_GENERIC_VDSO
        csky: kprobe: Fixup code in simulate without 'long'
        csky: Fixup swapon
        csky: pgtable.h: Coding convention
        csky: Fixup _PAGE_ACCESSED for default pgprot
        csky: remove unused including <linux/version.h>
        csky: Fix a size determination in gpr_get()
        csky: Reconstruct VDSO framework
        csky: mm: abort uaccess retries upon fatal signal
        csky: Sync riscv mm/fault.c for easy maintenance
        csky: Fixup do_page_fault parent irq status
        csky: Add faulthandler_disabled() check
        csky: Fixup update_mmu_cache called with user io mapping
        csky: Fixup FAULT_FLAG_XXX param for handle_mm_fault
        csky: Add show_tlb for CPU_CK860 debug
        csky: Fix TLB maintenance synchronization problem
        csky: Add kmemleak support
        ...
      cd278456
    • Linus Torvalds's avatar
      Merge tag 'riscv-for-linus-5.12-mw1' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux · 7d19ea5e
      Linus Torvalds authored
      Pull more RISC-V updates from Palmer Dabbelt:
       "A pair of patches that slipped through the cracks:
      
         - enable CPU hotplug in the defconfigs
      
         - some cleanups to setup_bootmem
      
        There's also a single fix for some randconfig build failures:
      
         - make NUMA depend on SMP"
      
      * tag 'riscv-for-linus-5.12-mw1' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux:
        riscv: Cleanup setup_bootmem()
        RISC-V: Enable CPU Hotplug in defconfigs
        RISC-V: Make NUMA depend on SMP
      7d19ea5e
    • Linus Torvalds's avatar
      Merge tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi · 0b311e34
      Linus Torvalds authored
      Pull more SCSI updates from James Bottomley:
       "This is a few driver updates (iscsi, mpt3sas) that were still in the
        staging queue when the merge window opened (all committed on or before
        8 Feb) and some small bug fixes which came in during the merge window
        (all committed on 22 Feb)"
      
      * tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi: (30 commits)
        scsi: hpsa: Correct dev cmds outstanding for retried cmds
        scsi: sd: Fix Opal support
        scsi: target: tcmu: Fix memory leak caused by wrong uio usage
        scsi: target: tcmu: Move some functions without code change
        scsi: sd: sd_zbc: Don't pass GFP_NOIO to kvcalloc
        scsi: aic7xxx: Remove unused function pointer typedef ahc_bus_suspend/resume_t
        scsi: bnx2fc: Fix Kconfig warning & CNIC build errors
        scsi: ufs: Fix a duplicate dev quirk number
        scsi: aic79xx: Fix spelling of version
        scsi: target: core: Prevent underflow for service actions
        scsi: target: core: Add cmd length set before cmd complete
        scsi: iscsi: Drop session lock in iscsi_session_chkready()
        scsi: qla4xxx: Use iscsi_is_session_online()
        scsi: libiscsi: Reset max/exp cmdsn during recovery
        scsi: iscsi_tcp: Fix shost can_queue initialization
        scsi: libiscsi: Add helper to calculate max SCSI cmds per session
        scsi: libiscsi: Fix iSCSI host workq destruction
        scsi: libiscsi: Fix iscsi_task use after free()
        scsi: libiscsi: Drop taskqueuelock
        scsi: libiscsi: Fix iscsi_prep_scsi_cmd_pdu() error handling
        ...
      0b311e34
    • Linus Torvalds's avatar
      Merge tag 'xfs-5.12-merge-6' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux · 03dc748b
      Linus Torvalds authored
      Pull more xfs updates from Darrick Wong:
       "The most notable fix here prevents premature reuse of freed metadata
        blocks, and adding the ability to detect accidental nested
        transactions, which are not allowed here.
      
         - Restore a disused sysctl control knob that was inadvertently
           dropped during the merge window to avoid fstests regressions.
      
         - Don't speculatively release freed blocks from the busy list until
           we're actually allocating them, which fixes a rare log recovery
           regression.
      
         - Don't nest transactions when scanning for free space.
      
         - Add an idiot^Wmaintainer light to detect nested transactions. ;)"
      
      * tag 'xfs-5.12-merge-6' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux:
        xfs: use current->journal_info for detecting transaction recursion
        xfs: don't nest transactions when scanning for eofblocks
        xfs: don't reuse busy extents on extent trim
        xfs: restore speculative_cow_prealloc_lifetime sysctl
      03dc748b
    • Linus Torvalds's avatar
      Merge tag 'block-5.12-2021-02-27' of git://git.kernel.dk/linux-block · 3ab6608e
      Linus Torvalds authored
      Pull more block updates from Jens Axboe:
       "A few stragglers (and one due to me missing it originally), and fixes
        for changes in this merge window mostly. In particular:
      
         - blktrace cleanups (Chaitanya, Greg)
      
         - Kill dead blk_pm_* functions (Bart)
      
         - Fixes for the bio alloc changes (Christoph)
      
         - Fix for the partition changes (Christoph, Ming)
      
         - Fix for turning off iopoll with polled IO inflight (Jeffle)
      
         - nbd disconnect fix (Josef)
      
         - loop fsync error fix (Mauricio)
      
         - kyber update depth fix (Yang)
      
         - max_sectors alignment fix (Mikulas)
      
         - Add bio_max_segs helper (Matthew)"
      
      * tag 'block-5.12-2021-02-27' of git://git.kernel.dk/linux-block: (21 commits)
        block: Add bio_max_segs
        blktrace: fix documentation for blk_fill_rw()
        block: memory allocations in bounce_clone_bio must not fail
        block: remove the gfp_mask argument to bounce_clone_bio
        block: fix bounce_clone_bio for passthrough bios
        block-crypto-fallback: use a bio_set for splitting bios
        block: fix logging on capacity change
        blk-settings: align max_sectors on "logical_block_size" boundary
        block: reopen the device in blkdev_reread_part
        block: don't skip empty device in in disk_uevent
        blktrace: remove debugfs file dentries from struct blk_trace
        nbd: handle device refs for DESTROY_ON_DISCONNECT properly
        kyber: introduce kyber_depth_updated()
        loop: fix I/O error on fsync() in detached loop devices
        block: fix potential IO hang when turning off io_poll
        block: get rid of the trace rq insert wrapper
        blktrace: fix blk_rq_merge documentation
        blktrace: fix blk_rq_issue documentation
        blktrace: add blk_fill_rwbs documentation comment
        block: remove superfluous param in blk_fill_rwbs()
        ...
      3ab6608e