->task_complete was added to serialised CQE posting by doing it from
the task context only (or fallback wq when the task is dead), and now we
can use that to avoid taking ->completion_lock while filling CQ entries.
The patch skips spinlocking only in two spots,
__io_submit_flush_completions() and flushing in io_aux_cqe, it's safer
and covers all cases we care about. Extra care is taken to force taking
the lock while queueing overflow entries.

It fundamentally relies on SINGLE_ISSUER to have only one task posting
events. It also need to take into account overflowed CQEs, flushing of
which happens in the cq wait path, and so this implementation also needs
DEFER_TASKRUN to limit waiters. For the same reason we disable it for
SQPOLL, and for IOPOLL as it won't benefit from it in any case.
DEFER_TASKRUN, SQPOLL and IOPOLL requirement may be relaxed in the
future.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/2a8c91fd82cfcdcc1d2e5bac7051fe2c183bda73.1670384893.git.asml.silence@gmail.com
[axboe: modify to apply]
Signed-off-by: Jens Axboe <axboe@kernel.dk>

While executing in a context of one io_uring instance msg_ring
manipulates another ring. We're trying to keep CQEs posting contained in
the context of the ring-owner task, use task_work to send the request to
the target ring's task when we're modifying its CQ or trying to install
a file. Note, we can't safely use io_uring task_work infra and have to
use task_work directly.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/4d76c7b28ed5d71b520de4482fbb7f660f21cd80.1670384893.git.asml.silence@gmail.com
[axboe: use TWA_SIGNAL_NO_IPI]
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Extract a helper called io_msg_install_complete() from io_msg_send_fd(),
will be used later.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/1500ca1054cc4286a3ee1c60aacead57fcdfa02a.1670384893.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

We don't need to take both uring_locks at once, msg_ring can be split in
two parts, first getting a file from the filetable of the first ring and
then installing it into the second one.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/a80ecc2bc99c3b3f2cf20015d618b7c51419a797.1670384893.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

Once we fallback a tw we want all requests to that task to be given to
the fallback wq so we dont run it in parallel with the last, i.e. post
PF_EXITING, tw run of the task.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/96f4987265c4312f376f206511c6af3e77aaf5ac.1670384893.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

Use task_work for completing rsrc removals, it'll be needed later for
spinlock optimisations.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/cbba5d53a11ee6fc2194dacea262c1d733c8b529.1670384893.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

Multishot are posting CQEs outside of the normal request completion
path, which is usually done from within a task work handler. However, it
might be not the case when it's yet to be polled but has been punted to
io-wq. Make it abide ->task_complete and push it to the polling path
when executed by io-wq.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/d7714aaff583096769a0f26e8e747759e556feb1.1670384893.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

This patch adds ctx->task_complete flag. If set, we'll complete all
requests in the context of the original task. Note, this extends to
completion CQE posting only but not io_kiocb cleanup / free, e.g. io-wq
may free the requests in the free calllback. This flag will be used
later for optimisations purposes.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/21ece72953f76bb2e77659a72a14326227ab6460.1670384893.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

The only way to fail overflowed CQEs flush is for CQ to be fully packed.
There is one place checking for flush failures, i.e. io_cqring_wait(),
but we limit the number to be waited for by the CQ size, so getting a
failure automatically means that we're done with waiting.

Don't check for failures, rarely but they might spuriously fail CQ
waiting with -EBUSY.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/6b720a45c03345655517f8202cbd0bece2848fb2.1670384893.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

After io_ring_ctx_wait_and_kill() is called there should be no users
poking into rings and so there is no need to post CQEs. So, instead of
trying to post overflowed CQEs into the CQ, drop them. Also, do it
in io_ring_exit_work() in a loop to reduce the number of contexts it
can be executed from and even when it struggles to quiesce the ring we
won't be leaving memory allocated for longer than needed.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/26d13751155a735a3029e24f8d9ca992f810419d.1670384893.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

msg_ring will fail the request if it can't lock rings, instead punt it
to io-wq as was originally intended.

Cc: stable@vger.kernel.org
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/4697f05afcc37df5c8f89e2fe6d9c7c19f0241f9.1670384893.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

We should not be messing with req->file outside of core paths. Clearing
it makes msg_ring non reentrant, i.e. luckily io_msg_send_fd() fails the
request on failed io_double_lock_ctx() but clearly was originally
intended to do retries instead.

Cc: stable@vger.kernel.org
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/e5ac9edadb574fe33f6d727cb8f14ce68262a684.1670384893.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

Reshuffle issue flags to keep normal flags separate from the uring_cmd
ctx-setup like flags. Shift the second type to the second byte so it's
easier to add new ones in the future.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/d6e4696c883943082d248716f4cd568f37b17a74.1669821213.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

There is no need to reinit data and install a new rsrc node every time
we get a task_work, it's detrimental, just execute it and conitnue
waiting.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/3895d3344164cd9b3a0bbb24a6e357e20a13434b.1669821213.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

Do a little bit of refactoring of io_rsrc_ref_quiesce(), flatten the
data refs checks and so get rid of a conditional weird unlock-else-break
construct.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/d21283e9f88a77612c746ed526d86fe3bfb58a70.1669821213.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

There is one newly added place when we lock ring with io_cq_lock() but
unlocking is hand coded calling spin_unlock directly. It's ugly and
troublesome in the long run. Make it consistent with the other completion
locking.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/4ca4f0564492b90214a190cd5b2a6c76522de138.1669821213.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

Merge apoll and regular poll tw handlers, it will help with inlining.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/482e59edb9fc81bd275fdbf486837330fb27120a.1669821213.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

Don't try to complete requests if their refs are broken and we've got
a warning, it's much better to drop them and potentially leaking than
double freeing.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/31edf9f96f05d03ab62c114508a231a2dce434cb.1669821213.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

ctx is only used by io_poll_check_events() for multishot poll CQE
posting, don't save it on stack in advance.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/552c1771f8a0e7688afdb4f538ead245f53e80e7.1669821213.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

The fast path in io_poll_check_events() is when we have only one
(i.e. master) reference. Move all verification, cancellations
checks, edge case handling and so on under a common if.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/8c21c5d5e027e32dc553705e88796dec79ff6f93.1669821213.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

We don't need to worry about checking PF_EXITING in io_poll_issue().
task works using the function should take care of it and never try to
resubmit / retry if the task is dying.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/2e9dc998dc07507c759a0c9cb5d2fbea0710d58c.1669821213.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

* for-6.2/io_uring: (41 commits)
  io_uring: keep unlock_post inlined in hot path
  io_uring: don't use complete_post in kbuf
  io_uring: spelling fix
  io_uring: remove io_req_complete_post_tw
  io_uring: allow multishot polled reqs to defer completion
  io_uring: remove overflow param from io_post_aux_cqe
  io_uring: add lockdep assertion in io_fill_cqe_aux
  io_uring: make io_fill_cqe_aux static
  io_uring: add io_aux_cqe which allows deferred completion
  io_uring: allow defer completion for aux posted cqes
  io_uring: defer all io_req_complete_failed
  io_uring: always lock in io_apoll_task_func
  io_uring: remove iopoll spinlock
  io_uring: iopoll protect complete_post
  io_uring: inline __io_req_complete_put()
  io_uring: remove io_req_tw_post_queue
  io_uring: use io_req_task_complete() in timeout
  io_uring: hold locks for io_req_complete_failed
  io_uring: add completion locking for iopoll
  io_uring: kill io_cqring_ev_posted() and __io_cq_unlock_post()
  ...

Pull vfs fix from Al Viro:
 "Amir's copy_file_range() fix"

* tag 'pull-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs:
  vfs: fix copy_file_range() averts filesystem freeze protection

Pull USB fixes from Greg KH:
 "Here are some small USB fixes for 6.1-rc7 that resolve some reported
  problems:

   - cdnsp driver fixes for reported problems

   - dwc3 fixes for some small reported problems

   - uvc gadget driver fix for reported regression

  All of these have been in linux-next with no reported problems"

* tag 'usb-6.1-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb:
  usb: cdnsp: fix issue with ZLP - added TD_SIZE = 1
  usb: dwc3: gadget: Clear ep descriptor last
  usb: dwc3: exynos: Fix remove() function
  usb: cdnsp: Fix issue with Clear Feature Halt Endpoint
  usb: dwc3: gadget: Disable GUSB2PHYCFG.SUSPHY for End Transfer
  usb: gadget: uvc: also use try_format in set_format

Pull char/misc driver fixes from Greg KH:
 "Here are some small driver fixes for 6.1-rc7, they include:

   - build warning fix for the vdso when using new versions of grep

   - iio driver fixes for reported issues

   - small nvmem driver fixes

   - fpga Kconfig fix

   - interconnect dt binding fix

  All of these have been in linux-next with no reported issues"

* tag 'char-misc-6.1-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc:
  lib/vdso: use "grep -E" instead of "egrep"
  nvmem: lan9662-otp: Change return type of lan9662_otp_wait_flag_clear()
  nvmem: rmem: Fix return value check in rmem_read()
  fpga: m10bmc-sec: Fix kconfig dependencies
  dt-bindings: iio: adc: Remove the property "aspeed,trim-data-valid"
  iio: adc: aspeed: Remove the trim valid dts property.
  iio: core: Fix entry not deleted when iio_register_sw_trigger_type() fails
  iio: accel: bma400: Fix memory leak in bma400_get_steps_reg()
  iio: light: rpr0521: add missing Kconfig dependencies
  iio: health: afe4404: Fix oob read in afe4404_[read|write]_raw
  iio: health: afe4403: Fix oob read in afe4403_read_raw
  iio: light: apds9960: fix wrong register for gesture gain
  dt-bindings: interconnect: qcom,msm8998-bwmon: Correct SC7280 CPU compatible

Pull timer fix from Borislav Petkov:

 - Return the proper timer register width (31 bits) for a 32-bit signed
   register in order to avoid a timer interrupt storm on ARM XGene-1
   hardware running in NO_HZ mode

* tag 'timers_urgent_for_v6.1_rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  clocksource/drivers/arm_arch_timer: Fix XGene-1 TVAL register math error

Pull objtool fix from Borislav Petkov:

 - Handle different output of readelf on different distros running
   ppc64le which confuses faddr2line's function offsets conversion

* tag 'objtool_urgent_for_v6.1_rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  scripts/faddr2line: Fix regression in name resolution on ppc64le

Pull x86 fixes from Borislav Petkov:

 - ioremap: mask out the bits which are not part of the physical address
   *after* the size computation is done to prevent any hypothetical
   ioremap failures

 - Change the MSR save/restore functionality during suspend to rely on
   flags denoting that the related MSRs are actually supported vs
   reading them and assuming they are (an Atom one allows reading but
   not writing, thus breaking this scheme at resume time)

 - prevent IV reuse in the AES-GCM communication scheme between SNP
   guests and the AMD secure processor

* tag 'x86_urgent_for_v6.1_rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  x86/ioremap: Fix page aligned size calculation in __ioremap_caller()
  x86/pm: Add enumeration check before spec MSRs save/restore setup
  x86/tsx: Add a feature bit for TSX control MSR support
  virt/sev-guest: Prevent IV reuse in the SNP guest driver

Pull perf fixes from Borislav Petkov:
 "Two more fixes to the perf sigtrap handling:

   - output the address in the sample only when it has been requested

   - handle the case where user-only events can hit in kernel and thus
     upset the sigtrap sanity checking"

* tag 'perf_urgent_for_v6.1_rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  perf: Consider OS filter fail
  perf: Fixup SIGTRAP and sample_flags interaction

Pull kvm fixes from Paolo Bonzini:
 "x86:

   - Fixes for Xen emulation. While nobody should be enabling it in the
     kernel (the only public users of the feature are the selftests),
     the bug effectively allows userspace to read arbitrary memory.

   - Correctness fixes for nested hypervisors that do not intercept INIT
     or SHUTDOWN on AMD; the subsequent CPU reset can cause a
     use-after-free when it disables virtualization extensions. While
     downgrading the panic to a WARN is quite easy, the full fix is a
     bit more laborious; there are also tests. This is the bulk of the
     pull request.

   - Fix race condition due to incorrect mmu_lock use around
     make_mmu_pages_available().

  Generic:

   - Obey changes to the kvm.halt_poll_ns module parameter in VMs not
     using KVM_CAP_HALT_POLL, restoring behavior from before the
     introduction of the capability"

* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm:
  KVM: Update gfn_to_pfn_cache khva when it moves within the same page
  KVM: x86/xen: Only do in-kernel acceleration of hypercalls for guest CPL0
  KVM: x86/xen: Validate port number in SCHEDOP_poll
  KVM: x86/mmu: Fix race condition in direct_page_fault
  KVM: x86: remove exit_int_info warning in svm_handle_exit
  KVM: selftests: add svm part to triple_fault_test
  KVM: x86: allow L1 to not intercept triple fault
  kvm: selftests: add svm nested shutdown test
  KVM: selftests: move idt_entry to header
  KVM: x86: forcibly leave nested mode on vCPU reset
  KVM: x86: add kvm_leave_nested
  KVM: x86: nSVM: harden svm_free_nested against freeing vmcb02 while still in use
  KVM: x86: nSVM: leave nested mode on vCPU free
  KVM: Obey kvm.halt_poll_ns in VMs not using KVM_CAP_HALT_POLL
  KVM: Avoid re-reading kvm->max_halt_poll_ns during halt-polling
  KVM: Cap vcpu->halt_poll_ns before halting rather than after

Pull cifs fixes from Steve French:
 "Two small cifs/smb3 client fixes:

   - an unlock missing in an error path in copychunk_range found by
     xfstest 476

   - a fix for a use after free in a debug code path"

* tag '6.1-rc6-smb3-fixes' of git://git.samba.org/sfrench/cifs-2.6:
  cifs: fix missing unlock in cifs_file_copychunk_range()
  cifs: Use after free in debug code

Merge tag 'kbuild-fixes-v6.1-4' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild

Pull Kbuild fixes from Masahiro Yamada:

 - Fix CC_HAS_ASM_GOTO_TIED_OUTPUT test in Kconfig

 - Fix noisy "No such file or directory" message when
   KBUILD_BUILD_VERSION is passed

 - Include rust/ in source tarballs

 - Fix missing FORCE for ARCH=nios2 builds

* tag 'kbuild-fixes-v6.1-4' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild:
  nios2: add FORCE for vmlinuz.gz
  scripts: add rust in scripts/Makefile.package
  kbuild: fix "cat: .version: No such file or directory"
  init/Kconfig: fix CC_HAS_ASM_GOTO_TIED_OUTPUT test with dash

Add FORCE to placate a warning from make:

arch/nios2/boot/Makefile:24: FORCE prerequisite is missing

Fixes: 2fc8483f ("nios2: Build infrastructure")
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Reviewed-by: Masahiro Yamada <masahiroy@kernel.org>

Pull nfsd fix from Chuck Lever:

 - Fix rare data corruption on READ operations

* tag 'nfsd-6.1-6' of git://git.kernel.org/pub/scm/linux/kernel/git/cel/linux:
  NFSD: Fix reads with a non-zero offset that don't end on a page boundary

Pull power supply fixes from Sebastian Reichel:

 - rk817: Two error handling fixes

 - ip5xxx: fix inter overflow in current calculation

 - ab8500: fix thermal zone probing

* tag 'for-v6.1-rc' of git://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-power-supply:
  power: supply: ab8500: Defer thermal zone probe
  power: supply: ip5xxx: Fix integer overflow in current_now calculation
  power: supply: rk817: Change rk817_chg_cur_to_reg to int
  power: supply: rk817: check correct variable

Pull block fixes from Jens Axboe:

 - A few fixes for s390 sads (Stefan, Colin)

 - Ensure that ublk doesn't reorder requests, as that can be problematic
   on devices that need specific ordering (Ming)

 - Fix a queue reference leak in disk allocation handling (Christoph)

* tag 'block-6.1-2022-11-25' of git://git.kernel.dk/linux:
  ublk_drv: don't forward io commands in reserve order
  s390/dasd: fix possible buffer overflow in copy_pair_show
  s390/dasd: fix no record found for raw_track_access
  s390/dasd: increase printing of debug data payload
  s390/dasd: Fix spelling mistake "Ivalid" -> "Invalid"
  blk-mq: fix queue reference leak on blk_mq_alloc_disk_for_queue failure

Pull io_uring fixes from Jens Axboe:

 - A few poll related fixes. One fixing a race condition between poll
   cancelation and trigger, and one making the overflow handling a bit
   more robust (Lin, Pavel)

 - Fix an fput() for error handling in the direct file table (Lin)

 - Fix for a regression introduced in this cycle, where we don't always
   get TIF_NOTIFY_SIGNAL cleared appropriately (me)

* tag 'io_uring-6.1-2022-11-25' of git://git.kernel.dk/linux:
  io_uring: clear TIF_NOTIFY_SIGNAL if set and task_work not available
  io_uring/poll: fix poll_refs race with cancelation
  io_uring/filetable: fix file reference underflow
  io_uring: make poll refs more robust
  io_uring: cmpxchg for poll arm refs release

Pull zonefs fixes from Damien Le Moal:

 - Fix a race between zonefs module initialization of sysfs attribute
   directory and mounting a drive (from Xiaoxu).

 - Fix active zone accounting in the rare case of an IO error due to a
   zone transition to offline or read-only state (from me).

* tag 'zonefs-6.1-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/dlemoal/zonefs:
  zonefs: Fix active zone accounting
  zonefs: Fix race between modprobe and mount

Merge tag 'regulator-fix-v6.1-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regulator

Pull regulator fixes from Mark Brown:
 "This is more changes than I'd like this late although the diffstat is
  still fairly small, I kept on holding off as new fixes came in to give
  things time to soak in -next but should probably have tagged and sent
  an additional pull request earlier.

  There's some relatively large fixes to the twl6030 driver to fix
  issues with the TWL6032 variant which resulted from some work on the
  core TWL6030 driver, a couple of fixes for error handling paths
  (mostly in the core), and a nice stability fix for the sgl51000 driver
  that's been pulled out of a BSP"

* tag 'regulator-fix-v6.1-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regulator:
  regulator: twl6030: fix get status of twl6032 regulators
  regulator: twl6030: re-add TWL6032_SUBCLASS
  regulator: slg51000: Wait after asserting CS pin
  regulator: core: fix UAF in destroy_regulator()
  regulator: rt5759: fix OOB in validate_desc()
  regulator: core: fix kobject release warning and memory leak in regulator_register()