It turns out the *_defined entries of bch_io_opts are only used in one
place - in the xattr get path - and there we immediately convert to a
bch_opts struct, which also has the *_defined entries.

This patch changes bch2_inode_opts_to_opts() to go directly from
bch_inode_unpacked to bch_opts, which is a minor simplification and will
also let us slim down struct bch_io_opts in another patch.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

This provides an inlined version of bch2_subvolume_get() and uses it in
bch2_subvolume_get_snapshot(), since this is the version that's used all
over the place and in fast paths (e.g. IO paths).
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

This is only called in two places, and when it's used we use it in a
tight loop - it's definitely worth inlining.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

This just removes a redundant comparison - there's more work we could do
here to remove some redundant copying.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

Convert some non-critical asserts in long-stable code to debug asserts.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

Standard splitting out of the slow path from the fast path of a
function. We may follow this up in another patch with inlining the fast
path into btree_iter.c.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

If the last journal write didn't complete sucessfully due to a torn
write, we'll detect it as a checksum error. In that case, we should just
pretend that journal entry was never written.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

Print out the journal entries we read and will replay as soon as
possible - if we get an error walidating keys it's helpful to know where
it was in the journal.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

It turns out we need bch2_extent_trim_atomi() even when we're deleting
extents one at a time because it's possible for one reflink_p to
reference arbitrarily many reflink_v extents. This doesn't normally
happen, but the data move path can fragment existing extents in the
background.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

b->write_type needs to be set atomically with setting the
btree_node_need_write flag, so move it into b->flags.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

 - Centralize format strings in bcachefs.h
 - Add bch2_fmt_inum_offset() and related helpers
 - Switch error messages for inodes to also print out the offset, in
   bytes
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

Warnings ought to always have a format string/log message - makes them
considerably more useful.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

Previously, when we exited from the loop body with a break statement
_ret wouldn't have been assigned to yet, and we could spuriously return
a transaction restart error.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

This improves the bkey_format calculation when splitting btree nodes.
Previously, we'd use a format calculated for the original node for the
lower of the two new nodes.

This was particularly bad on sequential insertions, where we iteratively
split the last btree node, whos format has to include KEY_MAX.

Now, we calculate formats precisely for the keys the two new nodes will
contain. This also should make splitting a bit more efficient, since
we're only copying keys once (from the original node to the new node,
instead of new node, replacement node, then upper split).
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

trans->restarted is a positive error code, not the usual negative
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

This switches where we take quota reservations to be per bch_wirte_op
instead of per dio_write, so we can drop the quota reservation in the
same place as we call i_sectors_acct(), and only take/release
ei_quota_lock once.

In the future we'd like ei_quota_lock to not be a mutex, so that we can
avoid punting to process context before deliving write completions in
nocow mode.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

The genradix code can handle multiple threads trying to allocate at the
same time - we don't need the genradix_ptr_alloc() call to happen under
a lock.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

This fixes a regression from percpu freedlists in the btree key cache
code: in a rare error path, we were immediately freeing a bkey_cached
that had been used before and should've waited for an SRCU barrier.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

These were wrappers around atomic operations that verified that the
counter wasn't negative, but they're dead code - delete.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

 - Marking a non-static function as inline doesn't actually work and is
   now causing problems - drop that

 - Introduce BCACHEFS_LOG_PREFIX for when we want to prefix log messages
   with bcachefs (filesystem name)

 - Userspace doesn't have real percpu variables (maybe we can get this
   fixed someday), put an #ifdef around bch2_disk_reservation_add()
   fastpath
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

We have a unique lock used for controlling adding to the pagecache: the
lock has two states, where both states are shared - the lock may be held
multiple times for either state - but not both states at the same time.

This is exactly what we need for nocow mode locking, so this patch pulls
it out of fs.c into its own file.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

BCH_WRITE_FLUSH is a write flag that causes a journal flush.  It's only
used in the direct IO path, and this will allow for some consolidation
with the regular fsync path, which will help with the upcoming nocow
mode.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

 - factor out more slowpath code into non-inline function
 - use bch2_print_string_as_lines(), so our error message doesn't get
   truncated
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

Only used in one place, just inline it there.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

btree_path_copy() doesn't need to call
bch2_btree_path_check_sort_fast() - the newly allocated path will always
be in the correct position, post copy; also delete some redundant
branches from __bch2_btree_path_make_mut().
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

 - Don't call into bch2_encrypt_bio() when we're not encrypting
 - Pull slowpath out of trans_lock_write()
 - Make sure bc2h_trans_journal_res_get() gets inlined.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

 - With BCH_WRITE_SYNC, we no longer need the completion in struct
   dio_write
 - Pull out bch2_dio_write_copy_iov() into a separate non-inline
   function, it's code that doesn't run in the common case
 - Copy mapping and inode pointers into dio_write, avoiding pointer
   chasing at the start of bch2_dio_write_loop()
 - kthread_use_mm() is not needed in the common case; move it into
   bch2_dio_write_loop_async()
 - factor out various helpers from bch2_dio_write_loop() and rework
   control flow for better icache utilization

Other small optimizations:

 - bch2_keylist_free() is only used in one place, at the end of the
   bch2_write() path - drop the reinit
 - in bch2_disk_reservation_put(), check if res->sectors is nonzero
   before touching c->online_reserved, since that will likely be a cache
   miss
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

bcachefs: More DIO write path optimization

Better code prefetching (?)
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

This adds a new flag for the write path, BCH_WRITE_SYNC, and switches
the O_DIRECT write path to use it when we're not running asynchronously.

It runs the btree update after the write in the original thread's
context instead of a kworker, cutting context switches in half.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

Fixes for various checkpatch errors.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

Dead code, delete.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

This factors out a properly-documented helper for deciding when we want
to sort a btree node with MAX_BSETS bsets down to a single bset.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

This replaces sysfs btree_avg_write_size with btree_write_stats, which
now breaks out statistics by the source of the btree write.

Btree writes that are too small are a source of inefficiency, and
excessive btree resort overhead - this will let us see what's causing
them.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

Fixes fstests generic/648
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

Per fstests generic/275, on -ENOSPC we're supposed write until the
filesystem is full - i.e. do a partial write instead of failing the full
write.

This is a partial fix for the buffered write path: we'll still fail on a
page boundary.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

 - In the btree iterator code that overlays keys from the journal, we
   were incorrectly specifying level=0 instead of the btree_path's
   current level in a few places
 - When we didn't do journal replay, we shouldn't free the journal keys:
   this fixes cmd_list and cmd_dump, which run in norecovery mode
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

roundup_pow_of_two() is undefined for 0 - oops.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

Use __func__ in error messages that refer to function name, and do so
more uniformly - suggested by checkpatch.pl
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

This adds an inlined version of bch2_bkey_cmp_packed(), and uses it in
bch2_sort_keys(), where it's part of the inner loop.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>