bcachefs: Fix a write buffer flush deadlock

We're not supposed to block if BTREE_INSERT_JOURNAL_RECLAIM && watermark
!= BCH_WATERMARK_reclaim.

This should really be a separate BTREE_INSERT_NONBLOCK flag - add some
comments to that effect, it's not important for this patch.

btree write buffer flush depends on this behaviour though - the first
loop tries to flush sequentially, which doesn't free up space in the
journal optimally. If that can't proceed we bail out and flush in
journal order - that won't work if we're blocked instead of returning an
error.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

fs/bcachefs/btree_update_interior.c

@@ -1158,6 +1158,17 @@ bch2_btree_update_start(struct btree_trans *trans, struct btree_path *path,
 	    bch2_err_matches(ret, ENOMEM)) {
 		struct closure cl;
 
+		/*
+		 * XXX: this should probably be a separate BTREE_INSERT_NONBLOCK
+		 * flag
+		 */
+		if (bch2_err_matches(ret, ENOSPC) &&
+		    (flags & BTREE_INSERT_JOURNAL_RECLAIM) &&
+		    watermark != BCH_WATERMARK_reclaim) {
+			ret = -BCH_ERR_journal_reclaim_would_deadlock;
+			goto err;
+		}
+
 		closure_init_stack(&cl);
 
 		do {

fs/bcachefs/btree_update_leaf.c

@@ -958,6 +958,10 @@ int bch2_trans_commit_error(struct btree_trans *trans, unsigned flags,
 			bch2_replicas_delta_list_mark(c, trans->fs_usage_deltas));
 		break;
 	case -BCH_ERR_journal_res_get_blocked:
+		/*
+		 * XXX: this should probably be a separate BTREE_INSERT_NONBLOCK
+		 * flag
+		 */
 		if ((flags & BTREE_INSERT_JOURNAL_RECLAIM) &&
 		    (flags & BCH_WATERMARK_MASK) != BCH_WATERMARK_reclaim) {
 			ret = -BCH_ERR_journal_reclaim_would_deadlock;