- 15 Mar, 2024 1 commit
-
-
Darrick J. Wong authored
Fix some inconsistencies in the xmbuf tracepoints -- they should be reporting the major/minor of the filesystem that they're associated with, so that we have some clue on whose behalf the xmbuf was created. Fix the xmbuf_free tracepoint to report the same. Don't call the trace function until the xmbuf is fully initialized. Fixes: 5076a604 ("xfs: support in-memory buffer cache target") Signed-off-by:
"Darrick J. Wong" <djwong@kernel.org> Reviewed-by:
Christoph Hellwig <hch@lst.de> Signed-off-by:
Chandan Babu R <chandanbabu@kernel.org>
-
- 07 Mar, 2024 1 commit
-
-
Dave Chinner authored
Chandan reported a AGI/AGF lock order hang on xfs/168 during recent testing. The cause of the problem was the task running xfs_growfs to shrink the filesystem. A failure occurred trying to remove the free space from the btrees that the shrink would make disappear, and that meant it ran the error handling for a partial failure. This error path involves restoring the per-ag block reservations, and that requires calculating the amount of space needed to be reserved for the free inode btree. The growfs operation hung here: [18679.536829] down+0x71/0xa0 [18679.537657] xfs_buf_lock+0xa4/0x290 [xfs] [18679.538731] xfs_buf_find_lock+0xf7/0x4d0 [xfs] [18679.539920] xfs_buf_lookup.constprop.0+0x289/0x500 [xfs] [18679.542628] xfs_buf_get_map+0x2b3/0xe40 [xfs] [18679.547076] xfs_buf_read_map+0xbb/0x900 [xfs] [18679.562616] xfs_trans_read_buf_map+0x449/0xb10 [xfs] [18679.569778] xfs_read_agi+0x1cd/0x500 [xfs] [18679.573126] xfs_ialloc_read_agi+0xc2/0x5b0 [xfs] [18679.578708] xfs_finobt_calc_reserves+0xe7/0x4d0 [xfs] [18679.582480] xfs_ag_resv_init+0x2c5/0x490 [xfs] [18679.586023] xfs_ag_shrink_space+0x736/0xd30 [xfs] [18679.590730] xfs_growfs_data_private.isra.0+0x55e/0x990 [xfs] [18679.599764] xfs_growfs_data+0x2f1/0x410 [xfs] [18679.602212] xfs_file_ioctl+0xd1e/0x1370 [xfs] trying to get the AGI lock. The AGI lock was held by a fstress task trying to do an inode allocation, and it was waiting on the AGF lock to allocate a new inode chunk on disk. Hence deadlock. The fix for this is for the growfs code to hold the AGI over the transaction roll it does in the error path. It already holds the AGF locked across this, and that is what causes the lock order inversion in the xfs_ag_resv_init() call. Reported-by:
Dave Chinner <dchinner@redhat.com> Reviewed-by:
Gao Xiang <hsiangkao@linux.alibaba.com> Reviewed-by:
-
- 29 Feb, 2024 2 commits
-
-
Akira Yokosawa authored
Use the form of *@param which kernel-doc recognizes now. This resolves the warnings from "make htmldocs" as reported in [1]. Reported-by:
Stephen Rothwell <sfr@canb.auug.org.au> Link: [1] https://lore.kernel.org/r/20240223153636.41358be5@canb.auug.org.au/Acked-by:
Akira Yokosawa <akiyks@gmail.com> Signed-off-by:
-
Akira Yokosawa authored
In kernel-doc comments, unary operator * collides with Sphinx/ docutil's markdown for emphasizing. This resulted in additional warnings from "make htmldocs": WARNING: Inline emphasis start-string without end-string. , as reported recently [1]. Those have been worked around either by escaping * (like \*param) or by using inline-literal form of ``*param``, both of which are specific to Sphinx/docutils. Such workarounds are against the kenrel-doc's ideal and should better be avoided. Instead, add "*" to the list of unary operators kernel-doc recognizes and make the form of *@param available in kernel-doc comments. Reported-by:Jonathan Corbet <corbet@lwn.net> Signed-off-by:
-
- 28 Feb, 2024 2 commits
-
-
Dave Chinner authored
The xfs_log_vec items are allocated by xlog_kvmalloc(), and so need to be freed with kvfree. This was missed when coverting from the kmem_free() API. Fixes: 49292576 ("xfs: convert kmem_free() for kvmalloc users to kvfree()") Reported-by:
-
Dave Chinner authored
This was missed in the conversion from KM* flags. Reported-by:
Dan Carpenter <dan.carpenter@linaro.org> Fixes: 10634530 ("xfs: convert kmem_zalloc() to kzalloc()") Signed-off-by:
-
- 26 Feb, 2024 1 commit
-
-
Darrick J. Wong authored
When the kernel is in lockdown mode, debugfs will only show files that are world-readable and cannot be written, mmaped, or used with ioctl. That more or less describes the scrub stats file, except that the permissions are wrong -- they should be 0444, not 0644. You can't write the stats file, so the 0200 makes no sense. Meanwhile, the clear_stats file is only writable, but it got mode 0400 instead of 0200, which would make more sense. Fix both files so that they make sense. Fixes: d7a74cad ("xfs: track usage statistics of online fsck") Signed-off-by:
-
- 24 Feb, 2024 19 commits
-
-
Darrick J. Wong authored
Per the comment in the error case of xfs_reflink_recover_cow, zero out any error (after shutting down the log) so that we actually kill any new intent items that might have gotten logged by later recovery steps. Discovered by xfs/434, which few people actually seem to run. Fixes: 2c1e31ed ("xfs: place intent recovery under NOFS allocation context") Signed-off-by:
-
Chandan Babu R authored
Merge tag 'symlink-cleanups-6.9_2024-02-23' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux into xfs-6.9-mergeC xfs: clean up symbolic link code This series cleans up a few bits of the symbolic link code as needed for future projects. Online repair requires the ability to commit fixed fork-based filesystem metadata such as directories, xattrs, and symbolic links atomically, so we need to rearrange the symlink code before we land the atomic extent swapping. Accomplish this by moving the remote symlink target block code and declarations to xfs_symlink_remote.[ch]. Signed-off-by:
-
Chandan Babu R authored
Merge tag 'expand-bmap-intent-usage_2024-02-23' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux into xfs-6.9-mergeC xfs: support attrfork and unwritten BUIs In preparation for atomic extent swapping and the online repair functionality that wants atomic extent swaps, enhance the BUI code so that we can support deferred work on the extended attribute fork and on unwritten extents. Signed-off-by:
-
Chandan Babu R authored
Merge tag 'realtime-bmap-intents-6.9_2024-02-23' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux into xfs-6.9-mergeC xfs: widen BUI formats to support realtime Atomic extent swapping (and later, reverse mapping and reflink) on the realtime device needs to be able to defer file mapping and extent freeing work in much the same manner as is required on the data volume. Make the BUI log items operate on rt extents in preparation for atomic swapping and realtime rmap. Signed-off-by:
-
Chandan Babu R authored
Merge tag 'bmap-intent-cleanups-6.9_2024-02-23' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux into xfs-6.9-mergeC xfs: bmap log intent cleanups The next major target of online repair are metadata that are persisted in blocks mapped by a file fork. In other words, we want to repair directories, extended attributes, symbolic links, and the realtime free space information. For file-based metadata, we assume that the space metadata is correct, which enables repair to construct new versions of the metadata in a temporary file. We then need to swap the file fork mappings of the two files atomically. With this patchset, we begin constructing such a facility based on the existing bmap log items and a new extent swap log item. This series cleans up a few parts of the file block mapping log intent code before we start adding support for realtime bmap intents. Most of it involves cleaning up tracepoints so that more of the data extraction logic ends up in the tracepoint code and not the tracepoint call site, which should reduce overhead further when tracepoints are disabled. There is also a change to pass bmap intents all the way back to the bmap code instead of unboxing the intent values and re-boxing them after the _finish_one function completes. Signed-off-by:
-
Chandan Babu R authored
Merge tag 'repair-refcount-scalability-6.9_2024-02-23' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux into xfs-6.9-mergeC xfs: reduce refcount repair memory usage The refcountbt repair code has serious memory usage problems when the block sharing factor of the filesystem is very high. This can happen if a deduplication tool has been run against the filesystem, or if the fs stores reflinked VM images that have been aging for a long time. Recall that the original reference counting algorithm walks the reverse mapping records of the filesystem to generate reference counts. For any given block in the AG, the rmap bag structure contains the all rmap records that cover that block; the refcount is the size of that bag. For online repair, the bag doesn't need the owner, offset, or state flag information, so it discards those. This halves the record size, but the bag structure still stores one excerpted record for each reverse mapping. If the sharing count is high, this will use a LOT of memory storing redundant records. In the extreme case, 100k mappings to the same piece of space will consume 100k*16 bytes = 1.6M of memory. For offline repair, the bag stores the owner values so that we know which inodes need to be marked as being reflink inodes. If a deduplication tool has been run and there are many blocks within a file pointing to the same physical space, this will stll use a lot of memory to store redundant records. The solution to this problem is to deduplicate the bag records when possible by adding a reference count to the bag record, and changing the bag add function to detect an existing record to bump the refcount. In the above example, the 100k mappings will now use 24 bytes of memory. These lookups can be done efficiently with a btree, so we create a new refcount bag btree type (inside of online repair). This is why we refactored the btree code in the previous patchset. The btree conversion also dramatically reduces the runtime of the refcount generation algorithm, because the code to delete all bag records that end at a given agblock now only has to delete one record instead of (using the example above) 100k records. As an added benefit, record deletion now gives back the unused xfile space, which it did not do previously. Signed-off-by:
-
Chandan Babu R authored
Merge tag 'repair-rmap-btree-6.9_2024-02-23' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux into xfs-6.9-mergeC xfs: online repair of rmap btrees We have now constructed the four tools that we need to scan the filesystem looking for reverse mappings: an inode scanner, hooks to receive live updates from other writer threads, the ability to construct btrees in memory, and a btree bulk loader. This series glues those three together, enabling us to scan the filesystem for mappings and keep it up to date while other writers run, and then commit the new btree to disk atomically. To reduce the size of each patch, the functionality is left disabled until the end of the series and broken up into three patches: one to create the mechanics of scanning the filesystem, a second to transition to in-memory btrees, and a third to set up the live hooks. Signed-off-by:
-
Chandan Babu R authored
Merge tag 'in-memory-btrees-6.9_2024-02-23' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux into xfs-6.9-mergeC xfs: support in-memory btrees Online repair of the reverse-mapping btrees presens some unique challenges. To construct a new reverse mapping btree, we must scan the entire filesystem, but we cannot afford to quiesce the entire filesystem for the potentially lengthy scan. For rmap btrees, therefore, we relax our requirements of totally atomic repairs. Instead, repairs will scan all inodes, construct a new reverse mapping dataset, format a new btree, and commit it before anyone trips over the corruption. This is exactly the same strategy as was used in the quotacheck and nlink scanners. Unfortunately, the xfarray cannot perform key-based lookups and is therefore unsuitable for supporting live updates. Luckily, we already a data structure that maintains an indexed rmap recordset -- the existing rmap btree code! Hence we port the existing btree and buffer target code to be able to create a btree using the xfile we developed earlier. Live hooks keep the in-memory btree up to date for any resources that have already been scanned. This approach is not maximally memory efficient, but we can use the same rmap code that we do everywhere else, which provides improved stability without growing the code base even more. Note that in-memory btree blocks are always page sized. This patchset modifies the kernel xfs buffer cache to be capable of using a xfile (aka a shmem file) as a backing device. It then augments the btree code to support creating btree cursors with buffers that come from a buftarg other than the data device (namely an xfile-backed buftarg). For the userspace xfs buffer cache, we instead use a memfd or an O_TMPFILE file as a backing device. Signed-off-by:
-
Chandan Babu R authored
Merge tag 'buftarg-cleanups-6.9_2024-02-23' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux into xfs-6.9-mergeC xfs: buftarg cleanups Clean up the buffer target code in preparation for adding the ability to target tmpfs files. That will enable the creation of in memory btrees. Signed-off-by:
-
Chandan Babu R authored
Merge tag 'btree-readahead-cleanups-6.9_2024-02-23' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux into xfs-6.9-mergeC xfs: btree readahead cleanups Minor cleanups for the btree block readahead code. Signed-off-by:
-
Chandan Babu R authored
Merge tag 'btree-check-cleanups-6.9_2024-02-23' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux into xfs-6.9-mergeC xfs: btree check cleanups Minor cleanups for the btree block pointer checking code. Signed-off-by:
-
Chandan Babu R authored
Merge tag 'btree-remove-btnum-6.9_2024-02-23' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux into xfs-6.9-mergeC xfs: remove bc_btnum from btree cursors From Christoph Hellwig, This series continues the migration of btree geometry information out of the cursor structure and into the ops structure. This time around, we replace the btree type enumeration (btnum) with an explicit name string in the btree ops structure. This enables easy creation of /any/ new btree type without having to mess with libxfs. Signed-off-by:
-
Chandan Babu R authored
Merge tag 'btree-geometry-in-ops-6.9_2024-02-23' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux into xfs-6.9-mergeC xfs: move btree geometry to ops struct This patchset prepares the generic btree code to allow for the creation of new btree types outside of libxfs. The end goal here is for online fsck to be able to create its own in-memory btrees that will be used to improve the performance (and reduce the memory requirements of) the refcount btree. To enable this, I decided that the btree ops structure is the ideal place to encode all of the geometry information about a btree. The btree ops struture already contains the buffer ops (and hence the btree block magic numbers) as well as the key and record sizes, so it doesn't seem all that farfetched to encode the XFS_BTREE_ flags that determine the geometry (ROOT_IN_INODE, LONG_PTRS, etc). The rest of the patchset cleans up the btree functions that initialize btree blocks and btree buffers. The bulk of this work is to replace btree geometry related function call arguments with a single pointer to the ops structure, and then clean up everything else around that. As a side effect, we rename the functions. Later, Christoph Hellwig and I merged together a bunch more cleanups that he wanted to do for a while. All the btree geometry information is now in the btree ops structure, we've created an explicit btree type (ag, inode, mem) and moved the per-btree type information to a separate union. Signed-off-by:
-
Chandan Babu R authored
Merge tag 'repair-fscounters-6.9_2024-02-23' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux into xfs-6.9-mergeC xfs: online repair for fs summary counters A longstanding deficiency in the online fs summary counter scrubbing code is that it hasn't any means to quiesce the incore percpu counters while it's running. There is no way to coordinate with other threads are reserving or freeing free space simultaneously, which leads to false error reports. Right now, if the discrepancy is large, we just sort of shrug and bail out with an incomplete flag, but this is lame. For repair activity, we actually /do/ need to stabilize the counters to get an accurate reading and install it in the percpu counter. To improve the former and enable the latter, allow the fscounters online fsck code to perform an exclusive mini-freeze on the filesystem. The exclusivity prevents userspace from thawing while we're running, and the mini-freeze means that we don't wait for the log to quiesce, which will make both speedier. Signed-off-by:
-
Chandan Babu R authored
Merge tag 'indirect-health-reporting-6.9_2024-02-23' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux into xfs-6.9-mergeC xfs: indirect health reporting This series enables the XFS health reporting infrastructure to remember indirect health concerns when resources are scarce. For example, if a scrub notices that there's something wrong with an inode's metadata but memory reclaim needs to free the incore inode, we want to record in the perag data the fact that there was some inode somewhere with an error. The perag structures never go away. The first two patches in this series set that up, and the third one provides a means for xfs_scrub to tell the kernel that it can forget the indirect problem report. Signed-off-by:
-
Chandan Babu R authored
Merge tag 'corruption-health-reports-6.9_2024-02-23' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux into xfs-6.9-mergeC xfs: report corruption to the health trackers Any time that the runtime code thinks it has found corrupt metadata, it should tell the health tracking subsystem that the corresponding part of the filesystem is sick. These reports come primarily from two places -- code that is reading a buffer that fails validation, and higher level pieces that observe a conflict involving multiple buffers. This patchset uses automated scanning to update all such callsites with a mark_sick call. Doing this enables the health system to record problem observed at runtime, which (for now) can prompt the sysadmin to run xfs_scrub, and (later) may enable more targetted fixing of the filesystem. Note: Earlier reviewers of this patchset suggested that the verifier functions themselves should be responsible for calling _mark_sick. In a higher level language this would be easily accomplished with lambda functions and closures. For the kernel, however, we'd have to create the necessary closures by hand, pass them to the buf_read calls, and then implement necessary state tracking to detach the xfs_buf from the closure at the necessary time. This is far too much work and complexity and will not be pursued further. Signed-off-by:
-
Chandan Babu R authored
Merge tag 'scrub-nlinks-6.9_2024-02-23' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux into xfs-6.9-mergeC xfs: online repair of file link counts Now that we've created the infrastructure to perform live scans of every file in the filesystem and the necessary hook infrastructure to observe live updates, use it to scan directories to compute the correct link counts for files in the filesystem, and reset those link counts. This patchset creates a tailored readdir implementation for scrub because the regular version has to cycle ILOCKs to copy information to userspace. We can't cycle the ILOCK during the nlink scan and we don't need all the other VFS support code (maintaining a readdir cursor and translating XFS structures to VFS structures and back) so it was easier to duplicate the code. Signed-off-by:
-
Chandan Babu R authored
Merge tag 'repair-quotacheck-6.9_2024-02-23' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux into xfs-6.9-mergeC xfs: online repair of quota counters This series uses the inode scanner and live update hook functionality introduced in the last patchset to implement quotacheck on a live filesystem. The quotacheck scrubber builds an incore copy of the dquot resource usage counters and compares it to the live dquots to report discrepancies. If the user chooses to repair the quota counters, the repair function visits each incore dquot to update the counts from the live information. The live update hooks are key to keeping the incore copy up to date. Signed-off-by:
-
Chandan Babu R authored
Merge tag 'repair-inode-mode-6.9_2024-02-23' of https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux into xfs-6.9-mergeC xfs: repair inode mode by scanning dirs One missing piece of functionality in the inode record repair code is figuring out what to do with a file whose mode is so corrupt that we cannot tell us the type of the file. Originally this was done by guessing the mode from the ondisk inode contents, but Christoph didn't like that because it read from data fork block 0, which could be user controlled data. Therefore, I've replaced all that with a directory scanner that looks for any dirents that point to the file with the garbage mode. If so, the ftype in the dirent will tell us exactly what mode to set on the file. Since users cannot directly write to the ftype field of a dirent, this should be safe. Signed-off-by:
-
- 22 Feb, 2024 14 commits
-
-
Darrick J. Wong authored
Move xfs_symlink_write_target to xfs_symlink_remote.c so that kernel and mkfs can share the same function. Signed-off-by:
-
Darrick J. Wong authored
Move xfs_readlink_bmap_ilocked to xfs_symlink_remote.c so that the swapext code can use it to convert a remote format symlink back to shortform format after a metadata repair. While we're at it, fix a broken printf prefix. Signed-off-by:
-
Darrick J. Wong authored
Move declarations for libxfs symlink functions into a separate header file like we do for most everything else. Signed-off-by:
-
Darrick J. Wong authored
The deferred bmap work state and the log item can transmit unwritten state, so the XFS_BMAP_MAP handler must map in extents with that unwritten state. Signed-off-by:
-
Darrick J. Wong authored
The deferred bmap update log item has always supported the attr fork, so plumb this in so that higher layers can access this. Signed-off-by:
-
Darrick J. Wong authored
Now that we have reflink on the realtime device, bmap intent items have to support remapping extents on the realtime volume. Signed-off-by:
-
Darrick J. Wong authored
Extend the bmap update (BUI) log items with a new realtime flag that indicates that the updates apply against a realtime file's data fork. We'll wire up the actual code later. Signed-off-by:
-
Darrick J. Wong authored
Add a helper to translate from the item list head to the attr_intent item structure and use it so shorten assignments and avoid the need for extra local variables. Inspired-by:
-
Darrick J. Wong authored
When XFS_BMAPI_REMAP is passed to bunmapi, that means that we want to remove part of a block mapping without touching the allocator. For realtime files with rtextsize > 1, that also means that we should skip all the code that changes a partial remove request into an unwritten extent conversion. IOWs, bunmapi in this mode should handle removing the mapping from the rt file and nothing else. Note that XFS_BMAPI_REMAP callers are required to decrement the reference count and/or free the space manually. Signed-off-by:
-
Darrick J. Wong authored
Move the code that adds the incore xfs_bmap_item deferred work data to a transaction live with the BUI log item code. This means that the file mapping code no longer has to know about the inner workings of the BUI log items. As a consequence, we can hide the _get_group helper. Signed-off-by:
-
Darrick J. Wong authored
Reuse xfs_bmap_update_cancel_item to put the AG/RTG and free the item in a few places that currently open code the logic. Inspired-by:
-
Darrick J. Wong authored
Add a helper to translate from the item list head to the bmap_intent structure and use it so shorten assignments and avoid the need for extra local variables. Inspired-by:
-
Darrick J. Wong authored
Remove this single-use helper. Signed-off-by:
-
Darrick J. Wong authored
Pass the incore bmap structure to the tracepoints instead of open-coding the argument passing. Signed-off-by:
-
