1. 02 Jul, 2017 1 commit
  2. 28 Jun, 2017 9 commits
  3. 24 Jun, 2017 1 commit
    • Brian Foster's avatar
      xfs: free uncommitted transactions during log recovery · 39775431
      Brian Foster authored
      Log recovery allocates in-core transaction and member item data
      structures on-demand as it processes the on-disk log. Transactions
      are allocated on first encounter on-disk and stored in a hash table
      structure where they are easily accessible for subsequent lookups.
      Transaction items are also allocated on demand and are attached to
      the associated transactions.
      
      When a commit record is encountered in the log, the transaction is
      committed to the fs and the in-core structures are freed. If a
      filesystem crashes or shuts down before all in-core log buffers are
      flushed to the log, however, not all transactions may have commit
      records in the log. As expected, the modifications in such an
      incomplete transaction are not replayed to the fs. The in-core data
      structures for the partial transaction are never freed, however,
      resulting in a memory leak.
      
      Update xlog_do_recovery_pass() to first correctly initialize the
      hash table array so empty lists can be distinguished from populated
      lists on function exit. Update xlog_recover_free_trans() to always
      remove the transaction from the list prior to freeing the associated
      memory. Finally, walk the hash table of transaction lists as the
      last step before it goes out of scope and free any transactions that
      may remain on the lists. This prevents a memory leak of partial
      transactions in the log.
      Signed-off-by: default avatarBrian Foster <bfoster@redhat.com>
      Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
      Reviewed-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      39775431
  4. 20 Jun, 2017 7 commits
  5. 19 Jun, 2017 22 commits
    • Darrick J. Wong's avatar
      xfs: separate function to check if inode shares extents · ea7cdd7b
      Darrick J. Wong authored
      Separate the "clear reflink flag" function into one function that checks
      if the flag is needed, and a second function that checks and clears the
      flag.  The inode scrub code will want to check the necessity of the flag
      without clearing it.
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Reviewed-by: default avatarBrian Foster <bfoster@redhat.com>
      ea7cdd7b
    • Darrick J. Wong's avatar
      xfs: reflink find shared should take a transaction · 92ff7285
      Darrick J. Wong authored
      Adapt _reflink_find_shared to take an optional transaction pointer.  The
      inode scrubber code will need to decide (within transaction context) if
      a file has shared blocks.  To avoid buffer deadlocks, we must pass the
      tp through to this function's utility calls.
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Reviewed-by: default avatarBrian Foster <bfoster@redhat.com>
      92ff7285
    • Darrick J. Wong's avatar
      xfs: check if an inode is cached and allocated · 378f681c
      Darrick J. Wong authored
      Check the inode cache for a particular inode number.  If it's in the
      cache, check that it's not currently being reclaimed.  If it's not being
      reclaimed, return zero if the inode is allocated.  This function will be
      used by various scrubbers to decide if the cache is more up to date
      than the disk in terms of checking if an inode is allocated.
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Reviewed-by: default avatarBrian Foster <bfoster@redhat.com>
      378f681c
    • Darrick J. Wong's avatar
      xfs: export _inobt_btrec_to_irec and _ialloc_cluster_alignment for scrub · e936945e
      Darrick J. Wong authored
      Create a function to extract an in-core inobt record from a generic
      btree_rec union so that scrub will be able to check inobt records
      and check inode block alignment.
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Reviewed-by: default avatarBrian Foster <bfoster@redhat.com>
      e936945e
    • Darrick J. Wong's avatar
      xfs: plumb in needed functions for range querying of various btrees · 118bb47e
      Darrick J. Wong authored
      Plumb in the pieces (init_high_key, diff_two_keys) necessary to call
      query_range on the inode space and block mapping btrees and to extract
      raw btree records.  This will eventually be used by the inobt and bmbt
      scrubbers.
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Reviewed-by: default avatarBrian Foster <bfoster@redhat.com>
      118bb47e
    • Darrick J. Wong's avatar
      xfs: export various function for the online scrubber · 26788097
      Darrick J. Wong authored
      Export various internal functions so that the online scrubber can use
      them to check the state of metadata.
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Reviewed-by: default avatarBrian Foster <bfoster@redhat.com>
      26788097
    • Darrick J. Wong's avatar
      xfs: always compile the btree inorder check functions · 38dee376
      Darrick J. Wong authored
      The btree record and key inorder check functions will be used by the
      btree scrubber code, so make sure they're always built.
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Reviewed-by: default avatarBrian Foster <bfoster@redhat.com>
      38dee376
    • Darrick J. Wong's avatar
      xfs: remove double-underscore integer types · c8ce540d
      Darrick J. Wong authored
      This is a purely mechanical patch that removes the private
      __{u,}int{8,16,32,64}_t typedefs in favor of using the system
      {u,}int{8,16,32,64}_t typedefs.  This is the sed script used to perform
      the transformation and fix the resulting whitespace and indentation
      errors:
      
      s/typedef\t__uint8_t/typedef __uint8_t\t/g
      s/typedef\t__uint/typedef __uint/g
      s/typedef\t__int\([0-9]*\)_t/typedef int\1_t\t/g
      s/__uint8_t\t/__uint8_t\t\t/g
      s/__uint/uint/g
      s/__int\([0-9]*\)_t\t/__int\1_t\t\t/g
      s/__int/int/g
      /^typedef.*int[0-9]*_t;$/d
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
      c8ce540d
    • Darrick J. Wong's avatar
      xfs: optimize _btree_query_all · 5a4c7334
      Darrick J. Wong authored
      Don't bother wandering our way through the leaf nodes when the caller
      issues a query_all; just zoom down the left side of the tree and walk
      rightwards along level zero.
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Reviewed-by: default avatarBrian Foster <bfoster@redhat.com>
      5a4c7334
    • Brian Foster's avatar
      xfs: remove bli from AIL before release on transaction abort · 3d4b4a3e
      Brian Foster authored
      When a buffer is modified, logged and committed, it ultimately ends
      up sitting on the AIL with a dirty bli waiting for metadata
      writeback. If another transaction locks and invalidates the buffer
      (freeing an inode chunk, for example) in the meantime, the bli is
      flagged as stale, the dirty state is cleared and the bli remains in
      the AIL.
      
      If a shutdown occurs before the transaction that has invalidated the
      buffer is committed, the transaction is ultimately aborted. The log
      items are flagged as such and ->iop_unlock() handles the aborted
      items. Because the bli is clean (due to the invalidation),
      ->iop_unlock() unconditionally releases it. The log item may still
      reside in the AIL, however, which means the I/O completion handler
      may still run and attempt to access it. This results in assert
      failure due to the release of the bli while still present in the AIL
      and a subsequent NULL dereference and panic in the buffer I/O
      completion handling. This can be reproduced by running generic/388
      in repetition.
      
      To avoid this problem, update xfs_buf_item_unlock() to first check
      whether the bli is aborted and if so, remove it from the AIL before
      it is released. This ensures that the bli is no longer accessed
      during the shutdown sequence after it has been freed.
      Signed-off-by: default avatarBrian Foster <bfoster@redhat.com>
      Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
      Reviewed-by: default avatarCarlos Maiolino <cmaiolino@redhat.com>
      Reviewed-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      3d4b4a3e
    • Brian Foster's avatar
      xfs: release bli from transaction properly on fs shutdown · 79e641ce
      Brian Foster authored
      If a filesystem shutdown occurs with a buffer log item in the CIL
      and a log force occurs, the ->iop_unpin() handler is generally
      expected to tear down the bli properly. This entails freeing the bli
      memory and releasing the associated hold on the buffer so it can be
      released and the filesystem unmounted.
      
      If this sequence occurs while ->bli_refcount is elevated (i.e.,
      another transaction is open and attempting to modify the buffer),
      however, ->iop_unpin() may not be responsible for releasing the bli.
      Instead, the transaction may release the final ->bli_refcount
      reference and thus xfs_trans_brelse() is responsible for tearing
      down the bli.
      
      While xfs_trans_brelse() does drop the reference count, it only
      attempts to release the bli if it is clean (i.e., not in the
      CIL/AIL). If the filesystem is shutdown and the bli is sitting dirty
      in the CIL as noted above, this ends up skipping the last
      opportunity to release the bli. In turn, this leaves the hold on the
      buffer and causes an unmount hang. This can be reproduced by running
      generic/388 in repetition.
      
      Update xfs_trans_brelse() to handle this shutdown corner case
      correctly. If the final bli reference is dropped and the filesystem
      is shutdown, remove the bli from the AIL (if necessary) and release
      the bli to drop the buffer hold and ensure an unmount does not hang.
      Signed-off-by: default avatarBrian Foster <bfoster@redhat.com>
      Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
      Reviewed-by: default avatarCarlos Maiolino <cmaiolino@redhat.com>
      Reviewed-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      79e641ce
    • Arnd Bergmann's avatar
      xfs: avoid harmless gcc-7 warnings · 0cbe48cc
      Arnd Bergmann authored
      gcc-7 flags the use of integer math inside of a condition
      as a potential bug:
      
      fs/xfs/xfs_bmap_util.c: In function 'xfs_swap_extents_check_format':
      fs/xfs/xfs_bmap_util.c:1619:8: error: '<<' in boolean context, did you mean '<' ? [-Werror=int-in-bool-context]
      fs/xfs/xfs_bmap_util.c:1629:8: error: '<<' in boolean context, did you mean '<' ? [-Werror=int-in-bool-context]
      
      There is already a helper function for testing the di_forkoff
      field for zero, so let's use that instead to shut up the warning.
      Signed-off-by: default avatarArnd Bergmann <arnd@arndb.de>
      Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
      Reviewed-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      0cbe48cc
    • Shan Hai's avatar
      xfs: remove lsn relevant fields from xfs_trans structure and its users · f990fc5a
      Shan Hai authored
      The t_lsn is not used anymore and the t_commit_lsn is used as a tmp
      storage for the checkpoint sequence number only in the current code.
      
      And the start/commit lsn are tracked as a transaction group tag in
      the xfs_cil_ctx instead of a single transaction, so remove them from
      the xfs_trans structure and their users to match with the design.
      Signed-off-by: default avatarShan Hai <shan.hai@oracle.com>
      Reviewed-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      f990fc5a
    • Christoph Hellwig's avatar
      xfs: remove XFS_HSIZE · 3398a400
      Christoph Hellwig authored
      XFS_HSIZE is an extremly confusing way to calculate the size of handle_t.
      Given that handle_t always only had two sizes, and one of them isn't
      even covered by XFS_HSIZE to start with just remove the macro and use
      a constant sizeof expression.
      
      Note that XFS_HSIZE isn't used in xfsprogs, xfsdump or xfstests either.
      Signed-off-by: default avatarChristoph Hellwig <hch@lst.de>
      Reviewed-by: default avatarEric Sandeen <sandeen@sandeen.net>
      Reviewed-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      3398a400
    • Brian Foster's avatar
      xfs: dump transaction usage details on log reservation overrun · d4ca1d55
      Brian Foster authored
      If a transaction log reservation overrun occurs, the ticket data
      associated with the reservation is dumped in xfs_log_commit_cil().
      This occurs long after the transaction items and details have been
      removed from the transaction and effectively lost. This limited set
      of ticket data provides very little information to support debugging
      transaction overruns based on the typical report.
      
      To improve transaction log reservation overrun reporting, create a
      helper to dump transaction details such as log items, log vector
      data, etc., as well as the underlying ticket data for the
      transaction. Move the overrun detection from xfs_log_commit_cil() to
      xlog_cil_insert_items() so it occurs prior to migration of the
      logged items to the CIL. Call the new helper such that it is able to
      dump this transaction data before it is lost.
      
      Also, warn on overrun to provide callstack context for the offending
      transaction and include a few additional messages from
      xlog_cil_insert_items() to display the reservation consumed locally
      for overhead such as log vector headers, split region headers and
      the context ticket. This provides a complete general breakdown of
      the reservation consumption of a transaction when/if it happens to
      overrun the reservation.
      Signed-off-by: default avatarBrian Foster <bfoster@redhat.com>
      Reviewed-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      d4ca1d55
    • Brian Foster's avatar
      xfs: refactor xlog_cil_insert_items() to facilitate transaction dump · e2f23426
      Brian Foster authored
      Transaction reservation overrun detection currently occurs too late
      to print useful information about the offending transaction.
      Ideally, the transaction data is printed before the associated log
      items are moved from the transaction to the CIL, which occurs in
      xlog_cil_insert_items(), such that details of the items logged by
      the transaction are available for analysis.
      
      Refactor xlog_cil_insert_items() to facilitate moving tx overrun
      detection to this function. Update the function to track each bit of
      extra log reservation stolen from the transaction (i.e., such as for
      the CIL context ticket) and perform the log item migration as the
      last operation before the CIL lock is released. This creates a
      context where the transaction reservation consumption has been fully
      calculated when the log items are moved to the CIL. This patch makes
      no functional changes.
      Signed-off-by: default avatarBrian Foster <bfoster@redhat.com>
      Reviewed-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      e2f23426
    • Brian Foster's avatar
      xfs: separate shutdown from ticket reservation print helper · 7d2d5653
      Brian Foster authored
      xlog_print_tic_res() pre-dates delayed logging and the committed
      items list (CIL) and thus retains some factoring warts, such as hard
      coded function names in the output and the fact that it induces a
      shutdown.
      
      In preparation for more detailed logging of regular transaction
      overrun situations, refactor xlog_print_tic_res() to be slightly
      more generic. Reword some of the warning messages and pull the
      shutdown into the callers.
      Signed-off-by: default avatarBrian Foster <bfoster@redhat.com>
      Reviewed-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      7d2d5653
    • Brian Foster's avatar
      xfs: define fatal assert build time tunable · 1040960e
      Brian Foster authored
      While configurable at runtime, the DEBUG mode assert failure
      behavior is usually either desired or not for a particular
      situation. For example, developers using kernel modules may prefer
      for fatal asserts to remain disabled across module reloads while QE
      engineers doing broad regression testing may prefer to have fatal
      asserts enabled on boot to facilitate data collection for bug
      reports.
      
      To provide a compromise/convenience for developers, create a Kconfig
      option that sets the default value of the DEBUG mode 'bug_on_assert'
      sysfs tunable. The default behavior remains to trigger kernel BUGs
      on assert failures to preserve existing behavior across kernel
      configuration updates with DEBUG mode enabled.
      Signed-off-by: default avatarBrian Foster <bfoster@redhat.com>
      Reviewed-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      1040960e
    • Brian Foster's avatar
      xfs: define bug_on_assert debug mode sysfs tunable · ccdab3d6
      Brian Foster authored
      In DEBUG mode, assert failures unconditionally trigger a kernel BUG.
      This is useful in diagnostic situations to panic a system and
      collect detailed state information at the time of a failure.
      
      This can also cause problems in cases where DEBUG mode code is
      desired but it is preferable not trigger kernel BUGs on assert
      failure. For example, during development of new code or during
      certain xfstests tests that intentionally cause corruption and test
      the kernel for survival (but otherwise may expect to trigger assert
      failures).
      
      To provide additional flexibility, create the
      <sysfs>/fs/xfs/debug/bug_on_assert tunable to configure assert
      failure behavior at runtime. This tunable is only available in DEBUG
      mode and is enabled by default to preserve existing default
      behavior. When disabled, assert failures in DEBUG mode result in
      kernel warnings.
      Signed-off-by: default avatarBrian Foster <bfoster@redhat.com>
      Reviewed-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      ccdab3d6
    • Darrick J. Wong's avatar
      xfs: try to avoid blowing out the transaction reservation when bunmaping a shared extent · e1a4e37c
      Darrick J. Wong authored
      In a pathological scenario where we are trying to bunmapi a single
      extent in which every other block is shared, it's possible that trying
      to unmap the entire large extent in a single transaction can generate so
      many EFIs that we overflow the transaction reservation.
      
      Therefore, use a heuristic to guess at the number of blocks we can
      safely unmap from a reflink file's data fork in an single transaction.
      This should prevent problems such as the log head slamming into the tail
      and ASSERTs that trigger because we've exceeded the transaction
      reservation.
      
      Note that since bunmapi can fail to unmap the entire range, we must also
      teach the deferred unmap code to roll into a new transaction whenever we
      get low on reservation.
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      [hch: random edits, all bugs are my fault]
      Signed-off-by: default avatarChristoph Hellwig <hch@lst.de>
      e1a4e37c
    • Darrick J. Wong's avatar
      xfs: refactor dir2 leaf readahead shadow buffer cleverness · d205a7d0
      Darrick J. Wong authored
      Currently, the dir2 leaf block getdents function uses a complex state
      tracking mechanism to create a shadow copy of the block mappings and
      then uses the shadow copy to schedule readahead.  Since the read and
      readahead functions are perfectly capable of reading the mappings
      themselves, we can tear all that out in favor of a simpler function that
      simply keeps pushing the readahead window further out.
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Reviewed-by: default avatarBrian Foster <bfoster@redhat.com>
      Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
      d205a7d0
    • Brian Foster's avatar
      xfs: push buffer of flush locked dquot to avoid quotacheck deadlock · 7912e7fe
      Brian Foster authored
      Reclaim during quotacheck can lead to deadlocks on the dquot flush
      lock:
      
       - Quotacheck populates a local delwri queue with the physical dquot
         buffers.
       - Quotacheck performs the xfs_qm_dqusage_adjust() bulkstat and
         dirties all of the dquots.
       - Reclaim kicks in and attempts to flush a dquot whose buffer is
         already queud on the quotacheck queue. The flush succeeds but
         queueing to the reclaim delwri queue fails as the backing buffer is
         already queued. The flush unlock is now deferred to I/O completion
         of the buffer from the quotacheck queue.
       - The dqadjust bulkstat continues and dirties the recently flushed
         dquot once again.
       - Quotacheck proceeds to the xfs_qm_flush_one() walk which requires
         the flush lock to update the backing buffers with the in-core
         recalculated values. It deadlocks on the redirtied dquot as the
         flush lock was already acquired by reclaim, but the buffer resides
         on the local delwri queue which isn't submitted until the end of
         quotacheck.
      
      This is reproduced by running quotacheck on a filesystem with a
      couple million inodes in low memory (512MB-1GB) situations. This is
      a regression as of commit 43ff2122 ("xfs: on-stack delayed write
      buffer lists"), which removed a trylock and buffer I/O submission
      from the quotacheck dquot flush sequence.
      
      Quotacheck first resets and collects the physical dquot buffers in a
      delwri queue. Then, it traverses the filesystem inodes via bulkstat,
      updates the in-core dquots, flushes the corrected dquots to the
      backing buffers and finally submits the delwri queue for I/O. Since
      the backing buffers are queued across the entire quotacheck
      operation, dquot reclaim cannot possibly complete a dquot flush
      before quotacheck completes.
      
      Therefore, quotacheck must submit the buffer for I/O in order to
      cycle the flush lock and flush the dirty in-core dquot to the
      buffer. Add a delwri queue buffer push mechanism to submit an
      individual buffer for I/O without losing the delwri queue status and
      use it from quotacheck to avoid the deadlock. This restores
      quotacheck behavior to as before the regression was introduced.
      Reported-by: default avatarMartin Svec <martin.svec@zoner.cz>
      Signed-off-by: default avatarBrian Foster <bfoster@redhat.com>
      Reviewed-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      7912e7fe