• Kirill Smelkov's avatar
    go/zodb/fs1tools: Verify · 5a26fb31
    Kirill Smelkov authored
    Add utility to verify FileStorage data for consistency.
    To verify we just need to iterate through all records, because
    FileStorage driver performs all consistency checks by itself.
    
    Mimic normal output to be the same as in fstest from ZODB/py.
    Example runs of fstest.py and `fs1 verify` on a broken file:
    
        $ python ~/src/wendelin/z/ZODB/src/ZODB/scripts/fstest.py -v 1.fs
                 4: transaction tid 0x03e044f6448c8022 #0
               213: transaction tid 0x03e044f646e044bb #1
        1.fs has data records that extend beyond the transaction record; end at 466
    
        $ fs1 verify -v 1.fs
                 4: transaction tid 0x03e044f6448c8022 #0
               213: transaction tid 0x03e044f646e044bb #1
        2021/05/24 12:43:37 fsverify: 1.fs: 1.fs: transaction record @355: -> (iter data): 1.fs: data record @416: check: data record [..., 466) overlaps txn boundary [..., 458)
    
    As can be seen, in fs1 case, the error contains more details: [start, end) of
    both mismatching transaction and data records.
    
    In addition to fstest-like verbosity, add progress-mode, where % of total
    completion is printed in a style similar to one used by `fs1 verify-index`.
    
    The Go-based implementation is also faster even when data is on HDD. For
    example on a 73GB database provided by @jerome[1] fsrefs.py takes ~15 minutes
    to run and occupy ~70-100% of CPU. On the other hand `fs1 verify` takes ~7
    minutes to run and occupy ~ 20-30% of CPU.
    
    Tests pending.
    
    [1] nexedi/zodbtools!19 (comment 129480)
    5a26fb31
verify.go 4.68 KB