• Jim Fulton's avatar
    Simplify MVCC by determining transaction start time using lastTransaction. · 227953b9
    Jim Fulton authored
    This implements: https://github.com/zopefoundation/ZODB/issues/50
    
    Rather than watching invalidations, simply use 1 + the storages
    lastTransaction, which is equivalent to but much simpler than waiting
    for the first invalidation after a transaction starts.
    
    More importantly, it means we can always use loadBefore and get away
    from load.  We no longer have to worry about ordering of invalidations
    and load() results.
    
    Much thanks to NEO for pointing the way toward this simplification!
    
    Implementing this initially caused a deadlock, because DB.open()
    called Connection.open() while holding a database lock and
    Connection.open() now calls IStotage.lastTransaction(), which acquires a
    storage lock. (It's not clear that lastTransaction() really needs a
    storage lock.)  Meanwhile, IStotage.tpc_finish() calls a DB function
    that requires the DB lock while holding the storage lock.  Fixing this
    required moving the call to Connection.open() outside the region where
    the DB lock was held.
    
    To debug the problem above, I greatly improved lock-debugging
    support. Now all of the ZODB code imports Lock, RLock and Condition
    from ZODB.utils. If the DEBUG_LOCKING is set to a non-empty value,
    then these are wrapped in such a way that debugging information is
    printed as they are used. This made spotting the nature of the
    deadlock easier.
    
    Of course, a change this basic broke lots of tests. Most of the
    breakage arises from the fact that connections now call
    lastTransaction on storages at transaction boundaries.  Lots of tests
    didn't clean up databases and connections properly.  I fixed many
    tests, but ultimately gave up and added some extra cleanup code that
    violated transaction-manager underware (and the underware's privates)
    to clear transaction synchonizers in test setup and tear-down.  I plan
    to add a transaction manager API for this and to use it in a
    subsequent PR.
    
    This tests makes database and connection hygiene a bit more important,
    especially for tests, because a connection will continue to interact
    with storages if it isn't properly closed, which can lead to errors if
    the storage is closed.  I chose not to swallow these errors in
    Connection, choosing rather to clean up tests.
    
    The thread debugging and test changes make this PR larger than I would
    have liked. Apologies in advance to the reviewers.
    227953b9
blobstorage_packing.txt 5.17 KB