1. 09 Jun, 2016 1 commit
    • Jim Fulton's avatar
      Be more careful about calling close callbacks · 36df7e06
      Jim Fulton authored
      By resetting self.__onCloseCallbacks before calling the close
      callbacks to forestall infinite recursion (as can be the case when a
      close callback is the database close methos).
      36df7e06
  2. 07 Jun, 2016 3 commits
    • Jim Fulton's avatar
      Fixed maxtid, copying value from ZEO. · baee84a6
      Jim Fulton authored
      Changed it to the value corresponding to the maximim *signed*
      big-endian 64-bit integer, because of LxBTrees.
      
      Note that this value isn't used anywhere in ZODB yet.
      
      Maybe it should be.
      baee84a6
    • Jim Fulton's avatar
      typo · 5bb8a810
      Jim Fulton authored
      5bb8a810
    • Jim Fulton's avatar
      Clear transaction syncs earlier · 7a75ba7a
      Jim Fulton authored
      To mitigate errors during tearDown.
      
      Lots of tests are sloppy about closing connection (because they could
      be), but storages tend to get closed and transactions aborted. This
      has the effedt of calling afterCompletion on connections which tries
      to call storage lastTransaction, which causes errors.
      7a75ba7a
  3. 06 Jun, 2016 3 commits
  4. 05 Jun, 2016 1 commit
  5. 31 May, 2016 1 commit
    • Jim Fulton's avatar
      Merge pull request #62 from... · 6fe36b98
      Jim Fulton authored
      Merge pull request #62 from zopefoundation/dont-depend-on-implementation-detail-in-tpc_finish-test-assertions
      
      Fixed a test that depended on some off behavior of tpc_finish in some…
      6fe36b98
  6. 30 May, 2016 1 commit
  7. 17 May, 2016 2 commits
  8. 12 May, 2016 1 commit
  9. 09 May, 2016 3 commits
  10. 05 May, 2016 5 commits
  11. 04 May, 2016 7 commits
    • Jim Fulton's avatar
      OK, copying loadBefore was the right fix to the checkPackLotsWhileWriting failure · d90a0243
      Jim Fulton authored
      Even though the previous commit otherwise made MVCCMappingStorage and
      IMVCCStorage handling better.
      d90a0243
    • Jim Fulton's avatar
      Better IMVCCStorage support · 2729f3a9
      Jim Fulton authored
      The previous commit, made in anger, made a test pass, but wasn't
      really the right fix.
      
      This commit fixes MVCCMappingStorage's loadBefore implementation by
      fixing handling of the internal _ltid variable so that it's updated
      during poll_invalidations.  This allowed the base class version of
      loadBefore to be used and, I'm 97% sure has the right semantics.
      Fixing this revealed a problem with the Connection changes.
      
      Fixed Connection.py to poll for invalidations before computing
      _txn_time by calling lastTransaction, so as to get a current value.
      We still apply invalidations after computing _txn_time, so that
      persistent classes can be loaded correctly when they are invalidated.
      This was accomplished by weaving _flush_invalidations into
      newTransaction.
      2729f3a9
    • Jim Fulton's avatar
      On Python 2 and Mac OS X, checkFlushAfterTruncate was failing. · 86e62dae
      Jim Fulton authored
      Changed to always empty the file pool.  I doubt that just flushing on
      Python 2 was much of an optimization, and ... Python 2. :)
      86e62dae
    • Jim Fulton's avatar
      When creating a "new_instance", copy loadBefore, as well s pack. · f2922e4c
      Jim Fulton authored
      This is needed because pack temporarily deletes data from the data
      structure that loadBefore reads.
      
      Note that in the new scheme of things, we avoid using load, but
      instead use loadBefore.  This we bypass the _data_snapshot
      shenanigans, which makes me think we aren't testing what we should be,
      but I have no confidence that the machinery in MVCCMappingStorage
      actually reflects anything meaningful.
      f2922e4c
    • Jim Fulton's avatar
      Removed debugging print · 69e78b3f
      Jim Fulton authored
      69e78b3f
    • Jim Fulton's avatar
      1b6f7156
    • Jim Fulton's avatar
      Simplify MVCC by determining transaction start time using lastTransaction. · 227953b9
      Jim Fulton authored
      This implements: https://github.com/zopefoundation/ZODB/issues/50
      
      Rather than watching invalidations, simply use 1 + the storages
      lastTransaction, which is equivalent to but much simpler than waiting
      for the first invalidation after a transaction starts.
      
      More importantly, it means we can always use loadBefore and get away
      from load.  We no longer have to worry about ordering of invalidations
      and load() results.
      
      Much thanks to NEO for pointing the way toward this simplification!
      
      Implementing this initially caused a deadlock, because DB.open()
      called Connection.open() while holding a database lock and
      Connection.open() now calls IStotage.lastTransaction(), which acquires a
      storage lock. (It's not clear that lastTransaction() really needs a
      storage lock.)  Meanwhile, IStotage.tpc_finish() calls a DB function
      that requires the DB lock while holding the storage lock.  Fixing this
      required moving the call to Connection.open() outside the region where
      the DB lock was held.
      
      To debug the problem above, I greatly improved lock-debugging
      support. Now all of the ZODB code imports Lock, RLock and Condition
      from ZODB.utils. If the DEBUG_LOCKING is set to a non-empty value,
      then these are wrapped in such a way that debugging information is
      printed as they are used. This made spotting the nature of the
      deadlock easier.
      
      Of course, a change this basic broke lots of tests. Most of the
      breakage arises from the fact that connections now call
      lastTransaction on storages at transaction boundaries.  Lots of tests
      didn't clean up databases and connections properly.  I fixed many
      tests, but ultimately gave up and added some extra cleanup code that
      violated transaction-manager underware (and the underware's privates)
      to clear transaction synchonizers in test setup and tear-down.  I plan
      to add a transaction manager API for this and to use it in a
      subsequent PR.
      
      This tests makes database and connection hygiene a bit more important,
      especially for tests, because a connection will continue to interact
      with storages if it isn't properly closed, which can lead to errors if
      the storage is closed.  I chose not to swallow these errors in
      Connection, choosing rather to clean up tests.
      
      The thread debugging and test changes make this PR larger than I would
      have liked. Apologies in advance to the reviewers.
      227953b9
  12. 29 Apr, 2016 2 commits
  13. 28 Apr, 2016 2 commits
  14. 27 Apr, 2016 4 commits
    • Julien Muchembled's avatar
      Changelog for PR #52 · 06df0eba
      Julien Muchembled authored
      06df0eba
    • Julien Muchembled's avatar
      Fix possible data corruption after FileStorage is truncated to roll back a transaction · 028b1922
      Julien Muchembled authored
      Multi-threaded IO support, which is new to ZODB 3.10, allows clients to read
      data (load & loadBefore) even after tpc_vote has started to write a new
      transaction to disk. This is done by using different 'file' objects.
      
      Issues start when a transaction is rolled back after data has been appended
      (using the writing file object). Truncating is not enough because the FilePool
      may have been used concurrently to read the end of the last transaction:
      file objects have their own read buffers which, in this case, may also contain
      the beginning of the aborted transaction.
      
      So a solution is to invalidate read buffers whenever they may contain wrong
      data. This patch does it on truncation, which happens rarely enough to not
      affect performance.
      
      We discovered this bug in the following conditions:
      - ZODB splitted in several FileStorage
      - many conflicts in the first committed DB, but always resolved
      - unresolved conflict in another DB
      If the transaction is replayed with success (no more conflict in the other DB),
      a subsequent load of the object that could be resolved in the first DB may, for
      example, return a wrong serial (tid of the aborted transaction) if the layout
      of the committed transaction matches that of the aborted one.
      
      The bug usually manifests with POSKeyError & CorruptedDataError exceptions in
      ZEO logs, for example while trying to resolve a conflict (and restarting the
      transaction does not help, causing Site Errors in Zope). But theorically,
      this could also cause silent corruption or unpickling errors at client side.
      028b1922
    • Julien Muchembled's avatar
      Changelog for PR #54 · b83ac1c2
      Julien Muchembled authored
      b83ac1c2
    • Julien Muchembled's avatar
      Merge pull request #54 from zopefoundation/deps-cleanup · eebae5a2
      Julien Muchembled authored
      Cleanup dependencies, remove ZEO leftovers
      eebae5a2
  15. 25 Apr, 2016 2 commits
  16. 04 Apr, 2016 1 commit
    • Jason Madden's avatar
      patch time.time in historical_connections to avoid windows failures; fix race... · 6b707535
      Jason Madden authored
      patch time.time in historical_connections to avoid windows failures; fix race condition in pool.availableGC
      
      * patch time.time in historical_connections to avoid windows failures.
      
      * Also use a mock version of datetime.datetime.utcnow.
      
      We can't patch the method in-place because it's an extension type.
      
      * synchronize sleep with time
      
      * increment by less than the pool timeout to avoid a race condition.
      
      * Avoid mutating the dict as we iter over it.
      
      Seen in Python 3.4 tests:
      
      File "/home/travis/build/zopefoundation/ZODB/src/ZODB/historical_connections.txt", line 263, in historical_connections.txt
      Failed example:
          conn = db.open()
      Exception raised:
          Traceback (most recent call last):
            File "/opt/python/3.4.2/lib/python3.4/doctest.py", line 1324, in __run
              compileflags, 1), test.globs)
            File "<doctest historical_connections.txt[0]>", line 1, in <module>
              conn = db.open()
            File "/home/travis/build/zopefoundation/ZODB/src/ZODB/DB.py", line 761, in open
              self.historical_pool.availableGC()
            File "/home/travis/build/zopefoundation/ZODB/src/ZODB/DB.py", line 277, in availableGC
              for key, pool in self.pools.items():
          RuntimeError: dictionary changed size during iteration
      6b707535
  17. 18 Mar, 2016 1 commit