1. 13 Sep, 2016 3 commits
    • Kirill Smelkov's avatar
      persistent: On deactivate release in-slots objects too · 9ddbacb0
      Kirill Smelkov authored
      ( This is backport of https://github.com/zopefoundation/persistent/pull/44
        to ZODB-3.10 )
      
      On ._p_deactivate() and ._p_invalidate(), when an object goes to ghost
      state, objects referenced by all its attributes, except related to
      persistence machinery, are released, this way freeing memory (if they
      were referenced only from going-to-ghost object).
      
      That's the idea - an object in ghost state is simply a stub, which loads
      its content on first access (via hooking into get/set attr) while
      occupying minimal memory in not-yet-loaded state.
      
      However the above is not completely true right now, as currently on
      ghostification only object's .__dict__ is released, while in-slots objects
      are retained attached to ghost object staying in RAM:
      
          ---- 8< ----
          from ZODB import DB
          from persistent import Persistent
          import gc
      
          db = DB(None)
          jar = db.open()
      
          class C:
              def __init__(self, v):
                  self.v = v
              def __del__(self):
                  print 'released (%s)' % self.v
      
          class P1(Persistent):
              pass
      
          class P2(Persistent):
              __slots__ = ('aaa')
      
          p1 = P1()
          jar.add(p1)
          p1.aaa = C(1)
      
          p2 = P2()
          jar.add(p2)
          p2.aaa = C(2)
      
          p1._p_invalidate()
          # "released (1)" is printed
      
          p2._p_invalidate()
          gc.collect()
          # "released (2)" is NOT printed     <--
          ---- 8< ----
      
      So teach ghostify() & friends to release objects in slots to free-up
      memory when an object goes to ghost state.
      
      NOTE PyErr_Occurred() added after ghostify() calls because
      pickle_slotnames() can raise an error, but we do not want to change
      ghostify() prototype for backward compatibility reason - as it is used
      in cPersistenceCAPIstruct.
      
      ( I hit this bug with wendelin.core which uses proxies to load
        data from DB to virtual memory manager and then deactivate proxy right
        after load has been completed:
      
        https://lab.nexedi.com/nexedi/wendelin.core/blob/f7803634/bigfile/file_zodb.py#L239
        https://lab.nexedi.com/nexedi/wendelin.core/blob/f7803634/bigfile/file_zodb.py#L295 )
      9ddbacb0
    • Julien Muchembled's avatar
      Changelog for PR #98 · 524c2714
      Julien Muchembled authored
      524c2714
    • Julien Muchembled's avatar
      Call _p_resolveConflict() even if a conflicting change doesn't change the state · 589f327a
      Julien Muchembled authored
      This reverts to the behaviour of 3.10.3 and older.
      
      (cherry picked from commit b74eef76)
      
      Conflicts:
      	src/ZODB/tests/testconflictresolution.py
      589f327a
  2. 28 Apr, 2016 2 commits
    • Julien Muchembled's avatar
      Changelog for PR #52 · 64f3b58f
      Julien Muchembled authored
      64f3b58f
    • Julien Muchembled's avatar
      Fix possible data corruption after FileStorage is truncated to roll back a transaction · 2ed12f35
      Julien Muchembled authored
      Multi-threaded IO support, which is new to ZODB 3.10, allows clients to read
      data (load & loadBefore) even after tpc_vote has started to write a new
      transaction to disk. This is done by using different 'file' objects.
      
      Issues start when a transaction is rolled back after data has been appended
      (using the writing file object). Truncating is not enough because the FilePool
      may have been used concurrently to read the end of the last transaction:
      file objects have their own read buffers which, in this case, may also contain
      the beginning of the aborted transaction.
      
      So a solution is to invalidate read buffers whenever they may contain wrong
      data. This patch does it on truncation, which happens rarely enough to not
      affect performance.
      
      We discovered this bug in the following conditions:
      - ZODB splitted in several FileStorage
      - many conflicts in the first committed DB, but always resolved
      - unresolved conflict in another DB
      If the transaction is replayed with success (no more conflict in the other DB),
      a subsequent load of the object that could be resolved in the first DB may, for
      example, return a wrong serial (tid of the aborted transaction) if the layout
      of the committed transaction matches that of the aborted one.
      
      The bug usually manifests with POSKeyError & CorruptedDataError exceptions in
      ZEO logs, for example while trying to resolve a conflict (and restarting the
      transaction does not help, causing Site Errors in Zope). But theorically,
      this could also cause silent corruption or unpickling errors at client side.
      
      (cherry picked from commit 028b1922)
      
      Conflicts:
      	src/ZODB/FileStorage/FileStorage.py
      2ed12f35
  3. 17 Mar, 2016 4 commits
  4. 16 Mar, 2016 1 commit
  5. 10 Mar, 2014 3 commits
  6. 03 Mar, 2014 2 commits
  7. 03 Feb, 2014 6 commits
  8. 23 Dec, 2013 1 commit
  9. 21 Oct, 2013 2 commits
  10. 17 Sep, 2013 3 commits
  11. 11 Jun, 2012 1 commit
  12. 25 May, 2012 1 commit
  13. 19 May, 2012 2 commits
  14. 02 May, 2012 3 commits
  15. 19 Nov, 2011 1 commit
  16. 18 Nov, 2011 1 commit
    • Jim Fulton's avatar
      Bug Fixed · 0eabedc1
      Jim Fulton authored
        Conflict resolution failed when state included cross-database
        persistent references with classes that couldn't be imported.
      0eabedc1
  17. 17 Nov, 2011 2 commits
  18. 16 Nov, 2011 1 commit
    • Jim Fulton's avatar
      Bug Fixed: · 0cd1074f
      Jim Fulton authored
        Conflict resolution failed when state included persistent references
        with classes that couldn't be imported.
      0cd1074f
  19. 12 Apr, 2011 1 commit