1. 31 Jul, 2020 1 commit
  2. 29 Jul, 2020 3 commits
    • Kirill Smelkov's avatar
      pack: Clear all non-current entries after pack · 32c9a7af
      Kirill Smelkov authored
      Else those non-current entries can be used to serve a loadBefore request
      with data, while, after pack that loadBefore request must return "data
      deleted" if requested object has current revision >= packtime.
      
      Fixes checkPackVSConnectionGet from ZODB from https://github.com/zopefoundation/ZODB/pull/322
      which, without this patch fails as e.g.
      
          Failure in test checkPackVSConnectionGet (ZEO.tests.testZEO.MappingStorageTests)
          Traceback (most recent call last):
            File "/usr/lib/python2.7/unittest/case.py", line 329, in run
              testMethod()
            File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/PackableStorage.py", line 636, in checkPackVSConnectionGet
              raises(ReadConflictError, conn1.get, oid)
            File "/usr/lib/python2.7/unittest/case.py", line 473, in assertRaises
              callableObj(*args, **kwargs)
            File "/usr/lib/python2.7/unittest/case.py", line 116, in __exit__
              "{0} not raised".format(exc_name))
          AssertionError: ReadConflictError not raised
      32c9a7af
    • Kirill Smelkov's avatar
      changes: Add changelog entry for #160 · 3de995f7
      Kirill Smelkov authored
      3de995f7
    • Kirill Smelkov's avatar
      Include both modified and just created objects into invalidations · ab86bd72
      Kirill Smelkov authored
      Starting from 1999 (b3805a2f "just getting started") only modified - not
      just created - objects were included into ZEO invalidation messages:
      
      https://github.com/zopefoundation/ZEO/commit/b3805a2f#diff-52fb76aaf08a1643cdb8fdaf69e37802R126-R127
      
      In 2000 this behaviour was further changed to not send invalidation
      message at all if the only objects a transaction has were the created ones:
      
      https://github.com/zopefoundation/ZEO/commit/230ffbe8#diff-52fb76aaf08a1643cdb8fdaf69e37802L163-R163
      
      In 2016 the latter was reconsidered as bug and fixed in ZEO5 because
      ZODB5 relies more heavily on MVCC semantic and needs to be notified
      about every transaction committed to storage to be able to properly
      update ZODB.Connection view:
      
      https://github.com/zopefoundation/ZEO/commit/02943acd#diff-52fb76aaf08a1643cdb8fdaf69e37802L889-R834
      https://github.com/zopefoundation/ZEO/commit/9613f09b
      
      In 2020, with this patch, I'm proposing to reconsider initial "send only
      modified, not created objects" as bug, and include both modified and
      just created objects into invalidation messages at least for the
      following reasons:
      
      - a ZODB client (not necessarily native ZODB/py client) can maintain
        raw cache for the storage. If such client tries to load an oid at
        database view when that object did not existed yet, gets "no object"
        reply and stores that information into raw cache, to properly invalidate
        the cache it needs an invalidation message from ZODB server that
        _includes_ created object.
      
      - tools like `zodb watch` [1,2,3] don't work properly (give incorrect output)
        if not all objects modified/created by a transaction are included into
        invalidation messages.
      
      - similarly to `zodb watch`, a monitoring tool, that would want to be
        notified of all created/modified objects, won't see full
        database-change picture, and so won't work properly without knowing
        which objects were created.
      
      - wendelin.core 2 - which builds data from ZODB BTrees and data objects
        into virtual filesystem - needs to get invalidation messages with both
        modified and created objects to properly implement its own lazy
        invalidation and isolation protocol for file blocks in OS cache: when
        a block of file is accessed, all clients, that have this block mmaped,
        need to be notified and asked to remmap that block into particular
        revision of the file depending on a client's view of the filesystem and
        database [4,5].
      
        To compute to where a client needs to remmap the block, WCFS server
        (that in turn acts as ZODB client wrt ZEO/NEO server), needs to be able
        to see whether client's view of the filesystem is before object creation
        (and then ask that client to pin that block to hole), or after creation
        (and then ask the client to pin that block to corresponding revision).
      
        This computation needs ZODB server to send invalidation messages in
        full: with both modified and just created objects.
      
      The patch is simple - it removes `if serial != b"\0\0\0\0\0\0\0\0"`
      before queuing oid into ZEOStorage.invalidated, and adjusts the tests
      correspondingly. From my point of view and experience, in practice, this
      patch should not cause any compatibility break nor performance regressions.
      
      Thanks beforehand,
      Kirill
      
      /cc @jimfulton
      
      [1] https://lab.nexedi.com/kirr/neo/blob/ea53a795/go/zodb/zodbtools/watch.go
      [2] neo@e0d59f5d
      [3] neo@c41c2907
      
      [4] https://lab.nexedi.com/kirr/wendelin.core/blob/1efb5876/wcfs/wcfs.go#L94-182
      [5] https://lab.nexedi.com/kirr/wendelin.core/blob/1efb5876/wcfs/client/wcfs.h#L20-71
      ab86bd72
  3. 28 Jul, 2020 1 commit
  4. 25 Jul, 2020 2 commits
  5. 24 Jul, 2020 2 commits
  6. 11 Jun, 2020 1 commit
  7. 29 Mar, 2020 1 commit
  8. 27 Mar, 2020 1 commit
  9. 16 Mar, 2020 3 commits
  10. 13 Mar, 2020 2 commits
  11. 13 Dec, 2019 2 commits
  12. 26 Aug, 2019 2 commits
  13. 31 Jul, 2019 1 commit
  14. 26 Apr, 2019 2 commits
  15. 25 Apr, 2019 4 commits
  16. 11 Apr, 2019 1 commit
  17. 09 Feb, 2019 2 commits
  18. 23 Jan, 2019 2 commits
  19. 17 Dec, 2018 1 commit
  20. 03 Dec, 2018 1 commit
  21. 25 Nov, 2018 2 commits
  22. 09 Nov, 2018 1 commit
  23. 02 Oct, 2018 1 commit
  24. 28 Mar, 2018 1 commit