1. 04 May, 2023 1 commit
  2. 18 Apr, 2023 3 commits
    • Dieter Maurer's avatar
      Merge pull request #380 from zopefoundation/racetest · 30c861dc
      Dieter Maurer authored
      `racetest` improvement
      30c861dc
    • dieter's avatar
      ca41ccd1
    • Kirill Smelkov's avatar
      racetest: Review 2 · 2d456463
      Kirill Smelkov authored
      - Rename `TestGroup` -> `TestWorkGroup`. I originally named it as just
        WorkGroup similarly to sync.WorkGroup in pygolang, but later added
        "Test" prefix to highlight the difference that this class manages
        working group of threads that take part in tests, instead of group of
        arbitrary threads. However in that process I dropped the "Work" part
        which turned the name to be somewhat ambiguous: it is not clear whether
        "TestGroup" is a group of threads serving one test, or a group of
        separate tests without any relation to threading. To remove the
        ambiguity let's restore the "Work" in the name so that both "Test" and
        "WorkGroup" are there.
      
      - Review test_racetest.py a bit:
      
        * make it a bit more robust by increasing "ok" timeout from 0.1s to 10s.
          in my experience 0.1s is too little and will regularly hit "timeout"
          when CI machines are overloaded. I'm still somewhat uncomfortable
          with 0.1s timeout we left in tests that exercise timeout handling,
          but let's leave it as is for now and see how it goes.
      
        * add test for cases when only one thread fails out of many, and when
          several threads fail too.
      
        * minor cosmetics.
      
      - spellcheck.
      2d456463
  3. 17 Apr, 2023 3 commits
  4. 15 Apr, 2023 1 commit
  5. 14 Apr, 2023 1 commit
    • Kirill Smelkov's avatar
      racetest: Review · f2335127
      Kirill Smelkov authored
      - Factor common logic to spawn and run group of test threads into
        TestGroup. This way the extra checks and robustness improvements, that
        Dieter just added to check_race_external_invalidate_vs_disconnect,
        become available to all tests in racetest module.
      
        Rework moved code so that nwork is not fixed beforehand and test threads
        can be added dynamically.
      
      - fix waiting logic in Finished:
        * on py2 Condition.wait does not return True/False as it does on py3 -
          we need to manually inspect the state.
        * fix race for when wait is called with already met condition:
          previously in such case it was waiting indefinitely and reporting
          failure on timeout
      
      - rename Finished to well-established concept of WaitGroup and adjust
        its interface accordingly (see
        https://pkg.go.dev/sync#WaitGroup,
        https://lab.nexedi.com/nexedi/pygolang/blob/master/golang/sync.py and
        https://lab.nexedi.com/nexedi/pygolang/blob/39dde7eb/golang/sync.cpp#L153-200)
      
      - no need to wrap try/except with additional try/finally, as
        try/except/finally works out of the box.
      f2335127
  6. 13 Apr, 2023 4 commits
  7. 28 Mar, 2023 1 commit
  8. 12 Mar, 2023 2 commits
  9. 06 Feb, 2023 1 commit
  10. 03 Feb, 2023 5 commits
  11. 09 Nov, 2022 4 commits
  12. 08 Nov, 2022 4 commits
  13. 02 Sep, 2022 1 commit
  14. 01 Sep, 2022 1 commit
  15. 31 Aug, 2022 3 commits
  16. 18 Aug, 2022 1 commit
  17. 15 Aug, 2022 4 commits
    • Kirill Smelkov's avatar
      racetest: Add test for disconnect / invalidation race · fc20f3aa
      Kirill Smelkov authored
      This currently fails for ZEO (see Bug2 in
      https://github.com/zopefoundation/ZEO/issues/209 for details):
      
          (z-dev) kirr@deca:~/src/wendelin/z/ZEO5$ ZEO_MTACCEPTOR=1 zope-testrunner -fvvvx --test-path=src -t check_race_external_invalidate_vs_disconnect
          /home/kirr/src/wendelin/venv/z-dev/bin/zope-testrunner traceio=True
          /home/kirr/src/wendelin/z/ZEO5/src/ZEO/StorageServer.py:51: DeprecationWarning: The mtacceptor module is deprecated and will be removed in ZEO version 6.
            'in ZEO version 6.', DeprecationWarning)
          Running tests at level 1
          Running .BlobAdaptedFileStorageTests tests:
            Set up .BlobAdaptedFileStorageTests in 0.000 seconds.
            Running:
           check_race_external_invalidate_vs_disconnect (ZEO.tests.testZEO.BlobAdaptedFileStorageTests) (1.889 s)
      
          Failure in test check_race_external_invalidate_vs_disconnect (ZEO.tests.testZEO.BlobAdaptedFileStorageTests)
          Traceback (most recent call last):
            File "/usr/lib/python2.7/unittest/case.py", line 329, in run
              testMethod()
            File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/racetest.py", line 357, in check_race_external_invalidate_vs_disconnect
              T2ObjectsInc2Phase())
            File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/util.py", line 400, in _
              return f(*argv, **kw)
            File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/racetest.py", line 446, in _check_race_xxx_vs_external_disconnect
              self.fail('\n\n'.join([_ for _ in failure if _]))
            File "/usr/lib/python2.7/unittest/case.py", line 410, in fail
              raise self.failureException(msg)
          AssertionError: T15: obj1 (6) - obj2(4) != phase (1)
          obj1._p_serial: 0x03ea4cc505486777  obj2._p_serial: 0x03ea4cc503413799  phase._p_serial: 0x03ea4cc505486777
          zconn_at: 0x03ea4cc505486777  # approximated as max(serials)
          zstor.loadBefore(obj1, @zconn.at)       ->  serial: 0x03ea4cc505486777  next_serial: None
          zstor.loadBefore(obj2, @zconn.at)       ->  serial: 0x03ea4cc504a74099  next_serial: None
          zstor.loadBefore(phase, @zconn.at)      ->  serial: 0x03ea4cc505486777  next_serial: None
          zstor._cache.clear()
          zstor.loadBefore(obj1, @zconn.at)       ->  serial: 0x03ea4cc505486777  next_serial: None
          zstor.loadBefore(obj2, @zconn.at)       ->  serial: 0x03ea4cc504a74099  next_serial: 0x03ea4cc506104155
          zstor.loadBefore(phase, @zconn.at)      ->  serial: 0x03ea4cc505486777  next_serial: 0x03ea4cc506104155
      
          T51: obj1 (6) - obj2(4) != phase (1)
          obj1._p_serial: 0x03ea4cc505486777  obj2._p_serial: 0x03ea4cc503413799  phase._p_serial: 0x03ea4cc505486777
          zconn_at: 0x03ea4cc505486777  # approximated as max(serials)
          zstor.loadBefore(obj1, @zconn.at)       ->  serial: 0x03ea4cc505486777  next_serial: None
          zstor.loadBefore(obj2, @zconn.at)       ->  serial: 0x03ea4cc503413799  next_serial: None
          zstor.loadBefore(phase, @zconn.at)      ->  serial: 0x03ea4cc505486777  next_serial: None
          zstor._cache.clear()
          zstor.loadBefore(obj1, @zconn.at)       ->  serial: 0x03ea4cc505486777  next_serial: None
          zstor.loadBefore(obj2, @zconn.at)       ->  serial: 0x03ea4cc504a74099  next_serial: None
          zstor.loadBefore(phase, @zconn.at)      ->  serial: 0x03ea4cc505486777  next_serial: None
      fc20f3aa
    • Kirill Smelkov's avatar
      racetest: Factor-out functionality to open another connection to the storage to common function · fa844159
      Kirill Smelkov authored
      We will need it in soon for the next test.
      fa844159
    • Kirill Smelkov's avatar
      racetest: Show relevant details about storage state on failure · c593c7d5
      Kirill Smelkov authored
      Suggested by @d-maurer.
      
      Example failure before this patch:
      
          Failure in test check_race_load_vs_external_invalidate (ZEO.tests.testZEO.BlobAdaptedFileStorageTests)
          Traceback (most recent call last):
            File "/usr/lib/python2.7/unittest/case.py", line 329, in run
              testMethod()
            File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/racetest.py", line 200, in check_race_load_vs_external_invalidate
              return self._check_race_load_vs_external_invalidate(T2ObjectsInc())
            File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/util.py", line 400, in _
              return f(*argv, **kw)
            File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/racetest.py", line 304, in _check_race_load_vs_external_invalidate
              self.fail([_ for _ in failure if _])
            File "/usr/lib/python2.7/unittest/case.py", line 410, in fail
              raise self.failureException(msg)
          AssertionError: ['T3: obj1 (1)  !=  obj2 (24)\n', 'T4: obj1 (1)  !=  obj2 (24)\n', 'T5: obj1 (1)  !=  obj2 (24)\n', 'T6: obj1 (1)  !=  obj2 (24)\n', 'T7: obj1 (2)  !=  obj2 (24)\n']
      
      Example failure after this patch:
      
          Failure in test check_race_load_vs_external_invalidate (ZEO.tests.testZEO.BlobAdaptedFileStorageTests)
          Traceback (most recent call last):
            File "/usr/lib/python2.7/unittest/case.py", line 329, in run
              testMethod()
            File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/racetest.py", line 202, in check_race_load_vs_external_invalidate
              return self._check_race_load_vs_external_invalidate(T2ObjectsInc())
            File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/util.py", line 400, in _
              return f(*argv, **kw)
            File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/racetest.py", line 307, in _check_race_load_vs_external_invalidate
              self.fail('\n\n'.join([_ for _ in failure if _]))
            File "/usr/lib/python2.7/unittest/case.py", line 410, in fail
              raise self.failureException(msg)
          AssertionError: T0: obj1 (1)  !=  obj2 (25)
          obj1._p_serial: 0x03ea4be4ea558066  obj2._p_serial: 0x03ea4be4eaac7077
          zconn_at: 0x03ea4be4eaac7077  # approximated as max(serials)
          zstor.loadBefore(obj1, @zconn.at)       ->  serial: 0x03ea4be4ea558066  next_serial: None
          zstor.loadBefore(obj2, @zconn.at)       ->  serial: 0x03ea4be4eaac7077  next_serial: None
          zstor._cache.clear()
          zstor.loadBefore(obj1, @zconn.at)       ->  serial: 0x03ea4be4eaac7077  next_serial: None
          zstor.loadBefore(obj2, @zconn.at)       ->  serial: 0x03ea4be4eaac7077  next_serial: None
      
          T2: obj1 (1)  !=  obj2 (25)
          obj1._p_serial: 0x03ea4be4ea558066  obj2._p_serial: 0x03ea4be4eaac7077
          zconn_at: 0x03ea4be4eaac7077  # approximated as max(serials)
          zstor.loadBefore(obj1, @zconn.at)       ->  serial: 0x03ea4be4ea558066  next_serial: None
          zstor.loadBefore(obj2, @zconn.at)       ->  serial: 0x03ea4be4eaac7077  next_serial: None
          zstor._cache.clear()
          zstor.loadBefore(obj1, @zconn.at)       ->  serial: 0x03ea4be4eaac7077  next_serial: None
          zstor.loadBefore(obj2, @zconn.at)       ->  serial: 0x03ea4be4eaac7077  next_serial: None
      
          T3: obj1 (26)  !=  obj2 (25)
          obj1._p_serial: 0x03ea4be4ec6af5cc  obj2._p_serial: 0x03ea4be4eaac7077
          zconn_at: 0x03ea4be4ec6af5cc  # approximated as max(serials)
          zstor.loadBefore(obj1, @zconn.at)       ->  serial: 0x03ea4be4ec6af5cc  next_serial: None
          zstor.loadBefore(obj2, @zconn.at)       ->  serial: 0x03ea4be4eaac7077  next_serial: 0x03ea4be4ec962b00
          zstor._cache.clear()
          zstor.loadBefore(obj1, @zconn.at)       ->  serial: 0x03ea4be4ec6af5cc  next_serial: 0x03ea4be4ec962b00
          zstor.loadBefore(obj2, @zconn.at)       ->  serial: 0x03ea4be4ec6af5cc  next_serial: 0x03ea4be4ec962b00
      c593c7d5
    • Kirill Smelkov's avatar
      racetest: Refactor tests to work via combining testing model with checkers · a03c4778
      Kirill Smelkov authored
      Introduce the notion of testing model to represent initial state,
      eveolution step and maintained invariant in the database.
      
      Refactor tests to accept arbitrary such model and to drive them through
      particular testing scenarious that try to hit specific concurrency
      issues.
      
      Not only this deduplicates the code a bit, but it also will allow us to
      run different models through existing and future checkers. Soon we will
      need to run a model with 3 variables (obj1, obj2 and phase) instead of
      only 2 (obj1 and obj2) for invalidate / disconnect test.
      
      Please see added module-level documentation for overview.
      a03c4778