1. 17 Dec, 2020 6 commits
  2. 16 Dec, 2020 6 commits
  3. 09 Dec, 2020 6 commits
  4. 08 Dec, 2020 1 commit
  5. 30 Nov, 2020 5 commits
    • Kirill Smelkov's avatar
      go/internal/xtesting: DrvTestWatch: Verify that watching works with empty transactions · 463ef9ad
      Kirill Smelkov authored
      It was not the case with ZEO4 server because we did not had
      kirr/ZEO@bf80d23d yet.
      463ef9ad
    • Kirill Smelkov's avatar
      go/zeo: tests: Terminate spawned processses with SIGTERM instead of SIGKILL · b516190c
      Kirill Smelkov authored
      For ZEO it is not strictly required, but for upcoming NEO for example
      NEOCluster needs to shutdown gracefully, else there are processes left for e.g.
      storage nodes and they dump somthing as below on the terminal after tests
      completion:
      
          === RUN   TestLoad
          2020/10/21 14:33:00 zodb: FIXME: open ../zodb/storage/fs1/testdata/1.fs: raw cache is not ready for invalidations -> NoCache forced
          === RUN   TestLoad/py
          I: runneo.py: /tmp/neo445013868/1: Started master(s): 127.0.0.1:24661
          WARNING: This is not the recommended way to import data to NEO: you should use the Importer backend instead.
          NEO also does not implement IStorageRestoreable interface, which means that undo information is not preserved when using this tool: conflict resolution could happen when undoing an old transaction.
          Migrating from ../zodb/storage/fs1/testdata/1.fs to 127.0.0.1:24661
          Migration done in 0.19877
          --- PASS: TestLoad (0.75s)
              --- PASS: TestLoad/py (0.74s)
          PASS
          ok      lab.nexedi.com/kirr/neo/go/neo  0.749s
          (neo) (z-dev) (g.env) kirr@deco:~/src/neo/src/lab.nexedi.com/kirr/neo/go/neo$ Traceback (most recent call last):
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/tests/functional/__init__.py", line 182, in start
              getattr(neo.scripts,  command).main()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/scripts/neostorage.py", line 66, in main
              app.run()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/app.py", line 147, in run
              self._run()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/app.py", line 178, in _run
              self.doOperation()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/app.py", line 266, in doOperation
              poll()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/app.py", line 87, in _poll
              self.em.poll(1)
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/lib/event.py", line 155, in poll
              self._poll(blocking)
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/lib/event.py", line 253, in _poll
              timeout_object.onTimeout()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/lib/event.py", line 259, in onTimeout
              on_timeout()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/database/manager.py", line 207, in _deferredCommit
              self.commit()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/database/manager.py", line 193, in commit
              self._commit()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/database/sqlite.py", line 90, in _commit
              retry_if_locked(self.conn.commit)
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/database/sqlite.py", line 45, in retry_if_locked
              return f(*args)
          OperationalError: disk I/O error
      b516190c
    • Kirill Smelkov's avatar
      go/zodb/zeo: Factor-out verifying empty database into xtesting.DrvTestEmptyDB · 8fafde00
      Kirill Smelkov authored
      And use it to verify other storages.
      8fafde00
    • Kirill Smelkov's avatar
    • Kirill Smelkov's avatar
  6. 27 Nov, 2020 1 commit
  7. 15 Nov, 2020 1 commit
  8. 05 Nov, 2020 9 commits
  9. 04 Nov, 2020 5 commits
    • Kirill Smelkov's avatar
      Merge branch 'master' into t · 58689cb6
      Kirill Smelkov authored
      * master:
        go/neo/neonet: Increase signal/noise in dump of packets with decode problems
        fixup! go/neo/proto: Switch enum encoding from int32 to int8
        go/zodb/zeo: Load: loadBefore RPC can return None to indicate non-existing object
        go/internal/xtesting: DrvTestLoad: Verify load of non-existing and not-yet-created objects
      58689cb6
    • Kirill Smelkov's avatar
      go/neo/neonet: Increase signal/noise in dump of packets with decode problems · 774f12c3
      Kirill Smelkov authored
      We recently hit "decode: buffer overflow" errors due to mismatch in
      between NEO/go and NEO/py (see previous patch). With dumpio=true that
      problem was showing itself as
      
          127.0.0.1:50624 > 127.0.0.1:41863: .5 GetObject &{0000000000000000 ffffffffffffffff 0285cbac258bf266}
          127.0.0.1:50624 < 127.0.0.1:41863: .5 (proto.Error) decode: buffer overflow; #24 [24]: 00 00 00 03 00 00 00 10 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 Error ACK ;  [24]tail: 00 00 00 03 00 00 00 10 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30
              xtesting.go:306: load 0285cbac258bf265:0000000000000000: returned err unexpected:
                  have: neo://1@127.0.0.1:24078: load 0285cbac258bf265:0000000000000000: 127.0.0.1:50624 - 127.0.0.1:41863 .5: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:24078: load 0285cbac258bf265:0000000000000000: 0000000000000000: object was not yet created
      
      Here after printing error and dumping all bytes from packet payload, it
      was printing again message type, message value from to-be-decoded place
      (which is zero-initialized) and data bytes again.
      -> Fix it not to print anything after dump of payload data:
      
          127.0.0.1:60518 > 127.0.0.1:46719: .5 GetObject &{0000000000000000 ffffffffffffffff 0285cbac258bf266}
          127.0.0.1:60518 < 127.0.0.1:46719: .5 (Error) decode: buffer overflow; #24 [24]: 00 00 00 03 00 00 00 10 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30
              xtesting.go:306: load 0285cbac258bf265:0000000000000000: returned err unexpected:
                  have: neo://1@127.0.0.1:27853: load 0285cbac258bf265:0000000000000000: 127.0.0.1:60518 - 127.0.0.1:46719 .5: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:27853: load 0285cbac258bf265:0000000000000000: 0000000000000000: object was not yet created
      774f12c3
    • Kirill Smelkov's avatar
      fixup! go/neo/proto: Switch enum encoding from int32 to int8 · 05463172
      Kirill Smelkov authored
      In commit 5f13cc85 I changed enum encodings from int32 to int8, but did
      not noticed that NEO/py commit 52db5607 ("protocol: a single byte is
      more than enough to encode enums") despite specified intent and
      ErrorCodes being marked with @Enum, changed encoding only for fields
      that are marked as PEnum in structures. In NEO/py the Error.code field
      is still marked as PNumber which encodes as 32-bit integer on the wire.
      -> Fix it back.
      
      With recent xtesting.DrvTestLoad update this error was manifesting itself as (on @t branch):
      
      --- FAIL: TestLoad (2.08s)
          --- FAIL: TestLoad/py (2.07s)
              xtesting.go:306: load 0285cbac258bf265:0000000000000000: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac258bf265:0000000000000000: 127.0.0.1:42288 - 127.0.0.1:37109 .5: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac258bf265:0000000000000000: 0000000000000000: object was not yet created
              xtesting.go:306: load 0285cbac3d0369e5:0000000000000001: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac3d0369e5:0000000000000001: 127.0.0.1:42288 - 127.0.0.1:37109 .13: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac3d0369e5:0000000000000001: 0000000000000001: object was not yet created
              xtesting.go:306: load 0285cbac41b4e832:0000000000000002: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac41b4e832:0000000000000002: 127.0.0.1:42288 - 127.0.0.1:37109 .21: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac41b4e832:0000000000000002: 0000000000000002: object was not yet created
              xtesting.go:306: load 0285cbac4666667f:0000000000000003: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac4666667f:0000000000000003: 127.0.0.1:42288 - 127.0.0.1:37109 .29: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac4666667f:0000000000000003: 0000000000000003: object was not yet created
              xtesting.go:306: load 0285cbac4fc96318:0000000000000004: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac4fc96318:0000000000000004: 127.0.0.1:42288 - 127.0.0.1:37109 .41: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac4fc96318:0000000000000004: 0000000000000004: object was not yet created
              xtesting.go:306: load 0285cbac547ae165:0000000000000005: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac547ae165:0000000000000005: 127.0.0.1:42288 - 127.0.0.1:37109 .49: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac547ae165:0000000000000005: 0000000000000005: object was not yet created
              xtesting.go:306: load 0285cbac628f5c4b:0000000000000006: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac628f5c4b:0000000000000006: 127.0.0.1:42288 - 127.0.0.1:37109 .65: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac628f5c4b:0000000000000006: 0000000000000006: object was not yet created
              xtesting.go:306: load 0285cbaca444447f:0000000000000007: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbaca444447f:0000000000000007: 127.0.0.1:42288 - 127.0.0.1:37109 .125: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbaca444447f:0000000000000007: 0000000000000007: object was not yet created
              xtesting.go:306: load 0285cbacbbbbbbff:0000000000000008: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbacbbbbbbff:0000000000000008: 127.0.0.1:42288 - 127.0.0.1:37109 .149: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbacbbbbbbff:0000000000000008: 0000000000000008: object was not yet created
              xtesting.go:306: load 0285cbad80da7498:0000000000000009: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbad80da7498:0000000000000009: 127.0.0.1:42288 - 127.0.0.1:37109 .269: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbad80da7498:0000000000000009: 0000000000000009: object was not yet created
              xtesting.go:331: load 7fffffffffffffff:000000000000000a: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 7fffffffffffffff:000000000000000a: 127.0.0.1:42288 - 127.0.0.1:37109 .295: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 7fffffffffffffff:000000000000000a: 000000000000000a: no such object
      05463172
    • Kirill Smelkov's avatar
      go/zodb/zeo: Load: loadBefore RPC can return None to indicate non-existing object · f8696942
      Kirill Smelkov authored
      Before this patch and with updated DrvTestLoad (see previous patch) it was failing as:
      
      --- FAIL: TestLoad (0.52s)
          --- FAIL: TestLoad/py/msgpack=false (0.25s)
              xtesting.go:306: load 0285cbac258bf265:0000000000000000: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac258bf265:0000000000000000: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac258bf265:0000000000000000: 0000000000000000: no such object
              xtesting.go:306: load 0285cbac3d0369e5:0000000000000001: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac3d0369e5:0000000000000001: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac3d0369e5:0000000000000001: 0000000000000001: no such object
              xtesting.go:306: load 0285cbac41b4e832:0000000000000002: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac41b4e832:0000000000000002: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac41b4e832:0000000000000002: 0000000000000002: no such object
              xtesting.go:306: load 0285cbac4666667f:0000000000000003: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac4666667f:0000000000000003: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac4666667f:0000000000000003: 0000000000000003: no such object
              xtesting.go:306: load 0285cbac4fc96318:0000000000000004: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac4fc96318:0000000000000004: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac4fc96318:0000000000000004: 0000000000000004: no such object
              xtesting.go:306: load 0285cbac547ae165:0000000000000005: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac547ae165:0000000000000005: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac547ae165:0000000000000005: 0000000000000005: no such object
              xtesting.go:306: load 0285cbac628f5c4b:0000000000000006: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac628f5c4b:0000000000000006: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac628f5c4b:0000000000000006: 0000000000000006: no such object
              xtesting.go:306: load 0285cbaca444447f:0000000000000007: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbaca444447f:0000000000000007: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbaca444447f:0000000000000007: 0000000000000007: no such object
              xtesting.go:306: load 0285cbacbbbbbbff:0000000000000008: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbacbbbbbbff:0000000000000008: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbacbbbbbbff:0000000000000008: 0000000000000008: no such object
              xtesting.go:306: load 0285cbad80da7498:0000000000000009: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbad80da7498:0000000000000009: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbad80da7498:0000000000000009: 0000000000000009: no such object
          --- FAIL: TestLoad/py/msgpack=true (0.26s)
              xtesting.go:306: load 0285cbac258bf265:0000000000000000: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac258bf265:0000000000000000: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac258bf265:0000000000000000: 0000000000000000: no such object
              xtesting.go:306: load 0285cbac3d0369e5:0000000000000001: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac3d0369e5:0000000000000001: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac3d0369e5:0000000000000001: 0000000000000001: no such object
              xtesting.go:306: load 0285cbac41b4e832:0000000000000002: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac41b4e832:0000000000000002: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac41b4e832:0000000000000002: 0000000000000002: no such object
              xtesting.go:306: load 0285cbac4666667f:0000000000000003: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac4666667f:0000000000000003: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac4666667f:0000000000000003: 0000000000000003: no such object
              xtesting.go:306: load 0285cbac4fc96318:0000000000000004: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac4fc96318:0000000000000004: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac4fc96318:0000000000000004: 0000000000000004: no such object
              xtesting.go:306: load 0285cbac547ae165:0000000000000005: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac547ae165:0000000000000005: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac547ae165:0000000000000005: 0000000000000005: no such object
              xtesting.go:306: load 0285cbac628f5c4b:0000000000000006: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac628f5c4b:0000000000000006: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac628f5c4b:0000000000000006: 0000000000000006: no such object
              xtesting.go:306: load 0285cbaca444447f:0000000000000007: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbaca444447f:0000000000000007: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbaca444447f:0000000000000007: 0000000000000007: no such object
              xtesting.go:306: load 0285cbacbbbbbbff:0000000000000008: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbacbbbbbbff:0000000000000008: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbacbbbbbbff:0000000000000008: 0000000000000008: no such object
              xtesting.go:306: load 0285cbad80da7498:0000000000000009: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbad80da7498:0000000000000009: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbad80da7498:0000000000000009: 0000000000000009: no such object
      f8696942
    • Kirill Smelkov's avatar
      go/internal/xtesting: DrvTestLoad: Verify load of non-existing and not-yet-created objects · 4acadb94
      Kirill Smelkov authored
      This currently fails on ZEO and NEO. Those storages will be fixed in the
      following patches (NEO only on @t branch for now).
      4acadb94