1. 28 Jun, 2018 7 commits
  2. 27 Jun, 2018 17 commits
  3. 26 Jun, 2018 6 commits
  4. 17 Apr, 2018 3 commits
    • Kirill Smelkov's avatar
    • Kirill Smelkov's avatar
      tests: Explicitly close ZODB connections for places with warnings found by previous patch · 01b995a4
      Kirill Smelkov authored
      bigfile/tests/test_filezodb.py ........W: testdb: teardown: <Connection at 7f8fe2b43b90> left not closed by test code; opened by:
        ...
        File "/home/kirr/src/wendelin/wendelin.core/bigfile/tests/test_filezodb.py", line 754, in test_bigfile_zblk1_zdata_reuse
          _test_bigfile_zblk1_zdata_reuse()
        File "/home/kirr/src/wendelin/wendelin.core/bigfile/tests/test_filezodb.py", line 759, in _test_bigfile_zblk1_zdata_reuse
          root = dbopen()
        File "/home/kirr/src/wendelin/wendelin.core/bigfile/tests/test_filezodb.py", line 47, in dbopen
          return testdb.dbopen()
        File "/home/kirr/src/wendelin/wendelin.core/lib/testing.py", line 188, in dbopen
          self.connv.append( (weakref.ref(conn), ''.join(traceback.format_stack())) )
      
      lib/tests/test_zodb.py .W: testdb: teardown: <Connection at 7f8fe26f13d0> left not closed by test code; opened by:
        ...
        File "/home/kirr/src/wendelin/wendelin.core/lib/tests/test_zodb.py", line 49, in test_deactivate_btree
          root = dbopen()
        File "/home/kirr/src/wendelin/wendelin.core/lib/tests/test_zodb.py", line 30, in dbopen
          return testdb.dbopen()
        File "/home/kirr/src/wendelin/wendelin.core/lib/testing.py", line 188, in dbopen
          self.connv.append( (weakref.ref(conn), ''.join(traceback.format_stack())) )
      01b995a4
    • Kirill Smelkov's avatar
      tests: Force-close ZODB connections in teardown, that testing code forgot to explicitly close · 5a5ed2c7
      Kirill Smelkov authored
      If a test forgets to explicitly close ZODB connection it was using, this
      connection stays alive in transaction synchronizers (it is a weakset),
      and continues to be used on e.g. transaction.commit() when all
      synchronizers are invoked. This could lead to crashes like below when
      underlying ZODB storage was closed by test module teardown and testing
      moved on to another test module:
      
          $ WENDELIN_CORE_TEST_DB="<neo>" py.test  bigfile/tests/test_filezodb.py::test_bigfile_zblk1_zdata_reuse lib/tests/test_zodb.py
          ======= test session starts ========
          platform linux2 -- Python 2.7.14+, pytest-3.5.0, py-1.5.3, pluggy-0.6.0
          rootdir: /home/kirr/src/wendelin/wendelin.core, inifile:
          collected 2 items
      
          bigfile/tests/test_filezodb.py .                                     [ 50%]
          lib/tests/test_zodb.py F                                             [100%]
      
          ______ test_deactivate_btree _______
      
              def test_deactivate_btree():
                  root = dbopen()
                  # init btree with many leaf nodes
                  leafv = []
                  root['btree'] = B = IOBTree()
                  for i in range(10000):
                      B[i] = xi = XInt(i)
                      leafv.append(xi)
          >       transaction.commit()
      
          lib/tests/test_zodb.py:56:
          _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
          ../venv/z5/local/lib/python2.7/site-packages/transaction/_manager.py:131: in commit
              return self.get().commit()
          ../venv/z5/local/lib/python2.7/site-packages/transaction/_transaction.py:316: in commit
              self._synchronizers.map(lambda s: s.afterCompletion(self))
          ../venv/z5/local/lib/python2.7/site-packages/transaction/weakset.py:62: in map
              f(elt)
          ../venv/z5/local/lib/python2.7/site-packages/transaction/_transaction.py:316: in <lambda>
              self._synchronizers.map(lambda s: s.afterCompletion(self))
          ../venv/z5/local/lib/python2.7/site-packages/ZODB/Connection.py:757: in afterCompletion
              self.newTransaction(transaction, False)
          ../venv/z5/local/lib/python2.7/site-packages/ZODB/Connection.py:737: in newTransaction
              invalidated = self._storage.poll_invalidations()
          ../venv/z5/local/lib/python2.7/site-packages/ZODB/mvccadapter.py:131: in poll_invalidations
              self._start = p64(u64(self._storage.lastTransaction()) + 1)
          _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
      
          self = <neo.client.Storage.Storage object at 0x7ffa1be8d410>
      
              def lastTransaction(self):
                  # Used in ZODB unit tests
          >       return self.app.last_tid
          E       AttributeError: 'NoneType' object has no attribute 'last_tid'
      
          ../../neo/src/lab.nexedi.com/kirr/neo/neo/client/Storage.py:181: AttributeError
      
      where NEO's Storage.app is None because the storage was closed.
      
      ----
      
      To avoid such kind of failures make sure TestDB.teardown() always closes
      all ZODB connections that were ever opened via TestDB.dbopen().
      
      Add a warning about such force-closing with information about corresponding
      connection and code place that created it, so that it is easy to
      understand which test needs a fix.
      
      /suggested-by @jm
      5a5ed2c7
  5. 16 Apr, 2018 4 commits
    • Kirill Smelkov's avatar
      wendelin.core v0.12 · 72df6eb6
      Kirill Smelkov authored
      72df6eb6
    • Kirill Smelkov's avatar
      tox: v↑ everything · 0dbf3c44
      Kirill Smelkov authored
      - start testing NEO with ZODB4 and ZODB5. ZODB4-*-NEO works while
        ZODB5-*-NEO currently fails [1].
      - update NumPy to latest releases.
      - update Python 3 to latest releases.
      0dbf3c44
    • Kirill Smelkov's avatar
      tests: Teach to run with ZEO5 · 808b59b7
      Kirill Smelkov authored
      Since 7fc4ec66 (tests: Allow to test with ZEO & NEO ZODB storages) we
      can run the tests with either FileStorage, ZEO or NEO. But ZEO test
      adapter started to fail with ZEO5:
      
      	self = <wendelin.lib.testing.TestDB_ZEO object at 0x7f1feb5091d0>
      
      	    def setup(self):
      	        port  = self.zeo_forker.get_port()
      	        zconf = self.zeo_forker.ZEOConfig(('', port))
      	        self.addr, self.adminaddr, self.pid, self.path = \
      	>               self.zeo_forker.start_zeo_server(zeo_conf=zconf, port=port)
      	E       ValueError: need more than 2 values to unpack
      
      This is because in ZEO5 forker.start_zeo_server() was reworked to return only
      addr and stop closure instead of returning all details and relying on caller to
      implement stop itself.
      
      Adapt the test to detect ZEO5 and use new calling convention.
      808b59b7
    • Kirill Smelkov's avatar
      tox/ZODB3: Limit ZConfig < 3.2.0 · c96daa2f
      Kirill Smelkov authored
      With ZConfig==3.2.0 (released 2017-06-22) ZEO started to get file names
      as unicode, which eventually breaks on assert in ZODB3.FileStorage code:
      
          Traceback (most recent call last):
            File "/home/kirr/src/wendelin/r/work/venv/local/lib/python2.7/site-packages/ZEO/tests/zeoserver.py", line 215, in <module>
              main()
            File "/home/kirr/src/wendelin/r/work/venv/local/lib/python2.7/site-packages/ZEO/tests/zeoserver.py", line 188, in main
              server = ZEO.runzeo.create_server({"1": storage}, zo)
            File "/home/kirr/src/wendelin/r/work/venv/local/lib/python2.7/site-packages/ZEO/runzeo.py", line 366, in create_server
              auth_realm = options.auth_realm,
            File "/home/kirr/src/wendelin/r/work/venv/local/lib/python2.7/site-packages/ZEO/StorageServer.py", line 890, in __init__
              self._setup_invq(name, storage)
            File "/home/kirr/src/wendelin/r/work/venv/local/lib/python2.7/site-packages/ZEO/StorageServer.py", line 928, in _setup_invq
              self.invq[name] = list(lastInvalidations(self.invq_bound))
            File "/home/kirr/src/wendelin/r/work/venv/local/lib/python2.7/site-packages/ZODB/FileStorage/FileStorage.py", line 1238, in lastInvalidations
              for trans in FileIterator(self._file_name, pos=pos)]
            File "/home/kirr/src/wendelin/r/work/venv/local/lib/python2.7/site-packages/ZODB/FileStorage/FileStorage.py", line 1640, in __init__
              assert isinstance(filename, str), `filename`
          AssertionError: u'/home/kirr/src/wendelin/r/work/wendelin.core/Data.fs'
      
      -> So for ZODB3 pin ZConfig to ZConfig <3.2.0 which is known to work with ZODB3.
      c96daa2f
  6. 12 Apr, 2018 1 commit
    • Kirill Smelkov's avatar
      bigarray: ArrayRef utility · e9d61a89
      Kirill Smelkov authored
      ArrayRef is a reference to NumPy array.
      
      The reference is represented by root array object and instructions how to
      create original array as some view of the root.
      
      Such reference could be useful in situations where one needs to pass arrays
      between processes and instead of copying array data, leverage the fact that
      top-level array, for example ZBigArray, is already persisted separately, and
      only send small amount of information referencing data in question.
      
      Use `ArrayRef(array)` to create reference to an ndarray.
      
      Use `.deref()` to convert ArrayRef to pointed array object.
      
      NOTE
      
      don't send ArrayRef unconditionally - for example when array object is
      small regular ndarray with also regular, but big, root ndarray, sending
      ArrayRef will send whole data for root object, not for small leaf.
      
      Sending ArrayRef only makes sense when root object is known to be already
      persisted by other means, for example something like below in ZODB context:
      
      ```python
        aref = ArrayRef(a)
        if isinstance(aref.root, Persistent):
            send aref
        else:
            send a 
      ```
      
      Please see individual patches for more details.
      
      /reviewed-on nexedi/wendelin.core!6
      e9d61a89
  7. 02 Apr, 2018 2 commits
    • Kirill Smelkov's avatar
      bigarray: ArrayRef support for BigArray · 450ad804
      Kirill Smelkov authored
      Rationale
      ---------
      
      Array reference could be useful in situations where one needs to pass arrays
      between processes and instead of copying array data, leverage the fact that
      top-level array, for example ZBigArray, is already persisted separately, and
      only send small amount of information referencing data in question.
      
      Implementation
      --------------
      
      BigArray is not regular NumPy array and so needs explicit support in
      ArrayRef code to find root object and indices. This patch adds such
      support via the following way:
      
      - when BigArray.__getitem__ creates VMA, it remembers in the VMA
        the top-level BigArray object under which this VMA was created.
      
      - when ArrayRef is finding root, it can detect such VMAs, because it will
        be pointed to by the most top regular ndarray's .base, and in turn gets
        top-level BigArray object from the VMA.
      
      - further all indices computations are performed, similarly to complete regular
        ndarrays case, on ndarrays root and a. But in the end .lo and .hi are
        adjusted for the corresponding offset of where root is inside whole
        BigArray.
      
      - there is no need to adjust .deref() at all.
      
      For remembering information into a VMA and also to be able to get
      (readonly) its mapping addresses _bigfile.c extension has to be extended
      a bit. Since we are now storing arbitrary python object attached to
      PyVMA - it can create cycles - and so PyVMA accordingly adjusted to
      support cyclic garbage collector.
      
      Please see the patch itself for more details and comments.
      450ad804
    • Kirill Smelkov's avatar
      bigarray: Add ArrayRef utility · d53371b6
      Kirill Smelkov authored
      ArrayRef is a tool to find out for a NumPy array its top-level root
      parent and remember instructions how to recreate original array from
      the root. For example if
      
      	root = arange(1E7)
      	z = root[1000:2000]
      	a = z[10:20]
      
      `ArrayRef(a)` will find out that the root array for `a` is `root` and
      that `a` occupies 1010:1020 bytes in it. The vice versa operation is
      also possible, for example given
      
      	aref = ArrayRef(a)
      
      it is possible to restore original `a` from `aref`:
      
      	a_ = aref.deref()
      	assert array_equal(a_, a)
      
      the restoration works without copying by creating appropriate view of
      root.
      
      ArrayRef should work reliably for arrays of arbitrary dimensions,
      strides etc - even fancy arrays created via stride tricks such as arrays
      whose elements overlap each other should be supported.
      
      This patch adds ArrayRef with support for regular ndarrays only.
      
      The next patch will add ArrayRef support for BigArray and description
      for ArrayRef rationale.
      d53371b6