An error occurred fetching the project authors.
- 25 Nov, 2022 3 commits
-
-
Levin Zimmermann authored
This patch allows using WCFS with a NEO or ZEO storage which is reachable by a URL which contains an ipv6 host. Without this patch the following example doesn't work: >>> from wendelin.lib.zodb import dbopen >>> root = dbopen("neo://cluster-name@[::1]:2051") >>> # "abc" points to a ZBigArray >>> root["abc"][0] It doesn't work because the parser missed adding square brackets around ipv6 hosts, due to which unparsing the resulting URL resulted in a wrong interpretation where a port starts. This patch furthermore amends 'test_zstor_2zurl' to test ZEO and NEO storages with ipv6 hosts. --- /reviewed-by @kirr /reviewed-on nexedi/wendelin.core!13
-
Levin Zimmermann authored
This patch adds comprehensive tests for 'wendelin.lib.zodb.zstor_2zurl'. Before this patch only one related test existed ('test_zurlstable'). This test only lightly checked correct functionality of 'zstor_2zurl'. Therefore we added the new tests 'test_zstor_2zurl' and 'test_zurlsamedb'. The new tests only cover existing functionality. --- Co-authored-by: kirr /reviewed-by @kirr /reviewed-on nexedi/wendelin.core!13
-
Levin Zimmermann authored
Before this patch 'zsync(storage)' was effectless for ZEO storages, it didn't synchronize the client with the server. This patch fixes 'zsync', so that it also performs synchronization of ZEO clients. Background information: ======================= 2006 the sync mode of ZEO has been removed: nexedi/ZEO@629b0667 and only async mode was supported from then. This means, that the "sync" method of ZEO.ClientStorage was in fact effectless. In ZEO 5 the "server-sync" option has been added: https://github.com/zopefoundation/ZEO/pull/63 Setting this option to 'True' makes the 'sync' method performing a "server round trip, thus causing client to wait for outstanding invalidations" [1]. In this patch we imitate the effect of this flag for both ZEO 4 and ZEO 5. [1] https://github.com/zopefoundation/ZEO/blob/423cb8563be3e1ee0bb4297ee980d9b74f09c710/src/ZEO/ClientStorage.py#L225-L226 --- /reviewed-by @kirr /reviewed-on nexedi/wendelin.core!13
-
- 16 Nov, 2021 2 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
This way on plain ZODB4 the following non-wcfs tests will continue to pass test.py/fs-!wcfs test.py/zeo-!wcfs test.py/neo-!wcfs instead of failing as e.g. in here: https://nexedijs.erp5.net/#/test_result_module/20211116-123A66706 On plain ZODB4 WCFS-related functionality - which uses zconn_at - will continue to raise corresponding assertion in WCFS-related tests, as e.g. in https://nexedijs.erp5.net/#/test_result_module/20211116-123A66706/6
-
- 28 Oct, 2021 4 commits
-
-
Kirill Smelkov authored
In 3bd82127 (lib/zodb: Add zconn_at draft (ZODB5 only)) we added zconn_at function to find out as of which state a ZODB connection is viewing the database. That was ZODB5-only however. Let's add support for ZODB4 now - by requiring ZODB4-wc2 - a version of ZODB4 with MVCC backported from ZODB5: nexedi/ZODB!1 This makes wendelin.core to work on either ZODB5 or ZODB4-wc2, but not plain ZODB4. However as zconn_at will be used only for WCFS-integration, non-wcfs mode will continue to work on all ZODB5, ZODB4-wc2 and plain ZODB4. ZBigFile + WCFS client integration will use zconn_at to open WCFS connection that corresponds to ZODB connection. Preliminary history: kirr/wendelin.core@1c3b7750 X zconn_at for ZODB4
-
Kirill Smelkov authored
Add patch to ZODB.Connection to support callback on after database is closed. ZBigFile + WCFS client integration will use this callback to close WCFS connection when corresponding ZODB.DB is closed. Preliminary history: kirr/wendelin.core@a26d9659 X lib/zodb: Connection += onShutdownCallback
-
Kirill Smelkov authored
In 959ae2d0 (lib/zodb: Add patch to ZODB.Connection to support callback on connection DB view change) we added patch for ZODB.Connection to support callback when database view of the connection changes. At that time the patch was working for ZODB5 and ZODB4 was TODO. Let's add support for ZODB4 (both ZODB4 and ZODB4-wc2) now. As a reminder: ZBigFile + WCFS client integration will use this callback to keep WCFS connection in sync with ZODB connection. Preliminary history: kirr/wendelin.core@533a4cfa X onResyncCallback for ZODB4
-
Kirill Smelkov authored
In 6637d216 (lib/zodb: Add zstor_2zurl - way to convert a ZODB storage into URL to access it) we added zstor_2zurl function to convert a ZODB storage client object into an URL to access the storage. At that time the function knew how to understand FileStorage only. Let's add support for other storages that WCFS will need to support now. NEO URI scheme matches the one currently used on ZODB/go side. It semantically needs nexedi/neoppod!18 to be also applied to NEO/py side, but we do not care for now that that patch is not merged (yet, or forever) because extracted ZURL is used only with WCFS which uses NEO/go. NEO support also depends on custom patch to remember SSL credentials on NEO Client: kirr/neo@a2f192cb Some preliminary history: kirr/wendelin.core@5cb39463 fixup! X wcfs/zeo started to work locally kirr/wendelin.core@1cf3b228 X zstor_2zurl += NEO kirr/wendelin.core@7f8fa32a X lib/zodb: zstor_2zurl += NEO/SSL support kirr/wendelin.core@e26524df X wcfs, lib/zodb: DemoStorage support
-
- 01 Apr, 2021 1 commit
-
-
Kirill Smelkov authored
The tests verify that there is no concurrency bugs around load, Connection.open and invalidations. See e.g. https://github.com/zopefoundation/ZODB/issues/290 https://github.com/zopefoundation/ZEO/issues/155 By including the tests into wendelin.core, we will have CI coverage for all supported storages (FileStorage, ZEO, NEO), and for all supported ZODB (currently ZODB4, ZODB4-wc2 and ZODB5). ZEO5 is know to currently fail zloadrace. However, even though ZODB#290 was fixed, ZEO5 turned out to also fail on zopenrace: def test_zodb_zopenrace(): # exercises ZODB.Connection + particular storage implementation > zopenrace.main() lib/tests/test_zodb.py:382: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ <decorator-gen-1>:2: in main ??? ../../tools/go/pygolang/golang/__init__.py:103: in _ return f(*argv, **kw) lib/tests/testprog/zopenrace.py:115: in main test(zstor) <decorator-gen-2>:2: in test ??? ../../tools/go/pygolang/golang/__init__.py:103: in _ return f(*argv, **kw) lib/tests/testprog/zopenrace.py:201: in test wg.wait() golang/_sync.pyx:246: in golang._sync.PyWorkGroup.wait ??? golang/_sync.pyx:226: in golang._sync.PyWorkGroup.go.pyrunf ??? lib/tests/testprog/zopenrace.py:165: in T1 t1() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def t1(): transaction.begin() zconn = db.open() root = zconn.root() obj1 = root['obj1'] obj2 = root['obj2'] # obj1 - reload it from zstor # obj2 - get it from zconn cache obj1._p_invalidate() # both objects must have the same values i1 = obj1.i i2 = obj2.i if i1 != i2: > raise AssertionError("T1: obj1.i (%d) != obj2.i (%d)" % (i1, i2)) E AssertionError: T1: obj1.i (3) != obj2.i (2) lib/tests/testprog/zopenrace.py:156: AssertionError
-
- 08 Mar, 2021 1 commit
-
-
Kirill Smelkov authored
It's been a while since last ZODB3 3.10.7 release in 2016 and the last commit in upstream ZODB3 repository (3.10 branch) is from 2017. The world switched since then to ZODB4 and to ZODB5 after that. We were still requiring ZODB3, because ZODB3 3.11 egg was just a dependency on newer ZODB, ZEO, BTrees and persistent; and this way we could be supporting all ZODB3.10.x and ZODB4 and ZODB5 via ZODB3.11. However upcoming Wendelin.core 2, for its proper working, needs MVCC semantic as implemented in ZODB5. This forces us, even for ZODB4, to backport non-trivial bits from ZODB5 (see [1]). Maintaining ZODB3 support at this point becomes non-practical, because, to our knowledge, there is no wendelin.core user that plans to continue using ZODB3 without switching to at least ZODB4 in the near future. So goodbye ZODB3. Even though ZODB still stays with us, it gives a feeling similar to [2], because in 2014, when I was myself learning ZODB, it was through ZODB3 - still at the time when all ZODB bits were living together in one place. [1] nexedi/ZODB!1 [2] https://lists.osuosl.org/pipermail/darcs-users/2008-September/014095.html
-
- 15 Apr, 2020 3 commits
-
-
Kirill Smelkov authored
Wendelin.core 2 will need to spawn WCFS filesystem server that accesses the same ZODB database as the program that spawns it. The database argument passed to WCFS is passed in the form of URL[1,2]. Even though zodburi provides way to convert an URL into ZODB storage instance, there is currently no way for reverse operation - to convert ZODB storage instance into URL to access it(*). So we have to build it by our own. Provide zstor_2zurl stub that currently works for FileStorage only. ZEO and NEO support is TODO. In the future we might want to move this functionality into zodbtools/py. [1] https://lab.nexedi.com/nexedi/zodbtools/blob/a2e4dd23/zodbtools/help.py#L27-53 [2] https://lab.nexedi.com/kirr/neo/blob/3d909114/go/zodb/zodbtools/help.go#L25-51 (*) contrary to ZODB/go where this functionality is provided out of the box: https://godoc.org/lab.nexedi.com/kirr/neo/go/zodb#IStorage
-
Kirill Smelkov authored
Wendelin.core 2 will need to hook into when client ZODB.Connection changes its database view and readjust WCFS-level client connection accordingly. ZODB.Connection can change its view on either connection reopen, or even without reopen on start of new transaction. This patch implements ZODB.Connection.onResyncCallback for ZODB5 only. ZODB4 and ZODB3 support is TODO.
-
Kirill Smelkov authored
For wendelin.core v2 we need a way to know at which particular database state application-level ZODB connection is viewing the database. Knowing that state, WCFS client library will interact with WCFS filesystem server and, in simple terms, request the server to provide data as of that particular database state. Contrary to ZODB/go[1] ZODB/py does not provide the functionality to obtain DB state of connection view, so we have to build it ourselves. Let us call the function that for a client ZODB connection returns database state corresponding to its database view as zconn_at. It is relatively easy to implement zconn_at for ZODB5, since ZODB5 adopted MVCC uniformly and this patch does just that. However even with ZODB5 currently all released ZODB5 versions have race in Connection.open() vs invalidations[2], and so the first ZODB5 release with which zconn_at implemented here will work reliable should be upcoming ZODB 5.5.2 It is TODO to implement zconn_at for ZODB4 and ZODB3, which organize things differently. Please note what would happen if zconn_at gives, even a bit, incorrect answer: wcfs client will ask wcfs server to provide array data as of different database state compared to current on-client ZODB connection. This will result in that data accessed via ZBigArray will _not_ correspond to all other data accessed via regular ZODB mechanism. It is, in other words, would be a data corruptions. [1] https://godoc.org/lab.nexedi.com/kirr/neo/go/zodb#Connection [2] https://github.com/zopefoundation/ZODB/issues/290
-
- 01 Apr, 2020 1 commit
-
-
Kirill Smelkov authored
-
- 18 Dec, 2019 1 commit
-
-
Kirill Smelkov authored
It was from long-ago marked as "XXX move to common place".
-
- 12 Jul, 2019 1 commit
-
-
Kirill Smelkov authored
For tests this makes sure that if one test fails, it won't make following tests fail just because the next test will fail trying to lock test database. For regular code (demo_zbigarray.py) this is also a good thing to do - to always close the database irregardless of whether an exception was raised before program reached end of main. Pygolang becomes regular - not test only - dependency. Being regular dependency is currently required only by demo_zbigarray.py, but it will be also used in upcoming wcfs, so adding pygolang into wendelin.core dependencies aligns with the plan. dbclose now uses defer almost everywhere - there are still few places in tests, where one test function is opening/closing test database multiple times - those were not (yet ?) converted.
-
- 17 Apr, 2018 1 commit
-
-
Kirill Smelkov authored
bigfile/tests/test_filezodb.py ........W: testdb: teardown: <Connection at 7f8fe2b43b90> left not closed by test code; opened by: ... File "/home/kirr/src/wendelin/wendelin.core/bigfile/tests/test_filezodb.py", line 754, in test_bigfile_zblk1_zdata_reuse _test_bigfile_zblk1_zdata_reuse() File "/home/kirr/src/wendelin/wendelin.core/bigfile/tests/test_filezodb.py", line 759, in _test_bigfile_zblk1_zdata_reuse root = dbopen() File "/home/kirr/src/wendelin/wendelin.core/bigfile/tests/test_filezodb.py", line 47, in dbopen return testdb.dbopen() File "/home/kirr/src/wendelin/wendelin.core/lib/testing.py", line 188, in dbopen self.connv.append( (weakref.ref(conn), ''.join(traceback.format_stack())) ) lib/tests/test_zodb.py .W: testdb: teardown: <Connection at 7f8fe26f13d0> left not closed by test code; opened by: ... File "/home/kirr/src/wendelin/wendelin.core/lib/tests/test_zodb.py", line 49, in test_deactivate_btree root = dbopen() File "/home/kirr/src/wendelin/wendelin.core/lib/tests/test_zodb.py", line 30, in dbopen return testdb.dbopen() File "/home/kirr/src/wendelin/wendelin.core/lib/testing.py", line 188, in dbopen self.connv.append( (weakref.ref(conn), ''.join(traceback.format_stack())) )
-
- 24 Oct, 2017 1 commit
-
-
Kirill Smelkov authored
Relicense to GPLv3+ with wide exception for all Free Software / Open Source projects + Business options. Nexedi stack is licensed under Free Software licenses with various exceptions that cover three business cases: - Free Software - Proprietary Software - Rebranding As long as one intends to develop Free Software based on Nexedi stack, no license cost is involved. Developing proprietary software based on Nexedi stack may require a proprietary exception license. Rebranding Nexedi stack is prohibited unless rebranding license is acquired. Through this licensing approach, Nexedi expects to encourage Free Software development without restrictions and at the same time create a framework for proprietary software to contribute to the long term sustainability of the Nexedi stack. Please see https://www.nexedi.com/licensing for details, rationale and options.
-
- 14 Aug, 2016 1 commit
-
-
Kirill Smelkov authored
13c0c17c (bigfile/zodb: Format #1 which is optimized for small changes) used BTree to organize ZBlk1 block's chunks and for loadblkdata() added "TODO we are missing to free internal BTree structures on data load". nexedi/wendelin.core#3 besides other things showed that even when we deactivate ZData objects, we are still keeping them as ghosts occupying memory and the same for IOBucket objects. This all happens because there is no proper way to deactivate whole btree - including internal buckets objects. And since internal buckets are not deactivated, they stay in picklecache and thus hold a reference to ZData objects and ZData objects in turn, even if explicitly deactivated, stay in memory. We can fix this all via implementing whole-btree deactivation procedure. To do so we need to iterate over all btree buckets recursively, but unfortunately there is no BTree API to access/iterate btree's buckets. We can however still get reference to first top-level buckets via gc.get_referents(btree) and then scan buckets further without hacks. gc.get_referents(btree) is a hack, but - it works in O(1) (we only get pointers from btree, not scanning all gcable objects and deducing them) - it works reliable if we filter out non-interesting objects. So in the end it works. Before the patch loading more and more ZBlk1 data with objgraph instrumentation was showing itself like # Nobj δ wendelin.bigfile.file_zodb.ZData 7168 +512 BTrees.IOBTree.IOBucket 238 +17 BTrees.IOBTree.IOBTree 14 +1 and after this patch we now have BTrees.IOBTree.IOBTree 14 +1 we cannot remove that "IOBTree + 1", since ZBlk1 is holding direct reference on it (via .chunktab) and we have to keep ZBlk1 live with ._v_zfile and ._v_zblk set for invalidation to work. "+1 IOBtree" is however small - 144 bytes per 2M (= 0.006%) so we can neglect that the same way we neglect keeping ZBlk1 staying live for each block.
-