- 02 Jun, 2023 5 commits
-
-
Levin Zimmermann authored
This is the NEO/go patch part of kirr/neo@4c9414ea. The rationale for this move is described there: -------------------8>-------------------- For example option `compress` controls kind of compression that _client_ performs when saving data to server. Similarly cache-size, logfile and read-only adjust on-client behaviour, not server. From nexedi/neoppod!18 (comment 124725) : In general the most correct thing to do is: - use host part for where to connect (host:port, list of host ports, UNIX socket, etc) - use path part to identify a database or other on-server resource - use query part for parameters that are passed to remote server (e.g. `storage` in case of ZEO) - use fragment part for local parameters that are not passed to remote server (e.g. local `logfile`) - use credentials part for things required to authenticate/encrypt. To normalize an URL wcfs client would drop credentials and fragment, but keep host, path and query. Fragments are documented not to be sent to remote side and to be evaluated by local side only. -> Move options that control client behaviour to fragment. -------------------<8--------------------
-
Levin Zimmermann authored
Before this patch, the parser ignored options which were already supported by the client (for instance 'read-only') and even raised an error. But the client can already use this option: as a9246333 describes this should happen in the local storage URL parser. Furthermore not-yet-supported client options (for instance compress) broke the NEO client before this patch. Now these options only raise a warning which informs the user that they are ignored. Why? We want to use pre-complete NEO in real-world projects together with NEO/py clusters. Those real-world projects may already specify options which aren't supported by our NEO/go client yet. But it doesn't matters so much, because those options are mostly relevant for other NEO/py cluster clients (e.g. zope nodes). Instead of filtering those parameters before parsing them to NEO/go in a higher level (e.g. SlapOS), NEO/go should already support any valid NEO URL and raise warnings for not yet implemented features.
-
Levin Zimmermann authored
This test was missing so far. Particularly recent changes of the NEO URI scheme [1], but also problems with valid old URI [2] stressed out the necessity for comprehensive NEO URI parser tests. [1] kirr/neo@4c9414ea [2] 573514c6 (comment 184417)
-
Levin Zimmermann authored
With all the recent changes of the NEO URI scheme we need to reliably test the function which parses the URI and convert it into the different parameter. Testing is much simpler if we can only analyse how the URI parsing works. Therefore this patch moves NEO URI parsing to an external function.
-
Levin Zimmermann authored
We can't extract storage client options in general-purpose zodburi opener, because zodburi aren't standardized. Each storage may define its own syntax, therefore it is better if this is done decentralized and independent in each local storage driver.
-
- 24 May, 2023 2 commits
-
-
Kirill Smelkov authored
For example option `compress` controls kind of compression that _client_ performs when saving data to server. Similarly cache-size, logfile and read-only adjust on-client behaviour, not server. From nexedi/neoppod!18 (comment 124725) : In general the most correct thing to do is: - use host part for where to connect (host:port, list of host ports, UNIX socket, etc) - use path part to identify a database or other on-server resource - use query part for parameters that are passed to remote server (e.g. `storage` in case of ZEO) - use fragment part for local parameters that are not passed to remote server (e.g. local `logfile`) - use credentials part for things required to authenticate/encrypt. To normalize an URL wcfs client would drop credentials and fragment, but keep host, path and query. Fragments are documented not to be sent to remote side and to be evaluated by local side only. -> Move options that control client behaviour to fragment.
-
Kirill Smelkov authored
Because list of masters and cluster name must be already present in netloc and path. Previously e.g. neo://α,β,γ/db?master_nodes=a,b,c" would mean to use master nodes {a,b,c} not {α,β,γ}. Now it is treated as invalid URI to remove ambiguity. Same for cluster name.
-
- 18 Jan, 2023 3 commits
-
-
Kirill Smelkov authored
* master: go/neo: Expand user prefix in TLS key/cert paths go/neo/interal += xpath, xfilepath
-
Levin Zimmermann authored
This patch fixes a discrepancy between NEO/py and NEO/go: NEO/py expands the '~' and the '~username' prefix in the file path of the TLS certificate/key files [1]. This syntax is used in NEO/py SlapOS SR [2]. We need to fix this discrepancy in NEO/go in order to use TLS encryption with NEO + WCFS. [1] https://lab.nexedi.com/nexedi/neoppod/blob/7c539f0f/neo/lib/config.py#L149 and https://lab.nexedi.com/nexedi/neoppod/blob/fa63d856/neo/lib/app.py#L25-31 [2] https://lab.nexedi.com/nexedi/slapos/blob/397726e1/stack/erp5/instance-zodb-base.cfg.in#L18-20 and https://lab.nexedi.com/nexedi/slapos/blob/a8150a1a/software/neoppod/instance-neo-input-schema.json#L62 /reviewed-by @kirr /reviewed-on !1
-
Levin Zimmermann authored
The xfilepath package supports resolving filepaths with a user prefix to absolute paths: it converts '~' and '~username' to $HOME of user (as it's done by for instance bash). No builtin golang module supports this functionality [1]. We need this functionality in order to imitate the behaviour of NEO/py in NEO/go [2]. --- [1] https://stackoverflow.com/questions/47261719/how-can-i-resolve-a-relative-path-to-absolute-path-in-golang) [2] nexedi/slapos!1307 (comment 17574) /reviewed-by @kirr /reviewed-on !1
-
- 04 Nov, 2022 1 commit
-
-
Kirill Smelkov authored
-
- 03 Nov, 2022 1 commit
-
-
Kirill Smelkov authored
* master: go/neo/t/neotest: Use python -c 'print ...' in a way that works on both py2 and py3 go/neo/t/tzodb.py: Fix zhash for Python3
-
- 18 May, 2022 2 commits
-
-
Kirill Smelkov authored
Without parenthesis it was failing on py3: (neo) (py3.venv) (g.env) kirr@deca:~/src/neo/src/lab.nexedi.com/kirr/neo/go/neo/t$ ./neotest info-local date: Wed, 18 May 2022 11:05:50 +0300 xnode: kirr@deca.navytux.spb.ru (2401:5180:0:af::1 192.168.0.3 (+ 1·ipv4)) uname: Linux deca 5.10.0-13-amd64 #1 SMP Debian 5.10.106-1 (2022-03-17) x86_64 GNU/Linux cpu: Intel(R) Core(TM) i7-7600U CPU @ 2.80GHz File "<string>", line 1 print '%.2fGHz' % (400000 / 1E6) ^ SyntaxError: invalid syntax
-
Kirill Smelkov authored
On py3 dict.keys() returns iterator instead of list: $ ./tzodb.py zhash Traceback (most recent call last): File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/neo/t/./tzodb.py", line 141, in <module> main() File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/neo/t/./tzodb.py", line 138, in main zhash() File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/neo/t/./tzodb.py", line 59, in zhash optv, argv = getopt(sys.argv[2:], "h", ["help", "check=", "bench="] + hashRegistry.keys()) TypeError: can only concatenate list (not "dict_keys") to list
-
- 25 Nov, 2021 2 commits
-
-
Kirill Smelkov authored
* master: go/internal/xurl: New package (draft)
-
Kirill Smelkov authored
With xurl.ParseQuery being simpler analog of url.ParseQuery. Simpler: It returns regular map instead of url.Values by not allowing duplicates.
-
- 24 Nov, 2021 1 commit
-
-
Kirill Smelkov authored
Instead of requiring users to explicitly mark their tests as finished in particular subject-to-deadlock places, tracetest was fixed to unconditionally detect "extra sends" deadlocks automatically: go123@3b19f68c
-
- 19 Nov, 2021 2 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
- 12 Nov, 2021 1 commit
-
-
Kirill Smelkov authored
go get: upgraded github.com/fsnotify/fsnotify v1.4.10-0.20200417215612-7f4cf4dd2b52 => v1.5.1 go get: upgraded github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b => v1.0.0 go get: upgraded github.com/gwenn/gosqlite v0.0.0-20201008200117-82e079acf5b6 => v0.0.0-20211101095637-b18efb2e44c8 go get: upgraded github.com/gwenn/yacr v0.0.0-20200112083327-bbe82c1f4d60 => v0.0.0-20211101095056-492fb0c571bc go get: upgraded github.com/kisielk/og-rek v1.1.0 => v1.2.0 go get: upgraded github.com/soheilhy/cmux v0.1.5-0.20210205191134-5ec6847320e5 => v0.1.5 go get: upgraded github.com/tinylib/msgp v1.1.5 => v1.1.6 go get: upgraded golang.org/x/net v0.0.0-20210226172049-e18ecbb05110 => v0.0.0-20211111160137-58aab5ef257a go get: upgraded golang.org/x/sys v0.0.0-20210301091718-77cc2087c03b => v0.0.0-20211111213525-f221eed1c01e go get: upgraded golang.org/x/text v0.3.5 => v0.3.7 go get: upgraded lab.nexedi.com/kirr/go123 v0.0.0-20210128150852-c20e95f0f789 => v0.0.0-20210906140734-c9eb28d9e408 Highlights: - og-rek brings support for pickle protocol 5 - go123 brings in support for Go1.17
-
- 04 Oct, 2021 13 commits
-
-
Kirill Smelkov authored
* master: go/zodb/btree: Change V<op> family to also provide visited node key coverage on visit callback go/zodb/btree: Add KeyRange type go/zodb/btree: Introduce constants for min/max key value go/zodb/btree: tests: Don't forget to close storage go/zodb/btree: Cosmetics
-
Kirill Smelkov authored
WCFS needs to know key coverage for every visited node. WARNING: this is API change.
-
Kirill Smelkov authored
KeyRange represents [lo,hi) key range. It simplifies working with ranges of keys. We will use it in the next commit. KeyRange originated in WCFS and was copied from there: https://lab.nexedi.com/kirr/wendelin.core/blob/57be0126/wcfs/internal/xbtree/blib/keyrange.go
-
Kirill Smelkov authored
We were already using math.Min<Key> in one place, but the number of such places is going to increase. -> Keep min/max definition in only one place.
-
Kirill Smelkov authored
NOTE: db was already being closed in the test's code.
-
Kirill Smelkov authored
-
Kirill Smelkov authored
* t+btree: . . . . . . . X Start reworking BTree to provide keycov on visit callback
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
- 01 Oct, 2021 2 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
- 08 Sep, 2021 5 commits
-
-
Kirill Smelkov authored
Due to security concerns, go1.17, even though very reluctantly, similarly to Python broke backward compatibility and stopped to treat ';' as separator when parsing URL queries. We still want to accept ';' as separator in credential part. -> Work it around by replacing ';' -> to '&'. See https://golang.org/doc/go1.17#semicolons and https://github.com/golang/go/issues/25192 for context. See also: cf685fb5 (fixup! Y client: Fix URI scheme to move credentials out of query)
-
Kirill Smelkov authored
parse_qsl no longer treats ';' as valid query separator for security reason because most proxies did not do so and it was possible to poison proxy cache due to difference in query separator handling (see bugs.python.org/issue42967). To handle credentials we don't have any proxy here, and it is still perfectly valid to use ';' as credentials separator. -> Fix it with ';' -> '&' replace workaround, before feeding credentials string to parse_qsl. Amends: b9a42957.
-
Kirill Smelkov authored
* master: go/internal/xtesting: Add missing X go/zodb/{fs1,zeo}: ~staticcheck go/zodb/btree: Fix missing return on data-consistency error go/zodb, go/zodb/btree: Fix go generate after rename on zodbtools side
-
Kirill Smelkov authored
staticcheck says: xtesting.go:386:2: this value of err is never used (SA4006)
-
Kirill Smelkov authored
fs1/format.go:204:2: only the first constant in this group has an explicit type (SA9004) zeo/proto.go:56:2: only the first constant in this group has an explicit type (SA9004)
-