- 19 Jul, 2024 10 commits
-
-
Kirill Smelkov authored
Julien notes this is very likely unneeded: nexedi/neoppod!21 (diffs, comment 195929) We had it like this since 01a01c8c (client: Add support for zodburi), but I rechecked zodburi codebase now and it does not do any similar lowering anywhere. So drop support for case normalization in zurl options. /cc @levin.zimmermann /reviewed-by @jm /reviewed-on nexedi/neoppod!21 (cherry-picked from commit 798c9f25)
-
Kirill Smelkov authored
Unfortunately after creating SSL context it is not possible, or at least I could not find how, to retrieve original credentials with which the context was created. However wendelin.core needs to be able to take a client storage, reconstruct zurl to refer to that particular storage, and pass that zurl to wcfs, so that wcfs, in turn, could access the same ZODB database. Given a NEO client instance, it is already possible to retrieve master_nodes, cluster name, and detect whether SSL is being in use. However without being able to retrieve original SSL credentials, reconstructed zurl will not be full and wcfs won't be able to use exactly the same secrets as python part does. -> Help wendelin.core by remembering which ca/cert/key were used to build SSL context. This information is used by zstor_2zurl in wendelin.core here: https://lab.nexedi.com/nexedi/wendelin.core/blob/885b3556/lib/zodb.py#L390-418 /cc @levin.zimmermann /reviewed-by @jm /reviewed-on nexedi/neoppod!21
-
Kirill Smelkov authored
Similarly to how it is done with e.g. http:// and https:// - if neos:// is given TLS usage is forced and ca/cert/key must be there either in the URI itself, or in $NEO_CA, $NEO_CERT and $NEO_KEY environment variables mimicking the way how e.g. for https:// TLS credentials are taken from host environment, not from the uri. The latter might be usability convenience, but is also useful for WCFS which needs to be able to remove secrets from uri on zurl normalization. Please see discussion at nexedi/neoppod!18 (comment 184439) for details. /cc @levin.zimmermann /reviewed-by @jm /reviewed-on nexedi/neoppod!21 (cherry-picked from commit bc3e38ea)
-
Kirill Smelkov authored
Because list of masters and cluster name must be already present in netloc and path. Previously e.g. neo://db@α,β,γ?master_nodes=a,b,c" would mean to use master nodes {a,b,c} not {α,β,γ}. Now it is treated as invalid URI to remove ambiguity. Same for cluster name. /cc @levin.zimmermann /reviewed-by @jm /reviewed-on nexedi/neoppod!21 (cherry-picked from commit 22ccebd6)
-
Levin Zimmermann authored
This reverts commit a2f192cb. This has been merged upstream with nexedi/neoppod@17af7f27. We should rather cherry-pick upstream commit.
-
Levin Zimmermann authored
This reverts commit b9a42957. In nexedi/neoppod!18 and nexedi/neoppod!21 a common solution for a zurl format was found. This common format keeps credentials in the query, therefore we should revert patch b9a42957.
-
Levin Zimmermann authored
This reverts the py part of kirr/neo@8c974485. go parts of this patch are handled in 70c0a984.
-
Levin Zimmermann authored
This reverts commit cf685fb5. This used to be a divergence between NEO/py and NEO/go, however in nexedi/neoppod!18 and nexedi/neoppod!21 a common solution for a zurl format was found. This common format keeps credentials in the query, therefore we should revert patch cf685fb5.
-
Levin Zimmermann authored
This reverts commit 6047f893 in order to replace it with py upstream commit nexedi/neoppod@22ccebd6.
-
Levin Zimmermann authored
NEO/go and NEO/py zurl format diverged over time: - 8c974485 However with nexedi/neoppod!21 a common solution was found. From there, this patch aims to adjust NEO/go zurl format to be in sync with NEO/py zurl format again.
-
- 02 Feb, 2024 3 commits
-
-
Kirill Smelkov authored
* master: go/zodb: Handle common options in zurl in generic layer
-
Kirill Smelkov authored
/reviewed-by @kirr /reviewed-on !4 * kirr/t+new-uri: Revert "Y client: Adjust URI scheme to move client-specific options to fragment" fixup! client.go: Fix URI client option parsing for supported + unsupported options client.go: Fix URI client option parsing for supported + unsupported options fixup! client_test: Add tests for NEO URI parser client_test: Add tests for NEO URI parser fixup! client: Refactor openClientByURL for easier testing client: Refactor openClientByURL for easier testing Y go/zodb: Handle common options in zurl in generic layer
-
Kirill Smelkov authored
Offload drivers from handling options such as ?read-only=1 and force them to deal with such options only via DriverOptions, never zurl. See added comment for details. /reviewed-by @levin.zimmermann /reviewed-on !4
-
- 29 Jan, 2024 8 commits
-
-
Levin Zimmermann authored
This reverts commit 4c9414ea. This patch was added at a time when nexedi/neoppod!18 wasn't resolved yet, but we already wanted to proceed with WCFS. Now the NEO MR is resolved and we decided to mostly leave the NEO zurl as it was originally implemented in nexedi/neoppod!6. This means we don't need this patch anymore which changed the NEO zurl format.
-
Kirill Smelkov authored
readonly is handled by common zodb.OpenDriver.
-
Levin Zimmermann authored
Before this patch, the parser ignored options which were already supported by the client (for instance 'read-only') and even raised an error. But the client can already use this option: as levin.zimmermann/neoppod@a9246333 describes this should happen in the local storage URL parser. Furthermore not-yet-supported client options (for instance compress) broke the NEO client before this patch. Now these options only raise a warning which informs the user that they are ignored. Why? We want to use pre-complete NEO in real-world projects together with NEO/py clusters. Those real-world projects may already specify options which aren't supported by our NEO/go client yet. But it doesn't matters so much, because those options are mostly relevant for other NEO/py cluster clients (e.g. zope nodes). Instead of filtering those parameters before parsing them to NEO/go in a higher level (e.g. SlapOS), NEO/go should already support any valid NEO URL and raise warnings for not yet implemented features.
-
Kirill Smelkov authored
- use simplified parseURL signature - DriverOptions are not passed nor changed there. - read-only is handled by generic zodb layer not neo.parseURL .
-
Levin Zimmermann authored
This test was missing so far. Particularly recent changes of the NEO URI scheme [1], but also problems with valid old URI [2] stressed out the necessity for comprehensive NEO URI parser tests. [1] 4c9414ea [2] levin.zimmermann/neoppod@573514c6 (comment 184417)
-
Kirill Smelkov authored
- no need to pass DriverOptions into parseURL - it is only zurl that is parsed, and also DriverOptions should not be changed by the opener. - no need to document "If anything fails within this process an error and nil are returned." because that is standard omnipresent Go convention.
-
Levin Zimmermann authored
With all the recent changes of the NEO URI scheme we need to reliably test the function which parses the URI and convert it into the different parameter. Testing is much simpler if we can only analyse how the URI parsing works. Therefore this patch moves NEO URI parsing to an external function.
-
Kirill Smelkov authored
Offload drivers from handling options such as ?read-only=1 and force them to deal with such options only via DriverOptions, never zurl. See added commend for details.
-
- 22 Aug, 2023 1 commit
-
-
Kirill Smelkov authored
I missed the following build failure in go/neo/cmd: # lab.nexedi.com/kirr/neo/go/neo/cmd/neo ./storage.go:128:37: cannot use master (variable of type string) as []string value in argument to neo.NewStorage
-
- 02 Aug, 2023 8 commits
-
-
Levin Zimmermann authored
See kirr/neo!2 for discussion, context and details. /reviewed-by @kirr * t-with-multiple-master-nodes: fixup! client_test: Add nmaster={1,2} to test matrix fixup! client_test: Support test cluster /w >1 master fixup! TalkMaster: Switch master if dialed M is secondary fixup! Node: Add support for NEO cluster with > 1 master fixup! Dial: Catch NotPrimaryMaster & return custom error fixup! proto: Implement Error for NotPrimaryMaster fixup! proto.NotPrimaryMaster: Fix .Primary data type (2) fixup! proto.NotPrimaryMaster: Fix .Primary data type (1) client_test: Add nmaster={1,2} to test matrix client_test: Support test cluster /w >1 master proto.NotPrimaryMaster: Fix .Primary data type TalkMaster: Switch master if dialed M is secondary Dial: Catch NotPrimaryMaster & return custom error proto: Implement Error for NotPrimaryMaster openClientByURL: Fix for >1 master (split URL host) Client.URL: Fix incomplete URL if > 1 master nodes Node: Add support for NEO cluster with > 1 master
-
Kirill Smelkov authored
Actually do test nmaster=2 case.
-
Kirill Smelkov authored
- show all options in error context and ran test kind - skip nmaster > 1 for NEO/go as that is currently not implemented on NEO/go server - use json for interacting with runneo.py, so that we can use whatever builtin type for any argument without hardcoding ad-hoc handling of specific arguments inside runneo.py. - adjust comments + cosmetics
-
Kirill Smelkov authored
- validate received NotPrimaryMaster - use Address.String() instead of printf with format that works only for ipv6 - add some logging, comments and TODO
-
Kirill Smelkov authored
- no need to keep Node.MasterAddr anymore - the address of current PM is managed by TalkMaster and is provided as part of operational context to user functions that TalkMaster runs. - correct docstrings. - cosmetics.
-
Kirill Smelkov authored
Expect NotPrimaryMaster only if we are trying to connect to a master.
-
Kirill Smelkov authored
Provide details in the error message.
-
Kirill Smelkov authored
Change .Promary type from int8 back to int32. 5d93e434 says that .Primary type is not NodeID. That is true, but changing it to int8 was a mistake: 1. PSignedNull is explicitly defined to come with '!l' struct code, which according to https://docs.python.org/3/library/struct.html#module-struct comes as 4-bytes integer on the wire: https://lab.nexedi.com/nexedi/neoppod/blob/v1.12-13-gf2ea4be2/neo/lib/protocol.py#L560-562 2. verifying this via serializing NotPrimaryMaster on NEO/py also confirms that .Primary occupies 4 bytes, not one: In [1]: from neo.lib.protocol import NotPrimaryMaster In [2]: NotPrimaryMaster(0x01020304, [('m111', 111), ('m222', 222)])._body Out[2]: '\x01\x02\x03\x04\x00\x00\x00\x02\x00\x00\x00\x04m111\x00o\x00\x00\x00\x04m222\x00\xde' ^^^^^^^^^^^^^^^^ NOTE NOTE NOTE -> change .Primary type back to being 4-bytes integer, but to int32 instead of NodeID because, as 5d93e434 correctly says, .Primary comes as array index, not a node ID. The following place of NEO/py code explicitly confirms this: https://lab.nexedi.com/nexedi/neoppod/blob/v1.12-13-gf2ea4be2/neo/master/handlers/identification.py#L155-159 Add corresponding test.
-
- 01 Aug, 2023 1 commit
-
-
Kirill Smelkov authored
Rerun `go generate`. As the diff in zproto-marshal.go shows changing NotPrimaryMaster.Primary type from NodeID to int8 actually does make a difference. This happens because NodeID type is based on int32 and changing that to int8 changes how NotPrimaryMaster structure is layed out in memory and on the wire. The changes to zproto-marshal.go in 5d93e434 seem to be done by hand and not matching the change to proto.go even though head of zproto-marshal.go says // Code generated by protogen.go; DO NOT EDIT.
-
- 18 Jul, 2023 9 commits
-
-
Levin Zimmermann authored
Tests should work with both one master or more than one masters.
-
Levin Zimmermann authored
Now it's possible to run client tests against a NEO cluster which has more than one master nodes. We need this adjustement in order to test NEO/go client modification in order to support more than one master node.
-
Levin Zimmermann authored
The '.Primary' attribute of the 'NotPrimaryMaster' packet has been assigned to 'NodeID' data type. This is incorrect, because the data doesn't represent the ID of the node, but an index of the '.KnownMasterList' [1]. In the old protocol NEO/py therefore also used 'PSignedNull' instead of 'PUUID' [2]. This patches fixes the data type of '.Primary' and uses 'int8' instead of 'NodeID'. Technically this doesn't make any difference, but semantically for human beings the code is easier to understand now. [1] https://lab.nexedi.com/nexedi/neoppod/blob/c6453626/neo/lib/handler.py#L161 [2] https://lab.nexedi.com/nexedi/neoppod/blob/c6453626/neo/lib/protocol.py#L716
-
Levin Zimmermann authored
When connecting to a master node, the client needs to try a different master if the initially tried one is a secondary master node. This statement wasn't implemented yet before this patch and therefore it was good luck if the initally tried master was the primary one - and the connection worked - or if it was a secondary master - and the client got stuck in re-trying the same node forever. This patch makes NEO/go usage with clusters of more than one master therefore much more stable.
-
Levin Zimmermann authored
After initial handshake a NEO node checks the identification of its peer by sending the 'RequestIdentification' packet. In case the peer is a secondary master it responds with 'NotPrimaryMaster'. Before this patch 'Dial' ignored the 'NotPrimaryMaster' packet and simply returned a general error. Now - after this patch - 'Dial' returns an instance of 'proto.NotPrimaryMaster' (which implements 'Error'). This helps a caller to correctly handle the secondary-master-case, which otherwise is impossible to differentiate from any other error possibility.
-
Levin Zimmermann authored
When a client receives 'NotPrimaryMaster' from a secondary master, the situation is similar to the situation when we receive an error: the other node tells us, don't connect with me, connect with someone else. Finally the peer even closes the connection. Due to this similarity in structure (& because it helps us later to teach NEO/go to correctly handle 'NotPrimaryMaster' with minimal changes), we implement 'Error' for 'proto.NotPrimaryMaster'. Now 'NotPrimaryMaster' can be treated like an error.
-
Levin Zimmermann authored
In a NEO URI more than one master node can be specified, because a NEO cluster may have more masters than one. But before this patch 'openClientByURL' always assumed that the given URL only specifies one master. Now the host is split into potentially > 1 master nodes. It therefore works now in the same way as the Python implementation [1]. [1] https://lab.nexedi.com/nexedi/neoppod/blob/342168cd/neo/client/zodburi.py#L64
-
Levin Zimmermann authored
Before this patch Client.URL didn't contain more than one master node. This can be problematic in case we have a cluster with > 1 master nodes and the printed master is a secondary master (which may be down). In this case the user who trusts the "URL" attribute to connect to the cluster won't succeed, because the primary master can't be reached.
-
Levin Zimmermann authored
Some NEO clusters have more than one master to gain a higher availability. Before this patch NEO/go Node type only handled one master address. This commit adjusts the node type and related bits so that it can support more than one master node.
-