1. 20 Aug, 2024 6 commits
    • Levin Zimmermann's avatar
      proto/msgpack: Fix handshake magic · 0e3681c3
      Levin Zimmermann authored
      This fixes the basic NEO handshake in msgpack encoding.
      
      Without this fix NEO/go client test fails with a timeout. In NEO/go log
      we can see
      
      ```
      Log file created at: 2024/08/20 12:50:11
      Running on machine: ubuntu
      Binary: Built with gc go1.18.1 for linux/amd64
      Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
      I0820 12:50:11.300574 1137321 client.go:418] neo: open neo://1@127.0.0.1:23004: [
      I0820 12:50:11.301130 1137321 mastered.go:119] C?: talk master(127.0.0.1:23004): [
      I0820 12:50:11.301140 1137321 connect.go:115] C?: talk master(127.0.0.1:23004): dial 127.0.0.1:23004 (MASTER): [
      I0820 12:50:11.303723 1137321 connect.go:119] C?: talk master(127.0.0.1:23004): dial 127.0.0.1:23004 (MASTER): ] (127.0.0.1:42388 - 127.0.0.1:23004: handshake (client): rx hello reply: unexpected EOF)
      W0820 12:50:11.303729 1137321 mastered.go:127] C?: talk master(127.0.0.1:23004): dial 127.0.0.1:23004 (MASTER): 127.0.0.1:42388 - 127.0.0.1:23004: handshake (client): rx hello reply: unexpected EOF
      I0820 12:50:12.304076 1137321 connect.go:115] C?: talk master(127.0.0.1:23004): dial 127.0.0.1:23004 (MASTER): [
      I0820 12:50:12.306096 1137321 connect.go:119] C?: talk master(127.0.0.1:23004): dial 127.0.0.1:23004 (MASTER): ] (127.0.0.1:42410 - 127.0.0.1:23004: handshake (client): rx hello reply: unexpected EOF)
      W0820 12:50:12.306102 1137321 mastered.go:127] C?: talk master(127.0.0.1:23004): dial 127.0.0.1:23004 (MASTER): 127.0.0.1:42410 - 127.0.0.1:23004: handshake (client): rx hello reply: unexpected EOF
      I0820 12:50:13.306311 1137321 connect.go:115] C?: talk master(127.0.0.1:23004): dial 127.0.0.1:23004 (MASTER): [
      I0820 12:50:13.307818 1137321 connect.go:119] C?: talk master(127.0.0.1:23004): dial 127.0.0.1:23004 (MASTER): ] (127.0.0.1:56834 - 127.0.0.1:23004: handshake (client): rx hello reply: unexpected EOF)
      W0820 12:50:13.307824 1137321 mastered.go:127] C?: talk master(127.0.0.1:23004): dial 127.0.0.1:23004 (MASTER): 127.0.0.1:56834 - 127.0.0.1:23004: handshake (client): rx hello reply: unexpected EOF
      ```
      
      And in NEO/py logs we can see:
      
      ```
      2024-08-20 12:51:35.5152 DEBUG     M1         Rejecting non-NEO <ServerConnection(nid=None, address=127.0.0.1:35404, handler=IdentificationHandler, fd=18, server) at 7f9417467510>
      2024-08-20 12:51:35.5153 DEBUG     M1         connection closed for <ServerConnection(nid=None, address=127.0.0.1:35404, handler=IdentificationHandler, closed, server) at 7f9417467510>
      2024-08-20 12:51:36.5161 DEBUG     M1         accepted a connection from 127.0.0.1:35408
      2024-08-20 12:51:36.5168 DEBUG     M1         connection completed for <ServerConnection(nid=None, address=127.0.0.1:35408, handler=IdentificationHandler, fd=18, server) at 7f9417467210> (from 127.0.0
      ```
      
      NEO/py defines its handshake packet here:
      
      https://lab.nexedi.com/nexedi/neoppod/-/blob/9d0bf97a1/neo/lib/protocol.py#L27
      
      It's encoded form can be simply revealed with the help of a python shell:
      
      ```python
      >>> from neo.lib import protocol
      >>> protocol.HANDSHAKE_PACKET
      '\x92\xa3NEO\x00'
      ```
      0e3681c3
    • Levin Zimmermann's avatar
      go/neo/proto: Update py test data after protocol change · 74b0904e
      Levin Zimmermann authored
      nexedi/neoppod@9d0bf97a
      changed the msgcode increment logic, therefore we have to re-create the
      test data.
      74b0904e
    • Levin Zimmermann's avatar
      go/neo/client_test: NEO/py now only supports M encoding · 4161dd45
      Levin Zimmermann authored
      With the switch to the msgpack encoding, the NEO/py server
      now only supports the M encoding.
      4161dd45
    • Julien Muchembled's avatar
      protocol: switch to msgpack for packet serialization · 711bd878
      Julien Muchembled authored
      Not only for performance reasons (at least 3% faster) but also because of
      several ugly things in the way packets were defined:
      - packet field names, which are only documentary; for roots fields,
        they even just duplicate the packet names
      - a lot of repetitions for packet names, and even confusion between the name
        of the packet definition and the name of the actual notify/request packet
      - the need to implement field types for anything, like PByte to support new
        compression formats, since PBoolean is not enough
      
      neo/lib/protocol.py is now much smaller.
      711bd878
    • Levin Zimmermann's avatar
      go/neo/neonet: DialLink: Fix SIGSEGV in case client handshake fails · b89b447c
      Levin Zimmermann authored
      In case the last 'handshakeClient' call returns an error, 'DialLink'
      returns 'link = nil, err = nil'. Callers of 'DialLink' then don't
      recognize that 'link' is 'nil', as it's the convention to only check if
      'err' is 'nil', which leads to a 'segmentation violation' as soon as
      subsequent code tries to access fields of 'link':
      
      ```
      panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x14 pc=0x7087ae]
      
      goroutine 5 [running]:
      lab.nexedi.com/kirr/neo/go/neo/neonet.(*NodeLink).NewConn(0x0)
      	/srv/slapgrid/slappart82/srv/runner/instance/slappart6/software_release/parts/wendelin.core/wcfs/neo/go/neo/neonet/connection.go:404 +0x4e
      lab.nexedi.com/kirr/neo/go/neo/xneo.Dial.func1()
      	/srv/slapgrid/slappart82/srv/runner/instance/slappart6/software_release/parts/wendelin.core/wcfs/neo/go/neo/xneo/connect.go:138 +0x52
      lab.nexedi.com/kirr/neo/go/internal/xio.WithCloseOnErrCancel.func2()
      	/srv/slapgrid/slappart82/srv/runner/instance/slappart6/software_release/parts/wendelin.core/wcfs/neo/go/internal/xio/xio.go:114 +0x6a
      created by lab.nexedi.com/kirr/neo/go/internal/xio.WithCloseOnErrCancel in goroutine 21
      	/srv/slapgrid/slappart82/srv/runner/instance/slappart6/software_release/parts/wendelin.core/wcfs/neo/go/internal/xio/xio.go:109 +0x1ad
      ```
      
      This patch fixes this issue so that now 'err' and 'link' are never both
      'nil' again.
      
      /reviewed-by @kirr
      /reviewed-on kirr/neo!10
      b89b447c
    • Kirill Smelkov's avatar
      go/neo/neonet: Demonstrate DialLink misbehaviour when all handshake attempts fail · 12b3e2af
      Kirill Smelkov authored
      Levin found that when all handshake attempts fail DialLink returns
      both link=nil and err=nil which breaks what callers expect and lead to
      segmentation fault when accessing that nil link.
      
      -> Add test to demonstrate the problem.
      
      With xfail removed that test currently fails as
      
          --- FAIL: TestDialLink_AllHandshakeErr (0.00s)
          panic: lab.nexedi.com/kirr/neo/go/neo/neonet.TestDialLink_AllHandshakeErr.gox.func4.1: lab.nexedi.com/kirr/neo/go/neo/neonet.TestDialLink_AllHandshakeErr.func2: DialLink to handshake-rejecting server:
          have: link=<nil> err=<nil>
          want: link=<nil> err=client:1 - server:2: handshake (client): unexpected EOF [recovered]
      
      We will fix the problem in the next patch.
      
      /reported-by @levin.zimmermann
      /reported-at kirr/neo!10
      12b3e2af
  2. 23 Jul, 2024 2 commits
  3. 21 Jul, 2024 1 commit
    • Kirill Smelkov's avatar
      X: Sync zurl format with NEO/py · 95572d6a
      Kirill Smelkov authored
      Hello Kirill,
      
      in nexedi/neoppod!18 and nexedi/neoppod!21 we could find a common solution for a zurl format that previously diverged between NEOgo and NEOpy. The purpose of this MR is to sync again NEOgo and NEOpy zurl format. After merging this, we can continue to sync NEO zurl format in 'wendelin.core' & 'slapos'. Then we finally have unified approach again, which simplifies understanding and reduces unnecessary mental overhead.
      
      As this is strongly related to nexedi/neoppod!21 I thought it'd be a good idea to generally reduce difference and to replace WIP commits with merged NEOpy upstream commits.
      
      Best, Levin
      
      /reviewed-by @kirr
      /reviewed-on kirr/neo!7
      
      * lev/sync-zurl:
        client: Don't allow oPtion_nAme in zurl
        app: Remember SSL credentials so that it is possible to retrieve them
        client: Allow to force TLS via neos:// scheme
        client: Don't allow master_nodes and name to be present in options
        Revert "."
        Revert "Y client: Fix URI scheme to move credentials out of query"
        Revert "X Adjust NEO/go to neo:// URL change + py fixups"
        Revert "fixup! Y client: Fix URI scheme to move credentials out of query"
        Revert "Y client: Don't allow master_nodes and name to be present in options"
        go/client/zurl: Sync format to py upstream
      95572d6a
  4. 19 Jul, 2024 10 commits
  5. 02 Feb, 2024 3 commits
  6. 29 Jan, 2024 8 commits
  7. 22 Aug, 2023 1 commit
  8. 02 Aug, 2023 8 commits
  9. 01 Aug, 2023 1 commit
    • Kirill Smelkov's avatar
      fixup! proto.NotPrimaryMaster: Fix .Primary data type (1) · 3cb82317
      Kirill Smelkov authored
      Rerun `go generate`. As the diff in zproto-marshal.go shows changing
      NotPrimaryMaster.Primary type from NodeID to int8 actually does make a
      difference. This happens because NodeID type is based on int32 and
      changing that to int8 changes how NotPrimaryMaster structure is layed
      out in memory and on the wire.
      
      The changes to zproto-marshal.go in 5d93e434 seem to be done by hand
      and not matching the change to proto.go even though head of
      zproto-marshal.go says
      
      	// Code generated by protogen.go; DO NOT EDIT.
      3cb82317