Package log provides logging with severity levels and integration with tasks.

Package task provides primitives to track tasks via contexts.

NEO/go server accepts preferred encoding of client from start, but
NEO/py server implements either N or M encoding and disconnects client
silently if handshake is not exactly what is expected.

However we can teach Dial to retry with different preferred options and
this way when connection to a NEO/py server, it will essentially
autodetect which encoding is used and accepted by remote peer.

This patch adds support for serializing packet frames with M encoding on
the wire. To do so it follows rules defined in

    	nexedi/neoppod@9d0bf97a
    	( nexedi/neoppod!11 )

Server handshake is reworked to autodetect client's preferred encoding.
Client always prefers 'N' for now.

This patch adds proto.Encoding('M') and teaches it to encode/decode
messages via MessagePack by the rules defined in

	nexedi/neoppod@9d0bf97a
	( nexedi/neoppod!11 )

It only adds support for messages serialization, without changing
proto.go to match changes in e.g. enums reordering, and without adding
support for MessagePack at link layer in neonet.

M-encoding was only tested for NEO/go-NEO/go, and was not yet tested for
NEO/go-NEO/py interoperation. There will be likely small mistakes
discovered on my side that should be hopefully easy to fix step by step
once we get to that phase.

It complements github.com/tinylib/msgp with things that neo/proto needs
to encode/decode messages.

It is noop for 'N' encoding, but will be non-empty for MessagePack.

genSliceHead and genMapHead will be reused for MsgPack encoding.
Changes in generated code are mostly due to move of where things are
typecasted to e.g. uint32 without any change in semantic.

For decode-family of functions protogen.go sticks to convention to name
target variable to where to assign the result as "assignto", not "path".

No change in generated code.

Encoding specifies a way to encode/decode NEO messages and packets.
Current way of how messages were encoded is called to be 'N' encoding.

This patch:

- adds proto.Encoding type
- changes MsgEncode and MsgDecode to be methods of Encoding
- renames thigs that are specific to 'N' encoding to have 'N' suffix
- changes tests to run a testcase agains vector of provided encodings.
  That vector is currently only ['N'].

And provide only single top-level entry-points to encode/decode
messages. As of now the entry points are just plain forwarding, but with
introducing of msgpack and encodings, they will take into account
through which encoding a message has to be encoded/decoded.

- Regrouping messages would change their message codes, thus introducing
  backward compatibility breakage.

- The protocol itself is currently kind-of documented in separate
  document here:

  https://neo.nexedi.com/P-NEO-Protocol.Specification.2019?portal_skin=CI_slideshow

It is internal detail that ideally should not get outside of proto
package. Don't clutter documentation of all messages with it.

-> For now access message code via proto.MsgCode(msg) from outside.

Generating code through sizer can already give answer if a type is fixed
size or not. And in practice we don't care at all if it will be a bit
slower compared to previous explicit type switch.

For current code it is just a small cleanup, and it will be handy to
support messagepack encoding as well.

No change in generated code.

Not used yet, but they will be needed for msgpack codec.

Besides proto.customCodec we will need to lookup more objects.

We previously used criteria that sizeof(typ.Elem()) == 1, but it would
also trigger on e.g. `struct { x byte }` which we would not want to
consider as byte array.

-> Clarify the test to explicitly test for typ.Elem() == byte.
No change in generated code.

Without the fix any pointer field of a struct would be silently skipped
without any serialization code generated for it. With the patch default
case triggers in and reports detail where such type is and in which
struct/field it is used.

Besides *mem.Buf - that is handled specially - we don't have pointers in
proto.go, so this currently does not affect generated code at all.

Noticed accidentally.

Without that, when lonet will be used in tests, port won't be correctly
extracted from an address.

With upcoming msgpack encoding packets might be with data < PktHeaderSize.
Adjust pktAlloc to unconditionally provide the invariant.

Preparatory step before we introduce support for msgpack encoding.

It is not used anywhere and would have to be reworked with upcoming
introduction of msgpack encoding. -> Simply remove that.

Just renaming.

This represents a neonet testing environment and in the future will be
used to run tests under different protocol versions, protocol encodings, etc.

For now it is noop wrapper around testing.T + t.bin that wraps []byte
and will be amended to return something different depending on t's
encoding.

Previously we were doing handshake symmetrically: both client and server
were transmitting hello and receiving peer's hello simultaneously.
However this does not allow server to adjust its behaviour depending on
which client (protocol version, protocol encoding, ...) is connecting to it.

-> Rework handshake so that client always sends its hello first, and
only then the server side replies. This matches actual NEO/py behaviour:

https://lab.nexedi.com/nexedi/neoppod/blob/v1.12-67-g261dd4b4/neo/lib/connector.py#L293-294

even though the "NEO protocol" states that

	Handshake transmissions are not ordered with respect to each other and can go in parallel.

	( https://neo.nexedi.com/P-NEO-Protocol.Specification.2019?portal_skin=CI_slideshow#/9/2 )

If I recall correctly that sentence was authored by me in 2018 based on
previous understanding of should-be full symmetry in-between client and
server.

However soon we are going to teach server sides to autodetect client
encoding and adjust server to talk to client via its preferred way.
This needs handshake for client and server to be differentiated.

The protocol needs to be adjusted as well. However I'm not sure it is
going to happen...

This will be handy to test underlying error for e.g. EOF with
errors.Cause or errors.Is in case that underlying error is also error
wrapper.

We were returning packet with tail read for next packet data.
Previously decoding was forgiving, but upcoming rework for msgpack
support will add check to catch packets with overlong payload, e.g.

    === RUN   TestEmptyDB/py/!ssl
    I: runneo.py: /tmp/neo214694750/1 !ssl: started master(s): 127.0.0.1:30621
    127.0.0.1:42266 > 127.0.0.1:30621: .1 RequestIdentification &{CLIENT ?(0)0  1 ø [] []}
    127.0.0.1:42266 < 127.0.0.1:30621: (N: decode header: len(payload) != msgLen) 00 00 00 01 80 01 00 00 00 09 00 f0 00 00 01 e0 00 00 01 00 00 00 00 00 06 00 00 00 58 41 d7 ff 69 82 0a 91 25 00 00 00 03 02 00 00 00 00 e0 00 00 01 02 41 d7 ff 69 82 0a 78 ff 01 00 00 00 09 31 32 37 2e 30 2e 30 2e 31 96 f7 00 00 00 01 02 41 d7 ff 69 81 c6 65 f5 00 00 00 00 09 31 32 37 2e 30 2e 30 2e 31 77 9d f0 00 00 01 02 ff ff ff ff ff ff ff ff 00 00 00 02 00 0a 00 00 00 19 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 01 00 00 00 01 00 00 00 01 01

-> Fix is: pkt.data should be data[:pktLen], not data[:n] since n is how
much we have read at all.

The utility to create Networker suitable for interoperating with nodes
in a NEO cluster, be it TCP, TLS/TCP, TLS/lonet, etc. See added comments
for details.

See ConfigForP2P comment for details.

xnet.Networker added Close recently and is now required to be explicitly
closed by the user:

go123@01a77bb9

See kirr/go123@515a6d14

See

	go123@3354b401
and

	go123@b03d65ff

The wrapping logic in LinkListener goes away because Accept from xnet
now supports cancellation.

It was not the case with ZEO4 server because we did not had
ZEO@bf80d23d yet.

For ZEO it is not strictly required, but for upcoming NEO for example
NEOCluster needs to shutdown gracefully, else there are processes left for e.g.
storage nodes and they dump somthing as below on the terminal after tests
completion:

    === RUN   TestLoad
    2020/10/21 14:33:00 zodb: FIXME: open ../zodb/storage/fs1/testdata/1.fs: raw cache is not ready for invalidations -> NoCache forced
    === RUN   TestLoad/py
    I: runneo.py: /tmp/neo445013868/1: Started master(s): 127.0.0.1:24661
    WARNING: This is not the recommended way to import data to NEO: you should use the Importer backend instead.
    NEO also does not implement IStorageRestoreable interface, which means that undo information is not preserved when using this tool: conflict resolution could happen when undoing an old transaction.
    Migrating from ../zodb/storage/fs1/testdata/1.fs to 127.0.0.1:24661
    Migration done in 0.19877
    --- PASS: TestLoad (0.75s)
        --- PASS: TestLoad/py (0.74s)
    PASS
    ok      lab.nexedi.com/kirr/neo/go/neo  0.749s
    (neo) (z-dev) (g.env) kirr@deco:~/src/neo/src/lab.nexedi.com/kirr/neo/go/neo$ Traceback (most recent call last):
      File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/tests/functional/__init__.py", line 182, in start
        getattr(neo.scripts,  command).main()
      File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/scripts/neostorage.py", line 66, in main
        app.run()
      File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/app.py", line 147, in run
        self._run()
      File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/app.py", line 178, in _run
        self.doOperation()
      File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/app.py", line 266, in doOperation
        poll()
      File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/app.py", line 87, in _poll
        self.em.poll(1)
      File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/lib/event.py", line 155, in poll
        self._poll(blocking)
      File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/lib/event.py", line 253, in _poll
        timeout_object.onTimeout()
      File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/lib/event.py", line 259, in onTimeout
        on_timeout()
      File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/database/manager.py", line 207, in _deferredCommit
        self.commit()
      File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/database/manager.py", line 193, in commit
        self._commit()
      File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/database/sqlite.py", line 90, in _commit
        retry_if_locked(self.conn.commit)
      File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/database/sqlite.py", line 45, in retry_if_locked
        return f(*args)
    OperationalError: disk I/O error

And use it to verify other storages.

See nexedi/nxdtest@53064e71

Go through zrpc and add error context to every function that can create
error. This is similar to what NEO does in neonet. The reason to do this
was the following obscure error from wcfs:

    E1105 11:32:33.295497   24639 wcfs.go:2505] zwatch zeo://:28359: EOF

which, after this patch, is now

    E1105 12:53:15.922052   30731 wcfs.go:2510] zwatch zeo://:27024: zlink 127.0.0.1:47768 - 127.0.0.1:27024: recvPkt: EOF

NEO was already doing this, but let's use xio.NoEOF everywhere for
uniformity.

If remote peer closes the link half-way of packet data - turn EOF into
ErrUnexpectedEOF.