This will be used in the next patch to handle invalidateTransaction
messages.

ZEO5 adds way for messages to be encoded via either pickles or MessagePack.
However until now we were always using pickles.
Let's add msgpack support to be able to e.g. use wire encoding that
server prefers.

MsgPack support is almost fully localized in encoding.
We use tinylib/msgp runtime routines to decode/encode msg fields with known types,
and shamaton/msgpack to decode/encode msg.arg, which is interface{},
because msgp does not generally work for arbitrary reflections.

For msgpack=true, tests state is the same as with pickles: handshake
works, but load fails when verifying that Load returns correct error for
deleted object:

    TestLoad/py/msgpack=false: xtesting.go:272: load 0285cbacc06d3a4c:0000000000000007: returned err unexpected:
        have: /tmp/zeo170183943/1.fs.zeosock: load 0285cbacc06d3a4c:0000000000000007: 0000000000000007: no such object
        want: /tmp/zeo170183943/1.fs.zeosock: load 0285cbacc06d3a4c:0000000000000007: 0000000000000007: object was deleted @0285cbacc06d3a4c
    TestLoad/py/msgpack=false: xtesting.go:272: load 0285cbad858bf2e6:0000000000000006: returned err unexpected:
        have: /tmp/zeo170183943/1.fs.zeosock: load 0285cbad858bf2e6:0000000000000006: 0000000000000006: no such object
        want: /tmp/zeo170183943/1.fs.zeosock: load 0285cbad858bf2e6:0000000000000006: 0000000000000006: object was deleted @0285cbad858bf2e6
    TestLoad/py/msgpack=false: xtesting.go:290: load 7fffffffffffffff:0000000000000007: returned err unexpected:
        have: /tmp/zeo170183943/1.fs.zeosock: load 7fffffffffffffff:0000000000000007: 0000000000000007: no such object
        want: /tmp/zeo170183943/1.fs.zeosock: load 7fffffffffffffff:0000000000000007: 0000000000000007: object was deleted @0285cbacc06d3a4c
    TestLoad/py/msgpack=false: xtesting.go:290: load 7fffffffffffffff:0000000000000006: returned err unexpected:
        have: /tmp/zeo170183943/1.fs.zeosock: load 7fffffffffffffff:0000000000000006: 0000000000000006: no such object
        want: /tmp/zeo170183943/1.fs.zeosock: load 7fffffffffffffff:0000000000000006: 0000000000000006: object was deleted @0285cbad858bf2e6

    TestLoad/py/msgpack=true: xtesting.go:272: load 0285cbacc06d3a4c:0000000000000007: returned err unexpected:
        have: /tmp/zeo247652538/1.fs.zeosock: load 0285cbacc06d3a4c:0000000000000007: 0000000000000007: no such object
        want: /tmp/zeo247652538/1.fs.zeosock: load 0285cbacc06d3a4c:0000000000000007: 0000000000000007: object was deleted @0285cbacc06d3a4c
    TestLoad/py/msgpack=true: xtesting.go:272: load 0285cbad858bf2e6:0000000000000006: returned err unexpected:
        have: /tmp/zeo247652538/1.fs.zeosock: load 0285cbad858bf2e6:0000000000000006: 0000000000000006: no such object
        want: /tmp/zeo247652538/1.fs.zeosock: load 0285cbad858bf2e6:0000000000000006: 0000000000000006: object was deleted @0285cbad858bf2e6
    TestLoad/py/msgpack=true: xtesting.go:290: load 7fffffffffffffff:0000000000000007: returned err unexpected:
        have: /tmp/zeo247652538/1.fs.zeosock: load 7fffffffffffffff:0000000000000007: 0000000000000007: no such object
        want: /tmp/zeo247652538/1.fs.zeosock: load 7fffffffffffffff:0000000000000007: 0000000000000007: object was deleted @0285cbacc06d3a4c
    TestLoad/py/msgpack=true: xtesting.go:290: load 7fffffffffffffff:0000000000000006: returned err unexpected:
        have: /tmp/zeo247652538/1.fs.zeosock: load 7fffffffffffffff:0000000000000006: 0000000000000006: no such object
        want: /tmp/zeo247652538/1.fs.zeosock: load 7fffffffffffffff:0000000000000006: 0000000000000006: object was deleted @0285cbad858bf2e6

This is due to https://github.com/zopefoundation/ZODB/issues/318

It is already documented in pktDecodeZ comment that flags is int|bool.
However until now we were decoding it only as int.

As is it was working, but it will fail when receiving e.g.
invalidateTransaction message, because ZEO/py server actually uses bool
when sending it:

    https://github.com/zopefoundation/ZEO/blob/5.2.1-20-gcb26281d/src/ZEO/asyncio/base.py#L139

Fix it.
This will be covered by watch tests, when watch support will be added in a followup patch.

Handle pickled lists as valid input when decoding tuples.

This is more uniform and will be followed by all data types in the next patch.
Here rename tid/oid pack/unpack routines correspondingly.
Add docstrings for X=tid|oid.

Keep information about which message encoding is used on the wire in
encoding type. Make pktDecode/pktEncode and data type conversion
utilities be methods of this type. For now there is only 'pickles'
encoding, but we'll soon introduce 'msgpack'.

Currently not everything related to pickles is localized in encoding -
we'll be moving more bits to encoding in the followup patches.

First step:

- move msg and msgFlags
- move pktDecode
- move functions to pack/unpack tid and oid

more to come.

The same behaviour as ZEO/py does.

Just renaming.

Starting from Go1.14 defer is no longer slow:
https://golang.org/doc/go1.14#runtime

See go123@515a6d14

As promised in b65f6d0f (go/zodb: Teach ZODB/go to access ZEO (draft))
let's start to add tests for ZEO/go client.

The client is tested against ZEO/py server.
Cover handshake and Load.

Load currently fails beause in ZODB/py POSKeyError does not allow to
distinguish whether it is no object at all or object exists and its data
was not found for tid_before.

    --- FAIL: TestLoad/py (0.22s)
        zeo_test.go:217: load 0285cbacc06d3a4c:0000000000000007: returned err unexpected:
            have: /tmp/zeo273044293/1.fs.zeosock: load 0285cbacc06d3a4c:0000000000000007: 0000000000000007: no such object
            want: /tmp/zeo273044293/1.fs.zeosock: load 0285cbacc06d3a4c:0000000000000007: 0000000000000007: object was deleted @0285cbacc06d3a4c
        zeo_test.go:217: load 0285cbad858bf2e6:0000000000000006: returned err unexpected:
            have: /tmp/zeo273044293/1.fs.zeosock: load 0285cbad858bf2e6:0000000000000006: 0000000000000006: no such object
            want: /tmp/zeo273044293/1.fs.zeosock: load 0285cbad858bf2e6:0000000000000006: 0000000000000006: object was deleted @0285cbad858bf2e6
        zeo_test.go:217: load 7fffffffffffffff:0000000000000006: returned err unexpected:
            have: /tmp/zeo273044293/1.fs.zeosock: load 7fffffffffffffff:0000000000000006: 0000000000000006: no such object
            want: /tmp/zeo273044293/1.fs.zeosock: load 7fffffffffffffff:0000000000000006: 0000000000000006: object was deleted @0285cbad858bf2e6
        zeo_test.go:217: load 7fffffffffffffff:0000000000000007: returned err unexpected:
            have: /tmp/zeo273044293/1.fs.zeosock: load 7fffffffffffffff:0000000000000007: 0000000000000007: no such object
            want: /tmp/zeo273044293/1.fs.zeosock: load 7fffffffffffffff:0000000000000007: 0000000000000007: object was deleted @0285cbacc06d3a4c

Problem reported upstream:

-> https://github.com/zopefoundation/ZODB/issues/318

No need to use trailing : suffix as xerr.Contextf automatically adds one.

This is utility to load whole content of a ZODB database.

It will be useful to load data to expect from e.g. dataok.fs and feed it
as txnvOk to DrvTestLoad when verifying another storage driver.

This is utility to create X function, similar to exc.Raisif, but that
would call t.Fatal instead.

We will soon need this infrastructure to test other storage drivers.

E.g. before:

    --- FAIL: TestLoad/py (0.22s)
        zeo_test.go:217: load 0285cbacc06d3a4c:0000000000000007: returned err unexpected: /tmp/zeo484021674/1.fs.zeosock: load 0285cbacc06d3a4c:0000000000000007: 0000000000000007: no such object  ; want: /tmp/zeo484021674/1.fs.zeosock: load 0285cbacc06d3a4c:0000000000000007: 0000000000000007: object was deleted @0285cbacc06d3a4c
        zeo_test.go:217: load 0285cbad858bf2e6:0000000000000006: returned err unexpected: /tmp/zeo484021674/1.fs.zeosock: load 0285cbad858bf2e6:0000000000000006: 0000000000000006: no such object  ; want: /tmp/zeo484021674/1.fs.zeosock: load 0285cbad858bf2e6:0000000000000006: 0000000000000006: object was deleted @0285cbad858bf2e6
        zeo_test.go:217: load 7fffffffffffffff:0000000000000006: returned err unexpected: /tmp/zeo484021674/1.fs.zeosock: load 7fffffffffffffff:0000000000000006: 0000000000000006: no such object  ; want: /tmp/zeo484021674/1.fs.zeosock: load 7fffffffffffffff:0000000000000006: 0000000000000006: object was deleted @0285cbad858bf2e6
        zeo_test.go:217: load 7fffffffffffffff:0000000000000007: returned err unexpected: /tmp/zeo484021674/1.fs.zeosock: load 7fffffffffffffff:0000000000000007: 0000000000000007: no such object  ; want: /tmp/zeo484021674/1.fs.zeosock: load 7fffffffffffffff:0000000000000007: 0000000000000007: object was deleted @0285cbacc06d3a4c

After:

    --- FAIL: TestLoad/py (0.22s)
        zeo_test.go:217: load 0285cbacc06d3a4c:0000000000000007: returned err unexpected:
            have: /tmp/zeo087978981/1.fs.zeosock: load 0285cbacc06d3a4c:0000000000000007: 0000000000000007: no such object
            want: /tmp/zeo087978981/1.fs.zeosock: load 0285cbacc06d3a4c:0000000000000007: 0000000000000007: object was deleted @0285cbacc06d3a4c
        zeo_test.go:217: load 0285cbad858bf2e6:0000000000000006: returned err unexpected:
            have: /tmp/zeo087978981/1.fs.zeosock: load 0285cbad858bf2e6:0000000000000006: 0000000000000006: no such object
            want: /tmp/zeo087978981/1.fs.zeosock: load 0285cbad858bf2e6:0000000000000006: 0000000000000006: object was deleted @0285cbad858bf2e6
        zeo_test.go:217: load 7fffffffffffffff:0000000000000007: returned err unexpected:
            have: /tmp/zeo087978981/1.fs.zeosock: load 7fffffffffffffff:0000000000000007: 0000000000000007: no such object
            want: /tmp/zeo087978981/1.fs.zeosock: load 7fffffffffffffff:0000000000000007: 0000000000000007: object was deleted @0285cbacc06d3a4c
        zeo_test.go:217: load 7fffffffffffffff:0000000000000006: returned err unexpected:
            have: /tmp/zeo087978981/1.fs.zeosock: load 7fffffffffffffff:0000000000000006: 0000000000000006: no such object
            want: /tmp/zeo087978981/1.fs.zeosock: load 7fffffffffffffff:0000000000000006: 0000000000000006: object was deleted @0285cbad858bf2e6

We will soon need this infrastructure to test other storage drivers.

This package already provides parts of infrastructure for ZODB testing
(see e.g. 46636b16 and 7f14e2cb), and will be used as the place for such
infrastructure.

Maybe in the future we'll rename it to something like ztesting, but
leaving package name as is for now.

See go123@515a6d14

See go123@515a6d14

Provide way for users to retrieve Connection -> DB from which it was opened.
Wendelin.core 2 might need this to open a second connection given a one.

Provide way for users to go from DB -> IStorage for which it was opened.
Wendelin.core 2 needs it.

Implement "TODO defer stor.Close()" that was sitting there in every subcommand.

Julien Muchembled notices that strictly speaking Head could return tid
that it receives during its call, not strictly as observed before the
call. In other words for which range might be returned it is like below:

           ...\\\\\\\\\\\↓      ↓////////////...
        ──────────────────────────────────────────
               <head call>      <sync call>

Fixes 151d8b79 (go/zodb: LastTid -> Sync + Head).

[1] https://github.com/zopefoundation/ZODB/pull/307#discussion_r436881665

This helps when events are printed and logged.

Paralleling nexedi/zodbtools@0b6f99da.

All test databases in */testdata/ stay as before bit-to-bit the same.

The only change is extra LF that is appended to end of 1.zdump.pyok due
to nexedi/zodbtools@624aeb09.

Correspondingly fix dump on zodbtools/go side.

The only update comes from

b@8c9698d6

Rerun `go generate` for this package. Updates come due to updated
`stringer`. zproto-marshal.go stays the same.

When trying to regenerate zproto-marshal.go on a fresh go workspace I
got:

    (neo) (z-dev) (g.env) kirr@deco:~/src/neo/src/lab.nexedi.com/kirr/neo/go/neo/proto$ go generate
    typecheck: ../../zodb/zodb.go:174:2: could not import lab.nexedi.com/kirr/go123/mem (can't find import: "lab.nexedi.com/kirr/go123/mem")
    exit status 1
    proto.go:57: running "sh": exit status 1

Digging around a bit this turned out to be due to that
importer.Default() is using packages in only binary installed form:

    https://github.com/golang/go/issues/19334#issuecomment-283430322
    https://github.com/golang/go/issues/11415

Fix it by using "source" importer which always looks at sources.

(zproto-marshal.go stays unchanged after `go generate` rerun)

So that Go1.13 and golang.org/x/xerrors understand the wrapping:
https://blog.golang.org/go1.13-errors

Preserve Cause for compatibility with github.com/pkg/errors.

This allows to use proper btree.Node type - not just zodb.IPersistent -
in places which expects a tree node.

More comments, typos, ...

If cache control changes, all objects needs to go through new cache
control as what was e.g. pinned before could be in other class
from the point of view of new cache control.

Tests pending...

We already have LiveCacheControl and policy to keep object state in live
cache. However that state is currently kept only until the object is
present in the cache, and the object is there currently only until there
are live pointers to the object from anywhere. If all such pointers go
away, LiveCache currently discards the object.

Add a way (PCachePinObject) to tell live cache that it must retain an
object even if there are no other pointers to it. This is needed for
Wendelin.core which semantically relies on some objects to be present in
live cache, even if in ghost state, for propagating ZODB invalidations
into OS pagecache:

https://lab.nexedi.com/kirr/wendelin.core/blob/000bf16359/wcfs/wcfs.go#L245

For symmetry add a way (PCacheDropObject) to tell live cache that it
must not retain an object. This makes sense for objects that are read in
large only once to avoid needlessly evicting all other objects while
making the room for what won't be soon used again.

For completeness add also a way (PCacheDropState) to tell live cache
that the state of this object should not be cached. This can make sense
even for objects that are pinned: for example Wendelin.core pins ZBlk
objects to rely on volatile metadata it attaches to them, but have to
drop ZBlk state that was loaded from database: that state is propagated
to be cached in OS pagecache, and keeping it also in ZODB live cache
would just waste RAM.

Finally add test to cover LiveCache/LiveCacheControl functionality in
detail.