In order to do that correctly, this commit contains several other changes:

When connecting to a primary master, a full node list always follows the
identification. For storage nodes, this means that they now know all nodes
during the RECOVERING phase.

The initial full node list now always contains a node tuple for:
- the server-side node (i.e. the primary master): on a master, this is
  done by always having a node describing itself in its node manager.
- the client-side node, to make sure it gets a id timestamp:
  now an admin node also receives a node for itself.

This keeps the connection fully functional when a handler raises an exception.

Commits like 7eb7cf1b
("Minimize the amount of work during tpc_finish") dropped what was done in
commit 07b48079
("Ignore some requests, based on connection state") to protect request handlers
when they respond.

This commit fixes this in a generic way.

The initial intention was to rely on stable sorting when several events have
the same key. For this to happen, sorting must not continue the comparison with
the second item of events.

This could lead to data corruption (conflict resolution with wrong base):

  FAIL: testNotifyReplicated (neo.tests.threaded.test.Test)
  ----------------------------------------------------------------------
  Traceback (most recent call last):
    File "neo/tests/threaded/__init__.py", line 1093, in wrapper
      return wrapped(self, cluster, *args, **kw)
    File "neo/tests/threaded/test.py", line 2019, in testNotifyReplicated
      self.assertEqual([15, 11, 13, 16], [r[x].value for x in 'abcd'])
    File "neo/tests/__init__.py", line 187, in assertEqual
      return super(NeoTestBase, self).assertEqual(first, second, msg=msg)
  failureException: Lists differ: [15, 11, 13, 16] != [19, 11, 13, 16]

  First differing element 0:
  15
  19

  - [15, 11, 13, 16]
  ?   ^

  + [19, 11, 13, 16]
  ?   ^

'aborted' could appear twice.

zodburi[1] provides a way to open ZODB storages by URL/URI. It already
has support for file:// zeo:// zconfig:// memory:// etc schemes out of
the box and third-party-to-ZODB storages can add support for their
schemes via providing zodburi.resolvers entrypoint.

For example relstorage and newtdb do this.

Let's also teach NEO to open itself via neo:// URI schema.

[1] http://docs.pylonsproject.org/projects/zodburi
[2] https://github.com/zodb/relstorage/blob/2.1a1-15-g68c8cf1/relstorage/zodburi_resolver.py
[3] https://github.com/newtdb/db/blob/0.5.2-1-gbd36e90/src/newt/db/zodburi.py

Commit 58d0b602 didn't fix the issue completely.
Storage space can be freed with --repair option.

This adds an expectedFailure test.

This is a follow up of commit 64afd7d2,
which focused on read accesses when there is no transaction activity.

This commit also includes a test to check a simpler scenario that the one
described in the previous commit.

Commit ad43dcd3 should have bumped it as well.

Unused but it is likely to be useful in the future.

The bug could lead to data corruption (if a partition is wrongly marked as
UP_TO_DATE) or crashes (assertion failure on either the storage or the master).

The protocol is extended to handle the following scenario:

    S                                    M
    partition 0 outdated
      <-- UnfinishedTransactions ------>
    replication of partition 0 ...
    partition 1 outdated
      --- UnfinishedTransactions ...
    ... replication finished
      --- ReplicationDone ...
                                         tweak
      <-- partition 1 discarded --------
                                         tweak
      <-- partition 1 outdated ---------
          ... UnfinishedTransactions -->
          ... ReplicationDone --------->

The master can't simply mark all outdated cells as being updatable when it
receives an UnfinishedTransactions packet.

After an attempt to read from a non-readable, which happens when a client has
a newer or older PT than storage's, the client now retries to read.

This bugfix is for all kinds of read-access except undoLog, which can still
report incomplete results.

This revert commit bddc1802,
to fix the following storage crash:

  Traceback (most recent call last):
    ...
    File "neo/lib/handler.py", line 72, in dispatch
      method(conn, *args, **kw)
    File "neo/storage/handlers/master.py", line 44, in notifyPartitionChanges
      app.pt.update(ptid, cell_list, app.nm)
    File "neo/lib/pt.py", line 231, in update
      assert node is not None, 'No node found for uuid ' + uuid_str(uuid)
  AssertionError: No node found for uuid S3

Partitition table updates must also be processed with InitializationHandler
when nodes remain in PENDING state because they're not added to the cluster.

In the worst case, with many clients trying to lock the same oids,
the cluster could enter in an infinite cascade of deadlocks.

Here is an overview with 3 storage nodes and 3 transactions:

 S1     S2     S3     order of locking tids          # abbreviations:
 l1     l1     l2     123                            #  l: lock
 q23    q23    d1q3   231                            #  d: deadlock triggered
 r1:l3  r1:l2  (r1)   # for S3, we still have l2     #  q: queued
 d2q1   q13    q13    312                            #  r: rebase

Above, we show what happens when a random transaction gets a lock just after
that another is rebased. Here, the result is that the last 2 lines are a
permutation of the first 2, and this can repeat indefinitely with bad luck.

This commit reduces the probability of deadlock by processing delayed
stores/checks in the order of their locking tid. In the above example,
S1 would give the lock to 2 when 1 is rebased, and 2 would vote successfully.

This fixes a bug that could to data corruption or crashes.

It becomes possible to answer with several packets:
- the last is the usual associated answer packet
- all other (previously sent) packets are notifications

Connection.send does not return the packet id anymore. This is not useful
enough, and the caller can inspect the sent packet (getId).

In reality, this was tested with
  taskset 1 neotestrunner ...

Traceback (most recent call last):
  ...
  File "neo/lib/handler.py", line 72, in dispatch
    method(conn, *args, **kw)
  File "neo/master/handlers/client.py", line 70, in askFinishTransaction
    conn.getPeerId(),
  File "neo/master/transactions.py", line 387, in prepare
    assert node_list, (ready, failed)
AssertionError: (set([]), frozenset([]))

Master log leading to the crash:
  PACKET    #0x0009 StartOperation                 > S1
  PACKET    #0x0004 BeginTransaction               < C1
  DEBUG     Begin <...>
  PACKET    #0x0004 AnswerBeginTransaction         > C1
  PACKET    #0x0001 NotifyReady                    < S1

It was wrong to process BeginTransaction before receiving NotifyReady.

The changes in the storage are cosmetics: the 'ready' attribute has become
redundant with 'operational'.