There are 2 objectives:
- Prevent randomly trying to connect to an unresponsive storage node, which
  impairs performances a lot. Note that this happens only when the master
  didn't notice the disconnection, so the node is still in running state in
  the node manager.
- Increase connection reuse, saving the cost of establishing a new
  connection and a slot in connection pool.
Randomisation should be kept to even out storage node use.

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2173 71dcc9de-d417-0410-9af5-da40c76e7ee4

Deadlock occurs between connection establishment (which holds the pool
lock and waits for an answer on its thread queue) and epoll thread, which
tries to unregister the connection from pool (trying to take pool lock)
before notifying the queue.
Unregistering a connection from pool does not require taking the pool lock,
as it is just meant to prevent connections from being created in parallel.

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2172 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2171 71dcc9de-d417-0410-9af5-da40c76e7ee4

A connection lost no more abort current transactions and clean the node
index (by calling abortFor). Do this job each time a transaction aborted
to avoid empty entries and memory leaks.

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2170 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2169 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2168 71dcc9de-d417-0410-9af5-da40c76e7ee4

Bug reported by Gaël Le Mignot: in a sample case he encountered,
"description" was a unicode object, causing encoding failure in packet
construction.
Python "str" is used in ZODB FileStorage code (see format.py), so just do
the same.

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2167 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2166 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2165 71dcc9de-d417-0410-9af5-da40c76e7ee4

ConnectorConnectionClosedException might get triggered when notifying a
node about transaction abortion, causing tpc_abort to fail eating pending
answers and clearing local_var.
Swallow all exceptions (various network problems translating to various
exceptions).

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2164 71dcc9de-d417-0410-9af5-da40c76e7ee4

This avoids sending multiple abort messages for each storage node (one per
partition it is assigned to).

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2163 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2162 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2161 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2160 71dcc9de-d417-0410-9af5-da40c76e7ee4

When a delayed store is processed, don't take the lock if the transaction
is no more registered (may have been aborted due to a timeout with the
delayed store).
Change the TransactionManager API: add register() that must be called before
any storeObject/storeTransaction operation.

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2159 71dcc9de-d417-0410-9af5-da40c76e7ee4

This requires the live debugger to be enabled. Only master, storage and
client side handle this signal.

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2158 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2157 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2156 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2155 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2154 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2153 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2152 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2151 71dcc9de-d417-0410-9af5-da40c76e7ee4

If the last storage node gets disconnected, an operation failure will
occur, so there is no risk of sending AnswerTransactionFinished to client
if some object was not stored.

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2150 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2149 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2148 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2147 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2146 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2145 71dcc9de-d417-0410-9af5-da40c76e7ee4

onStoreTimeout is called by poll thread, which causes a new local_var to be
implicitly created with a new queue (specific to poll thread, which isn't
supposed to have one).
To know the queue that should receive AnswerHasLock, reuse the one
registered to dispatcher to receive AnswerStoreObject.

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2144 71dcc9de-d417-0410-9af5-da40c76e7ee4

Make it optional, to suit "ping" use, but check that it's always passed
except in that special case.

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2143 71dcc9de-d417-0410-9af5-da40c76e7ee4

This fixes cases where a thread is expecting some answer from queue, and
a packet gets forgotten, as it would block until next expected packet
arrives (if there is no next packet, it will wait forever).

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2142 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2141 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2140 71dcc9de-d417-0410-9af5-da40c76e7ee4

- Merge sys.exit()
- Remove exit message
- Move out of loop the code that should run once
- Add an XXX

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2139 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2138 71dcc9de-d417-0410-9af5-da40c76e7ee4

This check was always false in the case of a client.
Move the 'node is None' condition in the potential scope.

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2137 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2136 71dcc9de-d417-0410-9af5-da40c76e7ee4

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2135 71dcc9de-d417-0410-9af5-da40c76e7ee4

Keep lock around _initNodeConnection() in connection pool to avoid concurrent
connections to the same storage node.

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2134 71dcc9de-d417-0410-9af5-da40c76e7ee4