More on collector 1847 (ZEO cache tracing).

simul.py hit NameErrors when startup cache verification
found data in the cache to invalidate.

The cache's loadBefore() implementation called _trace()
incorrectly, passing the tid where the version argument
belonged.

Part of Collector 1860 (the other part is in Zope).

There's no possiblity of rollback here, so no need to insist that the
data manager support rollbacks.

Plug leaks in the ZEO client cache.

ClientCache._evicted():  When deleting the last range of
non-current tids for an oid, remove the list from the
noncurrent dict instead of leaving an empty list sitting
there forever.  This also required adjusting the side-
effect dance in ClientCache._remove_noncurrent_revisions().

FileCache._makeroom():  This was the major leak -- it
neglected to remove an evicted object's Entry from the
key2entry dict.

  move things around so that zeo scripts are installed for
  standalone zodb releases but not for zope releases

Needs to import errno now.

Worm around suspected Windows socket bug in Windows trigger code.

See the thread starting at
 http://mail.zope.org/pipermail/zope/2005-July/160433.html
for gory details.

makeroom():  Simplify retrieve+del from filemap.

Removed the `key2size` OIBTree.

Maintaining this is a significant expense; the code didn't
make real use of it; and every time I've tried to exploit it
I've ended up with grossly too-large simulated hit rates.
Would probably take another day to figure out exactly why that
is, and the simulated rates are "good enough" now without it.

More words; update examples w/ new output.

The cache simulation seems good enough to be useful now,
although the toy app I wrote to generate a 500MB trace
file doesn't use MVCC in an essential way (some of the
MVCC simulation code is nevertheless exercised, since
an invalidation of current data in the presence of MVCC
always creates a cache record for the newly non-current
revision).

Still puzzling over what to do when the trace file records
a load hit but the simulated cache gets a miss.  The
old simulation code seemed to assume that a store for the same
oid would show up in the trace file next, and it could get
the info it needed about the missing object from the store
trace.  But that isn't true:  precisely because the load was
a hit in the trace file, the object isn't going to be stored
again "soon" in the trace file.

Here are some actual-vs-simulated hit rate results, for a
20MB cache, with a trace file covering about 9 million
loads, over 3 ZEO client (re)starts:

actual   simulated
------   ---------
  93.1        92.7
  79.8        79.0
  68.0        69.1
      
  81.4        81.1  overall

Since the simulated hit rates are both higher and lower
than the actual hit rates, that argues against a gross
systematic bias in the simulation (although there may
be several systematic biases in opposite directions).

Massive rewrite to make simul.py's circular cache aware of
MVCC.  It isn't blowing up, but it thinks the hit rate is
100% -- might have missed something there <wink>.

More steps on the way toward having useful cache simulation.

I think the primary hangup now is that simul.py doesn't know
anything about MVCC.  As a result, it thinks there's significatly
more free space in the cache than there really is, and that
probably accounts for it predicting significantly higher hit
rates than I actually see.

Lots of fiddling to work w/ current trace file format.

Output isn't obviously wholly insane, but isn't a reasonably
close enough match to stats.py output either yet.

stats.py:  cut the # of file reads by 1/3, and the # of
struct.unpack() calls by 1/2.

Various bugfixes elsewhere.

Many little bugfixes and improvements in stats.py.
This has survived several 100 MB of trace files
I generated over the last few days, so it's solid
now if not necessarily perfect.

Replaced simul.py with the much broader-ranging code
Jeremy and I were working on a couple years ago,
although it can't work with the current trace file
format (no real loss there -- the simul.py it's
replacing can't work with the current format either).

Stitch in ZConfig's docs from ZConfig.

Remove this -- it's a way out-of-date copy of the ZConfig
docs.  I'll stitch them back in from the ZConfig repo next,
via svn:externals.  Ideally, ZODB shouldn't have this as
part of its checkout tree at all, but that also has implications
for the way ZRS gets packaged, etc.

A large number of changes so that cache tracing isn't
100% broken -- I can get a trace file now, and stats.py
doesn't blow up anymore.  Don't yet know whether it's
reporting sensible things, but it's not obviously insane.
simul.py doesn't know about the current cache implementation
either yet.

ClientCache:  made it a new-style class.

FileCache:  Removed the reuse= argument.  IIRC, it made some sense
when different cache schemes were first being tried, but complicated
the logic now to no good end.

Nuked the maddening log warnings about reuse=True.  They were never
helpful and were often confusing.

Added info-level log messages to record the path of the client cache
file, whether it's persistent or temporary, and if it's persistent
whether we're creating or reusing it.

Minor changes to comments.

Massive rewrite of cache.txt, to match current reality.

Collector 1843:  IISet.keys: bad exception handling.

Bucket_rangeSearch():  De-obfuscated the calls to
Bucket_findRangeEnd(), so that they stop ignoring
the latter's error returns (a mind-bending combination
of embedded assignment nested in an UNLESS macro,
seemingly copy+paste'd so that the error occurred twice).

Port from ZODB 3.2.

Collector #1846:  If an uncommitted transaction was found, fsrecover.py
fell into an infinite loop.  Fixed that, and added a new test
(testUncommittedAtEnd) to ensure this stays fixed.

Subtransactions are deprecated in 3.5, and will go away
in 3.7.

Convert internal uses of subtxns to use savepoints instead.

I suspect BTrees/convert.py should be removed instead.

Revert extra copying in Connection._rollback().

Since TmpStore.reset() had to change too to copy the index anyway,
it no longer hurts that _rollback() clears the index (as a side
effect of self._cache.invalidate(src.index)).

Merge tim-savepoint branch.

Allow rollback to a given savepoint multiple times.

transaction/_transaction.py
    _remove_and_invalidate_after():  Don't remove the
    argument savepoint from the savepoint stack anymore.

    abort():  abort(True) relied on the subtxn savepoint
    being invalidated as a side effect of rolling back
    to it.  Invalidate it explicitly now.

    Savepoint.rollback():  Don't invalidate self.

ZODB/Connection.py
    Connection._rollback():  Pass a temp container to
    _cache.invalidate(), because that method clears the
    container as a side effect.  Clearing the index
    in a savepoint was a disaster if the savepoint was
    used again.

    TmpStore.reset():  Make a copy of the argument index.
    An alternative would be to make all callers make
    copies, but the only caller now thinks of its `state`
    argument as being opaque.

transaction/savepoint.txt
ZODB/tests/{testSubTransaction.py, testConnectionSavepoint.txt}
    Test the new semantics, and beef up the tests.

transaction/tests/savepointsample.py
    _rollback_savepoint():  Needed to copy the data.

English repairs.

Simplify.

Primarily, _tindex should have been a dict all along, pack()
worked too hard at the end, and loadEx() is a minor variant
of load().

load() and loadEx():  eliminate gratuitous differences.

Ideally, load() should call loadEx() instead, but we really
don't want "an extra" Python-level call here (heavily used).

loadEx():  deleted pointless call of self._read_txn_header().

ServerStub.loadEx() comments:  these were obviously wrong in
several ways, but I don't know the full truth.  Better to
say so up front than to leave them clearly wrong, though.

Collector 1831.

The BTree minKey() and maxKey() methods gave a misleading message if no key
satisfying the constraints existed in a non-empty tree.