http://permalink.gmane.org/gmane.comp.web.zope.zope3/13884

load() and loadEx():  eliminate gratuitous differences.

Ideally, load() should call loadEx() instead, but we really
don't want "an extra" Python-level call here (heavily used).

loadEx():  deleted pointless call of self._read_txn_header().

ServerStub.loadEx() comments:  these were obviously wrong in
several ways, but I don't know the full truth.  Better to
say so up front than to leave them clearly wrong, though.

Collector #1830.

In some error cases when reading a FileStorage index, the
code referenced an undefined global.

Get rid of all code defining or referencing t32.

Such code never made sense; it was all of the form

    t32 = 1L << 32
    if something < 0:
        something = t32 - something

which is a way to change negative integers into
gigantic positive integers greater than 4 billion, and
never made sense in context.  Possibly

        something = t32 + something

was intended, but that never made sense in context either.

In any case, `something` is invariably obtained from
struct.unpack using the "H" format code, and that never
returns a negative integer to begin with.

read_index():  Mechanical reformat for easier reading.

Collector 1822.

Make undo{Log,Info} arguments act like Python slice indices when
both are non-negative.  The code used to do that before ZODB 3.4a9,
but changed to match ZODB's UML documentation.  Alas, some
(untested) code in Zope relied on the actual behavior (see the
collector report).  Changed code, docs, and tests to bless the
old behavior in these cases.

DemoStorage.UndoLog:  this was wrong in several ways.  I'm still
unsure about why it skips "packed" transactions.  That doesn't seem
right, but I don't have time to wonder about that now.

Forward port from 2.7 branch.

UndoSearch._readnext():  Add the transaction size to the return value.

undoInfo() and undoLog() almost always returned wrong # of results.

Repaired, + new tests.

Port from ZODB 3.2.

DemoStorage:  Added implementations for registerDB() and new_oid().  As
Tres discovered the hard way, wrapping a ZEO client storage as a
DemoStorage base storage yields insane behavior otherwise.

BaseStorage.new_oid():  Rewrite to eliminate recursion, and hence also the
need for the undocumented and irregular `last=` argument.

Other:  removed the `last=` argument to new_oid() every place that felt
compelled to spread that insanity ;-).  Seriously, it served no purpose
at all elsewhere, and looks like people just cut 'n pasted in fear.

Added news blurbs about fsIndex improvements, and about that
.index files written by 3.4 won't be readable by earlier ZODBs.

This introduces a "multi-database" concept (a simplification
of Jim's Wiki proposal), and adds many interface definitions.

Work done during the PyCon 2005 ZODB sprint, by Christian
Theune, Jim Fulton and Tim Peters.

Add a record iteration protocol to FileStorage.  You can use the record iterator to iterate over all current revisions
of data pickles in the storage.

In order to support calling via ZEO, we don't implement this as an actual iterator.  An example of using the record iterator
protocol is as follows:

storage = FileStorage('anexisting.fs')
next_oid = None
while 1:
    oid, tid, data, next_oid = storage.record_iternext(next_oid)
    # do something with oid, tid and data
    if next_oid is None:
        break

The behavior of the iteration protocol is now to iterate over all current records in the database in ascending oid order, 
although this is not a promise to do so in the future.

Port from ZODB 3.2.

Change FileStorage .restore() and .store() to update max oid in use.

This is the last of the checkins to fix critical bugs involving rare cases
where a FileStorage could end up reusing old oids for new objects.

Port from ZODB 3.2.

Stop believing the maximum oid cached in a FileStorage's .index file.

This is a critical bugfix, although the problems it addresses are
(a) rare; and, (b) not entirely fixed yet (more checkins to come).

The true max oid is found efficiently now by exploiting the recently-added
fsIndex.maxKey() method (which was, of course, added for this purpose).

Also fix that the .index file could get updated on disk when the
FileStorage was opened in read-only mode.  The code was trying to prevent
this, but missed the most obvious rewrite path.

Incidentally improved many obsolete and/or incorrect comments.

Merge rev 27722 from Zope trunk.

Collector #1517: History tab for ZPT does not work

FileStorage.history() was reading the user, description,
and extension fields out of the object pickle, due to
starting the read at a wrong location.

and extension fields from a transaction was fatally confused,
usually reading them out of the object pickle by mistake.  This
caused several tools to display binary gibberish.

Collector 1327:  FileStorage init confused by time travel

If the system clock moved back in time more than 30 days between
a FileStorage close and (re)open, new txn ids could be smaller
than txn ids already in the database.  Plugged that hole.

fiddly code trying to prevent OverflowError.

conversion (this is an --all test, so the original committer probably
didn't run it).

Introduced a trivial new module (loglevels.py) to centralize creating
the custom log levels ZODB/ZEO want to use.

processing the messages because the format might have changed a little bit.

    FileStorageError:  The database has already been packed to a
    later time or no changes have been made since the last pack

exception.  Instead that message is logged (at INFO level), and
the pack attempt simply returns then (no pack is performed).

Incidentally, this should repair frequent reports of failure in the
new checkPackLotsWhileWriting test.  On non-Windows systems, it
seems that the worker thread often didn't get enough cycles to commit
a change between the main thread's attempts to run pack() (and so
the exception above got raised then).

failures in the ZEO FileStorage pack-while-writing tests, where they die
with
    CorruptedError: ... transaction with checkpoint flag set

By coincidence, the same kind of death during pack() was reported on
zodb-dev today(!).  The new checkPackLotsWhileWriting test provokes this
on my box reliably (failed each time it was run).

The problem appears to be that pack's idea of where a FileStorage
ends was obtained merely by seeking to the end of the file.  But if
a transaction is in progress, there can be an extra "not really there
yet" transaction at the end of the file, and when either buildPackIndex()
or findReachableFromFuture() bumped into a such a thing, the CorruptedError
above got raised.  This is always rare, and was exceedingly rare before
because only one pack per test was tried.  The new test tries packing
repeatedly while a thread keeps hammering the database, so is much more
likely to provoke the problem.

The fix amounts to passing FileStorage._pos to the pack code, telling
the latter directly where the legit transactions in the file end.

The only values _packt ever had were z64 and None.  The only thing _packt
was used for that appeared to make sense was to block undo while a pack
was in progress, and a bool works better for that.  tids were compared
to _packt in a few spots, but since the only values _packt could have
were in {z64, None}, the comparisons were doomed to (regardless of tid)
return "greater than" (z64) or to be undefined (None; although in recent
Pythons None is reliably less than objects of other types, so "greater
than" was the only outcome possible from these comparisons under 2.3.3).
Removed these comparisons, collapsing surrounding expressions to
equivalents taking into account that tid < self._packt was always false.

a two-state flag, with possible values z64 or None -- eh?).

However, fixing it so far *appears* to fix a bunch of shy FileStorage
pack[Now]WhileWriting test failures whose connection to loadBefore()
isn't obvious.  It's plausible that it's a real fix for those, just
not (yet) compelling.  It seems very likely to be a real fix for very
rare failures that have shown up only in ChrisM's overnight testrunner
reports (because those did have stuff related to loadBefore() in their
tracebacks).

The old undo was not used by Zope and shouldn't have been used by any
other client.  The newly named undo() is the preferred version of
undo.

XXX DemoStorage didn't implement transactionalUndo, so now it doesn't
implementation undo() at all.  We need to replace it with the demo
storage from ZODB4.

There are a few changes related to ZODB4 removal in this checkin.