GitLab

Connection objects were typically immortal because the threaded
transaction manager kept them in an ever-growing lists.  Reworked the
transaction manager internals to use a simple implementation of weak sets
instead.  This plugs all leaks in the test program attached to the
collector report (which was leaking about 100KB/sec on my box).

Forward-porting from Zope 2.7.

The checkFullTimeStamp() test was sensitive to unique mktime() behavior
on FreeBSD.  See:

http://lists.freebsd.org/pipermail/freebsd-standards/2003-November/000268.html 

The purpose of this test is to exercise ZODB's TimeStamp object, so got
rid of dependence on platform mktime() and time.timezone quirks --
TimeStamp works in GMT, so how mktime() treats tm_isdst should be
irrelevant in all TimeStamp tests.

Also added a comment about the highly non-obvious numeric characteristics
of TimeStamp's treatment of seconds (round-tripping is surprisingly
inaccurate, but for a real reason).

If the system clock moved back in time more than 30 days between
a FileStorage close and (re)open, new txn ids could be smaller
than txn ids already in the database.  Plugged that hole.

cacheMeanDeal, and cacheMeanDeac methods from the DB class.  They
were untested, undocumented, unused, and did nothing of use (the
first always returned an empty tuple, the rest always returned None).

of the most recent full backup, repozo.py failed to find the appropriate
files, erroneously claiming

    No files in repository before <specified time>

Repaired that.  Also made it much more robust against "junk files" the
user may create, or leave behind, in the backup directory.  Added test.

Collector #1330:  repozo.py -R can create corrupt .fs.
When looking for the backup files needed to recreate a Data.fs file,
repozo could (unintentionally) include its meta .dat files in the list,
or random files of any kind created by the user in the backup directory.
These would then get copied verbatim into the reconstructed file, filling
parts with junk.  Repaired by filtering the file list to include only
files with the data extensions repozo.py creates (.fs, .fsz, .deltafs,
and .deltafsz).  Thanks to James Henderson for the diagnosis.

This fixes it, based on an approach suggested by Toby
Dickenson.  The triggering condition wasn't entirely
sane, so was very rare:  a persistent object with a
__del__ method that referenced an attribute of self.

scan_gc_items():  Look at persistent objects accessed
while this is running at most once.

New test checkMinimizeTerminates():  This spawns a thread
that will in fact run forever if you don't recompile
cPickleCache.c.  The test suite will keep running, but
checkMinimizeTerminates will fail, and all future calls
to cacheMinimize() will be effectively ignored (so other
bad things may happen as a result).

The refcount reported for a ghost was one too small, because the code for
ghosts and non-ghosts was the same, and presumably "it's a feature" that
the code for non-ghosts deliberately lies about the true Python refcount,
in order (guessing here) to report what the count would have been if the
cPickleCache didn't exist.  But while the cPickleCache holds on to a real
reference to non-ghost objects, it does not hold a real reference to ghost
objects, so subtracting "an extra" count for all objects made it appear
that non-referenced ghosts exist in the cache (which doesn't actually
happen).

What a tangled web we weave ...

    FileStorageError:  The database has already been packed to a
    later time or no changes have been made since the last pack

exception.  Instead that message is logged (at INFO level), and
the pack attempt simply returns then (no pack is performed).

Incidentally, this should repair frequent reports of failure in the
new checkPackLotsWhileWriting test.  On non-Windows systems, it
seems that the worker thread often didn't get enough cycles to commit
a change between the main thread's attempts to run pack() (and so
the exception above got raised then).

failures in the ZEO FileStorage pack-while-writing tests, where they die
with
    CorruptedError: ... transaction with checkpoint flag set

By coincidence, the same kind of death during pack() was reported on
zodb-dev today(!).  The new checkPackLotsWhileWriting test provokes this
on my box reliably (failed each time it was run).

The problem appears to be that pack's idea of where a FileStorage
ends was obtained merely by seeking to the end of the file.  But if
a transaction is in progress, there can be an extra "not really there
yet" transaction at the end of the file, and when either buildPackIndex()
or findReachableFromFuture() bumped into a such a thing, the CorruptedError
above got raised.  This is always rare, and was exceedingly rare before
because only one pack per test was tried.  The new test tries packing
repeatedly while a thread keeps hammering the database, so is much more
likely to provoke the problem.

The fix amounts to passing FileStorage._pos to the pack code, telling
the latter directly where the legit transactions in the file end.