Partial savepoint review.

Added more comments.  Changed some comments to English.
Renamed some attributes and methods for clarity.
Switched to using a Python WeakKeyDictionary instead of
rolling our own out of a Python dict and raw weakrefs.

Port from ZODB 3.2.

Added new test checkSubtxnCommitDoesntGetInvalidations to
verify that a longstanding bug in subtransaction commit is
repaired.

Jim (Fulton) discovered this in ZODB 3.4's code, while implementing
savepoint/rollback.  Same bugs had been there at least since ZODB 3.1.

Also added news about the bug.

There is no logging level named EXCEPTION.

Original checkin msgs follow:

r30203 | jim | 2005-04-27 11:55:09 -0400 (Wed, 27 Apr 2005) | 2 lines
   M /ZODB/branches/3.4/src/transaction/_transaction.py

Fixed stupid bug.

r30200 | jim | 2005-04-27 07:20:56 -0400 (Wed, 27 Apr 2005) | 39 lines
   M /ZODB/branches/3.4/src/transaction/_transaction.py

Changed the strategy for managing savepoints.  The requirements
for savepoint management are:

- All savepoints for a transaction should be invalidated when the
  transaction commits or aborts

- If a savepoint is rolled back, then all savepoints after it within
  a transaction must be invalidated.

We previously implemented these requirements by organizing transaction
savepoints into a doubly linked list.  This was overkill.  We didn't
have need for such fine-grained ordering.  This strategy had the
disadvantage that it kept all savepoints around until the transaction
ended.  Savepoints could be expensive to keep and it's possible that
some applications could keep a lot of them.

The new stragey is to:

- Keep weak references to savepoints.  We can forget about savepoints
  that the application isn't using.  Any resources used by these
  savepoints can be freed.

(We have to keep a strong reference to the last savepoint used for
  a subtransaction.)

- We assign indexes to savepoints within a transaction.  When a
  savepoint is rolled back, in addition to invalidating that
  savepoint, we also invalidate savepoints with a higher index.

A side effect of this change is that code using the savepoint API
should interfere less with code using subtransactions.  Of course, we
really need to phase out code that uses subtransactions.

It is likely that we can leverage this change in strategy to speed
creation of ZODB connection savepoints.  Creating a ZODB connection
savepoint now requires copying the savepoint storage index.  This
index could become large.  If applications aren't holding on to old
savepoints, then it is possible that we could avoid this copy.

Trimmed trailing whitespace.

Repaired NEWS format; predicted the next release will be 3.4b1.

A ``getBeforeCommitHooks()`` method has been added to Transaction. It
returns an iterable producing the registered beforeCommit hooks.

Interface repairs, of many kinds.

Original checkin comments follow:

r30168 | tim_one | 2005-04-25 14:17:37 -0400 (Mon, 25 Apr 2005) | 2 lines
   M /ZODB/branches/3.4/NEWS.txt
   ...

An internal 3.4a5 release, to incorporate savepoint fixes.

r30165 | jim | 2005-04-25 12:29:28 -0400 (Mon, 25 Apr 2005) | 11 lines
   M /ZODB/branches/3.4/src/transaction/_transaction.py
   M /ZODB/branches/3.4/src/transaction/savepoint.txt
   M /ZODB/branches/3.4/src/transaction/tests/test_savepoint.py

Fixed a bug in savepoint rollback.  It's not enough to rollback
just the savepoint being rolled back because later savepoints
might involved data managers that hadn't joined when the savepoint
being rolled back was created.

Now, when a data manager joins and we have savepoints, we create a
data manager savepoint for the new data manager and add the
datamanager savepoint to all previous transaction savepoints.  Note
that this data manager savepoint can be a special savepoint that just
calls abort on the data manager when it is rolled back.

r30164 | tim_one | 2005-04-25 11:16:20 -0400 (Mon, 25 Apr 2005) | 2 lines
r30163 | tim_one | 2005-04-25 11:08:37 -0400 (Mon, 25 Apr 2005) | 2 lines
r30162 | tim_one | 2005-04-25 11:06:51 -0400 (Mon, 25 Apr 2005) | 2 lines
   M /ZODB/branches/3.4/src/transaction/interfaces.py

Grammar, spelling, English.
Close unterminated sentences.
Trim trailing whitespace.

r30161 | jim | 2005-04-25 10:51:16 -0400 (Mon, 25 Apr 2005) | 10 lines
   M /ZODB/branches/3.4/src/transaction/interfaces.py

Removed a "self" argument. self is normally not shown in interfaces.

Removed the freeme argument.  This argument is part of the
implementation, not the public interface.

Removed the subtransaction argument.  Although it is still supported,
it isn't part of the pblic interface.

Added missing documentation of the savepoint method.

r30160 | jim | 2005-04-25 10:41:08 -0400 (Mon, 25 Apr 2005) | 2 lines
   M /ZODB/branches/3.4/src/transaction/interfaces.py

Removed some stale discussion of subtransactions.

r30147 | jim | 2005-04-24 11:26:39 -0400 (Sun, 24 Apr 2005) | 7 lines
   M /ZODB/branches/3.4/src/transaction/_transaction.py
   M /ZODB/branches/3.4/src/transaction/savepoint.txt

Make transactions uncommitable if savepoint rollback fails.

Added demonstration of transaction non-commitability after savepoint
or savepoint rollback failure.

Updated "previous commit failed" error to "previous operation failed".

r30146 | jim | 2005-04-24 11:26:37 -0400 (Sun, 24 Apr 2005) | 5 lines
   M /ZODB/branches/3.4/src/ZODB/Connection.py
   M /ZODB/branches/3.4/src/transaction/interfaces.py
   M /ZODB/branches/3.4/src/transaction/tests/savepointsample.py

Refined interfaces to distinguish between data-manager savepoints and
transaction savepoints.

Updated some interface declarations.

r30145 | jim | 2005-04-24 10:48:15 -0400 (Sun, 24 Apr 2005) | 2 lines
   M /ZODB/branches/3.4/src/ZODB/tests/testConnectionSavepoint.py

added explanatory text

r30144 | jim | 2005-04-24 10:35:49 -0400 (Sun, 24 Apr 2005) | 2 lines
Changed paths:
   M /ZODB/branches/3.4/NEWS.txt

Updated to reflect savepoints.

Original checkin msgs follow:

r30131 | jim | 2005-04-23 23:33:29 -0400 (Sat, 23 Apr 2005) | 5 lines
   M /ZODB/branches/3.4/src/ZODB/Connection.py
   M /ZODB/branches/3.4/src/ZODB/tests/testConnectionSavepoint.py

Fixed a bug in commits with savepoints and changes since savepoints.
Once we start using savepoints, we need to make sure that all data are
committed through the savepoints. Otherwise, things can get committed
in the wrong order, leading to conflicts.

r30130 | jim | 2005-04-23 23:02:00 -0400 (Sat, 23 Apr 2005) | 6 lines
   M /ZODB/branches/3.4/src/ZODB/Connection.py
   M /ZODB/branches/3.4/src/ZODB/tests/testConnectionSavepoint.py

Fixed a bug that caused assertion errors if an object was added in a
savepoint, then modified and then aborted.

Also added missing code to clear registered objects when a savepoint
was rolled back.

r30129 | jim | 2005-04-23 21:29:02 -0400 (Sat, 23 Apr 2005) | 5 lines
   M /ZODB/branches/3.4/src/ZODB/Connection.py
   D /ZODB/branches/3.4/src/ZODB/TmpStore.py
   M /ZODB/branches/3.4/src/ZODB/tests/testConnection.py
   A /ZODB/branches/3.4/src/ZODB/tests/testConnectionSavepoint.py
   A /ZODB/branches/3.4/src/ZODB/tests/testConnectionSavepoint.txt
   M /ZODB/branches/3.4/src/ZODB/tests/testZODB.py
   M /ZODB/branches/3.4/src/ZODB/tests/test_datamanageradapter.py
   M /ZODB/branches/3.4/src/transaction/__init__.py
   M /ZODB/branches/3.4/src/transaction/_manager.py
   M /ZODB/branches/3.4/src/transaction/_transaction.py
   M /ZODB/branches/3.4/src/transaction/interfaces.py
   A /ZODB/branches/3.4/src/transaction/savepoint.txt
   A /ZODB/branches/3.4/src/transaction/tests/savepointsample.py
   M /ZODB/branches/3.4/src/transaction/tests/test_register_compat.py
   A /ZODB/branches/3.4/src/transaction/tests/test_savepoint.py

Added savepoints!

(And also added interfaces and rearranged some code to hopefully make
it easier to read.)

r30128 | jim | 2005-04-23 21:28:59 -0400 (Sat, 23 Apr 2005) | 2 lines
   M /ZODB/branches/3.4/src/transaction/tests/test_transaction.py

Removed some tests that son't make sense after the savepoont refactoring

r30127 | jim | 2005-04-23 21:28:57 -0400 (Sat, 23 Apr 2005) | 2 lines
   M /ZODB/branches/3.4/src/persistent/interfaces.py

Commented out mtime

ZEO version numbers the same (an auto-merge could do
very little of this correctly!).

Port ZEO changes for Windows from ZODB 3.2.

In addition, repaired the useless default logging in runzeo.py.

Here's Sidnei's Zope 2.7 checkin comment:

* Borrow Zope's 'Signal' mechanism for Windows, if available, to
  implement clean shutdown and log rotation handlers for Windows.

* Back to creating a .PID for ZEO, so external programs that wish to set
  the 'signal' can get the PID and therefore derive the signal name.
  Currently only necessary on Windows but created on all platforms which
  implement os.getpid(), as long as the 'pid-filename' option is set,
  or the 'INSTANCE_HOME' environment variable can be found.

Port from ZODB 3.2.

DemoStorage:  Added implementations for registerDB() and new_oid().  As
Tres discovered the hard way, wrapping a ZEO client storage as a
DemoStorage base storage yields insane behavior otherwise.

BaseStorage.new_oid():  Rewrite to eliminate recursion, and hence also the
need for the undocumented and irregular `last=` argument.

Other:  removed the `last=` argument to new_oid() every place that felt
compelled to spread that insanity ;-).  Seriously, it served no purpose
at all elsewhere, and looks like people just cut 'n pasted in fear.

Delayed news about old fsdump improvement.

This is revs 29961 and 29965.

test_cache():  Muck with Python's warnings module.

The test didn't work if run more than once (e.g., via Zope3's
test.py's "-N <n>" argument, or via ZODB's test.py's "-L"
argument), because the warnings module suppresses warnings
after the first from a single line.  Made the warnings module
report repeats of DeprecationWarnings for the duration of
the test.

Merge ZODB/branches/efge-beforeCommitHook.

This is Florent Guillaume's branch, giving transaction objects
a new beforeCommitHook() method, as proposed by Jim Fulton on
zodb-dev.  Some changes were made to the branch code (comments,
more tests, more words in the docs).

Merge the ZODB part of Zope/branches/jim-fix-zclasses.

Here are checkin msgs from the branch relating to ZODB code:

    r29872 | jim | 2005-04-04 07:04:39 -0400 (Mon, 04 Apr 2005) | 3 lines
    Changed paths:
       M /Zope/branches/jim-fix-zclasses/lib/python/ZODB/serialize.py

    For instances of persistent classes, save a class-module/clas-name
    tuple if the class has a non-empty module string.

    ------------------------------------------------------------------------
    r29871 | jim | 2005-04-04 07:04:33 -0400 (Mon, 04 Apr 2005) | 6 lines
    Changed paths:
       M /Zope/branches/jim-fix-zclasses/lib/python/ZODB/Connection.py

    Changed sub-transaction abort code to not invalidate created objects.
    There's really no point, because created objects will be unreachable
    after invalidating old objects.  Also, if a created object is
    non-ghostifiable, it will try to load it's state again, and then
    either it would fail or it would load non-committed state.

    ------------------------------------------------------------------------
    r29870 | jim | 2005-04-04 07:04:27 -0400 (Mon, 04 Apr 2005) | 3 lines
    Changed paths:
       A /Zope/branches/jim-fix-zclasses/lib/python/ZODB/persistentclass.py
       A /Zope/branches/jim-fix-zclasses/lib/python/ZODB/persistentclass.txt
       A /Zope/branches/jim-fix-zclasses/lib/python/ZODB/tests/testpersistentclass.py

    Added ZClass-independent test of (and possible base class for)
    persistent-class support machinery.

    ------------------------------------------------------------------------
    r29867 | jim | 2005-04-04 07:03:48 -0400 (Mon, 04 Apr 2005) | 2 lines
    Changed paths:
       M /Zope/branches/jim-fix-zclasses/lib/python/transaction/_manager.py

    Added missing arguments to commit and abort.

    ------------------------------------------------------------------------
    r29774 | jim | 2005-04-01 06:24:27 -0500 (Fri, 01 Apr 2005) | 3 lines
    Changed paths:
       M /Zope/branches/jim-fix-zclasses/lib/python/transaction/__init__.py

    Changed to use methods of a threaded manager directly, rather than
    through wrapper functions.

    ------------------------------------------------------------------------
    r29773 | jim | 2005-04-01 06:24:25 -0500 (Fri, 01 Apr 2005) | 3 lines
    Changed paths:
       M /Zope/branches/jim-fix-zclasses/lib/python/transaction/_manager.py

    Added commit and abort methods to transaction managers.
    This makes direcr use of managers simpler.

    ------------------------------------------------------------------------
    r29101 | jim | 2005-02-10 07:44:52 -0500 (Thu, 10 Feb 2005) | 2 lines
    Changed paths:
       M /Zope/branches/jim-fix-zclasses/lib/python/ZODB/Connection.py

    Removed an unused attribute.

    ------------------------------------------------------------------------
    r29100 | jim | 2005-02-10 07:42:35 -0500 (Thu, 10 Feb 2005) | 3 lines
    Changed paths:
       M /Zope/branches/jim-fix-zclasses/lib/python/transaction/_transaction.py

    Changed an XXX comment to an ordinary comment, explaining the relevent
    issue.

    ------------------------------------------------------------------------
    r29099 | jim | 2005-02-10 07:41:58 -0500 (Thu, 10 Feb 2005) | 3 lines
    Changed paths:
       M /Zope/branches/jim-fix-zclasses/lib/python/ZODB/Connection.py

    Changed some XXX comments to ordinary comments explaining the relevent
    issues.

    ------------------------------------------------------------------------
    r29071 | jim | 2005-02-07 07:36:05 -0500 (Mon, 07 Feb 2005) | 3 lines
    Changed paths:
       M /Zope/branches/jim-fix-zclasses/lib/python/transaction/_transaction.py

    Added a sanity check to avoid registration of objects without a
    manager. (Perhaps this should be an assert.)

    ------------------------------------------------------------------------
    r29067 | jim | 2005-02-07 07:35:56 -0500 (Mon, 07 Feb 2005) | 15 lines
    Changed paths:
       M /Zope/branches/jim-fix-zclasses/lib/python/ZODB/Connection.py

    Changed the strategy for handling invalidation of classes.
    No-longer use setklassstate.  Instead, just call _p_invalidate, as
    with any other object.  This changes didn't break any tests, so I
    assume that this was untested. :(

    Changed the strategy for invalidating objects.  Non-ghostifiable
    objects will load their state when they are invalidated.  We have to
    worry about other invalidations that come in while this is happening.
    See the comment in _flush_invalidations.

    We need to force all invalidations to take this into account, by going
    through _flush_invalidations.  I haven't done this yet, but I have
    left some XXX comments in places where it needs to be done to remind
    myself that this needs to be done.

    ------------------------------------------------------------------------
    r29066 | jim | 2005-02-07 07:35:54 -0500 (Mon, 07 Feb 2005) | 8 lines
    Changed paths:
       M /Zope/branches/jim-fix-zclasses/lib/python/persistent/cPickleCache.c

    Changed the strategy for handling invalidation of classes.
    No-longer use setklassstate.  Instead, just call _p_invalidate, as
    with any other object.  This changes didn't break any tests, so I
    assume that this was untested. :(

    Change invalidation to not swallow errors. (Swallowing errors here was a
    travesty!)

Port from 3.3 branch.

Port from Zope 2.7 branch.

The various flavors of the ``check2ZODBThreads`` and ``check7ZODBThreads``
tests are much less likely to suffer sproadic failures now.

modifiedInVersion:  remove set_trace() added during ZODB sprint.

checkOldStyleRoot:  Can't work in Zope3.

The Persistence module no longer exists in Zope3's idea of what
ZODB is, but this test relies on it.  Make it tolerate the
absence of Persistence.

This is news about ZEO protocol changes.  The promised
update of README.txt will have to be rewritten from
scratch for 3.4 (the new text in 3.3's README.txt isn't
quite right for the 3.4 release).

Rewrite ZEO protocol negotiation.

3.3 should have bumped the ZEO protocol number (new methods were
added for MVCC support), but didn't.  Untangling this is a mess.

the most common cases (commit() and abort()) -- it's not really
obvious that shortcut spelling exist for those.

Merge checkin made from a wrong project.

Believe it or not, ZopeUndo is ZEO code, not Zope code.

r29726 | slinkp | 2005-03-30 02:11:22 -0500 (Wed, 30 Mar 2005) | 3 lines
Changed paths:
   M /Zope/trunk/lib/python/ZopeUndo/Prefix.py
   M /Zope/trunk/lib/python/ZopeUndo/tests/testPrefix.py

Merged slinkp_1726_zopeundo: avoid showing undo transactions in
the wrong folder.

This always slept for 3 seconds, waiting for the storage to
disconnect.  This loses on two counts:

1. Since the timeout is set to 1 second, it typically sleeps
   longer than necessary.

2. Since there's no predicting thread and process scheduling,
   3 seconds isn't always long enough, and rare failures have
   been reported against this test.

Instead we do a polling loop now.  This typically succeeds in
a little more than a second, speeding the normal case.  The
loop will continue trying for up to a minute if not.