tagged

HISTORY.txt

-What's new in ZODB 3.8.0
-========================
-
-General
--------
-
-- The ZODB Storage APIs have been documented and cleaned up.
-
-- ZODB versions are now officially deprecated and support for them
-  will be removed in ZODB 3.9.  (They have been widely recognized as
-  deprecated for quite a while.)
-
-- Changed the automatic garbage collection when opening a connection to only
-  apply the garbage collections on those connections in the pool that are
-  closed. (This fixed issue 113932.)
-
-ZEO
----
-
-- (3.8a1) ZEO's strategoes for avoiding client cache verification were
-  improved in the case that servers are restarted.  Before, if
-  transactions were committed after the restart, clients that were up
-  to date or nearly up to date at the time of the restart and then
-  connected had to verify their caches.  Now, it is far more likely
-  that a client that reconnects soon after a server restart won't have
-  to verify its cache.
-
-- (3.8a1) Fixed a serious bug that could cause clients that disconnect from and
-  reconnect to a server to get bad invalidation data if the server
-  serves multiple storages with active writes.
-
-- (3.8a1) It is now theoretically possible to use a ClientStorage in a storage
-  server.  This might make it possible to offload read load from a
-  storage server at the cost of increasing write latency.  This should
-  increase write throughput by offloading reads from the final storage
-  server.  This feature is somewhat experimental.  It has tests, but
-  hasn't been used in production.
-
-Transactions
-------------
-
-- (3.8a1) Add a doom() and isDoomed() interface to the transaction module.
-
-  First step towards the resolution of
-  http://www.zope.org/Collectors/Zope3-dev/655
-
-  A doomed transaction behaves exactly the same way as an active transaction
-  but raises an error on any attempt to commit it, thus forcing an abort.
-
-  Doom is useful in places where abort is unsafe and an exception cannot be
-  raised.  This occurs when the programmer wants the code following the doom to
-  run but not commit. It is unsafe to abort in these circumstances as a
-  following get() may implicitly open a new transaction.
-
-  Any attempt to commit a doomed transaction will raise a DoomedTransaction
-  exception.
-
-- (3.8a1) Clean up the ZODB imports in transaction.
-
-  Clean up weird import dance with ZODB. This is unnecessary since the
-  transaction module stopped being imported in ZODB/__init__.py in rev 39622.
-
-- (3.8a1) Support for subtransactions has been removed in favor of
-  save points.
-
-Blobs
------
-
-- (3.8b1) Updated the Blob implementation in a number of ways.  Some
-  of these are backward incompatible with 3.8a1:
-
-  o The Blob class now lives in ZODB.blob
-
-  o The blob openDetached method has been replaced by the committed method.
-
-- (3.8a1) Added new blob feature. See the ZODB/Blobs directory for
-  documentation.
-
-  ZODB now handles (reasonably) large binary objects efficiently. Useful to
-  use from a few kilobytes to at least multiple hundred megabytes.
-
-BTrees
-------
-
-- (3.8a1) Added support for 64-bit integer BTrees as separate types.  
-
-  (For now, we're retaining compile-time support for making the regular
-  integer BTrees 64-bit.)
-
-- (3.8a1) Normalize names in modules so that BTrees, Buckets, Sets, and
-  TreeSets can all be accessed with those names in the modules (e.g.,
-  BTrees.IOBTree.BTree).  This is in addition to the older names (e.g.,
-  BTrees.IOBTree.IOBTree).  This allows easier drop-in replacement, which
-  can especially be simplify code for packages that want to support both
-  32-bit and 64-bit BTrees.
-
-- (3.8a1) Describe the interfaces for each module and actually declare
-  the interfaces for each.
-
-- (3.8a1) Fix module references so klass.__module__ points to the Python
-  wrapper module, not the C extension.
-
-- (3.8a1) introduce module families, to group all 32-bit and all 64-bit
-  modules.
-
-
 What's new in ZODB3 3.7.0
 ==========================
 Release date: 2007-04-20

NEWS.txt

-What's new in ZODB 3.9.0
+What's new on ZODB 3.8.0
 ========================
 
 General
 -------
 
-- (3.9.0a1) Document conflict resolution (see ZODB/ConflictResolution.txt).
+- The ZODB Storage APIs have been documented and cleaned up.
 
-- (3.9.0a1) Bugfix the situation in which comparing persistent objects (for
-  instance, as members in BTree set or keys of BTree) might cause data
-  inconsistency during conflict resolution.
-
-- (3.9.0a1) Support multidatabase references in conflict resolution.
+- ZODB versions are now officially deprecated and support for them
+  will be removed in ZODB 3.9.  (They have been widely recognized as
+  deprecated for quite a while.)
 
-- (3.9.0a1) Make it possible to examine oid and (in some situations) database
-  name of persistent object references during conflict resolution.
+- Changed the automatic garbage collection when opening a connection to only
+  apply the garbage collections on those connections in the pool that are
+  closed. (This fixed issue 113932.)
 
-- (3.9.0a1) Moved 'transaction' module out of ZODB.
-  ZODB depends upon this module, but it must be installed separately.
+- (3.8.0b3) Document conflict resolution (see ZODB/ConflictResolution.txt).
 
-- (3.9.0a1) ZODB installation now requires setuptools.
+- (3.8.0b3) Bugfix the situation in which comparing persistent objects (for
+  instance, as members in BTree set or keys of BTree) might cause data
+  inconsistency during conflict resolution.
 
-- (3.9.0a1) Added `offset` information to output of `fstail`
-  script. Added test harness for this script.
+- (3.8.0b3) Support multidatabase references in conflict resolution.
 
-- (3.9.0a1) Fixed bug 153316: persistent and BTrees were using `int`
-  for memory sizes which caused errors on x86_64 Intel Xeon machines
-  (using 64-bit Linux).
+- (3.8.0b3) Make it possible to examine oid and (in some situations) database
+  name of persistent object references during conflict resolution.
 
-- (3.9.0a1) Removed version support from connections and DB.  Versions
-  are still in the storages; this is an incremental step.
+- (3.8.0b3) Added missing data attribute for conflict errors.
 
-- (3.9.0a1) Added support for read-only, historical connections based
-  on datetimes or serials (TIDs).  See
-  src/ZODB/historical_connections.txt.
-  
-- (3.9.0a1) Fixed small bug that the Connection.isReadOnly method didn't
-  work after a savepoint.
+- (3.8.0b5) Fixed bug 153316: persistent and BTrees gave errors on x86_64
+  Intel XEON platforms.
 
 ZEO
 ---
 
-- (3.9.0a1) Bug #98275: Made ZEO cache more tolerant when invalidating current
+- (3.8.0b6) Bug #98275: Made ZEO cache more tolerant when invalidating current
   versions of objects.
 
-- (3.9.0a1) Fixed a serious bug that could cause client I/O to stop
+- (3.8.0b4, 3.8.0b5) Fixed a serious bug that could cause client I/O to stop
   (hang). This was accomonied by a critical log message along the
   lines of: "RuntimeError: dictionary changed size during iteration".
+  (In b4, the bug was only partially fixed.)
+
+- (3.8a1) ZEO's strategoes for avoiding client cache verification were
+  improved in the case that servers are restarted.  Before, if
+  transactions were committed after the restart, clients that were up
+  to date or nearly up to date at the time of the restart and then
+  connected had to verify their caches.  Now, it is far more likely
+  that a client that reconnects soon after a server restart won't have
+  to verify its cache.
+
+- (3.8a1) Fixed a serious bug that could cause clients that disconnect from and
+  reconnect to a server to get bad invalidation data if the server
+  serves multiple storages with active writes.
+
+- (3.8a1) It is now theoretically possible to use a ClientStorage in a storage
+  server.  This might make it possible to offload read load from a
+  storage server at the cost of increasing write latency.  This should
+  increase write throughput by offloading reads from the final storage
+  server.  This feature is somewhat experimental.  It has tests, but
+  hasn't been used in production.
 
 Transactions
 ------------
 
-- (3.9.0a1) 'transaction' module is not included in ZODB anymore.  It
-  is now just a ZODB dependency (via setuptools declarations).
+- (3.8a1) Add a doom() and isDoomed() interface to the transaction module.
+
+  First step towards the resolution of
+  http://www.zope.org/Collectors/Zope3-dev/655
+
+  A doomed transaction behaves exactly the same way as an active transaction
+  but raises an error on any attempt to commit it, thus forcing an abort.
+
+  Doom is useful in places where abort is unsafe and an exception cannot be
+  raised.  This occurs when the programmer wants the code following the doom to
+  run but not commit. It is unsafe to abort in these circumstances as a
+  following get() may implicitly open a new transaction.
+
+  Any attempt to commit a doomed transaction will raise a DoomedTransaction
+  exception.
+
+- (3.8a1) Clean up the ZODB imports in transaction.
+
+  Clean up weird import dance with ZODB. This is unnecessary since the
+  transaction module stopped being imported in ZODB/__init__.py in rev 39622.
+
+- (3.8a1) Support for subtransactions has been removed in favor of
+  save points.
 
 Blobs
 -----
 
-- (3.9.0a1) Fixed bug #127182: Blobs were subclassable which was not desired.
+- (3.8b5) Fixed bug #130459: Packing was broken by uncommitted blob data.
+
+- (3.8b4) Fixed bug #127182: Blobs were subclassable which was not desired.
 
-- (3.9.0a1) Fixed bug #126007: tpc_abort had untested code path that was
+- (3.8b3) Fixed bug #126007: tpc_abort had untested code path that was
   broken.
 
-- (3.9.0a1) Fixed bug #129921: getSize() function in BlobStorage could not
+- (3.8b3) Fixed bug #129921: getSize() function in BlobStorage could not
   deal with garbage files
 
-- (3.9.0a1) Fixed bug in which MVCC would not work for blobs.
+- (3.8b1) Updated the Blob implementation in a number of ways.  Some
+  of these are backward incompatible with 3.8a1:
+
+  o The Blob class now lives in ZODB.blob
+
+  o The blob openDetached method has been replaced by the committed method.
+
+- (3.8a1) Added new blob feature. See the ZODB/Blobs directory for
+  documentation.
+
+  ZODB now handles (reasonably) large binary objects efficiently. Useful to
+  use from a few kilobytes to at least multiple hundred megabytes.
 
 BTrees
 ------
 
-- 
+- (3.8a1) Added support for 64-bit integer BTrees as separate types.  
+
+  (For now, we're retaining compile-time support for making the regular
+  integer BTrees 64-bit.)
+
+- (3.8a1) Normalize names in modules so that BTrees, Buckets, Sets, and
+  TreeSets can all be accessed with those names in the modules (e.g.,
+  BTrees.IOBTree.BTree).  This is in addition to the older names (e.g.,
+  BTrees.IOBTree.IOBTree).  This allows easier drop-in replacement, which
+  can especially be simplify code for packages that want to support both
+  32-bit and 64-bit BTrees.
+
+- (3.8a1) Describe the interfaces for each module and actually declare
+  the interfaces for each.
+
+- (3.8a1) Fix module references so klass.__module__ points to the Python
+  wrapper module, not the C extension.
+
+- (3.8a1) introduce module families, to group all 32-bit and all 64-bit
+  modules.

buildout.cfg

 [buildout]
-develop = . transaction
+develop = .
 parts = test scripts
 find-links = http://download.zope.org/distribution/
 

doc/HOWTO-Blobs-NFS.txt

@@ -72,11 +72,11 @@ configuration should look like this::
         <filestorage>
             path $INSTANCE/var/Data.fs
         <filestorage>
-        blob-dir $SERVER/blobs
+        bob-dir $SERVER/blobs
     </blobstorage>
 
 (Remember to manually replace $SERVER and $CLIENT with the exported directory
-as accessible by either the ZEO server or the ZEO client.)
+as accessible bei either the ZEO server or the ZEO client.)
 
 Conclusion
 ----------

setup.py

@@ -20,7 +20,7 @@ to application logic.  ZODB includes features such as a plugable storage
 interface, rich transaction support, and undo.
 """
 
-VERSION = "3.8.0c1"
+VERSION = "3.8.0b6"
 
 # The (non-obvious!) choices for the Trove Development Status line:
 # Development Status :: 5 - Production/Stable
@@ -28,7 +28,6 @@ VERSION = "3.8.0c1"
 # Development Status :: 3 - Alpha
 
 classifiers = """\
-Development Status :: 4 - Beta
 Intended Audience :: Developers
 License :: OSI Approved :: Zope Public License
 Programming Language :: Python
@@ -38,9 +37,26 @@ Operating System :: Microsoft :: Windows
 Operating System :: Unix
 """
 
-from setuptools import setup
-
-entry_points = """
+try:
+    from setuptools import setup
+except ImportError:
+    from distutils.core import setup
+    extra = dict(
+        scripts = ["src/ZODB/scripts/fsdump.py",
+                   "src/ZODB/scripts/fsoids.py",
+                   "src/ZODB/scripts/fsrefs.py",
+                   "src/ZODB/scripts/fstail.py",
+                   "src/ZODB/scripts/fstest.py",
+                   "src/ZODB/scripts/repozo.py",
+                   "src/ZEO/scripts/zeopack.py",
+                   "src/ZEO/scripts/runzeo.py",
+                   "src/ZEO/scripts/zeopasswd.py",
+                   "src/ZEO/scripts/mkzeoinst.py",
+                   "src/ZEO/scripts/zeoctl.py",
+                   ],
+        )
+else:
+    entry_points = """
     [console_scripts]
     fsdump = ZODB.FileStorage.fsdump:main
     fsoids = ZODB.scripts.fsoids:main
@@ -53,8 +69,19 @@ entry_points = """
     mkzeoinst = ZEO.mkzeoinst:main
     zeoctl = ZEO.zeoctl:main
     """
-
-scripts = []
+    extra = dict(
+        install_requires = [
+            'zope.interface',
+            'zope.proxy',
+            'zope.testing',
+            'ZConfig',
+            'zdaemon',
+            ],
+        zip_safe = False,
+        entry_points = entry_points,
+        include_package_data = True,
+        )
+    scripts = []
 
 import glob
 import os
@@ -149,6 +176,7 @@ packages = ["BTrees", "BTrees.tests",
             "ZODB", "ZODB.FileStorage", "ZODB.tests",
                     "ZODB.scripts",
             "persistent", "persistent.tests",
+            "transaction", "transaction.tests",
             "ThreadedAsync",
             "ZopeUndo", "ZopeUndo.tests",
             ]
@@ -159,6 +187,8 @@ def copy_other_files(cmd, outputbase):
     extensions = ["*.conf", "*.xml", "*.txt", "*.sh"]
     directories = [
         "BTrees",
+        "transaction",
+        "transaction/tests",
         "persistent/tests",
         "ZEO",
         "ZEO/scripts",
@@ -210,24 +240,9 @@ class MyDistribution(Distribution):
         self.cmdclass['build_py'] = MyPyBuilder
         self.cmdclass['install_lib'] = MyLibInstaller
 
-def alltests():
-    # use the zope.testing testrunner machinery to find all the
-    # test suites we've put under ourselves
-    from zope.testing.testrunner import get_options
-    from zope.testing.testrunner import find_suites
-    from zope.testing.testrunner import configure_logging
-    configure_logging()
-    from unittest import TestSuite
-    here = os.path.abspath(os.path.dirname(sys.argv[0]))
-    args = sys.argv[:]
-    src = os.path.join(here, 'src')
-    defaults = ['--test-path', src]
-    options = get_options(args, defaults)
-    suites = list(find_suites(options))
-    return TestSuite(suites)
-
 doclines = __doc__.split("\n")
 
+
 setup(name="ZODB3",
       version=VERSION,
       maintainer="Zope Corporation",
@@ -244,35 +259,4 @@ setup(name="ZODB3",
       classifiers = filter(None, classifiers.split("\n")),
       long_description = "\n".join(doclines[2:]),
       distclass = MyDistribution,
-      test_suite="__main__.alltests", # to support "setup.py test"
-      tests_require = [
-        'zope.interface',
-        'zope.proxy',
-        'zope.testing',
-        'transaction',
-        'zdaemon',
-        ],
-      install_requires = [
-        'zope.interface',
-        'zope.proxy',
-        'zope.testing',
-        'ZConfig',
-        'zdaemon',
-        'transaction',
-        ],
-      zip_safe = False,
-      entry_points = """
-      [console_scripts]
-      fsdump = ZODB.FileStorage.fsdump:main
-      fsoids = ZODB.scripts.fsoids:main
-      fsrefs = ZODB.scripts.fsrefs:main
-      fstail = ZODB.scripts.fstail:Main
-      repozo = ZODB.scripts.repozo:main
-      zeopack = ZEO.scripts.zeopack:main
-      runzeo = ZEO.runzeo:main
-      zeopasswd = ZEO.zeopasswd:main
-      mkzeoinst = ZEO.mkzeoinst:main
-      zeoctl = ZEO.zeoctl:main
-      """,
-      include_package_data = True,
-      )
+      **extra)

src/ZEO/tests/testConnection.py

@@ -46,7 +46,7 @@ class MappingStorageConfig:
     def getConfig(self, path, create, read_only):
         return """<mappingstorage 1/>"""
 
-
+
 class FileStorageConnectionTests(
     FileStorageConfig,
     ConnectionTests.ConnectionTests,

src/ZEO/tests/test_cache.py

@@ -14,16 +14,11 @@
 """Basic unit tests for a multi-version client cache."""
 
 import os
-import random
 import tempfile
 import unittest
-import doctest
-import string
-import sys
 
 import ZEO.cache
-from ZODB.utils import p64, repr_to_oid
-
+from ZODB.utils import p64
 
 n1 = p64(1)
 n2 = p64(2)
@@ -31,60 +26,6 @@ n3 = p64(3)
 n4 = p64(4)
 n5 = p64(5)
 
-
-def hexprint(file):
-    file.seek(0)
-    data = file.read()
-    offset = 0
-    while data:
-        line, data = data[:16], data[16:]
-        printable = ""
-        hex = ""
-        for character in line:
-            if character in string.printable and not ord(character) in [12,13,9]:
-                printable += character
-            else:
-                printable += '.'
-            hex += character.encode('hex') + ' '
-        hex = hex[:24] + ' ' + hex[24:]
-        hex = hex.ljust(49)
-        printable = printable.ljust(16)
-        print '%08x  %s |%s|' % (offset, hex, printable)
-        offset += 16
-
-
-class ClientCacheDummy(object):
-
-    def __init__(self):
-        self.objects = {}
-
-    def _evicted(self, o):
-        if o.key in self.objects:
-            del self.objects[o.key]
-
-
-def oid(o):
-    repr = '%016x' % o
-    return repr_to_oid(repr)
-tid = oid
-
-
-class FileCacheFuzzing(unittest.TestCase):
-
-    def testFileCacheFuzzing(self):
-        cache_dummy = ClientCacheDummy()
-        fc = ZEO.cache.FileCache(maxsize=5000, fpath=None,
-                                      parent=cache_dummy)
-        for i in xrange(10000):
-            size = random.randint(0,5500)
-            obj = ZEO.cache.Object(key=(oid(i), oid(1)), version='',
-                                   data='*'*size, start_tid=oid(1),
-                                   end_tid=None)
-            fc.add(obj)
-        hexprint(fc.f)
-        fc.close()
-
-
 class CacheTests(unittest.TestCase):
 
     def setUp(self):
@@ -185,76 +126,6 @@ class CacheTests(unittest.TestCase):
         # TODO:  Need to make sure eviction of non-current data
         # and of version data are handled correctly.
 
-    def _run_fuzzing(self):
-        current_tid = 1
-        current_oid = 1
-        def log(*args):
-            #print args
-            pass
-        cache = self.fuzzy_cache
-        objects = self.fuzzy_cache_client.objects
-        for operation in xrange(10000):
-            op = random.choice(['add', 'access', 'remove', 'update', 'settid'])
-            if not objects:
-                op = 'add'
-            log(op)
-            if op == 'add':
-                current_oid += 1
-                key = (oid(current_oid), tid(current_tid))
-                object = ZEO.cache.Object(
-                    key=key, version='', data='*'*random.randint(1,60*1024),
-                    start_tid=tid(current_tid), end_tid=None)
-                assert key not in objects
-                log(key, len(object.data), current_tid)
-                cache.add(object)
-                if (object.size + ZEO.cache.OBJECT_HEADER_SIZE >
-                    cache.maxsize - ZEO.cache.ZEC3_HEADER_SIZE):
-                    assert key not in cache
-                else:
-                    objects[key] = object
-                    assert key in cache, key
-            elif op == 'access':
-                key = random.choice(objects.keys())
-                log(key)
-                object = objects[key]
-                found = cache.access(key)
-                assert object.data == found.data
-                assert object.key == found.key
-                assert object.size == found.size == (len(object.data)+object.TOTAL_FIXED_SIZE)
-            elif op == 'remove':
-                key = random.choice(objects.keys())
-                log(key)
-                cache.remove(key)
-                assert key not in cache
-                assert key not in objects
-            elif op == 'update':
-                key = random.choice(objects.keys())
-                object = objects[key]
-                log(key, object.key)
-                if not object.end_tid:
-                    object.end_tid = tid(current_tid)
-                    log(key, current_tid)
-                    cache.update(object)
-            elif op == 'settid':
-                current_tid += 1
-                log(current_tid)
-                cache.settid(tid(current_tid))
-        cache.close()
-
-    def testFuzzing(self):
-        random.seed()
-        seed = random.randint(0, sys.maxint)
-        random.seed(seed)
-        self.fuzzy_cache_client = ClientCacheDummy()
-        self.fuzzy_cache = ZEO.cache.FileCache(
-            random.randint(100, 50*1024), None, self.fuzzy_cache_client)
-        try:
-            self._run_fuzzing()
-        except:
-            print "Error in fuzzing with seed", seed
-            hexprint(self.fuzzy_cache.f)
-            raise
-
     def testSerialization(self):
         self.cache.store(n1, "", n2, None, "data for n1")
         self.cache.store(n2, "version", n2, None, "version data for n2")
@@ -281,10 +152,5 @@ class CacheTests(unittest.TestCase):
         eq(copy.current, self.cache.current)
         eq(copy.noncurrent, self.cache.noncurrent)
 
-
 def test_suite():
-    suite = unittest.TestSuite()
-    suite.addTest(unittest.makeSuite(CacheTests))
-    suite.addTest(unittest.makeSuite(FileCacheFuzzing))
-    suite.addTest(doctest.DocFileSuite('filecache.txt'))
-    return suite
+    return unittest.makeSuite(CacheTests)

src/ZEO/tests/zeo-fan-out.test

@@ -4,7 +4,7 @@ ZEO Fan Out
 We should be able to set up ZEO servers with ZEO clients.  Let's see
 if we can make it work.
 
-We'll use some helper functions.  The first is a helper that starts
+We'll use some helper functions.  The first is a helpter that starts
 ZEO servers for us and another one that picks ports.
 
 We'll start the first server:
@@ -16,7 +16,7 @@ We'll start the first server:
     ...    '<filestorage 1>\n  path fs\n</filestorage>\n', zconf0, port0)
 
 
-Then we'll start 2 others that use this one:
+Then we''ll start 2 others that use this one:
 
     >>> port1 = ZEO.tests.testZEO.get_port()
     >>> zconf1 = ZEO.tests.forker.ZEOConfig(('', port1))

src/ZEO/zrpc/connection.py

@@ -82,7 +82,10 @@ def client_loop():
                     continue
 
             if not (r or w or e):
-                for obj in client_map.itervalues():
+                # The line intentionally doesn't use iterators. Other
+                # threads can close dispatchers, causeing the socket
+                # map to shrink.
+                for obj in client_map.values():
                     if isinstance(obj, Connection):
                         # Send a heartbeat message as a reply to a
                         # non-existent message id.

src/ZODB/ExportImport.py

@@ -47,7 +47,7 @@ class ExportImport:
                 continue
             done_oids[oid] = True
             try:
-                p, serial = load(oid, '')
+                p, serial = load(oid, self._version)
             except:
                 logger.debug("broken reference for oid %s", repr(oid),
                              exc_info=True)
@@ -55,16 +55,16 @@ class ExportImport:
                 referencesf(p, oids)
                 f.writelines([oid, p64(len(p)), p])
 
-                if supports_blobs:
-                    if not isinstance(self._reader.getGhost(p), Blob):
-                        continue # not a blob
-                    
-                    blobfilename = self._storage.loadBlob(oid, serial)
-                    f.write(blob_begin_marker)
-                    f.write(p64(os.stat(blobfilename).st_size))
-                    blobdata = open(blobfilename, "rb")
-                    cp(blobdata, f)
-                    blobdata.close()
+            if supports_blobs:
+                if not isinstance(self._reader.getGhost(p), Blob):
+                    continue # not a blob
+                
+                blobfilename = self._storage.loadBlob(oid, serial)
+                f.write(blob_begin_marker)
+                f.write(p64(os.stat(blobfilename).st_size))
+                blobdata = open(blobfilename, "rb")
+                cp(blobdata, f)
+                blobdata.close()
             
         f.write(export_end_marker)
         return f
@@ -127,6 +127,8 @@ class ExportImport:
 
             return Ghost(oid)
 
+        version = self._version
+
         while 1:
             header = f.read(16)
             if header == export_end_marker:
@@ -178,9 +180,9 @@ class ExportImport:
 
             if blob_filename is not None:
                 self._storage.storeBlob(oid, None, data, blob_filename, 
-                                        '', transaction)
+                                        version, transaction)
             else:
-                self._storage.store(oid, None, data, '', transaction)
+                self._storage.store(oid, None, data, version, transaction)
 
 
 export_end_marker = '\377'*16

src/ZODB/POSException.py

@@ -17,10 +17,6 @@ $Id$"""
 
 from ZODB.utils import oid_repr, readable_tid_repr
 
-# BBB: We moved the two transactions to the transaction package
-from transaction.interfaces import TransactionError, TransactionFailedError
-
-
 def _fmt_undo(oid, reason):
     s = reason and (": %s" % reason) or ""
     return "Undo error %s%s" % (oid_repr(oid), s)
@@ -48,6 +44,18 @@ class POSKeyError(KeyError, POSError):
     def __str__(self):
         return oid_repr(self.args[0])
 
+class TransactionError(POSError):
+    """An error occurred due to normal transaction processing."""
+
+class TransactionFailedError(POSError):
+    """Cannot perform an operation on a transaction that previously failed.
+
+    An attempt was made to commit a transaction, or to join a transaction,
+    but this transaction previously raised an exception during an attempt
+    to commit it.  The transaction must be explicitly aborted, either by
+    invoking abort() on the transaction, or begin() on its transaction
+    manager.
+    """
 
 class ConflictError(TransactionError):
     """Two transactions tried to modify the same object at once.
@@ -98,12 +106,11 @@ class ConflictError(TransactionError):
             # avoid circular import chain
             from ZODB.utils import get_pickle_metadata
             self.class_name = "%s.%s" % get_pickle_metadata(data)
-##        else:
-##            if message != "data read conflict error":
-##                raise RuntimeError
 
         self.serials = serials
 
+        self.data = data
+
     def __str__(self):
         extras = []
         if self.oid:
@@ -234,10 +241,6 @@ class DanglingReferenceError(TransactionError):
         return "from %s to %s" % (oid_repr(self.referer),
                                   oid_repr(self.missing))
 
-
-############################################################################
-# Only used in storages; versions are no longer supported.
-
 class VersionError(POSError):
     """An error in handling versions occurred."""
 
@@ -250,7 +253,6 @@ class VersionLockError(VersionError, TransactionError):
     An attempt was made to modify an object that has been modified in an
     unsaved version.
     """
-############################################################################
 
 class UndoError(POSError):
     """An attempt was made to undo a non-undoable transaction."""
@@ -297,9 +299,6 @@ class ExportError(POSError):
 class Unsupported(POSError):
     """A feature was used that is not supported by the storage."""
 
-class ReadOnlyHistoryError(POSError):
-    """Unable to add or modify objects in an historical connection."""
-
 class InvalidObjectReference(POSError):
     """An object contains an invalid reference to another object.
 

src/ZODB/blob.py

@@ -490,6 +490,7 @@ class BlobStorage(SpecificationDecoratorBase):
         base_dir = self.fshelper.base_dir
         for oid, oid_path in self.fshelper.listOIDs():
             files = os.listdir(oid_path)
+
             for filename in files:
                 filepath = os.path.join(oid_path, filename)
                 whatever, serial = self.fshelper.splitBlobFilename(filepath)

src/ZODB/component.xml

@@ -180,22 +180,16 @@
         and exceeding twice pool-size connections causes a critical
         message to be logged.
       </description>
-    <key name="historical-pool-size" datatype="integer" default="3"/>
+    <key name="version-pool-size" datatype="integer" default="3"/>
       <description>
-        The expected maximum total number of historical connections
-        simultaneously open.
+        The expected maximum number of connections simultaneously open
+        per version.
       </description>
-    <key name="historical-cache-size" datatype="integer" default="1000"/>
+    <key name="version-cache-size" datatype="integer" default="100"/>
       <description>
-        Target size, in number of objects, of each historical connection's
+        Target size, in number of objects, of each version connection's
         object cache.
       </description>
-    <key name="historical-timeout" datatype="time-interval"
-         default="5m"/>
-      <description>
-        The minimum interval that an unused historical connection should be
-        kept.
-      </description>
     <key name="database-name" default="unnamed"/>
       <description>
         When multidatabases are in use, this is the name given to this

src/ZODB/config.py

@@ -68,6 +68,7 @@ def storageFromURL(url):
 def storageFromConfig(section):
     return section.open()
 
+
 class BaseConfig:
     """Object representing a configured storage or database.
 
@@ -98,9 +99,8 @@ class ZODBDatabase(BaseConfig):
             return ZODB.DB(storage,
                            pool_size=section.pool_size,
                            cache_size=section.cache_size,
-                           historical_pool_size=section.historical_pool_size,
-                           historical_cache_size=section.historical_cache_size,
-                           historical_timeout=section.historical_timeout,
+                           version_pool_size=section.version_pool_size,
+                           version_cache_size=section.version_cache_size,
                            database_name=section.database_name,
                            databases=databases)
         except:

src/ZODB/interfaces.py

@@ -34,10 +34,9 @@ class IConnection(Interface):
     loading objects from that Connection.  Objects loaded by one
     thread should not be used by another thread.
 
-    A Connection can be frozen to a serial--a transaction id, a single point in
-    history-- when it is created. By default, a Connection is not associated
-    with a serial; it uses current data. A Connection frozen to a serial is
-    read-only.
+    A Connection can be associated with a single version when it is
+    created.  By default, a Connection is not associated with a
+    version; it uses non-version data.
 
     Each Connection provides an isolated, consistent view of the
     database, by managing independent copies of objects in the
@@ -102,7 +101,8 @@ class IConnection(Interface):
 
         User Methods:
             root, get, add, close, db, sync, isReadOnly, cacheGC,
-            cacheFullSweep, cacheMinimize
+            cacheFullSweep, cacheMinimize, getVersion,
+            modifiedInVersion
 
         Experimental Methods:
             onCloseCallbacks
@@ -226,6 +226,9 @@ class IConnection(Interface):
         The root is a persistent.mapping.PersistentMapping.
         """
 
+    def getVersion():
+        """Returns the version this connection is attached to."""
+
     # Multi-database support.
 
     connections = Attribute(
@@ -322,7 +325,7 @@ class IStorageDB(Interface):
         there would be so many that it would be inefficient to do so.        
         """
 
-    def invalidate(transaction_id, oids):
+    def invalidate(transaction_id, oids, version=''):
         """Invalidate object ids committed by the given transaction
 
         The oids argument is an iterable of object identifiers.
@@ -353,15 +356,13 @@ class IDatabase(IStorageDB):
         entry.
         """)
 
-    def open(transaction_manager=None, serial=''):
+    def open(version='', transaction_manager=None):
         """Return an IConnection object for use by application code.
 
+        version: the "version" that all changes will be made
+            in, defaults to no version.
         transaction_manager: transaction manager to use.  None means
             use the default transaction manager.
-        serial: the serial (transaction id) of the database to open.
-            An empty string (the default) means to open it to the newest
-            serial. Specifying a serial results in a read-only historical
-            connection.
 
         Note that the connection pool is managed as a stack, to
         increase the likelihood that the connection's stack will
@@ -440,7 +441,7 @@ class IStorage(Interface):
         This is used soley for informational purposes.
         """
 
-    def history(oid, size=1):
+    def history(oid, version, size=1):
         """Return a sequence of history information dictionaries.
 
         Up to size objects (including no objects) may be returned.
@@ -456,6 +457,10 @@ class IStorage(Interface):
         tid
             The transaction identifier of the transaction that
             committed the version.
+        version
+            The version that the revision is in.  If the storage
+            doesn't support versions, then this must be an empty
+            string.
         user_name
             The user identifier, if any (or an empty string) of the
             user on whos behalf the revision was committed.
@@ -486,14 +491,18 @@ class IStorage(Interface):
         This is used soley for informational purposes.
         """
 
-    def load(oid):
-        """Load data for an object id
+    def load(oid, version):
+        """Load data for an object id and version
 
         A data record and serial are returned.  The serial is a
         transaction identifier of the transaction that wrote the data
         record.
 
-        A POSKeyError is raised if there is no record for the object id.
+        A POSKeyError is raised if there is no record for the object
+        id and version.
+
+        Storages that don't support versions must ignore the version
+        argument.
         """
 
     def loadBefore(oid, tid):
@@ -566,7 +575,7 @@ class IStorage(Interface):
         has a reasonable chance of being unique.
         """
 
-    def store(oid, serial, data, transaction):
+    def store(oid, serial, data, version, transaction):
         """Store data for the object id, oid.
 
         Arguments:
@@ -585,6 +594,11 @@ class IStorage(Interface):
         data
             The data record. This is opaque to the storage.
 
+        version
+            The version to store the data is.  If the storage doesn't
+            support versions, this should be an empty string and the
+            storage is allowed to ignore it.
+
         transaction
             A transaction object.  This should match the current
             transaction for the storage, set by tpc_begin.
@@ -693,7 +707,7 @@ class IStorageRestoreable(IStorage):
         #   failed to take into account records after the pack time.
         
 
-    def restore(oid, serial, data, prev_txn, transaction):
+    def restore(oid, serial, data, version, prev_txn, transaction):
         """Write data already committed in a separate database
 
         The restore method is used when copying data from one database
@@ -713,6 +727,9 @@ class IStorageRestoreable(IStorage):
              The record data.  This will be None if the transaction
              undid the creation of the object.
 
+        version
+             The version identifier for the record
+
         prev_txn
              The identifier of a previous transaction that held the
              object data.  The target storage can sometimes use this
@@ -729,6 +746,7 @@ class IStorageRecordInformation(Interface):
     """
 
     oid = Attribute("The object id")
+    version = Attribute("The version")
     data = Attribute("The data record")
 
 class IStorageTransactionInformation(Interface):
@@ -918,7 +936,7 @@ class IBlob(Interface):
 class IBlobStorage(Interface):
     """A storage supporting BLOBs."""
 
-    def storeBlob(oid, oldserial, data, blob, transaction):
+    def storeBlob(oid, oldserial, data, blob, version, transaction):
         """Stores data that has a BLOB attached."""
 
     def loadBlob(oid, serial):

src/ZODB/scripts/fstail.py

@@ -33,8 +33,8 @@ def main(path, ntxn):
         th.read_meta()
         print "%s: hash=%s" % (th.get_timestamp(),
                                binascii.hexlify(hash))
-        print ("user=%r description=%r length=%d offset=%d"
-               % (th.user, th.descr, th.length, th.get_data_offset()))
+        print ("user=%r description=%r length=%d"
+               % (th.user, th.descr, th.length))
         print
         th = th.prev_txn()
         i -= 1

src/ZODB/scripts/fstail.txt

-====================
-The `fstail` utility
-====================
-
-The `fstail` utility shows information for a FileStorage about the last `n`
-transactions:
-
-We have to prepare a FileStorage first:
-
-  >>> from ZODB.FileStorage import FileStorage
-  >>> from ZODB.DB import DB
-  >>> import transaction
-  >>> from tempfile import mktemp
-  >>> storagefile = mktemp()
-  >>> base_storage = FileStorage(storagefile)
-  >>> database = DB(base_storage)
-  >>> connection1 = database.open()
-  >>> root = connection1.root()
-  >>> root['foo'] = 1
-  >>> transaction.commit()
-
-Now lets have a look at the last transactions of this FileStorage:
-
-  >>> from ZODB.scripts.fstail import main
-  >>> main(storagefile, 5)
-  2007-11-10 15:18:48.543001: hash=b16422d09fabdb45d4e4325e4b42d7d6f021d3c3
-  user='' description='' length=138 offset=191
-  <BLANKLINE>
-  2007-11-10 15:18:48.543001: hash=b16422d09fabdb45d4e4325e4b42d7d6f021d3c3
-  user='' description='initial database creation' length=156 offset=52
-  <BLANKLINE>
-
-Now clean up the storage again:
-
-  >>> import os
-  >>> base_storage.close()
-  >>> os.unlink(storagefile)
-  >>> os.unlink(storagefile+'.index')
-  >>> os.unlink(storagefile+'.lock')
-  >>> os.unlink(storagefile+'.tmp')

src/ZODB/scripts/tests.py

@@ -11,21 +11,15 @@
 # FOR A PARTICULAR PURPOSE.
 #
 ##############################################################################
-"""Test harness for scripts.
+"""XXX short summary goes here.
 
 $Id$
 """
 import unittest
-import re
-from zope.testing import doctest, renormalizing
-
-checker = renormalizing.RENormalizing([
-    (re.compile('[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]+'),
-     '2007-11-10 15:18:48.543001'),
-    (re.compile('hash=[0-9a-f]{40}'),
-     'hash=b16422d09fabdb45d4e4325e4b42d7d6f021d3c3')])
+from zope.testing import doctest
 
 def test_suite():
     return unittest.TestSuite((
-        doctest.DocFileSuite('referrers.txt', 'fstail.txt', checker=checker),
+        doctest.DocFileSuite('referrers.txt'),
         ))
+

src/ZODB/serialize.py

@@ -371,7 +371,7 @@ class ObjectWriter:
             return oid
 
         # Note that we never get here for persistent classes.
-        # We'll use direct refs for normal classes.
+        # We'll use driect refs for normal classes.
 
         if database_name:
             return ['m', (database_name, oid, klass)]

src/ZODB/tests/VersionStorage.py

@@ -394,6 +394,153 @@ class VersionStorage:
         self._storage.tpc_finish(t)
         self.assertEqual(oids, [oid])
 
+    def checkPackVersions(self):
+        db = DB(self._storage)
+        cn = db.open(version="testversion")
+        root = cn.root()
+
+        obj = root["obj"] = MinPO("obj")
+        root["obj2"] = MinPO("obj2")
+        txn = transaction.get()
+        txn.note("create 2 objs in version")
+        txn.commit()
+
+        obj.value = "77"
+        txn = transaction.get()
+        txn.note("modify obj in version")
+        txn.commit()
+
+        # undo the modification to generate a mix of backpointers
+        # and versions for pack to chase
+        info = db.undoInfo()
+        db.undo(info[0]["id"])
+        txn = transaction.get()
+        txn.note("undo modification")
+        txn.commit()
+
+        snooze()
+        self._storage.pack(time.time(), referencesf)
+
+        db.commitVersion("testversion")
+        txn = transaction.get()
+        txn.note("commit version")
+        txn.commit()
+
+        cn = db.open()
+        root = cn.root()
+        root["obj"] = "no version"
+
+        txn = transaction.get()
+        txn.note("modify obj")
+        txn.commit()
+
+        self._storage.pack(time.time(), referencesf)
+
+    def checkPackVersionsInPast(self):
+        db = DB(self._storage)
+        cn = db.open(version="testversion")
+        root = cn.root()
+
+        obj = root["obj"] = MinPO("obj")
+        root["obj2"] = MinPO("obj2")
+        txn = transaction.get()
+        txn.note("create 2 objs in version")
+        txn.commit()
+
+        obj.value = "77"
+        txn = transaction.get()
+        txn.note("modify obj in version")
+        txn.commit()
+
+        t0 = time.time()
+        snooze()
+
+        # undo the modification to generate a mix of backpointers
+        # and versions for pack to chase
+        info = db.undoInfo()
+        db.undo(info[0]["id"])
+        txn = transaction.get()
+        txn.note("undo modification")
+        txn.commit()
+
+        self._storage.pack(t0, referencesf)
+
+        db.commitVersion("testversion")
+        txn = transaction.get()
+        txn.note("commit version")
+        txn.commit()
+
+        cn = db.open()
+        root = cn.root()
+        root["obj"] = "no version"
+
+        txn = transaction.get()
+        txn.note("modify obj")
+        txn.commit()
+
+        self._storage.pack(time.time(), referencesf)
+
+    def checkPackVersionReachable(self):
+        db = DB(self._storage)
+        cn = db.open()
+        root = cn.root()
+
+        names = "a", "b", "c"
+
+        for name in names:
+            root[name] = MinPO(name)
+            transaction.commit()
+
+        for name in names:
+            cn2 = db.open(version=name)
+            rt2 = cn2.root()
+            obj = rt2[name]
+            obj.value = MinPO("version")
+            transaction.commit()
+            cn2.close()
+
+        root["d"] = MinPO("d")
+        transaction.commit()
+        snooze()
+
+        self._storage.pack(time.time(), referencesf)
+        cn.sync()
+
+        # make sure all the non-version data is there
+        for name, obj in root.items():
+            self.assertEqual(name, obj.value)
+
+        # make sure all the version-data is there,
+        # and create a new revision in the version
+        for name in names:
+            cn2 = db.open(version=name)
+            rt2 = cn2.root()
+            obj = rt2[name].value
+            self.assertEqual(obj.value, "version")
+            obj.value = "still version"
+            transaction.commit()
+            cn2.close()
+
+        db.abortVersion("b")
+        txn = transaction.get()
+        txn.note("abort version b")
+        txn.commit()
+
+        t = time.time()
+        snooze()
+
+        L = db.undoInfo()
+        db.undo(L[0]["id"])
+        txn = transaction.get()
+        txn.note("undo abort")
+        txn.commit()
+
+        self._storage.pack(t, referencesf)
+
+        cn2 = db.open(version="b")
+        rt2 = cn2.root()
+        self.assertEqual(rt2["b"].value.value, "still version")
+
     def checkLoadBeforeVersion(self):
         eq = self.assertEqual
         oid = self._storage.new_oid()

src/ZODB/tests/blob_basic.txt

@@ -172,4 +172,3 @@ Blobs are not subclassable::
     Traceback (most recent call last):
         ...
     TypeError: Blobs do not support subclassing.
-

src/ZODB/tests/blob_connection.txt

@@ -15,8 +15,7 @@
 Connection support for Blobs tests
 ==================================
 
-Connections handle Blobs specially. To demonstrate that, we first need a Blob
-with some data:
+Connections handle Blobs specially. To demonstrate that, we first need a Blob with some data:
 
     >>> from ZODB.interfaces import IBlob
     >>> from ZODB.blob import Blob
@@ -26,16 +25,13 @@ with some data:
     >>> data.write("I'm a happy Blob.")
     >>> data.close()
 
-We also need a database with a blob supporting storage.  (We're going to use
-FileStorage rather than MappingStorage here because we will want ``loadBefore``
-for one of our examples.)
+We also need a database with a blob supporting storage:
 
-    >>> import ZODB.FileStorage
+    >>> from ZODB.MappingStorage import MappingStorage
     >>> from ZODB.blob import BlobStorage
     >>> from ZODB.DB import DB
     >>> from tempfile import mkdtemp
-    >>> base_storage = ZODB.FileStorage.FileStorage(
-    ...     'BlobTests.fs', create=True)
+    >>> base_storage = MappingStorage("test")
     >>> blob_dir = mkdtemp()
     >>> blob_storage = BlobStorage(blob_dir, base_storage)
     >>> database = DB(blob_storage)
@@ -55,55 +51,31 @@ calling the blob's open method:
     >>> root['anotherblob'] = anotherblob
     >>> nothing = transaction.commit()
 
-Getting stuff out of there works similarly:
+Getting stuff out of there works similar:
 
-    >>> transaction2 = transaction.TransactionManager()
-    >>> connection2 = database.open(transaction_manager=transaction2)
+    >>> connection2 = database.open()
     >>> root = connection2.root()
     >>> blob2 = root['myblob']
     >>> IBlob.providedBy(blob2)
     True
     >>> blob2.open("r").read()
     "I'm a happy Blob."
-    >>> transaction2.abort()
-
-MVCC also works.
-
-    >>> transaction3 = transaction.TransactionManager()
-    >>> connection3 = database.open(transaction_manager=transaction3)
-    >>> f = connection.root()['myblob'].open('w')
-    >>> f.write('I am an ecstatic Blob.')
-    >>> f.close()
-    >>> transaction.commit()
-    >>> connection3.root()['myblob'].open('r').read()
-    "I'm a happy Blob."
-
-    >>> transaction2.abort()
-    >>> transaction3.abort()
-    >>> connection2.close()
-    >>> connection3.close()
 
 You can't put blobs into a database that has uses a Non-Blob-Storage, though:
 
-    >>> from ZODB.MappingStorage import MappingStorage
     >>> no_blob_storage = MappingStorage()
     >>> database2 = DB(no_blob_storage)
-    >>> connection2 = database2.open(transaction_manager=transaction2)
-    >>> root = connection2.root()
+    >>> connection3 = database2.open()
+    >>> root = connection3.root()
     >>> root['myblob'] = Blob()
-    >>> transaction2.commit()        # doctest: +ELLIPSIS
+    >>> transaction.commit()        # doctest: +ELLIPSIS
     Traceback (most recent call last):
         ...
     Unsupported: Storing Blobs in <ZODB.MappingStorage.MappingStorage instance at ...> is not supported.
 
-    >>> transaction2.abort()
-    >>> connection2.close()
-
-After testing this, we don't need the storage directory and databases anymore:
+While we are testing this, we don't need the storage directory and
+databases anymore:
 
     >>> transaction.abort()
-    >>> connection.close()
     >>> database.close()
     >>> database2.close()
-    >>> blob_storage.close()
-    >>> base_storage.cleanup()

src/ZODB/tests/dbopen.txt

@@ -146,7 +146,7 @@ Reaching into the internals, we can see that db's connection pool now has
 two connections available for reuse, and knows about three connections in
 all:
 
-    >>> pool = db.pool
+    >>> pool = db._pools['']
     >>> len(pool.available)
     2
     >>> len(pool.all)
@@ -219,7 +219,7 @@ closed one out of the available connection stack.
     >>> conns = [db.open() for dummy in range(6)]
     >>> len(handler.records)  # 3 warnings for the "excess" connections
     3
-    >>> pool = db.pool
+    >>> pool = db._pools['']
     >>> len(pool.available), len(pool.all)
     (0, 6)
 
@@ -239,12 +239,12 @@ Closing connections adds them to the stack:
 Closing another one will purge the one with MARKER 0 from the stack
 (since it was the first added to the stack):
 
-    >>> [c.MARKER for c in pool.available.values()]
+    >>> [c.MARKER for c in pool.available]
     [0, 1, 2]
     >>> conns[0].close()  # MARKER 3
     >>> len(pool.available), len(pool.all)
     (3, 5)
-    >>> [c.MARKER for c in pool.available.values()]
+    >>> [c.MARKER for c in pool.available]
     [1, 2, 3]
 
 Similarly for the other two:
@@ -252,7 +252,7 @@ Similarly for the other two:
     >>> conns[1].close(); conns[2].close()
     >>> len(pool.available), len(pool.all)
     (3, 3)
-    >>> [c.MARKER for c in pool.available.values()]
+    >>> [c.MARKER for c in pool.available]
     [3, 4, 5]
 
 Reducing the pool size may also purge the oldest closed connections:
@@ -260,7 +260,7 @@ Reducing the pool size may also purge the oldest closed connections:
     >>> db.setPoolSize(2)  # gets rid of MARKER 3
     >>> len(pool.available), len(pool.all)
     (2, 2)
-    >>> [c.MARKER for c in pool.available.values()]
+    >>> [c.MARKER for c in pool.available]
     [4, 5]
 
 Since MARKER 5 is still the last one added to the stack, it will be the
@@ -297,7 +297,7 @@ Now open more connections so that the total exceeds pool_size (2):
 
     >>> conn1 = db.open()
     >>> conn2 = db.open()
-    >>> pool = db.pool
+    >>> pool = db._pools['']
     >>> len(pool.all), len(pool.available)  # all Connections are in use
     (3, 0)
 

src/ZODB/tests/testConnectionSavepoint.py

@@ -138,21 +138,6 @@ Verify all the values are as expected:
     >>> db.close()
 """
 
-def testIsReadonly():
-    """\
-The connection isReadonly method relies on the _storage to have an isReadOnly.
-We simply rely on the underlying storage method.
-
-    >>> import ZODB.tests.util
-    >>> db = ZODB.tests.util.DB()
-    >>> connection = db.open()
-    >>> root = connection.root()
-    >>> root['a'] = 1
-    >>> sp = transaction.savepoint()
-    >>> connection.isReadOnly()
-    False
-"""
-
 def test_suite():
     return unittest.TestSuite((
         doctest.DocFileSuite('testConnectionSavepoint.txt'),

src/ZODB/tests/testDB.py

@@ -14,7 +14,7 @@
 import os
 import time
 import unittest
-import datetime
+import warnings
 
 import transaction
 
@@ -35,6 +35,8 @@ class DBTests(unittest.TestCase):
         self.__path = os.path.abspath('test.fs')
         store = ZODB.FileStorage.FileStorage(self.__path)
         self.db = ZODB.DB(store)
+        warnings.filterwarnings(
+            'ignore', message='Versions are deprecated', module=__name__)
 
     def tearDown(self):
         self.db.close()
@@ -42,8 +44,8 @@ class DBTests(unittest.TestCase):
             if os.path.exists(self.__path+s):
                 os.remove(self.__path+s)
 
-    def dowork(self):
-        c = self.db.open()
+    def dowork(self, version=''):
+        c = self.db.open(version)
         r = c.root()
         o = r[time.time()] = MinPO(0)
         transaction.commit()
@@ -51,16 +53,85 @@ class DBTests(unittest.TestCase):
             o.value = MinPO(i)
             transaction.commit()
             o = o.value
-        serial = o._p_serial
-        root_serial = r._p_serial
         c.close()
-        return serial, root_serial
 
     # make sure the basic methods are callable
 
     def testSets(self):
         self.db.setCacheSize(15)
-        self.db.setHistoricalCacheSize(15)
+        self.db.setVersionCacheSize(15)
+
+    def test_removeVersionPool(self):
+        # Test that we can remove a version pool
+
+        # This is white box because we check some internal data structures
+
+        self.dowork()
+        self.dowork('v2')
+        c1 = self.db.open('v1')
+        c1.close() # return to pool
+        c12 = self.db.open('v1')
+        c12.close() # return to pool
+        self.assert_(c1 is c12) # should be same
+
+        pools = self.db._pools
+
+        self.assertEqual(len(pools), 3)
+        self.assertEqual(nconn(pools), 3)
+
+        self.db.removeVersionPool('v1')
+
+        self.assertEqual(len(pools), 2)
+        self.assertEqual(nconn(pools), 2)
+
+        c12 = self.db.open('v1')
+        c12.close() # return to pool
+        self.assert_(c1 is not c12) # should be different
+
+        self.assertEqual(len(pools), 3)
+        self.assertEqual(nconn(pools), 3)
+
+    def _test_for_leak(self):
+        self.dowork()
+        self.dowork('v2')
+        while 1:
+            c1 = self.db.open('v1')
+            self.db.removeVersionPool('v1')
+            c1.close() # return to pool
+
+    def test_removeVersionPool_while_connection_open(self):
+        # Test that we can remove a version pool
+
+        # This is white box because we check some internal data structures
+
+        self.dowork()
+        self.dowork('v2')
+        c1 = self.db.open('v1')
+        c1.close() # return to pool
+        c12 = self.db.open('v1')
+        self.assert_(c1 is c12) # should be same
+
+        pools = self.db._pools
+
+        self.assertEqual(len(pools), 3)
+        self.assertEqual(nconn(pools), 3)
+
+        self.db.removeVersionPool('v1')
+
+        self.assertEqual(len(pools), 2)
+        self.assertEqual(nconn(pools), 2)
+
+        c12.close() # should leave pools alone
+
+        self.assertEqual(len(pools), 2)
+        self.assertEqual(nconn(pools), 2)
+
+        c12 = self.db.open('v1')
+        c12.close() # return to pool
+        self.assert_(c1 is not c12) # should be different
+
+        self.assertEqual(len(pools), 3)
+        self.assertEqual(nconn(pools), 3)
 
     def test_references(self):
 

src/ZODB/tests/testZODB.py

@@ -136,6 +136,20 @@ class ZODBTests(unittest.TestCase):
     def checkExportImportAborted(self):
         self.checkExportImport(abort_it=True)
 
+    def checkVersionOnly(self):
+        # Make sure the changes to make empty transactions a no-op
+        # still allow things like abortVersion().  This should work
+        # because abortVersion() calls tpc_begin() itself.
+        conn = self._db.open("version")
+        try:
+            r = conn.root()
+            r[1] = 1
+            transaction.commit()
+        finally:
+            conn.close()
+        self._db.abortVersion("version")
+        transaction.commit()
+
     def checkResetCache(self):
         # The cache size after a reset should be 0.  Note that
         # _resetCache is not a public API, but the resetCaches()

src/ZODB/tests/testhistoricalconnections.py → src/ZODB/tests/test_misc.py

 ##############################################################################
 #
-# Copyright (c) 2007 Zope Corporation and Contributors.
+# Copyright (c) 2006 Zope Corporation and Contributors.
 # All Rights Reserved.
 #
 # This software is subject to the provisions of the Zope Public License,
@@ -11,34 +11,27 @@
 # FOR A PARTICULAR PURPOSE.
 #
 ##############################################################################
+"""Misc tests :)
 """
-$Id$
-"""
+
 import unittest
-from zope.testing import doctest, module
+from zope.testing import doctest
 
-def setUp(test):
-    module.setUp(test, 'historical_connections_txt')
+def conflict_error_retains_data_passed():
+    r"""
+    
+ConflictError can be passed a data record which it claims to retain as
+an attribute.
 
-def tearDown(test):
-    test.globs['db'].close()
-    test.globs['db2'].close()
-    test.globs['storage'].close()
-    test.globs['storage'].cleanup()
-    # the DB class masks the module because of __init__ shenanigans
-    DB_module = __import__('ZODB.DB', globals(), locals(), ['chicken'])
-    DB_module.time = test.globs['original_time']
-    module.tearDown(test)
+    >>> import ZODB.POSException
+    >>> 
+    >>> ZODB.POSException.ConflictError(data='cM\nC\n').data
+    'cM\nC\n'
+
+    """
 
 def test_suite():
     return unittest.TestSuite((
-        doctest.DocFileSuite('../historical_connections.txt',
-                             setUp=setUp,
-                             tearDown=tearDown,
-                             optionflags=doctest.INTERPRET_FOOTNOTES,
-                             ),
+        doctest.DocTestSuite(),
         ))
 
-if __name__ == '__main__':
-    unittest.main(defaultTest='test_suite')
-

src/transaction/DEPENDENCIES.cfg

+ZODB
+zope.interface

src/transaction/README.txt

+============
+Transactions
+============
+
+This package contains a generic transaction implementation for Python. It is
+mainly used by the ZODB, though.
+
+Note that the data manager API, ``transaction.interfaces.IDataManager``,
+is syntactically simple, but semantically complex.  The semantics
+were not easy to express in the interface. This could probably use
+more work.  The semantics are presented in detail through examples of
+a sample data manager in ``transaction.tests.test_SampleDataManager``.
+

src/transaction/__init__.py

+############################################################################
+#
+# Copyright (c) 2001, 2002, 2004 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+############################################################################
+"""Exported transaction functions.
+
+$Id$
+"""
+
+from transaction._transaction import Transaction
+from transaction._manager import TransactionManager, ThreadTransactionManager
+
+manager = ThreadTransactionManager()
+get = manager.get
+begin = manager.begin
+commit = manager.commit
+abort = manager.abort
+doom = manager.doom
+isDoomed = manager.isDoomed
+savepoint = manager.savepoint

src/transaction/_manager.py

+############################################################################
+#
+# Copyright (c) 2004 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+############################################################################
+"""A TransactionManager controls transaction boundaries.
+
+It coordinates application code and resource managers, so that they
+are associated with the right transaction.
+"""
+
+import thread
+
+from ZODB.utils import WeakSet, deprecated37
+
+from transaction._transaction import Transaction
+
+# Used for deprecated arguments.  ZODB.utils.DEPRECATED_ARGUMENT was
+# too hard to use here, due to the convoluted import dance across
+# __init__.py files.
+_marker = object()
+
+# We have to remember sets of synch objects, especially Connections.
+# But we don't want mere registration with a transaction manager to
+# keep a synch object alive forever; in particular, it's common
+# practice not to explicitly close Connection objects, and keeping
+# a Connection alive keeps a potentially huge number of other objects
+# alive (e.g., the cache, and everything reachable from it too).
+# Therefore we use "weak sets" internally.
+#
+
+# Call the ISynchronizer newTransaction() method on every element of
+# WeakSet synchs.
+# A transaction manager needs to do this whenever begin() is called.
+# Since it would be good if tm.get() returned the new transaction while
+# newTransaction() is running, calling this has to be delayed until after
+# the transaction manager has done whatever it needs to do to make its
+# get() return the new txn.
+def _new_transaction(txn, synchs):
+    if synchs:
+        synchs.map(lambda s: s.newTransaction(txn))
+
+# Important:  we must always pass a WeakSet (even if empty) to the Transaction
+# constructor:  synchronizers are registered with the TM, but the
+# ISynchronizer xyzCompletion() methods are called by Transactions without
+# consulting the TM, so we need to pass a mutable collection of synchronizers
+# so that Transactions "see" synchronizers that get registered after the
+# Transaction object is constructed.
+
+class TransactionManager(object):
+
+    def __init__(self):
+        self._txn = None
+        self._synchs = WeakSet()
+
+    def begin(self):
+        if self._txn is not None:
+            self._txn.abort()
+        txn = self._txn = Transaction(self._synchs, self)
+        _new_transaction(txn, self._synchs)
+        return txn
+
+    def get(self):
+        if self._txn is None:
+            self._txn = Transaction(self._synchs, self)
+        return self._txn
+
+    def free(self, txn):
+        assert txn is self._txn
+        self._txn = None
+
+    def registerSynch(self, synch):
+        self._synchs.add(synch)
+
+    def unregisterSynch(self, synch):
+        self._synchs.remove(synch)
+
+    def isDoomed(self):
+        return self.get().isDoomed()
+
+    def doom(self):
+        return self.get().doom()
+
+    def commit(self):
+        return self.get().commit()
+
+    def abort(self):
+        return self.get().abort()
+
+    def savepoint(self, optimistic=False):
+        return self.get().savepoint(optimistic)
+
+class ThreadTransactionManager(TransactionManager):
+    """Thread-aware transaction manager.
+
+    Each thread is associated with a unique transaction.
+    """
+
+    def __init__(self):
+        # _threads maps thread ids to transactions
+        self._txns = {}
+
+        # _synchs maps a thread id to a WeakSet of registered synchronizers.
+        # The WeakSet is passed to the Transaction constructor, because the
+        # latter needs to call the synchronizers when it commits.
+        self._synchs = {}
+
+    def begin(self):
+        tid = thread.get_ident()
+        txn = self._txns.get(tid)
+        if txn is not None:
+            txn.abort()
+
+        synchs = self._synchs.get(tid)
+        if synchs is None:
+            synchs = self._synchs[tid] = WeakSet()
+
+        txn = self._txns[tid] = Transaction(synchs, self)
+        _new_transaction(txn, synchs)
+        return txn
+
+    def get(self):
+        tid = thread.get_ident()
+        txn = self._txns.get(tid)
+        if txn is None:
+            synchs = self._synchs.get(tid)
+            if synchs is None:
+                synchs = self._synchs[tid] = WeakSet()
+            txn = self._txns[tid] = Transaction(synchs, self)
+        return txn
+
+    def free(self, txn):
+        tid = thread.get_ident()
+        assert txn is self._txns.get(tid)
+        del self._txns[tid]
+
+    def registerSynch(self, synch):
+        tid = thread.get_ident()
+        ws = self._synchs.get(tid)
+        if ws is None:
+            ws = self._synchs[tid] = WeakSet()
+        ws.add(synch)
+
+    def unregisterSynch(self, synch):
+        tid = thread.get_ident()
+        ws = self._synchs[tid]
+        ws.remove(synch)

src/transaction/tests/__init__.py

+#

src/transaction/tests/abstestIDataManager.py

+##############################################################################
+#
+# Copyright (c) 2001, 2002 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""Test cases for objects implementing IDataManager.
+
+This is a combo test between Connection and DB, since the two are
+rather incestuous and the DB Interface is not defined that I was
+able to find.
+
+To do a full test suite one would probably want to write a dummy
+storage that will raise errors as needed for testing.
+
+I started this test suite to reproduce a very simple error (tpc_abort
+had an error and wouldn't even run if called).  So it is *very*
+incomplete, and even the tests that exist do not make sure that
+the data actually gets written/not written to the storge.
+
+Obviously this test suite should be expanded.
+
+$Id$
+"""
+
+from unittest import TestCase
+
+class IDataManagerTests(TestCase, object):
+
+    def setUp(self):
+        self.datamgr = None # subclass should override
+        self.obj = None # subclass should define Persistent object
+        self.txn_factory = None
+
+    def get_transaction(self):
+        return self.txn_factory()
+
+    ################################
+    # IDataManager interface tests #
+    ################################
+
+    def testCommitObj(self):
+        tran = self.get_transaction()
+        self.datamgr.prepare(tran)
+        self.datamgr.commit(tran)
+
+    def testAbortTran(self):
+        tran = self.get_transaction()
+        self.datamgr.prepare(tran)
+        self.datamgr.abort(tran)

src/transaction/tests/doom.txt

+Dooming Transactions
+====================
+
+A doomed transaction behaves exactly the same way as an active transaction but
+raises an error on any attempt to commit it, thus forcing an abort.
+
+Doom is useful in places where abort is unsafe and an exception cannot be
+raised.  This occurs when the programmer wants the code following the doom to
+run but not commit. It is unsafe to abort in these circumstances as a following
+get() may implicitly open a new transaction.
+
+Any attempt to commit a doomed transaction will raise a DoomedTransaction
+exception.
+
+An example of such a use case can be found in
+zope/app/form/browser/editview.py.  Here a form validation failure must doom
+the transaction as committing the transaction may have side-effects. However,
+the form code must continue to calculate a form containing the error messages
+to return.
+
+For Zope in general, code running within a request should always doom
+transactions rather than aborting them. It is the responsibilty of the
+publication to either abort() or commit() the transaction. Application code can
+use savepoints and doom() safely.
+
+To see how it works we first need to create a stub data manager:
+
+    >>> from transaction.interfaces import IDataManager
+    >>> from zope.interface import implements
+    >>> class DataManager:
+    ...     implements(IDataManager)
+    ...     def __init__(self):
+    ...         self.attr_counter = {}
+    ...     def __getattr__(self, name):
+    ...         def f(transaction):
+    ...             self.attr_counter[name] = self.attr_counter.get(name, 0) + 1
+    ...         return f
+    ...     def total(self):
+    ...         count = 0
+    ...         for access_count in self.attr_counter.values():
+    ...             count += access_count
+    ...         return count
+    ...     def sortKey(self):
+    ...         return 1
+
+Start a new transaction:
+
+    >>> import transaction
+    >>> txn = transaction.begin()
+    >>> dm = DataManager()
+    >>> txn.join(dm)
+
+We can ask a transaction if it is doomed to avoid expensive operations. An
+example of a use case is an object-relational mapper where a pre-commit hook
+sends all outstanding SQL to a relational database for objects changed during
+the transaction. This expensive operation is not necessary if the transaction
+has been doomed. A non-doomed transaction should return False:
+
+    >>> txn.isDoomed()
+    False
+
+We can doom a transaction by calling .doom() on it:
+
+    >>> txn.doom()
+    >>> txn.isDoomed()
+    True
+
+We can doom it again if we like:
+
+    >>> txn.doom()
+
+The data manager is unchanged at this point:
+
+    >>> dm.total()
+    0
+
+Attempting to commit a doomed transaction any number of times raises a
+DoomedTransaction:
+
+    >>> txn.commit() # doctest: +ELLIPSIS
+    Traceback (most recent call last):
+        ...
+    DoomedTransaction
+    >>> txn.commit() # doctest: +ELLIPSIS
+    Traceback (most recent call last):
+        ...
+    DoomedTransaction
+
+But still leaves the data manager unchanged:
+
+    >>> dm.total()
+    0
+
+But the doomed transaction can be aborted:
+
+    >>> txn.abort()
+
+Which aborts the data manager:
+
+    >>> dm.total()
+    1
+    >>> dm.attr_counter['abort']
+    1
+
+Dooming the current transaction can also be done directly from the transaction
+module. We can also begin a new transaction directly after dooming the old one:
+
+    >>> txn = transaction.begin()
+    >>> transaction.isDoomed()
+    False
+    >>> transaction.doom()
+    >>> transaction.isDoomed()
+    True
+    >>> txn = transaction.begin()
+
+After committing a transaction we get an assertion error if we try to doom the
+transaction. This could be made more specific, but trying to doom a transaction
+after it's been committed is probably a programming error:
+
+    >>> txn = transaction.begin()
+    >>> txn.commit()
+    >>> txn.doom()
+    Traceback (most recent call last):
+        ...
+    AssertionError
+
+A doomed transaction should act the same as an active transaction, so we should
+be able to join it:
+
+    >>> txn = transaction.begin()
+    >>> txn.doom()
+    >>> dm2 = DataManager()
+    >>> txn.join(dm2)
+
+Clean up:
+
+    >>> txn = transaction.begin()
+    >>> txn.abort()