Commit 05128835 authored by Tim Peters's avatar Tim Peters

Merge the ZODB part of Zope/branches/jim-fix-zclasses.

This doesn't have a lot of visible effect on standalone ZODB use, but
is important for Zope 2.8.

Here are checkin msgs from the branch relating to ZODB code:

    r29872 | jim | 2005-04-04 07:04:39 -0400 (Mon, 04 Apr 2005) | 3 lines
    Changed paths:
       M /Zope/branches/jim-fix-zclasses/lib/python/ZODB/serialize.py

    For instances of persistent classes, save a class-module/clas-name
    tuple if the class has a non-empty module string.

    ------------------------------------------------------------------------
    r29871 | jim | 2005-04-04 07:04:33 -0400 (Mon, 04 Apr 2005) | 6 lines
    Changed paths:
       M /Zope/branches/jim-fix-zclasses/lib/python/ZODB/Connection.py

    Changed sub-transaction abort code to not invalidate created objects.
    There's really no point, because created objects will be unreachable
    after invalidating old objects.  Also, if a created object is
    non-ghostifiable, it will try to load it's state again, and then
    either it would fail or it would load non-committed state.

    ------------------------------------------------------------------------
    r29870 | jim | 2005-04-04 07:04:27 -0400 (Mon, 04 Apr 2005) | 3 lines
    Changed paths:
       A /Zope/branches/jim-fix-zclasses/lib/python/ZODB/persistentclass.py
       A /Zope/branches/jim-fix-zclasses/lib/python/ZODB/persistentclass.txt
       A /Zope/branches/jim-fix-zclasses/lib/python/ZODB/tests/testpersistentclass.py

    Added ZClass-independent test of (and possible base class for)
    persistent-class support machinery.

    ------------------------------------------------------------------------
    r29867 | jim | 2005-04-04 07:03:48 -0400 (Mon, 04 Apr 2005) | 2 lines
    Changed paths:
       M /Zope/branches/jim-fix-zclasses/lib/python/transaction/_manager.py

    Added missing arguments to commit and abort.

    ------------------------------------------------------------------------
    r29774 | jim | 2005-04-01 06:24:27 -0500 (Fri, 01 Apr 2005) | 3 lines
    Changed paths:
       M /Zope/branches/jim-fix-zclasses/lib/python/transaction/__init__.py

    Changed to use methods of a threaded manager directly, rather than
    through wrapper functions.

    ------------------------------------------------------------------------
    r29773 | jim | 2005-04-01 06:24:25 -0500 (Fri, 01 Apr 2005) | 3 lines
    Changed paths:
       M /Zope/branches/jim-fix-zclasses/lib/python/transaction/_manager.py

    Added commit and abort methods to transaction managers.
    This makes direcr use of managers simpler.

    ------------------------------------------------------------------------
    r29101 | jim | 2005-02-10 07:44:52 -0500 (Thu, 10 Feb 2005) | 2 lines
    Changed paths:
       M /Zope/branches/jim-fix-zclasses/lib/python/ZODB/Connection.py

    Removed an unused attribute.

    ------------------------------------------------------------------------
    r29100 | jim | 2005-02-10 07:42:35 -0500 (Thu, 10 Feb 2005) | 3 lines
    Changed paths:
       M /Zope/branches/jim-fix-zclasses/lib/python/transaction/_transaction.py

    Changed an XXX comment to an ordinary comment, explaining the relevent
    issue.

    ------------------------------------------------------------------------
    r29099 | jim | 2005-02-10 07:41:58 -0500 (Thu, 10 Feb 2005) | 3 lines
    Changed paths:
       M /Zope/branches/jim-fix-zclasses/lib/python/ZODB/Connection.py

    Changed some XXX comments to ordinary comments explaining the relevent
    issues.

    ------------------------------------------------------------------------
    r29071 | jim | 2005-02-07 07:36:05 -0500 (Mon, 07 Feb 2005) | 3 lines
    Changed paths:
       M /Zope/branches/jim-fix-zclasses/lib/python/transaction/_transaction.py

    Added a sanity check to avoid registration of objects without a
    manager. (Perhaps this should be an assert.)

    ------------------------------------------------------------------------
    r29067 | jim | 2005-02-07 07:35:56 -0500 (Mon, 07 Feb 2005) | 15 lines
    Changed paths:
       M /Zope/branches/jim-fix-zclasses/lib/python/ZODB/Connection.py

    Changed the strategy for handling invalidation of classes.
    No-longer use setklassstate.  Instead, just call _p_invalidate, as
    with any other object.  This changes didn't break any tests, so I
    assume that this was untested. :(

    Changed the strategy for invalidating objects.  Non-ghostifiable
    objects will load their state when they are invalidated.  We have to
    worry about other invalidations that come in while this is happening.
    See the comment in _flush_invalidations.

    We need to force all invalidations to take this into account, by going
    through _flush_invalidations.  I haven't done this yet, but I have
    left some XXX comments in places where it needs to be done to remind
    myself that this needs to be done.

    ------------------------------------------------------------------------
    r29066 | jim | 2005-02-07 07:35:54 -0500 (Mon, 07 Feb 2005) | 8 lines
    Changed paths:
       M /Zope/branches/jim-fix-zclasses/lib/python/persistent/cPickleCache.c

    Changed the strategy for handling invalidation of classes.
    No-longer use setklassstate.  Instead, just call _p_invalidate, as
    with any other object.  This changes didn't break any tests, so I
    assume that this was untested. :(

    Change invalidation to not swallow errors. (Swallowing errors here was a
    travesty!)
parent 13e8c879
......@@ -16,7 +16,7 @@
$Id$
"""
from doctest import DocTestSuite
from zope.testing.doctest import DocTestSuite
import pickle
from Persistence import Persistent
......
......@@ -15,7 +15,7 @@
$Id$
"""
import unittest
from doctest import DocTestSuite
from zope.testing.doctest import DocTestSuite
from Persistence import PersistentMapping
def test_basic_functionality():
......
......@@ -120,9 +120,9 @@ class Connection(ExportImport, object):
# will execute atomically by virtue of the GIL. But some storage
# might generate oids where hash or compare invokes Python code. In
# that case, the GIL can't save us.
self._inv_lock = threading.Lock()
self._invalidated = d = {}
self._invalid = d.has_key
# We intend to prevent committing a transaction in which
# ReadConflictError occurs. _conflicts is the set of oids that
......@@ -221,6 +221,26 @@ class Connection(ExportImport, object):
del obj._p_jar
del obj._p_oid
else:
# Note: If we invalidate a non-ghostifiable object
# (i.e. a persistent class), the object will
# immediately reread it's state. That means that the
# following call could result in a call to
# self.setstate, which, of course, must suceed.
# In general, it would be better if the read could be
# delayed until the start of the next transaction. If
# we read at the end of a transaction and if the
# object was invalidated during this transaction, then
# we'll read non-current data, which we'll discard
# later in transaction finalization. Unfortnately, we
# can only delay the read if this abort corresponds to
# a top-level-transaction abort. We can't tell if
# this is a top-level-transaction abort, so we have to
# go ahead and invalidate now. Fortunately, it's
# pretty unlikely that the object we are invalidating
# was invalidated by another thread, so the risk of a
# reread is pretty low.
self._cache.invalidate(oid)
self._tpc_cleanup()
......@@ -387,6 +407,22 @@ class Connection(ExportImport, object):
self._storage = self._tmp
self._tmp = None
# Note: If we invalidate a non-ghostifiable object (i.e. a
# persistent class), the object will immediately reread it's
# state. That means that the following call could result in a
# call to self.setstate, which, of course, must succeed. In
# general, it would be better if the read could be delayed
# until the start of the next transaction. If we read at the
# end of a transaction and if the object was invalidated
# during this transaction, then we'll read non-current data,
# which we'll discard later in transaction finalization. We
# could, theoretically queue this invalidation by calling
# self.invalidate. Unfortunately, attempts to make that
# change resulted in mysterious test failures. It's pretty
# unlikely that the object we are invalidating was invalidated
# by another thread, so the risk of a reread is pretty low.
# It's really not worth the effort to pursue this.
self._cache.invalidate(src._index.keys())
self._invalidate_creating(src._creating)
......@@ -415,12 +451,44 @@ class Connection(ExportImport, object):
def _flush_invalidations(self):
self._inv_lock.acquire()
try:
self._cache.invalidate(self._invalidated)
self._invalidated.clear()
# Non-ghostifiable objects may need to read when they are
# invalidated, so, we'll quickly just replace the
# invalidating dict with a new one. We'll then process
# the invalidations after freeing the lock *and* after
# reseting the time. This means that invalidations will
# happen after the start of the transactions. They are
# subject to conflict errors and to reading old data,
# TODO: There is a potential problem lurking for persistent
# classes. Suppose we have an invlidation of a persistent
# class and of an instance. If the instance is
# invalidated first and if the invalidation logic uses
# data read from the class, then the invalidation could
# be performed with state data. Or, suppose that there
# are instances of the class that are freed as a result of
# invalidating some object. Perhaps code in their __del__
# uses class data. Really, the only way to properly fix
# this is to, in fact, make classes ghostifiable. Then
# we'd have to reimplement attribute lookup to check the
# class state and, if necessary, activate the class. It's
# much worse than that though, because we'd also need to
# deal with slots. When a class is ghostified, we'd need
# to replace all of the slot operations with versions that
# reloaded the object when caled. It's hard to say which
# is better for worse. For now, it seems the risk of
# using a class while objects are being invalidated seems
# small enough t be acceptable.
invalidated = self._invalidated
self._invalidated = {}
self._txn_time = None
finally:
self._inv_lock.release()
# Now is a good time to collect some garbage
self._cache.invalidate(invalidated)
# Now is a good time to collect some garbage.
self._cache.incrgc()
def root(self):
......@@ -532,10 +600,26 @@ class Connection(ExportImport, object):
self._tpc_cleanup()
def tpc_abort(self, transaction):
"""Abort a transaction."""
if self._import:
self._import = None
self._storage.tpc_abort(transaction)
# Note: If we invalidate a non-justifiable object (i.e. a
# persistent class), the object will immediately reread it's
# state. That means that the following call could result in a
# call to self.setstate, which, of course, must succeed. In
# general, it would be better if the read could be delayed
# until the start of the next transaction. If we read at the
# end of a transaction and if the object was invalidated
# during this transaction, then we'll read non-current data,
# which we'll discard later in transaction finalization. We
# could, theoretically queue this invalidation by calling
# self.invalidate. Unfortunately, attempts to make that
# change resulted in mysterious test failures. It's pretty
# unlikely that the object we are invalidating was invalidated
# by another thread, so the risk of a reread is pretty low.
# It's really not worth the effort to pursue this.
self._cache.invalidate(self._modified)
self._invalidate_creating()
while self._added:
......@@ -630,7 +714,9 @@ class Connection(ExportImport, object):
# because we have to check again after the load anyway.
if (obj._p_oid in self._invalidated
and not myhasattr(obj, "_p_independent")):
and not myhasattr(obj, "_p_independent")
and not self._invalidated
):
# If the object has _p_independent(), we will handle it below.
self._load_before_or_conflict(obj)
return
......@@ -913,4 +999,3 @@ class Connection(ExportImport, object):
if dt is not DEPRECATED_ARGUMENT:
deprecated36("cacheMinimize() dt= is ignored.")
self._cache.minimize()
##############################################################################
#
# Copyright (c) 2004 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""Persistent Class Support
$Id$
"""
# Notes:
#
# Persistent classes are non-ghostable. This has some interesting
# ramifications:
#
# - When an object is invalidated, it must reload it's state
#
# - When an object is loaded from the database, it's state must be
# loaded. Unfortunately, there isn't a clear signal when an object is
# loaded from the database. This should probably be fixed.
#
# In the mean time, we need to infer. This should be viewed as a
# short term hack.
#
# Here's the strategy we'll use:
#
# - We'll have a need to be loaded flag that we'll set in
# __new__, through an extra argument.
#
# - When setting _p_oid and _p_jar, if both are set and we need to be
# loaded, then we'll load out state.
#
# - We'll use _p_changed is None to indicate that we're in this state.
#
class _p_DataDescr(object):
# Descr used as base for _p_ data. Data are stored in
# _p_class_dict.
def __init__(self, name):
self.__name__ = name
def __get__(self, inst, cls):
if inst is None:
return self
if '__global_persistent_class_not_stored_in_DB__' in inst.__dict__:
raise AttributeError, self.__name__
return inst._p_class_dict.get(self.__name__)
def __set__(self, inst, v):
inst._p_class_dict[self.__name__] = v
def __delete__(self, inst):
raise AttributeError, self.__name__
class _p_oid_or_jar_Descr(_p_DataDescr):
# Special descr for _p_oid and _p_jar that loads
# state when set if both are set and and _p_changed is None
#
# See notes above
def __set__(self, inst, v):
get = inst._p_class_dict.get
if v == get(self.__name__):
return
inst._p_class_dict[self.__name__] = v
jar = get('_p_jar')
if (jar is not None
and get('_p_oid') is not None
and get('_p_changed') is None
):
jar.setstate(inst)
class _p_ChangedDescr(object):
# descriptor to handle special weird emantics of _p_changed
def __get__(self, inst, cls):
if inst is None:
return self
return inst._p_class_dict['_p_changed']
def __set__(self, inst, v):
if v is None:
return
inst._p_class_dict['_p_changed'] = bool(v)
def __delete__(self, inst):
inst._p_invalidate()
class _p_MethodDescr(object):
"""Provide unassignable class attributes
"""
def __init__(self, func):
self.func = func
def __get__(self, inst, cls):
if inst is None:
return cls
return self.func.__get__(inst, cls)
def __set__(self, inst, v):
raise AttributeError, self.__name__
def __delete__(self, inst):
raise AttributeError, self.__name__
special_class_descrs = '__dict__', '__weakref__'
class PersistentMetaClass(type):
_p_jar = _p_oid_or_jar_Descr('_p_jar')
_p_oid = _p_oid_or_jar_Descr('_p_oid')
_p_changed = _p_ChangedDescr()
_p_serial = _p_DataDescr('_p_serial')
def __new__(self, name, bases, cdict, _p_changed=False):
cdict = dict([(k, v) for (k, v) in cdict.items()
if not k.startswith('_p_')])
cdict['_p_class_dict'] = {'_p_changed': _p_changed}
return super(PersistentMetaClass, self).__new__(
self, name, bases, cdict)
def __getnewargs__(self):
return self.__name__, self.__bases__, {}, None
__getnewargs__ = _p_MethodDescr(__getnewargs__)
def _p_maybeupdate(self, name):
get = self._p_class_dict.get
data_manager = get('_p_jar')
if (
(data_manager is not None)
and
(get('_p_oid') is not None)
and
(get('_p_changed') == False)
):
self._p_changed = True
data_manager.register(self)
def __setattr__(self, name, v):
if not ((name.startswith('_p_') or name.startswith('_v'))):
self._p_maybeupdate(name)
super(PersistentMetaClass, self).__setattr__(name, v)
def __delattr__(self, name):
if not ((name.startswith('_p_') or name.startswith('_v'))):
self._p_maybeupdate(name)
super(PersistentMetaClass, self).__delattr__(name)
def _p_deactivate(self):
# persistent classes can't be ghosts
pass
_p_deactivate = _p_MethodDescr(_p_deactivate)
def _p_invalidate(self):
# reset state
self._p_class_dict['_p_changed'] = None
self._p_jar.setstate(self)
_p_invalidate = _p_MethodDescr(_p_invalidate)
def __getstate__(self):
return (self.__bases__,
dict([(k, v) for (k, v) in self.__dict__.items()
if not (k.startswith('_p_')
or k.startswith('_v_')
or k in special_class_descrs
)
]),
)
__getstate__ = _p_MethodDescr(__getstate__)
def __setstate__(self, state):
self.__bases__, cdict = state
cdict = dict([(k, v) for (k, v) in cdict.items()
if not k.startswith('_p_')])
_p_class_dict = self._p_class_dict
self._p_class_dict = {}
to_remove = [k for k in self.__dict__
if ((k not in cdict)
and
(k not in special_class_descrs)
and
(k != '_p_class_dict')
)]
for k in to_remove:
delattr(self, k)
for k, v in cdict.items():
setattr(self, k, v)
self._p_class_dict = _p_class_dict
self._p_changed = False
__setstate__ = _p_MethodDescr(__setstate__)
def _p_activate(self):
self._p_jar.setstate(self)
_p_activate = _p_MethodDescr(_p_activate)
Persistent Classes
==================
NOTE: persistent classes are EXPERIMENTAL and, in some sense,
incomplete. This module exists largely to test changes made to
support Zope 2 ZClasses, with their historical flaws.
The persistentclass module provides a meta class that can be used to implement
persistent classes.
Persistent classes have the following properties:
- They cannot be turned into ghosts
- They can only contain picklable subobjects
- They don't live in regular file-system modules
Let's look at an example:
>>> def __init__(self, name):
... self.name = name
>>> def foo(self):
... return self.name, self.kind
>>> import ZODB.persistentclass
>>> class C:
... __metaclass__ = ZODB.persistentclass.PersistentMetaClass
... __init__ = __init__
... __module__ = '__zodb__'
... foo = foo
... kind = 'sample'
This example is obviously a bit contrived. In particular, we defined
the methods outside of the class. Why? Because all of the items in a
persistent class must be picklable. We defined the methods as global
functions to make them picklable.
Also note that we explictly set the module. Persistent classes don't
live in normal Python modules. Rather, they live in the database. We
use information in __module__ to record where in the database. When
we want to use a database, we will need to supply a custom class
factory to load instances of the class.
The class we created works a lot like other persistent objects. It
has standard standard persistent attributes:
>>> C._p_oid
>>> C._p_jar
>>> C._p_serial
>>> C._p_changed
False
Because we haven't saved the object, the jar, oid, and serial are all
None and it's not changed.
We can create and use instances of the class:
>>> c = C('first')
>>> c.foo()
('first', 'sample')
We can modify the class and none of the persistent attributes will
change because the object hasn't been saved.
>>> def bar(self):
... print 'bar', self.name
>>> C.bar = bar
>>> c.bar()
bar first
>>> C._p_oid
>>> C._p_jar
>>> C._p_serial
>>> C._p_changed
False
Now, we can store the class in a database. We're going to use an
explicit transaction manager so that we can show parallel transactions
without having to use threads.
>>> import transaction
>>> tm = transaction.TransactionManager()
>>> connection = some_database.open(txn_mgr=tm)
>>> connection.root()['C'] = C
>>> tm.commit()
Now, if we look at the persistence variables, we'll see that they have
values:
>>> C._p_oid
'\x00\x00\x00\x00\x00\x00\x00\x01'
>>> C._p_jar is not None
True
>>> C._p_serial is not None
True
>>> C._p_changed
False
Now, if we modify the class:
>>> def baz(self):
... print 'baz', self.name
>>> C.baz = baz
>>> c.baz()
baz first
We'll see that the class has changed:
>>> C._p_changed
True
If we abort the transaction:
>>> tm.abort()
Then the class will return to it's prior state:
>>> c.baz()
Traceback (most recent call last):
...
AttributeError: 'C' object has no attribute 'baz'
>>> c.bar()
bar first
We can open another connection and access the class there.
>>> tm2 = transaction.TransactionManager()
>>> connection2 = some_database.open(txn_mgr=tm2)
>>> C2 = connection2.root()['C']
>>> c2 = C2('other')
>>> c2.bar()
bar other
If we make changes without commiting them:
>>> C.bar = baz
>>> c.bar()
baz first
>>> C is C2
False
Other connections are unaffected:
>>> connection2.sync()
>>> c2.bar()
bar other
Until we commit:
>>> tm.commit()
>>> connection2.sync()
>>> c2.bar()
baz other
Similarly, we don't see changes made in other connections:
>>> C2.color = 'red'
>>> tm2.commit()
>>> c.color
Traceback (most recent call last):
...
AttributeError: 'C' object has no attribute 'color'
until we sync:
>>> connection.sync()
>>> c.color
'red'
Instances of Persistent Classes
-------------------------------
We can, of course, store instances of perstent classes in the
database:
>>> c.color = 'blue'
>>> connection.root()['c'] = c
>>> tm.commit()
>>> connection2.sync()
>>> connection2.root()['c'].color
'blue'
NOTE: If a non-persistent instance of a persistent class is copied,
the class may be copied as well. This is usually not the desired
result.
Persistent instances of persistent classes
------------------------------------------
Persistent instances of persistent classes are handled differently
than normal instances. When we copy a persistent instances of a
persistent class, we want to avoid copying the class.
Lets create a persistent class that subclasses Persistent:
>>> import persistent
>>> class P(persistent.Persistent, C):
... __module__ = '__zodb__'
... color = 'green'
>>> connection.root()['P'] = P
>>> import persistent.mapping
>>> connection.root()['obs'] = persistent.mapping.PersistentMapping()
>>> p = P('p')
>>> connection.root()['obs']['p'] = p
>>> tm.commit()
You might be wondering why we didn't just stick 'p' into the root
object. We created an intermediate persistent object instead. We are
storing persistent classes in the root object. To create a ghost for a
persistent instance of a persistent class, we need to be able to be
able to access the root object and it must be loaded first. If the
instance was in the root object, we'd be unable to create it while
loading the root object.
Now, if we try to load it, we get a broken oject:
>>> connection2.sync()
>>> connection2.root()['obs']['p']
<persistent broken __zodb__.P instance '\x00\x00\x00\x00\x00\x00\x00\x04'>
because the module, "__zodb__" can't be loaded. We need to provide a
class factory that knows about this special module. Here we'll supply a
sample class factory that looks up a class name in the database root
if the module is "__zodb__". It falls back to the normal class lookup
for other modules:
>>> from ZODB.broken import find_global
>>> def classFactory(connection, modulename, globalname):
... if modulename == '__zodb__':
... return connection.root()[globalname]
... return find_global(modulename, globalname)
>>> some_database.classFactory = classFactory
Normally, the classFactory should be set before a database is opened.
We'll reopen the connections we're using. We'll assign the old
connections to a variable first to prevent getting them from the
connection pool:
>>> old = connection, connection2
>>> connection = some_database.open(txn_mgr=tm)
>>> connection2 = some_database.open(txn_mgr=tm2)
Now, we can read the object:
>>> connection2.root()['obs']['p'].color
'green'
>>> connection2.root()['obs']['p'].color = 'blue'
>>> tm2.commit()
>>> connection.sync()
>>> p = connection.root()['obs']['p']
>>> p.color
'blue'
Copying
-------
If we copy an instance via export/import, the copy and the original
share the same class:
>>> file = connection.exportFile(p._p_oid)
>>> file.seek(0)
>>> cp = connection.importFile(file)
>>> cp.color
'blue'
>>> cp is not p
True
>>> cp.__class__ is p.__class__
True
XXX test abort of import
......@@ -32,14 +32,19 @@ object exists as a ghost, its state is passed to ``__setstate__``.
The class description can be in a variety of formats, in part to
provide backwards compatibility with earlier versions of Zope. The
two current formats for class description are:
four current formats for class description are:
1. type(obj)
2. type(obj), obj.__getnewargs__()
3. (module name, class name), None
7. (module name, class name), obj.__getnewargs__()
The second of these options is used if the object has a __getnewargs__()
method. It is intended to support objects like persistent classes that have
custom C layouts that are determined by arguments to __new__().
custom C layouts that are determined by arguments to __new__(). The
third and fourth (#3 & #7) apply to instances of a persistent class (which
means the class itself is persistent, not that it's a subclass of
Persistent).
The type object is usually stored using the standard pickle mechanism, which
involves the pickle GLOBAL opcode (giving the type's module and name as
......@@ -59,17 +64,17 @@ useful places to concentrate confusion about exactly which formats exist:
Earlier versions of Zope supported several other kinds of class
descriptions. The current serialization code reads these descriptions, but
does not write them. The four earlier formats are:
does not write them. The three earlier formats are:
3. (module name, class name), None
4. (module name, class name), __getinitargs__()
5. class, None
6. class, __getinitargs__()
Formats 4 and 6 are used only if the class defines a __getinitargs__()
method. Formats 5 and 6 are used if the class does not have a __module__
attribute (I'm not sure when this applies, but I think it occurs for some
but not all ZClasses).
method, but we really can't tell them apart from formats 7 and 2
(respectively). Formats 5 and 6 are used if the class does not have a
__module__ attribute (I'm not sure when this applies, but I think it occurs
for some but not all ZClasses).
Persistent references
......@@ -100,6 +105,8 @@ from ZODB import broken
from ZODB.broken import Broken
from ZODB.POSException import InvalidObjectReference
_oidtypes = str, type(None)
# Might to update or redo coptimizations to reflect weakrefs:
# from ZODB.coptimizations import new_persistent_id
......@@ -147,10 +154,10 @@ class ObjectWriter:
... _p_jar = jar
>>> writer = ObjectWriter(O)
Normally, object references include the oid and a cached
reference to the class. Having the class available allows
fast creation of the ghost, avoiding requiring an additional
database lookup.
Normally, object references include the oid and a cached named
reference to the class. Having the class information
available allows fast creation of the ghost, avoiding
requiring an additional database lookup.
>>> bob = P('bob')
>>> oid, cls = writer.persistent_id(bob)
......@@ -282,8 +289,16 @@ class ObjectWriter:
# It's possible that __getnewargs__ is degenerate and
# returns (), but we don't want to have to deghostify
# the object to find out.
# Note that this has the odd effect that, if the class has
# __getnewargs__ of its own, we'll lose the optimization
# of caching the class info.
return oid
# Note that we never get here for persistent classes.
# We'll use driect refs for normal classes.
return oid, klass
def serialize(self, obj):
......@@ -291,10 +306,25 @@ class ObjectWriter:
# We don't want to be fooled by proxies.
klass = type(obj)
# We want to serialize persistent classes by name if they have
# a non-None non-empty module so as not to have a direct
# ref. This is important when copying. We probably want to
# revisit this in the future.
newargs = getattr(obj, "__getnewargs__", None)
if (isinstance(getattr(klass, '_p_oid', 0), _oidtypes)
and klass.__module__):
# This is a persistent class with a non-empty module. This
# uses pickle format #3 or #7.
klass = klass.__module__, klass.__name__
if newargs is None:
meta = klass, None
else:
meta = klass, newargs()
elif newargs is None:
# Pickle format #1.
meta = klass
else:
# Pickle format #2.
meta = klass, newargs()
return self._dump(meta, obj.__getstate__())
......
......@@ -405,7 +405,7 @@ class Rollback(object):
def test_suite():
from doctest import DocTestSuite
from zope.testing.doctest import DocTestSuite
return DocTestSuite()
if __name__ == '__main__':
......
......@@ -20,7 +20,7 @@ import sys
import unittest
import persistent
import transaction
from doctest import DocTestSuite
from zope.testing.doctest import DocTestSuite
from ZODB.tests.util import DB
def test_integration():
......
......@@ -13,7 +13,7 @@
##############################################################################
"""Unit tests for the Connection class."""
import doctest
from zope.testing import doctest
import unittest
import warnings
......
......@@ -485,7 +485,7 @@ def testTimeTravelOnOpen():
"""
def test_suite():
import doctest
from zope.testing import doctest
suite = unittest.TestSuite()
for klass in [FileStorageTests, Corruption.FileStorageCorruptTests,
......
......@@ -121,7 +121,7 @@ class SerializerTestCase(unittest.TestCase):
def test_suite():
import doctest
from zope.testing import doctest
suite = unittest.makeSuite(SerializerTestCase)
suite.addTest(doctest.DocTestSuite("ZODB.serialize"))
return suite
......@@ -132,7 +132,7 @@ database state as of the last sub-transaction commit. There is
"""
import doctest
from zope.testing import doctest
def test_suite():
return doctest.DocTestSuite()
......@@ -13,7 +13,7 @@
##############################################################################
"""Test behavior of Connection plus cPickleCache."""
import doctest
from zope.testing import doctest
from persistent import Persistent
import transaction
......
......@@ -15,7 +15,7 @@
$Id$
"""
import unittest
from doctest import DocTestSuite
from zope.testing.doctest import DocTestSuite
from transaction._transaction import DataManagerAdapter
from ZODB.tests.sampledm import DataManager
......
......@@ -358,7 +358,7 @@ clearing the Connection's local transaction.
"""
import doctest
from zope.testing import doctest
def test_suite():
return doctest.DocTestSuite()
##############################################################################
#
# Copyright (c) 2004 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""ZClass tests
$Id$
"""
import os, sys
import unittest
import ZODB.tests.util
import transaction
from zope.testing import doctest
# XXX need to update files to get newer testing package
class FakeModule:
def __init__(self, name, dict):
self.__dict__ = dict
self.__name__ = name
def setUp(test):
test.globs['some_database'] = ZODB.tests.util.DB()
module = FakeModule('ZODB.persistentclass_txt', test.globs)
sys.modules[module.__name__] = module
def tearDown(test):
transaction.abort()
test.globs['some_database'].close()
del sys.modules['ZODB.persistentclass_txt']
def test_suite():
return unittest.TestSuite((
doctest.DocFileSuite("../persistentclass.txt",
setUp=setUp, tearDown=tearDown),
))
if __name__ == '__main__':
unittest.main(defaultTest='test_suite')
......@@ -115,7 +115,6 @@ typedef struct {
int klass_count; /* count of persistent classes */
PyObject *data; /* oid -> object dict */
PyObject *jar; /* Connection object */
PyObject *setklassstate; /* ??? */
int cache_size; /* target number of items in cache */
/* Most of the time the ring contains only:
......@@ -331,58 +330,48 @@ cc_minimize(ccobject *self, PyObject *args)
return lockgc(self, 0);
}
static void
static int
_invalidate(ccobject *self, PyObject *key)
{
static PyObject *_p_invalidate;
PyObject *v = PyDict_GetItem(self->data, key);
static PyObject *_p_invalidate = NULL;
PyObject *meth, *v;
v = PyDict_GetItem(self->data, key);
if (v == NULL)
return 0;
if (!_p_invalidate) {
if (_p_invalidate == NULL)
{
_p_invalidate = PyString_InternFromString("_p_invalidate");
if (!_p_invalidate) {
if (_p_invalidate == NULL)
{
/* It doesn't make any sense to ignore this error, but
the caller ignores all errors.
TODO: and why does it do that? This should be fixed
*/
PyErr_Clear();
return;
return -1;
}
}
if (!v)
return;
if (PyType_Check(v)) {
if (v->ob_refcnt <= 1 && PyType_Check(v)) {
/* This looks wrong, but it isn't. We use strong references to types
because they don't have the ring members.
The result is that we *never* remove classes unless
they are modified.
they are modified. We can fix this by using wekrefs uniformly.
*/
if (v->ob_refcnt <= 1) {
self->klass_count--;
if (PyDict_DelItem(self->data, key) < 0)
PyErr_Clear();
}
else {
v = PyObject_CallFunction(self->setklassstate, "O", v);
if (v)
Py_DECREF(v);
else
PyErr_Clear();
return PyDict_DelItem(self->data, key);
}
} else {
PyObject *meth, *err;
meth = PyObject_GetAttr(v, _p_invalidate);
if (!meth) {
PyErr_Clear();
return;
}
err = PyObject_CallObject(meth, NULL);
if (meth == NULL)
return -1;
v = PyObject_CallObject(meth, NULL);
Py_DECREF(meth);
if (!err)
PyErr_Clear();
}
return v == NULL ? -1 : 0;
}
static PyObject *
......@@ -391,16 +380,23 @@ cc_invalidate(ccobject *self, PyObject *inv)
PyObject *key, *v;
int i = 0;
if (PyDict_Check(inv)) {
if (PyDict_Check(inv))
{
while (PyDict_Next(inv, &i, &key, &v))
_invalidate(self, key);
{
if (_invalidate(self, key) < 0)
return NULL;
}
PyDict_Clear(inv);
}
else {
if (PyString_Check(inv))
_invalidate(self, inv);
{
if (_invalidate(self, inv) < 0)
return NULL;
}
else {
int l;
int l, r;
l = PyObject_Length(inv);
if (l < 0)
......@@ -409,8 +405,10 @@ cc_invalidate(ccobject *self, PyObject *inv)
key = PySequence_GetItem(inv, i);
if (!key)
return NULL;
_invalidate(self, key);
r = _invalidate(self, key);
Py_DECREF(key);
if (r < 0)
return NULL;
}
/* Dubious: modifying the input may be an unexpected side effect. */
PySequence_DelSlice(inv, 0, l);
......@@ -669,7 +667,7 @@ cc_init(ccobject *self, PyObject *args, PyObject *kwds)
if (!PyArg_ParseTuple(args, "O|i", &jar, &cache_size))
return -1;
self->setklassstate = self->jar = NULL;
self->jar = NULL;
self->data = PyDict_New();
if (self->data == NULL) {
Py_DECREF(self);
......@@ -686,11 +684,6 @@ cc_init(ccobject *self, PyObject *args, PyObject *kwds)
non-ghost objects.
*/
PyObject_GC_UnTrack((void *)self->data);
self->setklassstate = PyObject_GetAttrString(jar, "setklassstate");
if (self->setklassstate == NULL) {
Py_DECREF(self);
return -1;
}
self->jar = jar;
Py_INCREF(jar);
self->cache_size = cache_size;
......@@ -708,7 +701,6 @@ cc_dealloc(ccobject *self)
{
Py_XDECREF(self->data);
Py_XDECREF(self->jar);
Py_XDECREF(self->setklassstate);
PyObject_GC_Del(self);
}
......@@ -755,7 +747,6 @@ cc_clear(ccobject *self)
}
Py_XDECREF(self->jar);
Py_XDECREF(self->setklassstate);
while (PyDict_Next(self->data, &pos, &k, &v)) {
Py_INCREF(v);
......@@ -765,7 +756,6 @@ cc_clear(ccobject *self)
Py_XDECREF(self->data);
self->data = NULL;
self->jar = NULL;
self->setklassstate = NULL;
return 0;
}
......@@ -794,7 +784,6 @@ cc_traverse(ccobject *self, visitproc visit, void *arg)
}
VISIT(self->jar);
VISIT(self->setklassstate);
here = self->ring_home.r_next;
......
......@@ -40,7 +40,7 @@ def test_delitem():
"""
from doctest import DocTestSuite
from zope.testing.doctest import DocTestSuite
import unittest
def test_suite():
......
......@@ -398,5 +398,5 @@ class SampleOverridingGetattributeSetattrAndDelattr(Persistent):
def test_suite():
from doctest import DocTestSuite
from zope.testing.doctest import DocTestSuite
return DocTestSuite()
......@@ -11,11 +11,9 @@
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
from zope.testing import doctest
from persistent import Persistent
from zope.testing.doctestunit import DocFileSuite
class P(Persistent):
def __init__(self):
self.x = 0
......@@ -23,4 +21,4 @@ class P(Persistent):
self.x += 1
def test_suite():
return DocFileSuite("persistent.txt", globs={"P": P})
return doctest.DocFileSuite("persistent.txt", globs={"P": P})
......@@ -269,7 +269,7 @@ def test_pickling_w_slots_w_empty_dict():
"""
from doctest import DocTestSuite
from zope.testing.doctest import DocTestSuite
import unittest
def test_suite():
......
......@@ -15,7 +15,7 @@
$Id$
"""
import unittest
from doctest import DocTestSuite
from zope.testing.doctest import DocTestSuite
def test_suite():
return DocTestSuite('persistent.wref')
......
......@@ -20,18 +20,10 @@ from transaction._transaction import Transaction
from transaction._manager import TransactionManager, ThreadTransactionManager
manager = ThreadTransactionManager()
def get():
return manager.get()
def begin():
return manager.begin()
def commit(sub=False):
manager.get().commit(sub)
def abort(sub=False):
manager.get().abort(sub)
get = manager.get
begin = manager.begin
commit = manager.commit
abort = manager.abort
def get_transaction():
from ZODB.utils import deprecated36
......
......@@ -61,7 +61,13 @@ class TransactionManager(object):
def unregisterSynch(self, synch):
self._synchs.remove(synch)
class ThreadTransactionManager(object):
def commit(self, sub=False):
self.get().commit(sub)
def abort(self, sub=False):
self.get().abort(sub)
class ThreadTransactionManager(TransactionManager):
"""Thread-aware transaction manager.
Each thread is associated with a unique transaction.
......
......@@ -54,8 +54,10 @@ methods and support a second argument to tpc_begin().
The second argument to tpc_begin() indicates that a subtransaction
commit is beginning (if it is true). In a subtransaction, there is no
tpc_vote() call (I don't know why not). The tpc_finish()
or tpc_abort() call applies just to that subtransaction.
tpc_vote() call, because sub-transactions don't need 2-phase commit.
If a sub-transaction abort or commit fails, we can abort the outer
transaction. The tpc_finish() or tpc_abort() call applies just to
that subtransaction.
Once a resource manager is involved in a subtransaction, all
subsequent transactions will be treated as subtransactions until
......@@ -244,6 +246,8 @@ class Transaction(object):
# commit protocol.
manager = getattr(obj, "_p_jar", obj)
if manager is None:
raise ValueError("Register with no manager")
adapter = self._adapters.get(manager)
if adapter is None:
if myhasattr(manager, "commit_sub"):
......
......@@ -405,7 +405,7 @@ class Rollback(object):
def test_suite():
from doctest import DocTestSuite
from zope.testing.doctest import DocTestSuite
return DocTestSuite()
if __name__ == '__main__':
......
......@@ -148,7 +148,7 @@ class TestConnection:
obj.abort()
self.aborted.append(obj)
import doctest
from zope.testing import doctest
def test_suite():
return doctest.DocTestSuite()
......@@ -46,15 +46,11 @@ from ZODB.utils import positive_id
class TransactionTests(unittest.TestCase):
def setUp(self):
self.orig_tm = transaction.manager
transaction.manager = transaction.TransactionManager()
self.sub1 = DataObject()
self.sub2 = DataObject()
self.sub3 = DataObject()
self.nosub1 = DataObject(nost=1)
def tearDown(self):
transaction.manager = self.orig_tm
self.txn_mgr = transaction.TransactionManager()
self.sub1 = DataObject(self.txn_mgr)
self.sub2 = DataObject(self.txn_mgr)
self.sub3 = DataObject(self.txn_mgr)
self.nosub1 = DataObject(self.txn_mgr, nost=1)
# basic tests with two sub trans jars
# really we only need one, so tests for
......@@ -64,7 +60,7 @@ class TransactionTests(unittest.TestCase):
self.sub1.modify()
self.sub2.modify()
transaction.commit()
self.txn_mgr.commit()
assert self.sub1._p_jar.ccommit_sub == 0
assert self.sub1._p_jar.ctpc_finish == 1
......@@ -74,13 +70,13 @@ class TransactionTests(unittest.TestCase):
self.sub1.modify()
self.sub2.modify()
transaction.abort()
self.txn_mgr.abort()
assert self.sub2._p_jar.cabort == 1
def testTransactionNote(self):
t = transaction.get()
t = self.txn_mgr.get()
t.note('This is a note.')
self.assertEqual(t.description, 'This is a note.')
......@@ -94,12 +90,12 @@ class TransactionTests(unittest.TestCase):
self.sub1.modify()
self.sub2.modify()
transaction.commit(1)
self.txn_mgr.commit(1)
assert self.sub1._p_jar.ctpc_vote == 0
assert self.sub1._p_jar.ctpc_finish == 1
transaction.commit()
self.txn_mgr.commit()
assert self.sub1._p_jar.ccommit_sub == 1
assert self.sub1._p_jar.ctpc_vote == 1
......@@ -109,8 +105,8 @@ class TransactionTests(unittest.TestCase):
self.sub1.modify()
self.sub2.modify()
transaction.commit(1)
transaction.abort()
self.txn_mgr.commit(1)
self.txn_mgr.abort()
assert self.sub1._p_jar.ctpc_vote == 0
assert self.sub1._p_jar.cabort == 0
......@@ -118,12 +114,12 @@ class TransactionTests(unittest.TestCase):
def testMultipleSubTransactionCommitCommit(self):
self.sub1.modify()
transaction.commit(1)
self.txn_mgr.commit(1)
self.sub2.modify()
# reset a flag on the original to test it again
self.sub1.ctpc_finish = 0
transaction.commit(1)
self.txn_mgr.commit(1)
# this is interesting.. we go through
# every subtrans commit with all subtrans capable
......@@ -135,7 +131,7 @@ class TransactionTests(unittest.TestCase):
# add another before we do the entire txn commit
self.sub3.modify()
transaction.commit()
self.txn_mgr.commit()
# we did an implicit sub commit, is this impl artifact?
assert self.sub3._p_jar.ccommit_sub == 1
......@@ -161,12 +157,12 @@ class TransactionTests(unittest.TestCase):
# add it
self.sub1.modify()
transaction.commit(1)
self.txn_mgr.commit(1)
# add another
self.sub2.modify()
transaction.commit(1)
self.txn_mgr.commit(1)
assert self.sub1._p_jar.ctpc_vote == 0
assert self.sub1._p_jar.ctpc_finish > 0
......@@ -175,10 +171,10 @@ class TransactionTests(unittest.TestCase):
self.sub3.modify()
# abort the sub transaction
transaction.abort(1)
self.txn_mgr.abort(1)
# commit the container transaction
transaction.commit()
self.txn_mgr.commit()
assert self.sub3._p_jar.cabort == 1
assert self.sub1._p_jar.ccommit_sub == 1
......@@ -190,7 +186,7 @@ class TransactionTests(unittest.TestCase):
self.nosub1.modify()
transaction.commit()
self.txn_mgr.commit()
assert self.nosub1._p_jar.ctpc_finish == 1
......@@ -198,7 +194,7 @@ class TransactionTests(unittest.TestCase):
self.nosub1.modify()
transaction.abort()
self.txn_mgr.abort()
assert self.nosub1._p_jar.ctpc_finish == 0
assert self.nosub1._p_jar.cabort == 1
......@@ -221,7 +217,7 @@ class TransactionTests(unittest.TestCase):
self.sub1.modify(tracing='sub')
self.nosub1.modify(tracing='nosub')
transaction.commit(1)
self.txn_mgr.commit(1)
assert self.sub1._p_jar.ctpc_finish == 1
......@@ -229,7 +225,7 @@ class TransactionTests(unittest.TestCase):
# in a subtrans
assert self.nosub1._p_jar.ctpc_finish == 0
transaction.abort()
self.txn_mgr.abort()
assert self.nosub1._p_jar.cabort == 1
assert self.sub1._p_jar.cabort_sub == 1
......@@ -239,11 +235,11 @@ class TransactionTests(unittest.TestCase):
self.sub1.modify()
self.nosub1.modify()
transaction.commit(1)
self.txn_mgr.commit(1)
assert self.nosub1._p_jar.ctpc_vote == 0
transaction.commit()
self.txn_mgr.commit()
#assert self.nosub1._p_jar.ccommit_sub == 0
assert self.nosub1._p_jar.ctpc_vote == 1
......@@ -276,12 +272,12 @@ class TransactionTests(unittest.TestCase):
# add it
self.sub1.modify()
transaction.commit(1)
self.txn_mgr.commit(1)
# add another
self.nosub1.modify()
transaction.commit(1)
self.txn_mgr.commit(1)
assert self.sub1._p_jar.ctpc_vote == 0
assert self.nosub1._p_jar.ctpc_vote == 0
......@@ -291,7 +287,7 @@ class TransactionTests(unittest.TestCase):
self.sub2.modify()
# commit the container transaction
transaction.commit()
self.txn_mgr.commit()
# we did an implicit sub commit
assert self.sub2._p_jar.ccommit_sub == 1
......@@ -316,7 +312,7 @@ class TransactionTests(unittest.TestCase):
self.sub2.modify()
try:
transaction.abort()
self.txn_mgr.abort()
except TestTxnException: pass
assert self.nosub1._p_jar.cabort == 1
......@@ -330,7 +326,7 @@ class TransactionTests(unittest.TestCase):
self.sub1.modify(nojar=1)
try:
transaction.commit()
self.txn_mgr.commit()
except TestTxnException: pass
assert self.nosub1._p_jar.ctpc_finish == 0
......@@ -345,7 +341,7 @@ class TransactionTests(unittest.TestCase):
self.sub1.modify(nojar=1)
try:
transaction.commit()
self.txn_mgr.commit()
except TestTxnException: pass
assert self.nosub1._p_jar.ctpc_finish == 0
......@@ -371,7 +367,7 @@ class TransactionTests(unittest.TestCase):
self.sub1.modify(nojar=1)
try:
transaction.commit()
self.txn_mgr.commit()
except TestTxnException: pass
assert self.nosub1._p_jar.ctpc_abort == 1
......@@ -385,7 +381,7 @@ class TransactionTests(unittest.TestCase):
self.sub1.modify(nojar=1)
try:
transaction.commit()
self.txn_mgr.commit()
except TestTxnException:
pass
......@@ -402,19 +398,19 @@ class TransactionTests(unittest.TestCase):
# they come out of the dictionary.
self.sub1.modify()
transaction.commit(1)
self.txn_mgr.commit(1)
self.nosub1.modify()
self.sub2._p_jar = SubTransactionJar(errors='commit_sub')
self.sub2.modify(nojar=1)
transaction.commit(1)
self.txn_mgr.commit(1)
self.sub3.modify()
try:
transaction.commit()
self.txn_mgr.commit()
except TestTxnException:
pass
......@@ -441,17 +437,17 @@ class TransactionTests(unittest.TestCase):
self.sub1._p_jar = SubTransactionJar(errors='commit_sub')
self.sub1.modify(nojar=1)
transaction.commit(1)
self.txn_mgr.commit(1)
self.nosub1.modify()
self.sub2._p_jar = SubTransactionJar(errors='abort_sub')
self.sub2.modify(nojar=1)
transaction.commit(1)
self.txn_mgr.commit(1)
self.sub3.modify()
try:
transaction.commit()
self.txn_mgr.commit()
except TestTxnException, err:
pass
else:
......@@ -500,7 +496,8 @@ class TransactionTests(unittest.TestCase):
class DataObject:
def __init__(self, nost=0):
def __init__(self, txn_mgr, nost=0):
self.txn_mgr = txn_mgr
self.nost = nost
self._p_jar = None
......@@ -510,7 +507,7 @@ class DataObject:
self._p_jar = NoSubTransactionJar(tracing=tracing)
else:
self._p_jar = SubTransactionJar(tracing=tracing)
transaction.get().register(self)
self.txn_mgr.get().register(self)
class TestTxnException(Exception):
pass
......@@ -635,7 +632,7 @@ def test_join():
"""
def test_suite():
from doctest import DocTestSuite
from zope.testing.doctest import DocTestSuite
return unittest.TestSuite((
DocTestSuite(),
unittest.makeSuite(TransactionTests),
......
......@@ -16,7 +16,7 @@
$Id$
"""
import unittest
from doctest import DocTestSuite
from zope.testing.doctest import DocTestSuite
def test_suite():
return DocTestSuite('transaction.util')
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment