Commit 81f586c4 authored by Jeremy Hylton's avatar Jeremy Hylton

Merge ZEO2-branch to trunk.

parent d09403de
...@@ -10,20 +10,23 @@ ClientStorage ...@@ -10,20 +10,23 @@ ClientStorage
Creating a ClientStorage Creating a ClientStorage
At a minimum, a client storage requires an argument (named The ClientStorage requires at leats one argument, the address or
connection) giving connection information. This argument should be addresses of the server(s) to use. It accepts several other
a string, specifying a unix-domain socket file name, or a tuple optional keyword arguments.
consisting of a host and port. The host should be a string host
name or IP number. The port should be a numeric port number.
The ClientStorage constructor provides a number of additional The address argument can be one of:
options (arguments). The full list of arguments is:
- a tuple containing hostname and port number
- a string specifying the path to a Unix domain socket
- a sequence of the previous two
connection -- Connection information. If a sequence of addresses is specified, the client will use the
first server from the list that it can connect to.
This argument is either a string containing a socket file name The ClientStorage constructor provides a number of additional
or a tuple consisting of a string host name or ip number and an options (arguments). The full list of arguments is:
integer port.
storage -- The name of the storage to connect to. storage -- The name of the storage to connect to.
...@@ -33,7 +36,9 @@ ClientStorage ...@@ -33,7 +36,9 @@ ClientStorage
default name for both the server and client is '1'. default name for both the server and client is '1'.
cache_size -- The number of bytes to allow for the client cache. cache_size -- The number of bytes to allow for the client cache.
The default is 20,000,000. The default is 20,000,000. A large cache can significantly
increase the performance of a ZEO system. For applications that
have a large database, the default size may be too small.
For more information on client caches, see ClientCache.txt. For more information on client caches, see ClientCache.txt.
...@@ -54,10 +59,6 @@ ClientStorage ...@@ -54,10 +59,6 @@ ClientStorage
For more information on client cache files, see ClientCache.txt. For more information on client cache files, see ClientCache.txt.
debug -- If this is provided, it should be a non-empty string. It
indicates that client should log tracing and debugging
information, using zLOG.
var -- The directory in which persistent cache files should be var -- The directory in which persistent cache files should be
written. If this option is provided, it is unnecessary to written. If this option is provided, it is unnecessary to
set INSTANCE_HOME in __builtins__. set INSTANCE_HOME in __builtins__.
...@@ -82,6 +83,13 @@ ClientStorage ...@@ -82,6 +83,13 @@ ClientStorage
The default is 300 seconds. The default is 300 seconds.
wait_for_server_on_starup -- Indicate whether the ClientStorage wait -- Indicate whether the ClientStorage should block waiting
should block waiting for a storage server connection, or whether for a storage server connection, or whether it should proceed,
it should proceed, satisfying reads from the client cache. satisfying reads from the client cache.
read_only -- Open a read-only connection to the server. If the
client attempts to commit a transaction, it will get a
ReadOnlyError exception.
Each storage served by a ZEO server can be configured as either
read-write or read-only.
Zope Enterprize Objects Zope Enterprize Objects
ZEO 1.0 requires Python 2.0 when used without Zope. It depends on Installation
versions of asyncore and cPickle that were first released with
Python 2.0.
Put the ZEO package in a directory on your Python path. On a Unix ZEO 2.0 requires Python 2.1 or higher when used without Zope. If
system, you can use the site-packages directory of your Python lib you use Python 2.1, we recommend the latest minor release (2.1.3 as
directory. The ZEO package is the directory named ZEO that contains of this writing) because it includes a few bug fixes that affect
an __init__.py file. ZEO.
Starting (and configuring) the ZEO Server ZEO is packaged with distutils. To install it, run this command
from the top-level ZEO directory::
python setup.py install
The setup script will install the ZEO package in your Python
site-packages directory.
You can test ZEO before installing it with the test script::
To start the storage server, run the start.py script contained in python test.py -v
the ZEO package. You can run the script from the package
directory or copy it to a directory on your path. Run the script with the -h option for a full list of options. The
ZEO 2.0a1 release contains 87 unit tests on Unix.
Starting (and configuring) the ZEO Server
Specify the port number when you run the script:: To start the storage server, go to your Zope install directory and
run::
python ZEO/start.py -p port_number python lib/python/ZEO/start.py -p port_number
Or run start.py without arguments to see options. The options are This run the storage sever under zdaemon. zdaemon automatically
documented in start.txt. restarts programs that exit unexpectedly.
The server and the client don't have to be on the same machine. The server and the client don't have to be on the same machine.
If the server and client *are* on the same machine, then you can If they are on the same machine, then you can use a Unix domain
use a Unix domain socket:: socket::
python ZEO/start.py -U filename python lib/python/ZEO/start.py -U filename
The start script provides a number of options not documented here.
See doc/start.txt for more information.
Running a ZEO client Running a ZEO client
In your application, create a ClientStorage, rather than, say, a In your application, create a ClientStorage, rather than, say, a
FileStorage: FileStorage:
import ZODB, ZEO.ClientStorage import ZODB
Storage=ZEO.ClientStorage.ClientStorage(('',port_number)) from ZEO.ClientStorage import ClientStorage
db=ZODB.DB(Storage) Storage = ClientStorage(('', port_number))
db = ZODB.DB(Storage)
You can specify a host name (rather than '') if you want. The port You can specify a host name (rather than '') if you want. The port
number is, of course, the port number used to start the storage number is, of course, the port number used to start the storage
...@@ -43,38 +57,24 @@ Zope Enterprize Objects ...@@ -43,38 +57,24 @@ Zope Enterprize Objects
You can also give the name of a Unix domain socket file:: You can also give the name of a Unix domain socket file::
import ZODB, ZEO.ClientStorage import ZODB
Storage=ZEO.ClientStorage.ClientStorage(filename) from ZEO.ClientStorage import ClientStorage
db=ZODB.DB(Storage) Storage = ClientStorage(filename)
db = ZODB.DB(Storage)
There are a number of configuration options available for the There are a number of configuration options available for the
ClientStorage. See ClientStorage.txt for details. ClientStorage. See ClientStorage.txt for details.
If you want a persistent client cache which retains cache contents If you want a persistent client cache which retains cache contents
across ClientStorage restarts, you need to define the environment across ClientStorage restarts, you need to define the environment
variable, ZEO_CLIENT, to a unique name for the client. This is variable, ZEO_CLIENT, or set the client keyword argument to the
needed so that unique cache name files can be computed. Otherwise, constructor to a unique name for the client. This is needed so
the client cache is stored in temporary files which are removed when that unique cache name files can be computed. Otherwise, the
client cache is stored in temporary files which are removed when
the ClientStorage shuts down. the ClientStorage shuts down.
Dependencies on other modules Dependencies on other modules
- The module ThreadedAsync must be on the Python path. ZEO depends on other modules that are distributed with
StandaloneZODB and with Zope. You can download StandaloneZODB
- The zdaemon module is necessary if you want to run your from http://www.zope.org/Products/StandaloneZODB.
storage server as a daemon that automatically restarts itself
if there is a fatal error.
- The zLOG module provides a handy logging capability.
If you are using a version of Python before Python 2:
- ZServer should be in the Python path, or you should copy the
version of asyncore.py from ZServer (from Zope 2.2 or CVS) to
your Python path, or you should copy a version of a asyncore
from the medusa CVS tree to your Python path. A recent change
in asyncore is required.
- The version of cPickle from Zope, or from the python.org CVS
tree must be used. It has a hook to provide control over which
"global objects" (e.g. classes) may be pickled.
...@@ -2,30 +2,38 @@ Zope Enterprise Objects ...@@ -2,30 +2,38 @@ Zope Enterprise Objects
Installation Installation
ZEO 1.0 requires Zope 2.2 or higher. ZEO 2.0 requires Zope 2.4 or higher and Python 2.1 or higher.
If you use Python 2.1, we recommend the latest minor release
(2.1.3 as of this writing) because it includes a few bug fixes
that affect ZEO.
Put this package (the ZEO directory, without any wrapping directory Put the package (the ZEO directory, without any wrapping directory
included in a distribution) in your Zope lib/python. included in a distribution) in your Zope lib/python.
If you are using Python 1.5.2, the lib/python/ZODB directory must The setup.py script in the top-level ZEO directory can also be
contain a cPickle.so (Unix) or cPickle.pyd (Windows) file. In used. Run "python setup.py install --home=ZOPE" where ZOPE is the
many cases, the Zope installation process will not place this file top-level Zope directory.
in the right location. You may need to copy it from lib/python to
lib/python/ZODB. You can test ZEO before installing it with the test script::
python test.py -v
Run the script with the -h option for a full list of options. The
ZEO 2.0a1 release contains 87 unit tests on Unix.
Starting (and configuring) the ZEO Server Starting (and configuring) the ZEO Server
To start the storage server, go to your Zope install directory and:: To start the storage server, go to your Zope install directory and
run::
python lib/python/ZEO/start.py -p port_number python lib/python/ZEO/start.py -p port_number
(Run start without arguments to see options.) This run the storage sever under zdaemon. zdaemon automatically
restarts programs that exit unexpectedly.
Of course, the server and the client don't have to be on the same The server and the client don't have to be on the same machine.
machine. If they are on the same machine, then you can use a Unix domain
socket::
If the server and client *are* on the same machine, then you can use
a Unix domain socket::
python lib/python/ZEO/start.py -U filename python lib/python/ZEO/start.py -U filename
...@@ -38,10 +46,8 @@ Zope Enterprise Objects ...@@ -38,10 +46,8 @@ Zope Enterprise Objects
custom_zodb.py, in your Zope install directory, so that Zope uses a custom_zodb.py, in your Zope install directory, so that Zope uses a
ClientStorage:: ClientStorage::
import ZEO.ClientStorage from ZEO.ClientStorage import ClientStorage
Storage=ZEO.ClientStorage.ClientStorage(('',port_number)) Storage = ClientStorage(('', port_number))
(See the misc/custom_zodb.py for an example.)
You can specify a host name (rather than '') if you want. The port You can specify a host name (rather than '') if you want. The port
number is, of course, the port number used to start the storage number is, of course, the port number used to start the storage
...@@ -49,19 +55,20 @@ Zope Enterprise Objects ...@@ -49,19 +55,20 @@ Zope Enterprise Objects
You can also give the name of a Unix domain socket file:: You can also give the name of a Unix domain socket file::
import ZEO.ClientStorage from ZEO.ClientStorage import ClientStorage
Storage=ZEO.ClientStorage.ClientStorage(filename) Storage = ClientStorage(filename)
There are a number of configuration options available for the There are a number of configuration options available for the
ClientStorage. See doc/ClientStorage.txt for details. ClientStorage. See doc/ClientStorage.txt for details.
If you want a persistent client cache which retains cache contents If you want a persistent client cache which retains cache contents
across ClientStorage restarts, you need to define the environment across ClientStorage restarts, you need to define the environment
variable, ZEO_CLIENT, to a unique name for the client. This is variable, ZEO_CLIENT, or set the client keyword argument to the
needed so that unique cache name files can be computed. Otherwise, constructor to a unique name for the client. This is needed so
the client cache is stored in temporary files which are removed when that unique cache name files can be computed. Otherwise, the
client cache is stored in temporary files which are removed when
the ClientStorage shuts down. For example, to start two Zope the ClientStorage shuts down. For example, to start two Zope
processes with unique caches, use something like: processes with unique caches, use something like::
python z2.py -P8700 ZEO_CLIENT=8700 python z2.py -P8700 ZEO_CLIENT=8700
python z2.py -P8800 ZEO_CLIENT=8800 python z2.py -P8800 ZEO_CLIENT=8800
...@@ -74,9 +81,8 @@ Zope Enterprise Objects ...@@ -74,9 +81,8 @@ Zope Enterprise Objects
different clients have different software installed, the correct different clients have different software installed, the correct
state of the database is ambiguous. state of the database is ambiguous.
Starting in Zope 2.2, Zope will not modify the Zope database Zope will not modify the Zope database during product installation
during product installation if the environment variable ZEO_CLIENT if the environment variable ZEO_CLIENT is set.
is set.
Normally, Zope ZEO clients should be run with ZEO_CLIENT set so Normally, Zope ZEO clients should be run with ZEO_CLIENT set so
that product initialization is not performed. that product initialization is not performed.
......
...@@ -73,145 +73,183 @@ file 0 and file 1. ...@@ -73,145 +73,183 @@ file 0 and file 1.
""" """
__version__ = "$Revision: 1.22 $"[11:-2] __version__ = "$Revision: 1.23 $"[11:-2]
import os, tempfile import os
import sys
import tempfile
from struct import pack, unpack from struct import pack, unpack
from thread import allocate_lock from thread import allocate_lock
import zLOG
magic='ZEC0' import zLOG
from ZEO.ICache import ICache
def LOG(msg, level=zLOG.BLATHER): def log(msg, level=zLOG.INFO):
zLOG.LOG("ZEC", level, msg) zLOG.LOG("ZEC", level, msg)
magic='ZEC0'
class ClientCache: class ClientCache:
__implements__ = ICache
def __init__(self, storage='', size=20000000, client=None, var=None): def __init__(self, storage='', size=20000000, client=None, var=None):
# Allocate locks: # Allocate locks:
l=allocate_lock() L = allocate_lock()
self._acquire=l.acquire self._acquire = L.acquire
self._release=l.release self._release = L.release
if client: if client:
# Create a persistent cache # Create a persistent cache
if var is None: if var is None:
try: var=CLIENT_HOME try:
var = CLIENT_HOME
except: except:
try: var=os.path.join(INSTANCE_HOME,'var') try:
except: var=os.getcwd() var = os.path.join(INSTANCE_HOME, 'var')
except:
var = os.getcwd()
# Get the list of cache file names # Get the list of cache file names
self._p=p=map(lambda i, p=storage, var=var, c=client: self._p = p = map(lambda i, p=storage, var=var, c=client:
os.path.join(var,'c%s-%s-%s.zec' % (p, c, i)), os.path.join(var, 'c%s-%s-%s.zec' % (p, c, i)),
(0,1)) (0, 1))
# get the list of cache files # get the list of cache files
self._f=f=[None, None] self._f = f = [None, None]
# initialize cache serial numbers # initialize cache serial numbers
s=['\0\0\0\0\0\0\0\0', '\0\0\0\0\0\0\0\0'] s=['\0\0\0\0\0\0\0\0', '\0\0\0\0\0\0\0\0']
for i in 0,1: for i in 0, 1:
if os.path.exists(p[i]): if os.path.exists(p[i]):
fi=open(p[i],'r+b') fi = open(p[i],'r+b')
if fi.read(4)==magic: # Minimal sanity if fi.read(4) == magic: # Minimal sanity
fi.seek(0,2) fi.seek(0, 2)
if fi.tell() > 30: if fi.tell() > 30:
fi.seek(22) fi.seek(22)
s[i]=fi.read(8) s[i] = fi.read(8)
# If we found a non-zero serial, then use the file # If we found a non-zero serial, then use the file
if s[i] != '\0\0\0\0\0\0\0\0': f[i]=fi if s[i] != '\0\0\0\0\0\0\0\0':
fi=None f[i] = fi
fi = None
# Whoever has the larger serial is the current # Whoever has the larger serial is the current
if s[1] > s[0]: current=1 if s[1] > s[0]:
elif s[0] > s[1]: current=0 current = 1
elif s[0] > s[1]:
current = 0
else: else:
if f[0] is None: if f[0] is None:
# We started, open the first cache file # We started, open the first cache file
f[0]=open(p[0], 'w+b') f[0] = open(p[0], 'w+b')
f[0].write(magic) f[0].write(magic)
current=0 current = 0
f[1]=None f[1] = None
else: else:
self._f = f = [tempfile.TemporaryFile(suffix='.zec'), None] self._f = f = [tempfile.TemporaryFile(suffix='.zec'), None]
# self._p file names 'None' signifies unnamed temp files. # self._p file name 'None' signifies an unnamed temp file.
self._p = p = [None, None] self._p = p = [None, None]
f[0].write(magic) f[0].write(magic)
current=0 current = 0
self._limit=size/2 log("cache opened. current = %s" % current)
self._current=current
def close(self): self._limit = size / 2
try: self._current = current
self._f[self._current].close()
except (os.error, ValueError):
pass
def open(self): def open(self):
# XXX open is overloaded to perform two tasks for
# optimization reasons
self._acquire() self._acquire()
try: try:
self._index=index={} self._index=index={}
self._get=index.get self._get = index.get
serial={} serial = {}
f=self._f f = self._f
current=self._current current = self._current
if f[not current] is not None: if f[not current] is not None:
read_index(index, serial, f[not current], not current) read_index(index, serial, f[not current], not current)
self._pos=read_index(index, serial, f[current], current) self._pos = read_index(index, serial, f[current], current)
return serial.items() return serial.items()
finally: self._release() finally:
self._release()
def close(self):
for f in self._f:
if f is not None:
# In 2.1 on Windows, the TemporaryFileWrapper doesn't allow
# closing a file more than once.
try:
f.close()
except OSError:
pass
def verify(self, verifyFunc):
"""Call the verifyFunc on every object in the cache.
verifyFunc(oid, serialno, version)
"""
for oid, (s, vs) in self.open():
verifyFunc(oid, s, vs)
def invalidate(self, oid, version): def invalidate(self, oid, version):
self._acquire() self._acquire()
try: try:
p=self._get(oid, None) p = self._get(oid, None)
if p is None: return None if p is None:
f=self._f[p < 0] return None
ap=abs(p) f = self._f[p < 0]
ap = abs(p)
f.seek(ap) f.seek(ap)
h=f.read(8) h = f.read(8)
if h != oid: return if h != oid:
f.seek(8,1) # Dang, we shouldn't have to do this. Bad Solaris & NT return
f.seek(8, 1) # Dang, we shouldn't have to do this. Bad Solaris & NT
if version: if version:
f.write('n') f.write('n')
else: else:
del self._index[oid] del self._index[oid]
f.write('i') f.write('i')
finally: self._release() finally:
self._release()
def load(self, oid, version): def load(self, oid, version):
self._acquire() self._acquire()
try: try:
p=self._get(oid, None) p = self._get(oid, None)
if p is None: return None if p is None:
f=self._f[p < 0] return None
ap=abs(p) f = self._f[p < 0]
seek=f.seek ap = abs(p)
read=f.read seek = f.seek
read = f.read
seek(ap) seek(ap)
h=read(27) h = read(27)
if len(h)==27 and h[8] in 'nv' and h[:8]==oid: if len(h)==27 and h[8] in 'nv' and h[:8]==oid:
tlen, vlen, dlen = unpack(">iHi", h[9:19]) tlen, vlen, dlen = unpack(">iHi", h[9:19])
else: tlen=-1 else:
tlen = -1
if tlen <= 0 or vlen < 0 or dlen < 0 or vlen+dlen > tlen: if tlen <= 0 or vlen < 0 or dlen < 0 or vlen+dlen > tlen:
del self._index[oid] del self._index[oid]
return None return None
if h[8]=='n': if h[8]=='n':
if version: return None if version:
return None
if not dlen: if not dlen:
del self._index[oid] del self._index[oid]
return None return None
if not vlen or not version: if not vlen or not version:
if dlen: return read(dlen), h[19:] if dlen:
else: return None return read(dlen), h[19:]
else:
return None
if dlen: seek(dlen, 1) if dlen:
v=read(vlen) seek(dlen, 1)
v = read(vlen)
if version != v: if version != v:
if dlen: if dlen:
seek(-dlen-vlen, 1) seek(-dlen-vlen, 1)
...@@ -219,24 +257,25 @@ class ClientCache: ...@@ -219,24 +257,25 @@ class ClientCache:
else: else:
return None return None
dlen=unpack(">i", read(4))[0] dlen = unpack(">i", read(4))[0]
return read(dlen), read(8) return read(dlen), read(8)
finally: self._release() finally:
self._release()
def update(self, oid, serial, version, data): def update(self, oid, serial, version, data):
self._acquire() self._acquire()
try: try:
if version: if version:
# We need to find and include non-version data # We need to find and include non-version data
p=self._get(oid, None) p = self._get(oid, None)
if p is None: if p is None:
return self._store(oid, '', '', version, data, serial) return self._store(oid, '', '', version, data, serial)
f=self._f[p < 0] f = self._f[p < 0]
ap=abs(p) ap = abs(p)
seek=f.seek seek = f.seek
read=f.read read = f.read
seek(ap) seek(ap)
h=read(27) h = read(27)
if len(h)==27 and h[8] in 'nv' and h[:8]==oid: if len(h)==27 and h[8] in 'nv' and h[:8]==oid:
tlen, vlen, dlen = unpack(">iHi", h[9:19]) tlen, vlen, dlen = unpack(">iHi", h[9:19])
else: else:
...@@ -246,8 +285,8 @@ class ClientCache: ...@@ -246,8 +285,8 @@ class ClientCache:
return self._store(oid, '', '', version, data, serial) return self._store(oid, '', '', version, data, serial)
if dlen: if dlen:
p=read(dlen) p = read(dlen)
s=h[19:] s = h[19:]
else: else:
return self._store(oid, '', '', version, data, serial) return self._store(oid, '', '', version, data, serial)
...@@ -255,41 +294,47 @@ class ClientCache: ...@@ -255,41 +294,47 @@ class ClientCache:
else: else:
# Simple case, just store new data: # Simple case, just store new data:
self._store(oid, data, serial, '', None, None) self._store(oid, data, serial, '', None, None)
finally: self._release() finally:
self._release()
def modifiedInVersion(self, oid): def modifiedInVersion(self, oid):
self._acquire() self._acquire()
try: try:
p=self._get(oid, None) p = self._get(oid, None)
if p is None: return None if p is None:
f=self._f[p < 0] return None
ap=abs(p) f = self._f[p < 0]
seek=f.seek ap = abs(p)
read=f.read seek = f.seek
read = f.read
seek(ap) seek(ap)
h=read(27) h = read(27)
if len(h)==27 and h[8] in 'nv' and h[:8]==oid: if len(h)==27 and h[8] in 'nv' and h[:8]==oid:
tlen, vlen, dlen = unpack(">iHi", h[9:19]) tlen, vlen, dlen = unpack(">iHi", h[9:19])
else: tlen=-1 else:
tlen = -1
if tlen <= 0 or vlen < 0 or dlen < 0 or vlen+dlen > tlen: if tlen <= 0 or vlen < 0 or dlen < 0 or vlen+dlen > tlen:
del self._index[oid] del self._index[oid]
return None return None
if h[8]=='n': return None if h[8] == 'n':
return None
if not vlen: return '' if not vlen:
return ''
seek(dlen, 1) seek(dlen, 1)
return read(vlen) return read(vlen)
finally: self._release() finally:
self._release()
def checkSize(self, size): def checkSize(self, size):
self._acquire() self._acquire()
try: try:
# Make sure we aren't going to exceed the target size. # Make sure we aren't going to exceed the target size.
# If we are, then flip the cache. # If we are, then flip the cache.
if self._pos+size > self._limit: if self._pos + size > self._limit:
current=not self._current current = not self._current
self._current=current self._current = current
if self._p[current] is not None: if self._p[current] is not None:
# Persistent cache file: # Persistent cache file:
# Note that due to permission madness, waaa, # Note that due to permission madness, waaa,
...@@ -297,90 +342,100 @@ class ClientCache: ...@@ -297,90 +342,100 @@ class ClientCache:
# we open the new one. Waaaaaaaaaa. # we open the new one. Waaaaaaaaaa.
if self._f[current] is not None: if self._f[current] is not None:
self._f[current].close() self._f[current].close()
try: os.remove(self._p[current]) try:
except: pass os.remove(self._p[current])
self._f[current]=open(self._p[current],'w+b') except:
pass
self._f[current] = open(self._p[current],'w+b')
else: else:
# Temporary cache file: # Temporary cache file:
if self._f[current] is not None:
self._f[current].close()
self._f[current] = tempfile.TemporaryFile(suffix='.zec') self._f[current] = tempfile.TemporaryFile(suffix='.zec')
self._f[current].write(magic) self._f[current].write(magic)
self._pos=pos=4 self._pos = pos = 4
finally: self._release() finally:
self._release()
def store(self, oid, p, s, version, pv, sv): def store(self, oid, p, s, version, pv, sv):
self._acquire() self._acquire()
try: self._store(oid, p, s, version, pv, sv) try:
finally: self._release() self._store(oid, p, s, version, pv, sv)
finally:
self._release()
def _store(self, oid, p, s, version, pv, sv): def _store(self, oid, p, s, version, pv, sv):
if not s: if not s:
p='' p = ''
s='\0\0\0\0\0\0\0\0' s = '\0\0\0\0\0\0\0\0'
tlen=31+len(p) tlen = 31 + len(p)
if version: if version:
tlen=tlen+len(version)+12+len(pv) tlen = tlen + len(version) + 12 + len(pv)
vlen=len(version) vlen = len(version)
else: else:
vlen=0 vlen = 0
pos=self._pos stlen = pack(">I", tlen)
current=self._current # accumulate various data to write into a list
f=self._f[current] l = [oid, 'v', stlen, pack(">HI", vlen, len(p)), s]
f.seek(pos) if p:
stlen=pack(">I",tlen) l.append(p)
write=f.write
write(oid+'v'+stlen+pack(">HI", vlen, len(p))+s)
if p: write(p)
if version: if version:
write(version) l.extend([version,
write(pack(">I", len(pv))) pack(">I", len(pv)),
write(pv) pv, sv])
write(sv) l.append(stlen)
f = self._f[self._current]
write(stlen) f.seek(self._pos)
f.write("".join(l))
if current: self._index[oid]=-pos
else: self._index[oid]=pos if self._current:
self._index[oid] = - self._pos
else:
self._index[oid] = self._pos
self._pos=pos+tlen self._pos += tlen
def read_index(index, serial, f, current): def read_index(index, serial, f, current):
LOG("read_index(%s)" % f.name) seek = f.seek
seek=f.seek read = f.read
read=f.read pos = 4
pos=4 seek(0, 2)
size = f.tell()
while 1: while 1:
seek(pos) f.seek(pos)
h=read(27) h = read(27)
if len(h)==27 and h[8] in 'vni': if len(h)==27 and h[8] in 'vni':
tlen, vlen, dlen = unpack(">iHi", h[9:19]) tlen, vlen, dlen = unpack(">iHi", h[9:19])
else: else:
break tlen = -1
if tlen <= 0 or vlen < 0 or dlen < 0 or vlen+dlen > tlen: if tlen <= 0 or vlen < 0 or dlen < 0 or vlen + dlen > tlen:
break break
oid=h[:8] oid = h[:8]
if h[8]=='v' and vlen: if h[8]=='v' and vlen:
seek(dlen+vlen, 1) seek(dlen+vlen, 1)
vdlen=read(4) vdlen = read(4)
if len(vdlen) != 4: break if len(vdlen) != 4:
vdlen=unpack(">i", vdlen)[0] break
if vlen+dlen+42+vdlen > tlen: break vdlen = unpack(">i", vdlen)[0]
if vlen+dlen+42+vdlen > tlen:
break
seek(vdlen, 1) seek(vdlen, 1)
vs=read(8) vs = read(8)
if read(4) != h[9:13]: break if read(4) != h[9:13]:
else: vs=None break
else:
vs = None
if h[8] in 'vn': if h[8] in 'vn':
if current: index[oid]=-pos if current:
else: index[oid]=pos index[oid] = -pos
serial[oid]=h[-8:], vs else:
index[oid] = pos
serial[oid] = h[-8:], vs
else: else:
if serial.has_key(oid): if serial.has_key(oid):
# We have a record for this oid, but it was invalidated! # We have a record for this oid, but it was invalidated!
...@@ -388,22 +443,12 @@ def read_index(index, serial, f, current): ...@@ -388,22 +443,12 @@ def read_index(index, serial, f, current):
del index[oid] del index[oid]
pos=pos+tlen pos = pos + tlen
f.seek(pos) f.seek(pos)
try: f.truncate() try:
except: pass f.truncate()
except:
pass
return pos return pos
def main(files):
for file in files:
print file
index = {}
serial = {}
read_index(index, serial, open(file), 0)
print index.keys()
if __name__ == "__main__":
import sys
main(sys.argv[1:])
...@@ -13,185 +13,166 @@ ...@@ -13,185 +13,166 @@
############################################################################## ##############################################################################
"""Network ZODB storage client """Network ZODB storage client
""" """
__version__='$Revision: 1.40 $'[11:-2] __version__='$Revision: 1.41 $'[11:-2]
import struct, time, os, socket, string import cPickle
import tempfile, thread import os
from struct import pack, unpack import tempfile
from types import TupleType import threading
import time
import Invalidator, ExtensionClass from ZEO import ClientCache, ServerStub
import ThreadedAsync, Sync, zrpc, ClientCache from ZEO.TransactionBuffer import TransactionBuffer
from ZEO.Exceptions import Disconnected
from ZEO.zrpc.client import ConnectionManager
from ZODB import POSException, BaseStorage from ZODB import POSException
from ZODB.TimeStamp import TimeStamp from ZODB.TimeStamp import TimeStamp
from zLOG import LOG, PROBLEM, INFO, BLATHER
from ZEO.logger import zLogger def log2(type, msg, subsys="ClientStorage %d" % os.getpid()):
LOG(subsys, type, msg)
log = zLogger("ZEO Client")
try: try:
from ZODB.ConflictResolution import ResolvedSerial from ZODB.ConflictResolution import ResolvedSerial
except: except ImportError:
ResolvedSerial='rs' ResolvedSerial = 'rs'
class ClientStorageError(POSException.StorageError): class ClientStorageError(POSException.StorageError):
"""An error occured in the ZEO Client Storage""" """An error occured in the ZEO Client Storage"""
class UnrecognizedResult(ClientStorageError): class UnrecognizedResult(ClientStorageError):
"""A server call returned an unrecognized result """A server call returned an unrecognized result"""
"""
class ClientDisconnected(ClientStorageError):
"""The database storage is disconnected from the storage.
"""
class ClientStorage(ExtensionClass.Base, BaseStorage.BaseStorage):
_connected=_async=0
__begin='tpc_begin_sync'
def __init__(self, connection, storage='1', cache_size=20000000,
name='', client='', debug=0, var=None,
min_disconnect_poll=5, max_disconnect_poll=300,
wait_for_server_on_startup=1):
# Decide whether to use non-temporary files
client=client or os.environ.get('ZEO_CLIENT','')
self._connection=connection
self._storage=storage
self._debug=debug
self._wait_for_server_on_startup=wait_for_server_on_startup
self._info={'length': 0, 'size': 0, 'name': 'ZEO Client',
'supportsUndo':0, 'supportsVersions': 0,
}
if debug:
debug_log = log
else:
debug_log = None
self._call=zrpc.asyncRPC(connection, debug=debug_log,
tmin=min_disconnect_poll,
tmax=max_disconnect_poll)
name = name or str(connection)
self.closed = 0
self._tfile=tempfile.TemporaryFile()
self._oids=[]
self._serials=[]
self._seriald={}
ClientStorage.inheritedAttribute('__init__')(self, name)
self.__lock_acquire=self._lock_acquire
self._cache=ClientCache.ClientCache( class ClientDisconnected(ClientStorageError, Disconnected):
storage, cache_size, client=client, var=var) """The database storage is disconnected from the storage."""
def get_timestamp(prev_ts=None):
t = time.time()
t = apply(TimeStamp, (time.gmtime(t)[:5] + (t % 60,)))
if prev_ts is not None:
t = t.laterThan(prev_ts)
return t
ThreadedAsync.register_loop_callback(self.becomeAsync) class DisconnectedServerStub:
"""Raise ClientDisconnected on all attribute access."""
# IMPORTANT: Note that we aren't fully "there" yet. def __getattr__(self, attr):
# In particular, we don't actually connect to the server raise ClientDisconnected()
# until we have a controlling database set with registerDB
# below.
def registerDB(self, db, limit): disconnected_stub = DisconnectedServerStub()
"""Register that the storage is controlled by the given DB.
"""
# Among other things, we know that our data methods won't get class ClientStorage:
# called until after this call.
self.invalidator = Invalidator.Invalidator(db.invalidate, def __init__(self, addr, storage='1', cache_size=20000000,
self._cache.invalidate) name='', client='', var=None,
min_disconnect_poll=5, max_disconnect_poll=300,
wait=0, read_only=0):
def out_of_band_hook( self._server = disconnected_stub
code, args, self._is_read_only = read_only
get_hook={ self._storage = storage
'b': (self.invalidator.begin, 0),
'i': (self.invalidator.invalidate, 1),
'e': (self.invalidator.end, 0),
'I': (self.invalidator.Invalidate, 1),
'U': (self._commit_lock_release, 0),
's': (self._serials.append, 1),
'S': (self._info.update, 1),
}.get):
hook = get_hook(code, None) self._info = {'length': 0, 'size': 0, 'name': 'ZEO Client',
if hook is None: return 'supportsUndo':0, 'supportsVersions': 0}
hook, flag = hook
if flag: hook(args)
else: hook()
self._call.setOutOfBand(out_of_band_hook) self._tbuf = TransactionBuffer()
self._db = None
self._oids = []
# _serials: stores (oid, serialno) as returned by server
# _seriald: _check_serials() moves from _serials to _seriald,
# which maps oid to serialno
self._serials = []
self._seriald = {}
# Now that we have our callback system in place, we can self._basic_init(name or str(addr))
# try to connect
self._startup() # Decide whether to use non-temporary files
client = client or os.environ.get('ZEO_CLIENT', '')
self._cache = ClientCache.ClientCache(storage, cache_size,
client=client, var=var)
self._cache.open() # XXX open now? or later?
def _startup(self): self._rpc_mgr = ConnectionManager(addr, self,
tmin=min_disconnect_poll,
tmax=max_disconnect_poll)
if not self._call.connect(not self._wait_for_server_on_startup): # XXX What if we can only get a read-only connection and we
# want a read-write connection? Looks like the current code
# will block forever. (Future feature)
if wait:
self._rpc_mgr.connect(sync=1)
else:
if not self._rpc_mgr.attempt_connect():
self._rpc_mgr.connect()
# If we can't connect right away, go ahead and open the cache def _basic_init(self, name):
# and start a separate thread to try and reconnect. """Handle initialization activites of BaseStorage"""
log.problem("Failed to connect to storage") # XXX does anything depend on attr being __name__
self._cache.open() self.__name__ = name
thread.start_new_thread(self._call.connect,(0,))
# If the connect succeeds then this work will be done by # A ClientStorage only allows one client to commit at a time.
# notifyConnected # A client enters the commit state by finding tpc_tid set to
# None and updating it to the new transaction's id. The
# tpc_tid variable is protected by tpc_cond.
self.tpc_cond = threading.Condition()
self._transaction = None
def notifyConnected(self, s): # Prevent multiple new_oid calls from going out. The _oids
log.info("Connected to storage") # variable should only be modified while holding the
self._lock_acquire() # oid_cond.
try: self.oid_cond = threading.Condition()
# We let the connection keep coming up now that commit_lock = threading.Lock()
# we have the storage lock. This way, we know no calls self._commit_lock_acquire = commit_lock.acquire
# will be made while in the process of coming up. self._commit_lock_release = commit_lock.release
self._call.finishConnect(s) t = self._ts = get_timestamp()
self._serial = `t`
self._oid='\0\0\0\0\0\0\0\0'
if self.closed: def close(self):
return if self._tbuf is not None:
self._tbuf.close()
if self._cache is not None:
self._cache.close()
self._rpc_mgr.close()
self._connected=1 def registerDB(self, db, limit):
self._oids=[] """Register that the storage is controlled by the given DB."""
log2(INFO, "registerDB(%s, %s)" % (repr(db), repr(limit)))
self._db = db
# we do synchronous commits until we are sure that def is_connected(self):
# we have and are ready for a main loop. if self._server is disconnected_stub:
return 0
else:
return 1
# Hm. This is a little silly. If self._async, then def notifyConnected(self, c):
# we will really never do a synchronous commit. log2(INFO, "Connected to storage via %s" % repr(c))
# See below.
self.__begin='tpc_begin_sync'
self._call.message_output(str(self._storage)) # check the protocol version here?
### This seems silly. We should get the info asynchronously. stub = ServerStub.StorageServer(c)
# self._info.update(self._call('get_info'))
cached=self._cache.open() self._oids = []
### This is a little expensive for large caches
if cached:
self._call.sendMessage('beginZeoVerify')
for oid, (s, vs) in cached:
self._call.sendMessage('zeoVerify', oid, s, vs)
self._call.sendMessage('endZeoVerify')
finally: self._lock_release() # XXX Why is this synchronous? If it were async, verification
# would start faster.
stub.register(str(self._storage), self._is_read_only)
self._info.update(stub.get_info())
self.verify_cache(stub)
if self._async: # Don't make the server available to clients until after
import asyncore # validating the cache
self.becomeAsync(asyncore.socket_map) self._server = stub
def verify_cache(self, server):
server.beginZeoVerify()
self._cache.verify(server.zeoVerify)
server.endZeoVerify()
### Is there a race condition between notifyConnected and ### Is there a race condition between notifyConnected and
### notifyDisconnected? In Particular, what if we get ### notifyDisconnected? In Particular, what if we get
...@@ -201,370 +182,334 @@ class ClientStorage(ExtensionClass.Base, BaseStorage.BaseStorage): ...@@ -201,370 +182,334 @@ class ClientStorage(ExtensionClass.Base, BaseStorage.BaseStorage):
### notifyDisconnected had to get the instance lock. There's ### notifyDisconnected had to get the instance lock. There's
### nothing to gain by getting the instance lock. ### nothing to gain by getting the instance lock.
### Note that we *don't* have to worry about getting connected def notifyDisconnected(self):
### in the middle of notifyDisconnected, because *it's* log2(PROBLEM, "Disconnected from storage")
### responsible for starting the thread that makes the connection. self._server = disconnected_stub
def notifyDisconnected(self, ignored): def __len__(self):
log.problem("Disconnected from storage") return self._info['length']
self._connected=0
self._transaction=None def getName(self):
thread.start_new_thread(self._call.connect,(0,)) return "%s (%s)" % (self.__name__, "XXX")
if self._transaction is not None:
try:
self._commit_lock_release()
except:
pass
def becomeAsync(self, map): def getSize(self):
self._lock_acquire() return self._info['size']
def supportsUndo(self):
return self._info['supportsUndo']
def supportsVersions(self):
return self._info['supportsVersions']
def supportsTransactionalUndo(self):
try: try:
self._async=1 return self._info['supportsTransactionalUndo']
if self._connected: except KeyError:
self._call.setLoop(map, getWakeup()) return 0
self.__begin='tpc_begin'
finally: self._lock_release() def isReadOnly(self):
return self._is_read_only
def __len__(self): return self._info['length'] def _check_trans(self, trans, exc=None):
if self._transaction is not trans:
if exc is None:
return 0
else:
raise exc(self._transaction, trans)
return 1
def _check_tid(self, tid, exc=None):
if self.tpc_tid != tid:
if exc is None:
return 0
else:
raise exc(self.tpc_tid, tid)
return 1
def abortVersion(self, src, transaction): def abortVersion(self, src, transaction):
if transaction is not self._transaction: if self._is_read_only:
raise POSException.StorageTransactionError(self, transaction) raise POSException.ReadOnlyError()
self._lock_acquire() self._check_trans(transaction,
try: POSException.StorageTransactionError)
oids=self._call('abortVersion', src, self._serial) oids = self._server.abortVersion(src, self._serial)
vlen = pack(">H", len(src))
for oid in oids: for oid in oids:
self._tfile.write("i%s%s%s" % (oid, vlen, src)) self._tbuf.invalidate(oid, src)
return oids return oids
finally: self._lock_release()
def close(self):
self._lock_acquire()
try:
log.info("close")
self._call.closeIntensionally()
try:
self._tfile.close()
except os.error:
# On Windows, this can fail if it is called more than
# once, because it tries to delete the file each
# time.
pass
self._cache.close()
if self.invalidator is not None:
self.invalidator.close()
self.invalidator = None
self.closed = 1
finally: self._lock_release()
def commitVersion(self, src, dest, transaction): def commitVersion(self, src, dest, transaction):
if transaction is not self._transaction: if self._is_read_only:
raise POSException.StorageTransactionError(self, transaction) raise POSException.ReadOnlyError()
self._lock_acquire() self._check_trans(transaction,
try: POSException.StorageTransactionError)
oids=self._call('commitVersion', src, dest, self._serial) oids = self._server.commitVersion(src, dest, self._serial)
if dest: if dest:
vlen = pack(">H", len(src))
# just invalidate our version data # just invalidate our version data
for oid in oids: for oid in oids:
self._tfile.write("i%s%s%s" % (oid, vlen, src)) self._tbuf.invalidate(oid, src)
else: else:
vlen = pack(">H", len(dest))
# dest is '', so invalidate version and non-version # dest is '', so invalidate version and non-version
for oid in oids: for oid in oids:
self._tfile.write("i%s%s%s" % (oid, vlen, dest)) self._tbuf.invalidate(oid, dest)
return oids return oids
finally: self._lock_release()
def getName(self):
return "%s (%s)" % (
self.__name__,
self._connected and 'connected' or 'disconnected')
def getSize(self): return self._info['size']
def history(self, oid, version, length=1): def history(self, oid, version, length=1):
self._lock_acquire() return self._server.history(oid, version, length)
try: return self._call('history', oid, version, length)
finally: self._lock_release()
def loadSerial(self, oid, serial): def loadSerial(self, oid, serial):
self._lock_acquire() return self._server.loadSerial(oid, serial)
try: return self._call('loadSerial', oid, serial)
finally: self._lock_release()
def load(self, oid, version, _stuff=None): def load(self, oid, version, _stuff=None):
self._lock_acquire() p = self._cache.load(oid, version)
try: if p:
cache=self._cache return p
p = cache.load(oid, version) if self._server is None:
if p: return p raise ClientDisconnected()
p, s, v, pv, sv = self._call('zeoLoad', oid) p, s, v, pv, sv = self._server.zeoLoad(oid)
cache.checkSize(0) self._cache.checkSize(0)
cache.store(oid, p, s, v, pv, sv) self._cache.store(oid, p, s, v, pv, sv)
if not v or not version or version != v: if v and version and v == version:
if s: return p, s
raise KeyError, oid # no non-version data for this
return pv, sv return pv, sv
finally: self._lock_release() else:
if s:
return p, s
raise KeyError, oid # no non-version data for this
def modifiedInVersion(self, oid): def modifiedInVersion(self, oid):
self._lock_acquire() v = self._cache.modifiedInVersion(oid)
try: if v is not None:
v=self._cache.modifiedInVersion(oid) return v
if v is not None: return v return self._server.modifiedInVersion(oid)
return self._call('modifiedInVersion', oid)
finally: self._lock_release()
def new_oid(self, last=None): def new_oid(self, last=None):
self._lock_acquire() if self._is_read_only:
try: raise POSException.ReadOnlyError()
oids=self._oids # avoid multiple oid requests to server at the same time
if not oids: self.oid_cond.acquire()
oids[:]=self._call('new_oids') if not self._oids:
oids.reverse() self._oids = self._server.new_oids()
self._oids.reverse()
return oids.pop() self.oid_cond.notifyAll()
finally: self._lock_release() oid = self._oids.pop()
self.oid_cond.release()
return oid
def pack(self, t=None, rf=None, wait=0, days=0): def pack(self, t=None, rf=None, wait=0, days=0):
# Note that we ignore the rf argument. The server if self._is_read_only:
# will provide it's own implementation. raise POSException.ReadOnlyError()
if t is None: t=time.time() # rf argument ignored; server will provide it's own implementation
t=t-(days*86400) if t is None:
self._lock_acquire() t = time.time()
try: return self._call('pack', t, wait) t = t - (days * 86400)
finally: self._lock_release() return self._server.pack(t, wait)
def store(self, oid, serial, data, version, transaction): def _check_serials(self):
if transaction is not self._transaction:
raise POSException.StorageTransactionError(self, transaction)
self._lock_acquire()
try:
serial=self._call.sendMessage('storea', oid, serial,
data, version, self._serial)
write=self._tfile.write
buf = string.join(("s", oid,
pack(">HI", len(version), len(data)),
version, data), "")
write(buf)
if self._serials: if self._serials:
s=self._serials l = len(self._serials)
l=len(s) r = self._serials[:l]
r=s[:l] del self._serials[:l]
del s[:l] for oid, s in r:
d=self._seriald if isinstance(s, Exception):
for oid, s in r: d[oid]=s raise s
self._seriald[oid] = s
return r return r
return serial def store(self, oid, serial, data, version, transaction):
if self._is_read_only:
finally: self._lock_release() raise POSException.ReadOnlyError()
self._check_trans(transaction, POSException.StorageTransactionError)
self._server.storea(oid, serial, data, version, self._serial)
self._tbuf.store(oid, version, data)
return self._check_serials()
def tpc_vote(self, transaction): def tpc_vote(self, transaction):
self._lock_acquire()
try:
if transaction is not self._transaction: if transaction is not self._transaction:
return return
self._call('vote', self._serial) self._server.vote(self._serial)
return self._check_serials()
if self._serials:
s=self._serials
l=len(s)
r=s[:l]
del s[:l]
d=self._seriald
for oid, s in r: d[oid]=s
return r
finally: self._lock_release()
def supportsUndo(self):
return self._info['supportsUndo']
def supportsVersions(self):
return self._info['supportsVersions']
def supportsTransactionalUndo(self):
try:
return self._info['supportsTransactionalUndo']
except KeyError:
return 0
def tpc_abort(self, transaction): def tpc_abort(self, transaction):
self._lock_acquire() if transaction is not self._transaction:
return
try: try:
if transaction is not self._transaction: return self._server.tpc_abort(self._serial)
self._call('tpc_abort', self._serial) self._tbuf.clear()
self._transaction=None
self._tfile.seek(0)
self._seriald.clear() self._seriald.clear()
del self._serials[:] del self._serials[:]
self._commit_lock_release() finally:
finally: self._lock_release() self._transaction = None
self.tpc_cond.notify()
def tpc_begin(self, transaction): self.tpc_cond.release()
self._lock_acquire()
try: def tpc_begin(self, transaction, tid=None, status=' '):
if self._transaction is transaction: return self.tpc_cond.acquire()
while self._transaction is not None:
user=transaction.user if self._transaction == transaction:
desc=transaction.description # Our tpc_cond lock is re-entrant. It is allowable for a
ext=transaction._extension # client to call two tpc_begins in a row with the same
# transaction, and the second of these must be ignored. Our
# locking is safe because the acquire() above gives us a
# second lock on tpc_cond, and the following release() brings
# us back to owning just the one tpc_cond lock (acquired
# during the first of two consecutive tpc_begins).
self.tpc_cond.release()
return
self.tpc_cond.wait()
while 1: if self._server is None:
self._lock_release() self.tpc_cond.release()
self._commit_lock_acquire() self._transaction = None
self._lock_acquire() raise ClientDisconnected()
# We've got the local commit lock. Now get if tid is None:
# a (tentative) transaction time stamp. self._ts = get_timestamp(self._ts)
t=time.time() id = `self._ts`
t=apply(TimeStamp,(time.gmtime(t)[:5]+(t%60,))) else:
self._ts=t=t.laterThan(self._ts) self._ts = TimeStamp(tid)
id=`t` id = tid
self._transaction = transaction
try: try:
if not self._connected: r = self._server.tpc_begin(id,
raise ClientDisconnected( transaction.user,
"This action is temporarily unavailable.<p>") transaction.description,
r=self._call(self.__begin, id, user, desc, ext) transaction._extension,
tid, status)
except: except:
# XXX can't seem to guarantee that the lock is held here. # Client may have disconnected during the tpc_begin().
self._commit_lock_release() # Then notifyDisconnected() will have released the lock.
if self._server is not disconnected_stub:
self._transaction = None
self.tpc_cond.release()
raise raise
if r is None: break self._serial = id
# We have *BOTH* the local and distributed commit
# lock, now we can actually get ready to get started.
self._serial=id
self._tfile.seek(0)
self._seriald.clear() self._seriald.clear()
del self._serials[:] del self._serials[:]
self._transaction=transaction
finally: self._lock_release()
def tpc_finish(self, transaction, f=None): def tpc_finish(self, transaction, f=None):
self._lock_acquire() if transaction is not self._transaction:
return
try: try:
if transaction is not self._transaction: return if f is not None:
if f is not None: f() f()
self._call('tpc_finish', self._serial, self._server.tpc_finish(self._serial)
transaction.user,
transaction.description,
transaction._extension)
seriald=self._seriald r = self._check_serials()
if self._serials: assert r is None or len(r) == 0, "unhandled serialnos: %s" % r
s=self._serials
l=len(s) self._update_cache()
r=s[:l] finally:
del s[:l] self._transaction = None
for oid, s in r: seriald[oid]=s self.tpc_cond.notify()
self.tpc_cond.release()
tfile=self._tfile
seek=tfile.seek def _update_cache(self):
read=tfile.read # Iterate over the objects in the transaction buffer and
cache=self._cache # update or invalidate the cache.
size=tfile.tell() self._cache.checkSize(self._tbuf.get_size())
cache.checkSize(size) try:
seek(0) self._tbuf.begin_iterate()
i=0 except ValueError, msg:
while i < size:
opcode=read(1)
if opcode == "s":
oid=read(8)
s=seriald[oid]
h=read(6)
vlen, dlen = unpack(">HI", h)
if vlen: v=read(vlen)
else: v=''
p=read(dlen)
if len(p) != dlen:
raise ClientStorageError, ( raise ClientStorageError, (
"Unexpected end of file in client storage " "Unexpected error reading temporary file in "
"temporary file." "client storage: %s" % msg)
) while 1:
if s==ResolvedSerial: try:
t = self._tbuf.next()
except ValueError, msg:
raise ClientStorageError, (
"Unexpected error reading temporary file in "
"client storage: %s" % msg)
if t is None:
break
oid, v, p = t
if p is None: # an invalidation
s = None
else:
s = self._seriald[oid]
if s == ResolvedSerial or s is None:
self._cache.invalidate(oid, v) self._cache.invalidate(oid, v)
else: else:
self._cache.update(oid, s, v, p) self._cache.update(oid, s, v, p)
i=i+15+vlen+dlen self._tbuf.clear()
elif opcode == "i":
oid=read(8)
h=read(2)
vlen=unpack(">H", h)[0]
v=read(vlen)
self._cache.invalidate(oid, v)
i=i+11+vlen
seek(0)
self._transaction=None
self._commit_lock_release()
finally: self._lock_release()
def transactionalUndo(self, trans_id, trans): def transactionalUndo(self, trans_id, trans):
self._lock_acquire() if self._is_read_only:
try: raise POSException.ReadOnlyError()
if trans is not self._transaction: self._check_trans(trans, POSException.StorageTransactionError)
raise POSException.StorageTransactionError(self, transaction) oids = self._server.transactionalUndo(trans_id, self._serial)
oids = self._call('transactionalUndo', trans_id, self._serial)
for oid in oids: for oid in oids:
# write invalidation records with no version self._tbuf.invalidate(oid, '')
self._tfile.write("i%s\000\000" % oid)
return oids return oids
finally: self._lock_release()
def undo(self, transaction_id): def undo(self, transaction_id):
self._lock_acquire() if self._is_read_only:
try: raise POSException.ReadOnlyError()
oids=self._call('undo', transaction_id) # XXX what are the sync issues here?
cinvalidate=self._cache.invalidate oids = self._server.undo(transaction_id)
for oid in oids: for oid in oids:
cinvalidate(oid,'') self._cache.invalidate(oid, '')
return oids return oids
finally: self._lock_release()
def undoInfo(self, first=0, last=-20, specification=None): def undoInfo(self, first=0, last=-20, specification=None):
self._lock_acquire() return self._server.undoInfo(first, last, specification)
try:
return self._call('undoInfo', first, last, specification)
finally: self._lock_release()
def undoLog(self, first, last, filter=None): def undoLog(self, first, last, filter=None):
if filter is not None: return () if filter is not None:
return () # can't pass a filter to server
self._lock_acquire() return self._server.undoLog(first, last) # Eek!
try: return self._call('undoLog', first, last) # Eek!
finally: self._lock_release()
def versionEmpty(self, version): def versionEmpty(self, version):
self._lock_acquire() return self._server.versionEmpty(version)
try: return self._call('versionEmpty', version)
finally: self._lock_release()
def versions(self, max=None): def versions(self, max=None):
self._lock_acquire() return self._server.versions(max)
try: return self._call('versions', max)
finally: self._lock_release()
def sync(self): self._call.sync() # below are methods invoked by the StorageServer
def status(self): def serialnos(self, args):
self._call.sendMessage('status') self._serials.extend(args)
def getWakeup(_w=[]): def info(self, dict):
if _w: return _w[0] self._info.update(dict)
import trigger
t=trigger.trigger().pull_trigger def begin(self):
_w.append(t) self._tfile = tempfile.TemporaryFile(suffix=".inv")
return t self._pickler = cPickle.Pickler(self._tfile, 1)
self._pickler.fast = 1 # Don't use the memo
def invalidate(self, args):
# Queue an invalidate for the end the transaction
if self._pickler is None:
return
self._pickler.dump(args)
def end(self):
if self._pickler is None:
return
self._pickler.dump((0,0))
self._tfile.seek(0)
unpick = cPickle.Unpickler(self._tfile)
f = self._tfile
self._tfile = None
while 1:
oid, version = unpick.load()
if not oid:
break
self._cache.invalidate(oid, version=version)
self._db.invalidate(oid, version=version)
f.close()
def Invalidate(self, args):
for oid, version in args:
self._cache.invalidate(oid, version=version)
try:
self._db.invalidate(oid, version=version)
except AttributeError, msg:
log2(PROBLEM,
"Invalidate(%s, %s) failed for _db: %s" % (repr(oid),
repr(version),
msg))
##############################################################################
#
# Copyright (c) 2001, 2002 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE
#
##############################################################################
"""Stub for interface exported by ClientStorage"""
class ClientStorage:
def __init__(self, rpc):
self.rpc = rpc
def beginVerify(self):
self.rpc.callAsync('begin')
# XXX must rename the two invalidate messages. I can never
# remember which is which
def invalidate(self, args):
self.rpc.callAsync('invalidate', args)
def Invalidate(self, args):
self.rpc.callAsync('Invalidate', args)
def endVerify(self):
self.rpc.callAsync('end')
def serialnos(self, arg):
self.rpc.callAsync('serialnos', arg)
def info(self, arg):
self.rpc.callAsync('info', arg)
##############################################################################
#
# Copyright (c) 2001, 2002 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE
#
##############################################################################
"""Log a transaction's commit info during two-phase commit.
A storage server allows multiple clients to commit transactions, but
must serialize them as the actually execute at the server. The
concurrent commits are achieved by logging actions up until the
tpc_vote(). At that point, the entire transaction is committed on the
real storage.
"""
import cPickle
import tempfile
class CommitLog:
def __init__(self):
self.file = tempfile.TemporaryFile(suffix=".log")
self.pickler = cPickle.Pickler(self.file, 1)
self.pickler.fast = 1
self.stores = 0
self.read = 0
def tpc_begin(self, t, tid, status):
self.t = t
self.tid = tid
self.status = status
def store(self, oid, serial, data, version):
self.pickler.dump((oid, serial, data, version))
self.stores += 1
def get_loader(self):
self.read = 1
self.file.seek(0)
return self.stores, cPickle.Unpickler(self.file)
##############################################################################
#
# Copyright (c) 2001, 2002 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE
#
##############################################################################
"""Exceptions for ZEO."""
class Disconnected(Exception):
"""Exception raised when a ZEO client is disconnected from the
ZEO server."""
try:
from Interface import Base
except ImportError:
class Base:
# a dummy interface for use when Zope's is unavailable
pass
class ICache(Base):
"""ZEO client cache.
__init__(storage, size, client, var)
All arguments optional.
storage -- name of storage
size -- max size of cache in bytes
client -- a string; if specified, cache is persistent.
var -- var directory to store cache files in
"""
def open():
"""Returns a sequence of object info tuples.
An object info tuple is a pair containing an object id and a
pair of serialnos, a non-version serialno and a version serialno:
oid, (serial, ver_serial)
This method builds an index of the cache and returns a
sequence used for cache validation.
"""
def close():
"""Closes the cache."""
def verify(func):
"""Call func on every object in cache.
func is called with three arguments
func(oid, serial, ver_serial)
"""
def invalidate(oid, version):
"""Remove object from cache."""
def load(oid, version):
"""Load object from cache.
Return None if object not in cache.
Return data, serialno if object is in cache.
"""
def store(oid, p, s, version, pv, sv):
"""Store a new object in the cache."""
def update(oid, serial, version, data):
"""Update an object already in the cache.
XXX This method is called to update objects that were modified by
a transaction. It's likely that it is already in the cache,
and it may be possible for the implementation to operate more
efficiently.
"""
def modifiedInVersion(oid):
"""Return the version an object is modified in.
'' signifies the trunk.
Returns None if the object is not in the cache.
"""
def checkSize(size):
"""Check if adding size bytes would exceed cache limit.
This method is often called just before store or update. The
size is a hint about the amount of data that is about to be
stored. The cache may want to evict some data to make space.
"""
##############################################################################
#
# Copyright (c) 2001, 2002 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE
#
##############################################################################
"""Stub for interface exposed by StorageServer"""
class StorageServer:
def __init__(self, rpc):
self.rpc = rpc
def register(self, storage_name, read_only):
self.rpc.call('register', storage_name, read_only)
def get_info(self):
return self.rpc.call('get_info')
def get_size_info(self):
return self.rpc.call('get_size_info')
def beginZeoVerify(self):
self.rpc.callAsync('beginZeoVerify')
def zeoVerify(self, oid, s, sv):
self.rpc.callAsync('zeoVerify', oid, s, sv)
def endZeoVerify(self):
self.rpc.callAsync('endZeoVerify')
def new_oids(self, n=None):
if n is None:
return self.rpc.call('new_oids')
else:
return self.rpc.call('new_oids', n)
def pack(self, t, wait=None):
if wait is None:
self.rpc.call('pack', t)
else:
self.rpc.call('pack', t, wait)
def zeoLoad(self, oid):
return self.rpc.call('zeoLoad', oid)
def storea(self, oid, serial, data, version, id):
self.rpc.callAsync('storea', oid, serial, data, version, id)
def tpc_begin(self, id, user, descr, ext, tid, status):
return self.rpc.call('tpc_begin', id, user, descr, ext, tid, status)
def vote(self, trans_id):
return self.rpc.call('vote', trans_id)
def tpc_finish(self, id):
return self.rpc.call('tpc_finish', id)
def tpc_abort(self, id):
self.rpc.callAsync('tpc_abort', id)
def abortVersion(self, src, id):
return self.rpc.call('abortVersion', src, id)
def commitVersion(self, src, dest, id):
return self.rpc.call('commitVersion', src, dest, id)
def history(self, oid, version, length=None):
if length is not None:
return self.rpc.call('history', oid, version)
else:
return self.rpc.call('history', oid, version, length)
def load(self, oid, version):
return self.rpc.call('load', oid, version)
def loadSerial(self, oid, serial):
return self.rpc.call('loadSerial', oid, serial)
def modifiedInVersion(self, oid):
return self.rpc.call('modifiedInVersion', oid)
def new_oid(self, last=None):
if last is None:
return self.rpc.call('new_oid')
else:
return self.rpc.call('new_oid', last)
def store(self, oid, serial, data, version, trans):
return self.rpc.call('store', oid, serial, data, version, trans)
def transactionalUndo(self, trans_id, trans):
return self.rpc.call('transactionalUndo', trans_id, trans)
def undo(self, trans_id):
return self.rpc.call('undo', trans_id)
def undoLog(self, first, last):
# XXX filter not allowed across RPC
return self.rpc.call('undoLog', first, last)
def undoInfo(self, first, last, spec):
return self.rpc.call('undoInfo', first, last, spec)
def versionEmpty(self, vers):
return self.rpc.call('versionEmpty', vers)
def versions(self, max=None):
if max is None:
return self.rpc.call('versions')
else:
return self.rpc.call('versions', max)
...@@ -11,660 +11,550 @@ ...@@ -11,660 +11,550 @@
# FOR A PARTICULAR PURPOSE # FOR A PARTICULAR PURPOSE
# #
############################################################################## ##############################################################################
"""Network ZODB storage server
__version__ = "$Revision: 1.36 $"[11:-2] This server acts as a front-end for one or more real storages, like
file storage or Berkeley storage.
import asyncore, socket, string, sys, os XXX Need some basic access control-- a declaration of the methods
exported for invocation by the server.
"""
import asyncore
import cPickle import cPickle
from cPickle import Unpickler import os
from cStringIO import StringIO import sys
from thread import start_new_thread import threading
import time
from types import StringType from ZEO import ClientStub
from ZEO.CommitLog import CommitLog
from ZODB import POSException from ZEO.zrpc.server import Dispatcher
from ZODB.POSException import TransactionError, UndoError, VersionCommitError from ZEO.zrpc.connection import ManagedServerConnection, Delay
from ZODB.Transaction import Transaction
import zLOG
from ZODB.POSException import StorageError, StorageTransactionError, \
TransactionError, ReadOnlyError
from ZODB.referencesf import referencesf from ZODB.referencesf import referencesf
from ZODB.utils import U64 from ZODB.Transaction import Transaction
from ZODB.TmpStore import TmpStore
from ZEO import trigger
from ZEO import asyncwrap
from ZEO.smac import Disconnected, SizedMessageAsyncConnection
from ZEO.logger import zLogger, format_msg
class StorageServerError(POSException.StorageError):
pass
# We create a special fast pickler! This allows us # We create a special fast pickler! This allows us
# to create slightly more efficient pickles and # to create slightly more efficient pickles and
# to create them a tad faster. # to create them a tad faster.
pickler=cPickle.Pickler() pickler = cPickle.Pickler()
pickler.fast=1 # Don't use the memo pickler.fast = 1 # Don't use the memo
dump=pickler.dump dump = pickler.dump
log = zLogger("ZEO Server") def log(message, level=zLOG.INFO, label="ZEO Server:%s" % os.getpid(),
error=None):
class StorageServer(asyncore.dispatcher): zLOG.LOG(label, level, message, error=error)
def __init__(self, connection, storages): # a version of log that includes the storage name
def slog(storage, msg, level=zLOG.INFO, error=None, pid=os.getpid()):
name = getattr(storage, '__name__', None)
if name is None:
name = str(self.storage)
zLOG.LOG("ZEO Server:%s:%s" % (pid, name), level, msg, error=error)
class StorageServerError(StorageError):
pass
self.__storages=storages class StorageServer:
for n, s in storages.items(): def __init__(self, addr, storages, read_only=0):
init_storage(s) # XXX should read_only be a per-storage option? not yet...
# Create a waiting list to support the distributed commit lock. self.addr = addr
self.storages = storages
for s in storages.values():
s._waiting = [] s._waiting = []
self.read_only = read_only
self.__connections={} self.connections = {}
self.__get_connections=self.__connections.get self.dispatcher = Dispatcher(addr, factory=self.newConnection,
reuse_addr=1)
self._pack_trigger = trigger.trigger()
asyncore.dispatcher.__init__(self) def newConnection(self, sock, addr):
c = ManagedServerConnection(sock, addr, ZEOStorage(self), self)
if type(connection) is type(''): log("new connection %s: %s" % (addr, `c`))
self.create_socket(socket.AF_UNIX, socket.SOCK_STREAM) return c
try: os.unlink(connection)
except: pass def register(self, storage_id, proxy):
"""Register a connection's use with a particular storage.
This information is needed to handle invalidation.
"""
l = self.connections.get(storage_id)
if l is None:
l = self.connections[storage_id] = []
l.append(proxy)
def invalidate(self, conn, storage_id, invalidated=(), info=0):
for p in self.connections.get(storage_id, ()):
if invalidated and p is not conn:
p.client.Invalidate(invalidated)
else: else:
self.create_socket(socket.AF_INET, socket.SOCK_STREAM) p.client.info(info)
self.set_reuse_addr()
def close_server(self):
log.info('Listening on %s' % repr(connection)) # Close the dispatcher so that there are no new connections.
self.bind(connection) self.dispatcher.close()
self.listen(5) for storage in self.storages.values():
storage.close()
def register_connection(self, connection, storage_id): # Force the asyncore mainloop to exit by hackery, i.e. close
storage=self.__storages.get(storage_id, None) # every socket in the map. loop() will return when the map is
if storage is None: # empty.
log.error("Unknown storage_id: %s" % storage_id) for s in asyncore.socket_map.values():
connection.close()
return None, None
connections=self.__get_connections(storage_id, None)
if connections is None:
self.__connections[storage_id]=connections=[]
connections.append(connection)
return storage, storage_id
def unregister_connection(self, connection, storage_id):
connections=self.__get_connections(storage_id, None)
if connections:
n=[]
for c in connections:
if c is not connection:
n.append(c)
self.__connections[storage_id]=n
def invalidate(self, connection, storage_id, invalidated=(), info=0,
dump=dump):
for c in self.__connections[storage_id]:
if invalidated and c is not connection:
c.message_output('I'+dump(invalidated, 1))
if info:
c.message_output('S'+dump(info, 1))
def writable(self): return 0
def handle_read(self): pass
def readable(self): return 1
def handle_connect (self): pass
def handle_accept(self):
try: try:
r = self.accept() s.close()
if r is None:
return
sock, addr = r
except socket.error, err:
log.warning("accept() failed: %s" % err)
else:
ZEOConnection(self, sock, addr)
def status(self):
"""Log status information about connections and storages"""
lines = []
for storage_id, connections in self.__connections.items():
s = "Storage %s has %d connections" % (storage_id,
len(connections))
lines.append(s)
for c in connections:
lines.append("%s readable=%s writeable=%s" % (
c, c.readable(), c.writable()))
lines.append("\t" + c.stats())
log.info(string.join(lines, "\n"))
return _noreturn
storage_methods={}
for n in (
'get_info', 'abortVersion', 'commitVersion',
'history', 'load', 'loadSerial',
'modifiedInVersion', 'new_oid', 'new_oids', 'pack', 'store',
'storea', 'tpc_abort', 'tpc_begin', 'tpc_begin_sync',
'tpc_finish', 'undo', 'undoLog', 'undoInfo', 'versionEmpty', 'versions',
'transactionalUndo',
'vote', 'zeoLoad', 'zeoVerify', 'beginZeoVerify', 'endZeoVerify',
'status'
):
storage_methods[n]=1
storage_method=storage_methods.has_key
def find_global(module, name,
global_dict=globals(), silly=('__doc__',)):
try: m=__import__(module, global_dict, global_dict, silly)
except:
raise StorageServerError, (
"Couldn\'t import global module %s" % module)
try:
r=getattr(m, name)
except: except:
raise StorageServerError, ( pass
"Couldn\'t find global %s in module %s" % (name, module))
safe=getattr(r, '__no_side_effects__', 0)
if safe: return r
raise StorageServerError, 'Unsafe global, %s.%s' % (module, name)
_noreturn=[]
class ZEOConnection(SizedMessageAsyncConnection):
_transaction=None def close(self, conn):
__storage=__storage_id=None removed = 0
for sid, cl in self.connections.items():
if conn.obj in cl:
cl.remove(conn.obj)
removed = 1
class ZEOStorage:
def __init__(self, server):
self.server = server
self.client = None
self.__storage = None
self.__storage_id = "uninitialized"
self._transaction = None
def __init__(self, server, sock, addr): def notifyConnected(self, conn):
self.__server=server self.client = ClientStub.ClientStorage(conn)
self.status = server.status
self.__invalidated=[]
self.__closed=None
if __debug__:
debug = log
else:
debug = None
if __debug__:
# store some detailed statistics about method calls
self._last_method = None
self._t_begin = None
self._t_end = None
self._ncalls = 0
SizedMessageAsyncConnection.__init__(self, sock, addr, debug=debug)
self.logaddr = repr(addr) # form of addr suitable for logging
log.info('Connect %s %s' % (id(self), self.logaddr))
def stats(self):
# This method is called via the status() command. The stats
# are of limited use for the current command, because the
# actual invocation of status() will clobber the previous
# method's statistics.
#
# When there are multiple connections active, a new connection
# can always get detailed statistics about other connections.
if __debug__:
if self._last_method == "status":
return "method=status begin=%s end=... ncalls=%d" % (
self._t_begin, self._ncalls)
if self._t_end is not None and self._t_begin is not None:
delta = self._t_end - self._t_begin
else:
delta = -1
return "method=%s begin=%s end=%s delta=%.3f ncalls=%d" % (
self._last_method, self._t_begin, self._t_end, delta,
self._ncalls)
else:
return ""
def __repr__(self): def __repr__(self):
return "<ZEOConnection %s%s" % (`self.addr`, tid = self._transaction and repr(self._transaction.id)
# sort of messy way to add tag 'closed' to if self.__storage:
# connections that are closed stid = self.__storage._transaction and \
(self.__closed is None and '>' or ' closed>')) repr(self.__storage._transaction.id)
def close(self):
t=self._transaction
if (t is not None and self.__storage is not None and
self.__storage._transaction is t):
self.tpc_abort(t.id)
else: else:
self._transaction=None stid = None
self.__invalidated=[] name = self.__class__.__name__
return "<%s %X trans=%s s_trans=%s>" % (name, id(self), tid, stid)
self.__server.unregister_connection(self, self.__storage_id)
self.__closed=1 def _log(self, msg, level=zLOG.INFO, error=None, pid=os.getpid()):
SizedMessageAsyncConnection.close(self) zLOG.LOG("ZEO Server:%s:%s" % (pid, self.__storage_id),
log.info('Close %s' % id(self)) level, msg, error=error)
def message_input(self, message, def setup_delegation(self):
dump=dump, Unpickler=Unpickler, StringIO=StringIO, """Delegate several methods to the storage"""
None=None): self.undoInfo = self.__storage.undoInfo
if __debug__: self.undoLog = self.__storage.undoLog
self.versionEmpty = self.__storage.versionEmpty
self._t_begin = time.time() self.versions = self.__storage.versions
self._t_end = None self.history = self.__storage.history
self.load = self.__storage.load
if len(message) > 120: # XXX need constant from logger self.loadSerial = self.__storage.loadSerial
tmp = `message[:120]`
def _check_tid(self, tid, exc=None):
caller = sys._getframe().f_back.f_code.co_name
if self._transaction is None:
self._log("no current transaction: %s()" % caller,
zLOG.PROBLEM)
if exc is not None:
raise exc(None, tid)
else: else:
tmp = `message` return 0
log.trace("message_input %s" % tmp) if self._transaction.id != tid:
self._log("%s(%s) invalid; current transaction = %s" % \
if self.__storage is None: (caller, repr(tid), repr(self._transaction.id)),
if __debug__: zLOG.PROBLEM)
log.blather("register connection to %s from %s" % (message, if exc is not None:
self.logaddr)) raise exc(self._transaction.id, tid)
# This is the first communication from the client
self.__storage, self.__storage_id = (
self.__server.register_connection(self, message))
# Send info back asynchronously, so client need not ask
self.message_output('S'+dump(self.get_info(), 1))
return
try:
# Unpickle carefully.
unpickler=Unpickler(StringIO(message))
unpickler.find_global=find_global
args=unpickler.load()
name, args = args[0], args[1:]
if __debug__:
self._last_method = name
self._ncalls = self._ncalls + 1
log.debug("call %s%s from %s" % (name, format_msg(args),
self.logaddr))
if not storage_method(name):
log.warning("Invalid method name: %s" % name)
if __debug__:
self._t_end = time.time()
raise 'Invalid Method Name', name
if hasattr(self, name):
r=apply(getattr(self, name), args)
else: else:
r=apply(getattr(self.__storage, name), args) return 0
if r is _noreturn: return 1
if __debug__:
log.debug("no return to %s" % self.logaddr)
self._t_end = time.time()
return
except (UndoError, VersionCommitError), err:
if __debug__:
log.debug("return error %s to %s" % (err, self.logaddr))
self._t_end = time.time()
# These are normal usage errors. No need to log them.
self.return_error(sys.exc_info()[0], sys.exc_info()[1])
return
except:
if __debug__:
self._t_end = time.time()
log.error("error", error=sys.exc_info())
self.return_error(sys.exc_info()[0], sys.exc_info()[1])
return
if __debug__:
log.debug("return %s to %s" % (format_msg(r), self.logaddr))
self._t_end = time.time()
r=dump(r,1)
self.message_output('R'+r)
def return_error(self, err_type, err_value, type=type, dump=dump):
if type(err_value) is not type(self):
err_value = err_type, err_value
if __debug__: def register(self, storage_id, read_only):
log.trace("%s E: %s" % (id(self), `err_value`)) """Select the storage that this client will use
try: r=dump(err_value, 1) This method must be the first one called by the client.
except: """
# Ugh, must be an unpicklable exception self._log("register(%s, %s)" % (storage_id, read_only))
r=StorageServerError("Couldn't pickle error %s" % `r`) storage = self.server.storages.get(storage_id)
dump('',1) # clear pickler if storage is None:
r=dump(r,1) self._log("unknown storage_id: %s" % storage_id)
raise ValueError, "unknown storage: %s" % storage_id
self.message_output('E'+r) if not read_only and (self.server.read_only or storage.isReadOnly()):
raise ReadOnlyError()
self.__storage_id = storage_id
self.__storage = storage
self.setup_delegation()
self.server.register(storage_id, self)
self._log("registered storage %s: %s" % (storage_id, storage))
def get_info(self): def get_info(self):
storage=self.__storage return {'length': len(self.__storage),
info = { 'size': self.__storage.getSize(),
'length': len(storage), 'name': self.__storage.getName(),
'size': storage.getSize(), 'supportsUndo': self.__storage.supportsUndo(),
'name': storage.getName(), 'supportsVersions': self.__storage.supportsVersions(),
'supportsTransactionalUndo':
self.__storage.supportsTransactionalUndo(),
} }
for feature in ('supportsUndo',
'supportsVersions',
'supportsTransactionalUndo',):
if hasattr(storage, feature):
info[feature] = getattr(storage, feature)()
else:
info[feature] = 0
return info
def get_size_info(self): def get_size_info(self):
storage=self.__storage return {'length': len(self.__storage),
return { 'size': self.__storage.getSize(),
'length': len(storage),
'size': storage.getSize(),
} }
def zeoLoad(self, oid): def zeoLoad(self, oid):
if __debug__: v = self.__storage.modifiedInVersion(oid)
log.blather("zeoLoad(%s) %s" % (U64(oid), self.logaddr)) if v:
storage=self.__storage pv, sv = self.__storage.load(oid, v)
v=storage.modifiedInVersion(oid) else:
if v: pv, sv = storage.load(oid, v) pv = sv = None
else: pv=sv=None
try: try:
p, s = storage.load(oid,'') p, s = self.__storage.load(oid, '')
except KeyError: except KeyError:
if sv: if sv:
# Created in version, no non-version data # Created in version, no non-version data
p=s=None p = s = None
else: else:
raise raise
return p, s, v, pv, sv return p, s, v, pv, sv
def beginZeoVerify(self): def beginZeoVerify(self):
if __debug__: self.client.beginVerify()
log.blather("beginZeoVerify() %s" % self.logaddr)
self.message_output('bN.') def zeoVerify(self, oid, s, sv):
return _noreturn try:
p, os, v, pv, osv = self.zeoLoad(oid)
def zeoVerify(self, oid, s, sv, except: # except what?
dump=dump): return None
try: p, os, v, pv, osv = self.zeoLoad(oid)
except: return _noreturn
p=pv=None # free the pickles
if os != s: if os != s:
self.message_output('i'+dump((oid, ''),1)) self.client.invalidate((oid, ''))
elif osv != sv: elif osv != sv:
self.message_output('i'+dump((oid, v),1)) self.client.invalidate((oid, v))
return _noreturn
def endZeoVerify(self): def endZeoVerify(self):
if __debug__: self.client.endVerify()
log.blather("endZeoVerify() %s" % self.logaddr)
self.message_output('eN.')
return _noreturn
def new_oids(self, n=100): def modifiedInVersion(self, oid):
new_oid=self.__storage.new_oid return self.__storage.modifiedInVersion(oid)
if n < 0: n=1
r=range(n)
for i in r: r[i]=new_oid()
return r
def pack(self, t, wait=0): def pack(self, t, wait=0):
start_new_thread(self._pack, (t,wait)) t = threading.Thread(target=self._pack, args=(t, wait))
if wait: return _noreturn t.start()
def _pack(self, t, wait=0): def _pack(self, t, wait=0):
try: try:
log.blather('pack begin')
self.__storage.pack(t, referencesf) self.__storage.pack(t, referencesf)
log.blather('pack end')
except: except:
log.error( self._log('Pack failed for %s' % self.__storage_id,
'Pack failed for %s' % self.__storage_id, zLOG.ERROR,
error=sys.exc_info()) error=sys.exc_info())
if wait: if wait:
self.return_error(sys.exc_info()[0], sys.exc_info()[1]) raise
self.__server._pack_trigger.pull_trigger()
else:
if wait:
self.message_output('RN.')
self.__server._pack_trigger.pull_trigger()
else: else:
# XXX Why doesn't we broadcast on wait?
if not wait:
# Broadcast new size statistics # Broadcast new size statistics
self.__server.invalidate(0, self.__storage_id, (), self.server.invalidate(0, self.__storage_id, (),
self.get_size_info()) self.get_size_info())
def abortVersion(self, src, id): def new_oids(self, n=100):
t=self._transaction """Return a sequence of n new oids, where n defaults to 100"""
if t is None or id != t.id: if n < 0:
raise POSException.StorageTransactionError(self, id) n = 1
oids=self.__storage.abortVersion(src, t) return [self.__storage.new_oid() for i in range(n)]
a=self.__invalidated.append
for oid in oids: a((oid,src))
return oids
def commitVersion(self, src, dest, id): def undo(self, transaction_id):
t=self._transaction oids = self.__storage.undo(transaction_id)
if t is None or id != t.id: if oids:
raise POSException.StorageTransactionError(self, id) self.server.invalidate(self, self.__storage_id,
oids=self.__storage.commitVersion(src, dest, t) map(lambda oid: (oid, None, ''), oids))
a=self.__invalidated.append
for oid in oids:
a((oid,dest))
if dest: a((oid,src))
return oids return oids
return ()
def storea(self, oid, serial, data, version, id, def tpc_begin(self, id, user, description, ext, tid, status):
dump=dump): if self._transaction is not None:
if __debug__: if self._transaction.id == id:
log.blather("storea(%s, [%d], %s) %s" % (U64(oid), len(data), self._log("duplicate tpc_begin(%s)" % repr(id))
U64(id), self.logaddr)) return
try:
t=self._transaction
if t is None or id != t.id:
raise POSException.StorageTransactionError(self, id)
newserial=self.__storage.store(oid, serial, data, version, t)
except TransactionError, v:
# This is a normal transaction errorm such as a conflict error
# or a version lock or conflict error. It doen't need to be
# logged.
newserial=v
except:
# all errors need to be serialized to prevent unexpected
# returns, which would screw up the return handling.
# IOW, Anything that ends up here is evil enough to be logged.
log.error('store error', error=sys.exc_info())
newserial=sys.exc_info()[1]
else: else:
if serial != '\0\0\0\0\0\0\0\0': raise StorageTransactionError("Multiple simultaneous tpc_begin"
self.__invalidated.append((oid, version)) " requests from one client.")
try: r=dump((oid,newserial), 1) if self.__storage._transaction is None:
except: self.strategy = ImmediateCommitStrategy(self.__storage,
# We got a pickling error, must be because the self.client)
# newserial is an unpicklable exception. else:
r=StorageServerError("Couldn't pickle exception %s" % `newserial`) self.strategy = DelayedCommitStrategy(self.__storage,
dump('',1) # clear pickler self.wait)
r=dump((oid, r),1)
t = Transaction()
t.id = id
t.user = user
t.description = description
t._extension = ext
self.strategy.tpc_begin(t, tid, status)
self._transaction = t
def tpc_finish(self, id):
if not self._check_tid(id):
return
invalidated = self.strategy.tpc_finish()
if invalidated:
self.server.invalidate(self, self.__storage_id,
invalidated, self.get_size_info())
if not self._handle_waiting():
self._transaction = None
self.strategy = None
def tpc_abort(self, id):
if not self._check_tid(id):
return
self.strategy.tpc_abort()
if not self._handle_waiting():
self._transaction = None
self.strategy = None
# XXX handle new serialnos
self.message_output('s'+r) def storea(self, oid, serial, data, version, id):
return _noreturn self._check_tid(id, exc=StorageTransactionError)
self.strategy.store(oid, serial, data, version)
def vote(self, id): def vote(self, id):
t=self._transaction self._check_tid(id, exc=StorageTransactionError)
if t is None or id != t.id: return self.strategy.tpc_vote()
raise POSException.StorageTransactionError(self, id)
return self.__storage.tpc_vote(t)
def transactionalUndo(self, trans_id, id): def abortVersion(self, src, id):
if __debug__: self._check_tid(id, exc=StorageTransactionError)
log.blather("transactionalUndo(%s, %s) %s" % (trans_id, return self.strategy.abortVersion(src)
U64(id), self.logaddr))
t=self._transaction
if t is None or id != t.id:
raise POSException.StorageTransactionError(self, id)
return self.__storage.transactionalUndo(trans_id, self._transaction)
def undo(self, transaction_id): def commitVersion(self, src, dest, id):
if __debug__: self._check_tid(id, exc=StorageTransactionError)
log.blather("undo(%s) %s" % (transaction_id, self.logaddr)) return self.strategy.commitVersion(src, dest)
oids=self.__storage.undo(transaction_id)
if oids:
self.__server.invalidate(
self, self.__storage_id, map(lambda oid: (oid,None), oids)
)
return oids
return ()
# distributed commit lock support methods def transactionalUndo(self, trans_id, id):
self._check_tid(id, exc=StorageTransactionError)
# Only one client at a time can commit a transaction on a return self.strategy.transactionalUndo(trans_id)
# storage. If one client is committing a transaction, and a
# second client sends a tpc_begin(), then second client is queued. # When a delayed transaction is restarted, the dance is
# When the first transaction finishes, either by abort or commit, # complicated. The restart occurs when one ZEOStorage instance
# the request from the queued client must be handled. # finishes as a transaction and finds another instance is in the
# _waiting list.
# It is important that this code be robust. If a queued
# transaction is not restarted, the server will stop processing # XXX It might be better to have a mechanism to explicitly send
# new transactions. # the finishing transaction's reply before restarting the waiting
# transaction. If the restart takes a long time, the previous
# This lock is implemented by storing the queued requests in a # client will be blocked until it finishes.
# list on the storage object. The list contains:
# a callable object to resume request def wait(self):
# arguments to that object if self.__storage._transaction:
# a callable object to handle errors during resume d = Delay()
self.__storage._waiting.append((d, self))
# XXX I am not sure that the commitlock_resume() method is self._log("Transaction block waiting for storage. "
# sufficiently paranoid. "%d clients waiting." % len(self.__storage._waiting))
return d
def commitlock_suspend(self, resume, args, onerror): else:
self.__storage._waiting.append((resume, args, onerror)) self.restart()
log.blather("suspend %s. %d queued clients" % (resume.im_self,
len(self.__storage._waiting))) def _handle_waiting(self):
while self.__storage._waiting:
def commitlock_resume(self): delay, zeo_storage = self.__storage._waiting.pop(0)
waiting = self.__storage._waiting if self._restart(zeo_storage, delay):
while waiting:
resume, args, onerror = waiting.pop(0)
log.blather("resuming queued client %s, %d still queued" % (
resume.im_self, len(waiting)))
try:
if apply(resume, args):
break break
except Disconnected:
# A disconnected error isn't an unexpected error.
# There should be no need to log it, because the
# disconnect will have generated its own log event.
onerror()
except:
log.error(
"Unexpected error handling queued tpc_begin()",
error=sys.exc_info())
onerror()
def tpc_abort(self, id): def _restart(self, zeo_storage, delay):
if __debug__: # call the restart() method on the appropriate server
try: try:
log.blather("tpc_abort(%s) %s" % (U64(id), self.logaddr)) zeo_storage.restart(delay)
except: except:
print repr(id) self._log("Unexpected error handling waiting transaction",
raise level=zLOG.WARNING, error=sys.exc_info())
t = self._transaction zeo_storage.close()
if t is None or id != t.id: return 0
return else:
r = self.__storage.tpc_abort(t) return 1
self._transaction = None def restart(self, delay=None):
self.__invalidated = [] old_strategy = self.strategy
self.commitlock_resume() self.strategy = ImmediateCommitStrategy(self.__storage,
self.client)
resp = old_strategy.restart(self.strategy)
if delay is not None:
delay.reply(resp)
def unlock(self): # A ZEOStorage instance can use different strategies to commit a
if self.__closed: # transaction. The current implementation uses different strategies
return # depending on whether the underlying storage is available. These
self.message_output('UN.') # strategies implement the distributed commit lock.
def tpc_begin(self, id, user, description, ext):
if __debug__:
log.blather("tpc_begin(%s, %s, %s) %s" % (U64(id), `user`,
`description`,
self.logaddr))
t = self._transaction
if t is not None:
if id == t.id:
return
else:
raise StorageServerError(
"Multiple simultaneous tpc_begin requests from the same "
"client."
)
storage = self.__storage
if storage._transaction is not None:
self.commitlock_suspend(self.unlock, (), self.close)
return 1 # Return a flag indicating a lock condition.
assert id != 't'
self._transaction=t=Transaction()
t.id=id
t.user=user
t.description=description
t._extension=ext
storage.tpc_begin(t)
self.__invalidated=[]
def tpc_begin_sync(self, id, user, description, ext):
if self.__closed: return
t=self._transaction
if t is not None and id == t.id: return
storage=self.__storage
if storage._transaction is None:
self.try_again_sync(id, user, description, ext)
else:
self.commitlock_suspend(self.try_again_sync,
(id, user, description, ext),
self.close)
return _noreturn
def try_again_sync(self, id, user, description, ext):
storage=self.__storage
if storage._transaction is None:
self._transaction=t=Transaction()
t.id=id
t.user=user
t.description=description
storage.tpc_begin(t)
self.__invalidated=[]
self.message_output('RN.')
return 1 # If the underlying storage is availabe, start the commit immediately
# using the ImmediateCommitStrategy. If the underlying storage is not
# available because another client is committing a transaction, delay
# the commit as long as possible. At some point it will no longer be
# possible to delay; either the transaction will reach the vote stage
# or a synchronous method like transactionalUndo() will be called.
# When it is no longer possible to delay, the client must block until
# the storage is ready. Then we switch back to the immediate strategy.
def tpc_finish(self, id, user, description, ext): class ICommitStrategy:
if __debug__: """A class that describes that commit strategy interface.
log.blather("tpc_finish(%s) %s" % (U64(id), self.logaddr))
t = self._transaction
if id != t.id:
return
storage = self.__storage The commit strategy interface does not require the transaction
r = storage.tpc_finish(t) argument, except for tpc_begin(). The storage interface requires
the client to pass a transaction object/id to each transactional
method. The strategy does not; it requires the caller to only
call methods for a single transaction.
"""
# This isn't a proper Zope interface, because I don't want to
# introduce a dependency between ZODB and Zope interfaces.
self._transaction = None def tpc_begin(self, trans, tid, status): pass
if self.__invalidated:
self.__server.invalidate(self, self.__storage_id, def store(self, oid, serial, data, version): pass
self.__invalidated,
self.get_size_info()) def abortVersion(self, src): pass
self.__invalidated = []
def commitVersion(self, src, dest): pass
# the trans_id arg to transactionalUndo is not the current txn's id
def transactionalUndo(self, trans_id): pass
self.commitlock_resume() def tpc_vote(self): pass
def init_storage(storage): def tpc_abort(self): pass
if not hasattr(storage,'tpc_vote'): storage.tpc_vote=lambda *args: None
if __name__=='__main__': def tpc_finish(self): pass
import ZODB.FileStorage
name, port = sys.argv[1:3] class ImmediateCommitStrategy:
log.trace(format_msg(name, port)) """The storage is available so do a normal commit."""
def __init__(self, storage, client):
self.storage = storage
self.client = client
self.invalidated = []
self.serials = []
def tpc_begin(self, txn, tid, status):
self.txn = txn
self.storage.tpc_begin(txn, tid, status)
def tpc_vote(self):
# send all the serialnos as a batch
self.client.serialnos(self.serials)
return self.storage.tpc_vote(self.txn)
def tpc_finish(self):
self.storage.tpc_finish(self.txn)
return self.invalidated
def tpc_abort(self):
self.storage.tpc_abort(self.txn)
def store(self, oid, serial, data, version):
try: try:
port='', int(port) newserial = self.storage.store(oid, serial, data, version,
self.txn)
except TransactionError, err:
# Storage errors are passed to the client
newserial = err
except: except:
pass # Unexpected storage errors are logged and passed to the client
exc_info = sys.exc_info()
slog(self.storage, "store error: %s, %s" % exc_info[:2],
zLOG.ERROR, error=exc_info)
newserial = exc_info[1]
del exc_info
else:
if serial != "\0\0\0\0\0\0\0\0":
self.invalidated.append((oid, version))
try:
nil = dump(newserial, 1)
except:
msg = "Couldn't pickle storage exception: %s" % repr(newserial)
slog(self.storage, msg, zLOG.ERROR)
dump('', 1) # clear pickler
r = StorageServerError(msg)
newserial = r
self.serials.append((oid, newserial))
def commitVersion(self, src, dest):
oids = self.storage.commitVersion(src, dest, self.txn)
inv = [(oid, dest) for oid in oids]
self.invalidated.extend(inv)
if dest:
inv = [(oid, src) for oid in oids]
self.invalidated.extend(inv)
return oids
def abortVersion(self, src):
oids = self.storage.abortVersion(src, self.txn)
inv = [(oid, src) for oid in oids]
self.invalidated.extend(inv)
return oids
def transactionalUndo(self, trans_id):
oids = self.storage.transactionalUndo(trans_id, self.txn)
inv = [(oid, None) for oid in oids]
self.invalidated.extend(inv)
return oids
d = {'1': ZODB.FileStorage.FileStorage(name)} class DelayedCommitStrategy:
StorageServer(port, d) """The storage is unavailable, so log to a file."""
asyncwrap.loop()
def __init__(self, storage, block):
# the block argument is called when we can't delay any longer
self.storage = storage
self.block = block
self.log = CommitLog()
self.invalidated = []
# Store information about the call that blocks
self.name = None
self.args = None
def tpc_begin(self, txn, tid, status):
self.txn = txn
self.tid = tid
self.status = status
def store(self, oid, serial, data, version):
self.log.store(oid, serial, data, version)
def tpc_abort(self):
pass # just forget about this strategy
def tpc_finish(self):
raise RuntimeError, "Logic error. This method must not be called."
def tpc_vote(self):
self.name = "tpc_vote"
self.args = ()
return self.block()
def commitVersion(self, src, dest):
self.name = "commitVersion"
self.args = src, dest
return self.block()
def abortVersion(self, src):
self.name = "abortVersion"
self.args = src,
return self.block()
def transactionalUndo(self, trans_id):
self.name = "transactionalUndo"
self.args = trans_id,
return self.block()
def restart(self, new_strategy):
# called by the storage when the storage is available
new_strategy.tpc_begin(self.txn, self.tid, self.status)
loads, loader = self.log.get_loader()
for i in range(loads):
oid, serial, data, version = loader.load()
new_strategy.store(oid, serial, data, version)
meth = getattr(new_strategy, self.name)
return meth(*self.args)
##############################################################################
#
# Copyright (c) 2001, 2002 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE
#
##############################################################################
"""A TransactionBuffer store transaction updates until commit or abort.
A transaction may generate enough data that it is not practical to
always hold pending updates in memory. Instead, a TransactionBuffer
is used to store the data until a commit or abort.
"""
# A faster implementation might store trans data in memory until it
# reaches a certain size.
import tempfile
import cPickle
class TransactionBuffer:
def __init__(self):
self.file = tempfile.TemporaryFile(suffix=".tbuf")
self.count = 0
self.size = 0
# It's safe to use a fast pickler because the only objects
# stored are builtin types -- strings or None.
self.pickler = cPickle.Pickler(self.file, 1)
self.pickler.fast = 1
def close(self):
try:
self.file.close()
except OSError:
pass
def store(self, oid, version, data):
"""Store oid, version, data for later retrieval"""
self.pickler.dump((oid, version, data))
self.count += 1
# Estimate per-record cache size
self.size = self.size + len(data) + (27 + 12)
if version:
self.size = self.size + len(version) + 4
def invalidate(self, oid, version):
self.pickler.dump((oid, version, None))
self.count += 1
def clear(self):
"""Mark the buffer as empty"""
self.file.seek(0)
self.count = 0
self.size = 0
# unchecked constraints:
# 1. can't call store() after begin_iterate()
# 2. must call clear() after iteration finishes
def begin_iterate(self):
"""Move the file pointer in advance of iteration"""
self.file.flush()
self.file.seek(0)
self.unpickler = cPickle.Unpickler(self.file)
def next(self):
"""Return next tuple of data or None if EOF"""
if self.count == 0:
del self.unpickler
return None
oid_ver_data = self.unpickler.load()
self.count -= 1
return oid_ver_data
def get_size(self):
"""Return size of data stored in buffer (just a hint)."""
return self.size
...@@ -11,5 +11,3 @@ ...@@ -11,5 +11,3 @@
# FOR A PARTICULAR PURPOSE # FOR A PARTICULAR PURPOSE
# #
############################################################################## ##############################################################################
import fap
...@@ -14,11 +14,14 @@ ...@@ -14,11 +14,14 @@
"""Sized message async connections """Sized message async connections
""" """
__version__ = "$Revision: 1.16 $"[11:-2] __version__ = "$Revision: 1.17 $"[11:-2]
import asyncore, struct
from Exceptions import Disconnected
from zLOG import LOG, TRACE, ERROR, INFO, BLATHER
from types import StringType
import asyncore, string, struct, zLOG, sys, Acquisition
import socket, errno import socket, errno
from logger import zLogger
# Use the dictionary to make sure we get the minimum number of errno # Use the dictionary to make sure we get the minimum number of errno
# entries. We expect that EWOULDBLOCK == EAGAIN on most systems -- # entries. We expect that EWOULDBLOCK == EAGAIN on most systems --
...@@ -38,81 +41,103 @@ tmp_dict = {errno.EAGAIN: 0, ...@@ -38,81 +41,103 @@ tmp_dict = {errno.EAGAIN: 0,
expected_socket_write_errors = tuple(tmp_dict.keys()) expected_socket_write_errors = tuple(tmp_dict.keys())
del tmp_dict del tmp_dict
class SizedMessageAsyncConnection(Acquisition.Explicit, asyncore.dispatcher): class SizedMessageAsyncConnection(asyncore.dispatcher):
__super_init = asyncore.dispatcher.__init__
__super_close = asyncore.dispatcher.close
__closed = 1 # Marker indicating that we're closed
__append=None # Marker indicating that we're closed socket = None # to outwit Sam's getattr
socket=None # to outwit Sam's getattr READ_SIZE = 8096
def __init__(self, sock, addr, map=None, debug=None): def __init__(self, sock, addr, map=None, debug=None):
SizedMessageAsyncConnection.inheritedAttribute( self.addr = addr
'__init__')(self, sock, map) if debug is not None:
self.addr=addr
if debug is None and __debug__:
self._debug = zLogger("smac")
else:
self._debug = debug self._debug = debug
self.__state=None elif not hasattr(self, '_debug'):
self.__inp=None self._debug = __debug__ and 'smac'
self.__inpl=0 self.__state = None
self.__l=4 self.__inp = None # None, a single String, or a list
self.__output=output=[] self.__input_len = 0
self.__append=output.append self.__msg_size = 4
self.__pop=output.pop self.__output = []
self.__closed = None
def handle_read(self, self.__super_init(sock, map)
join=string.join, StringType=type(''), _type=type,
_None=None): # XXX avoid expensive getattr calls? Can't remember exactly what
# this comment was supposed to mean, but it has something to do
# with the way asyncore uses getattr and uses if sock:
def __nonzero__(self):
return 1
def handle_read(self):
# Use a single __inp buffer and integer indexes to make this
# fast.
try: try:
d=self.recv(8096) d=self.recv(8096)
except socket.error, err: except socket.error, err:
if err[0] in expected_socket_read_errors: if err[0] in expected_socket_read_errors:
return return
raise raise
if not d: return if not d:
return
input_len = self.__input_len + len(d)
msg_size = self.__msg_size
state = self.__state
inp=self.__inp inp = self.__inp
if inp is _None: if msg_size > input_len:
inp=d if inp is None:
elif _type(inp) is StringType: self.__inp = d
inp=[inp,d] elif type(self.__inp) is StringType:
self.__inp = [self.__inp, d]
else:
self.__inp.append(d)
self.__input_len = input_len
return # keep waiting for more input
# load all previous input and d into single string inp
if isinstance(inp, StringType):
inp = inp + d
elif inp is None:
inp = d
else: else:
inp.append(d) inp.append(d)
inp = "".join(inp)
inpl=self.__inpl+len(d) offset = 0
l=self.__l while (offset + msg_size) <= input_len:
msg = inp[offset:offset + msg_size]
while 1: offset = offset + msg_size
if state is None:
if l <= inpl:
# Woo hoo, we have enough data
if _type(inp) is not StringType: inp=join(inp,'')
d=inp[:l]
inp=inp[l:]
inpl=inpl-l
if self.__state is _None:
# waiting for message # waiting for message
l=struct.unpack(">i",d)[0] msg_size = struct.unpack(">i", msg)[0]
self.__state=1 state = 1
else:
l=4
self.__state=_None
self.message_input(d)
else: else:
break # not enough data msg_size = 4
state = None
self.message_input(msg)
self.__l=l self.__state = state
self.__inp=inp self.__msg_size = msg_size
self.__inpl=inpl self.__inp = inp[offset:]
self.__input_len = input_len - offset
def readable(self): return 1 def readable(self):
def writable(self): return not not self.__output return 1
def writable(self):
if len(self.__output) == 0:
return 0
else:
return 1
def handle_write(self): def handle_write(self):
output=self.__output output = self.__output
while output: while output:
v=output[0] v = output[0]
try: try:
n=self.send(v) n=self.send(v)
except socket.error, err: except socket.error, err:
...@@ -120,37 +145,33 @@ class SizedMessageAsyncConnection(Acquisition.Explicit, asyncore.dispatcher): ...@@ -120,37 +145,33 @@ class SizedMessageAsyncConnection(Acquisition.Explicit, asyncore.dispatcher):
break # we couldn't write anything break # we couldn't write anything
raise raise
if n < len(v): if n < len(v):
output[0]=v[n:] output[0] = v[n:]
break # we can't write any more break # we can't write any more
else: else:
del output[0] del output[0]
#break # waaa
def handle_close(self): def handle_close(self):
self.close() self.close()
def message_output(self, message, def message_output(self, message):
pack=struct.pack, len=len): if __debug__:
if self._debug is not None: if self._debug:
if len(message) > 40: if len(message) > 40:
m = message[:40]+' ...' m = message[:40]+' ...'
else: else:
m = message m = message
self._debug.trace('message_output %s' % `m`) LOG(self._debug, TRACE, 'message_output %s' % `m`)
append=self.__append
if append is None:
raise Disconnected("This action is temporarily unavailable.<p>")
append(pack(">i",len(message))+message) if self.__closed is not None:
raise Disconnected, (
"This action is temporarily unavailable."
"<p>"
)
# do two separate appends to avoid copying the message string
self.__output.append(struct.pack(">i", len(message)))
self.__output.append(message)
def close(self): def close(self):
if self.__append is not None: if self.__closed is None:
self.__append=None self.__closed = 1
SizedMessageAsyncConnection.inheritedAttribute('close')(self) self.__super_close()
class Disconnected(Exception):
"""The client has become disconnected from the server
"""
...@@ -11,14 +11,16 @@ ...@@ -11,14 +11,16 @@
# FOR A PARTICULAR PURPOSE # FOR A PARTICULAR PURPOSE
# #
############################################################################## ##############################################################################
"""Start the server storage. """Start the server storage.
""" """
__version__ = "$Revision: 1.32 $"[11:-2] __version__ = "$Revision: 1.33 $"[11:-2]
import sys, os, getopt, string import sys, os, getopt, string
import StorageServer
import asyncore
def directory(p, n=1): def directory(p, n=1):
d=p d=p
while n: while n:
...@@ -44,9 +46,11 @@ def get_storage(m, n, cache={}): ...@@ -44,9 +46,11 @@ def get_storage(m, n, cache={}):
def main(argv): def main(argv):
me=argv[0] me=argv[0]
sys.path[:]==filter(None, sys.path)
sys.path.insert(0, directory(me, 2)) sys.path.insert(0, directory(me, 2))
# XXX hack for profiling support
global unix, storages, zeo_pid, asyncore
args=[] args=[]
last='' last=''
for a in argv[1:]: for a in argv[1:]:
...@@ -77,14 +81,13 @@ def main(argv): ...@@ -77,14 +81,13 @@ def main(argv):
fs = os.path.join(var, 'Data.fs') fs = os.path.join(var, 'Data.fs')
usage = """%s [options] [filename] usage="""%s [options] [filename]
where options are: where options are:
-D -- Run in debug mode -D -- Run in debug mode
-d -- Generate detailed debug logging without running -d -- Set STUPD_LOG_SEVERITY to -300
in the foreground.
-U -- Unix-domain socket file to listen on -U -- Unix-domain socket file to listen on
...@@ -92,8 +95,8 @@ def main(argv): ...@@ -92,8 +95,8 @@ def main(argv):
The username to run the ZEO server as. You may want to run The username to run the ZEO server as. You may want to run
the ZEO server as 'nobody' or some other user with limited the ZEO server as 'nobody' or some other user with limited
resouces. The only works under Unix, and if the storage resouces. The only works under Unix, and if ZServer is
server is started by root. started by root.
-p port -- port to listen on -p port -- port to listen on
...@@ -116,30 +119,47 @@ def main(argv): ...@@ -116,30 +119,47 @@ def main(argv):
attr_name -- This is the name to which the storage object attr_name -- This is the name to which the storage object
is assigned in the module. is assigned in the module.
-P file -- Run under profile and dump output to file. Implies the
-s flag.
if no file name is specified, then %s is used. if no file name is specified, then %s is used.
""" % (me, fs) """ % (me, fs)
try: try:
opts, args = getopt.getopt(args, 'p:Ddh:U:sS:u:') opts, args = getopt.getopt(args, 'p:Dh:U:sS:u:P:d')
except getopt.error, err: except getopt.error, msg:
print err
print usage print usage
print msg
sys.exit(1) sys.exit(1)
port=None port = None
debug=detailed=0 debug = 0
host='' host = ''
unix=None unix =None
Z=1 Z = 1
UID='nobody' UID = 'nobody'
prof = None
detailed = 0
for o, v in opts: for o, v in opts:
if o=='-p': port=string.atoi(v) if o=='-p':
elif o=='-h': host=v port = int(v)
elif o=='-U': unix=v elif o=='-h':
elif o=='-u': UID=v host = v
elif o=='-D': debug=1 elif o=='-U':
elif o=='-d': detailed=1 unix = v
elif o=='-s': Z=0 elif o=='-u':
UID = v
elif o=='-D':
debug = 1
elif o=='-d':
detailed = 1
elif o=='-s':
Z = 0
elif o=='-P':
prof = v
if prof:
Z = 0
if port is None and unix is None: if port is None and unix is None:
print usage print usage
...@@ -153,9 +173,11 @@ def main(argv): ...@@ -153,9 +173,11 @@ def main(argv):
sys.exit(1) sys.exit(1)
fs=args[0] fs=args[0]
if debug: os.environ['Z_DEBUG_MODE']='1' __builtins__.__debug__=debug
if debug:
if detailed: os.environ['STUPID_LOG_SEVERITY']='-99999' os.environ['Z_DEBUG_MODE'] = '1'
if detailed:
os.environ['STUPID_LOG_SEVERITY'] = '-300'
from zLOG import LOG, INFO, ERROR from zLOG import LOG, INFO, ERROR
...@@ -243,7 +265,7 @@ def main(argv): ...@@ -243,7 +265,7 @@ def main(argv):
if not unix: unix=host, port if not unix: unix=host, port
ZEO.StorageServer.StorageServer(unix, storages) StorageServer.StorageServer(unix, storages)
try: try:
ppid, pid = os.getppid(), os.getpid() ppid, pid = os.getppid(), os.getpid()
...@@ -269,7 +291,6 @@ def main(argv): ...@@ -269,7 +291,6 @@ def main(argv):
asyncore.loop() asyncore.loop()
def rotate_logs(): def rotate_logs():
import zLOG import zLOG
if hasattr(zLOG.log_write, 'reinitialize'): if hasattr(zLOG.log_write, 'reinitialize'):
...@@ -292,29 +313,21 @@ def shutdown(storages, die=1): ...@@ -292,29 +313,21 @@ def shutdown(storages, die=1):
# unnecessary, since we now use so_reuseaddr. # unnecessary, since we now use so_reuseaddr.
for ignored in 1,2: for ignored in 1,2:
for socket in asyncore.socket_map.values(): for socket in asyncore.socket_map.values():
try: try: socket.close()
socket.close() except: pass
except:
pass
for storage in storages.values(): for storage in storages.values():
try: try: storage.close()
storage.close() finally: pass
except:
pass
try: try:
from zLOG import LOG, INFO from zLOG import LOG, INFO
LOG('ZEO Server', INFO, LOG('ZEO Server', INFO,
"Shutting down (%s)" % (die and "shutdown" or "restart") "Shutting down (%s)" % (die and "shutdown" or "restart")
) )
except: except: pass
pass
if die: if die: sys.exit(0)
sys.exit(0) else: sys.exit(1)
else:
sys.exit(1)
if __name__ == '__main__': if __name__=='__main__': main(sys.argv)
main(sys.argv)
##############################################################################
#
# Copyright (c) 2002 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE
#
##############################################################################
"""Tests of the distributed commit lock."""
import threading
from ZODB.Transaction import Transaction
from ZODB.tests.StorageTestBase import zodb_pickle, MinPO
import ZEO.ClientStorage
from ZEO.Exceptions import Disconnected
ZERO = '\0'*8
class DummyDB:
def invalidate(self, *args):
pass
class WorkerThread(threading.Thread):
# run the entire test in a thread so that the blocking call for
# tpc_vote() doesn't hang the test suite.
def __init__(self, storage, trans, method="tpc_finish"):
self.storage = storage
self.trans = trans
self.method = method
threading.Thread.__init__(self)
def run(self):
try:
self.storage.tpc_begin(self.trans)
oid = self.storage.new_oid()
self.storage.store(oid, ZERO, zodb_pickle(MinPO("c")), '', self.trans)
oid = self.storage.new_oid()
self.storage.store(oid, ZERO, zodb_pickle(MinPO("c")), '', self.trans)
self.storage.tpc_vote(self.trans)
if self.method == "tpc_finish":
self.storage.tpc_finish(self.trans)
else:
self.storage.tpc_abort(self.trans)
except Disconnected:
pass
class CommitLockTests:
# The commit lock tests verify that the storage successfully
# blocks and restarts transactions when there is content for a
# single storage. There are a lot of cases to cover.
# CommitLock1 checks the case where a single transaction delays
# other transactions before they actually block. IOW, by the time
# the other transactions get to the vote stage, the first
# transaction has finished.
def checkCommitLock1OnCommit(self):
self._storages = []
try:
self._checkCommitLock("tpc_finish", self._dosetup1, self._dowork1)
finally:
self._cleanup()
def checkCommitLock1OnAbort(self):
self._storages = []
try:
self._checkCommitLock("tpc_abort", self._dosetup1, self._dowork1)
finally:
self._cleanup()
def checkCommitLock2OnCommit(self):
self._storages = []
try:
self._checkCommitLock("tpc_finish", self._dosetup2, self._dowork2)
finally:
self._cleanup()
def checkCommitLock2OnAbort(self):
self._storages = []
try:
self._checkCommitLock("tpc_abort", self._dosetup2, self._dowork2)
finally:
self._cleanup()
def _cleanup(self):
for store, trans in self._storages:
store.tpc_abort(trans)
store.close()
self._storages = []
def _checkCommitLock(self, method_name, dosetup, dowork):
# check the commit lock when a client attemps a transaction,
# but fails/exits before finishing the commit.
# Start on transaction normally.
t = Transaction()
self._storage.tpc_begin(t)
# Start a second transaction on a different connection without
# blocking the test thread.
self._storages = []
for i in range(4):
storage2 = self._duplicate_client()
t2 = Transaction()
tid = `ZEO.ClientStorage.get_timestamp()` # XXX why?
dosetup(storage2, t2, tid)
if i == 0:
storage2.close()
else:
self._storages.append((storage2, t2))
oid = self._storage.new_oid()
self._storage.store(oid, ZERO, zodb_pickle(MinPO(1)), '', t)
self._storage.tpc_vote(t)
if method_name == "tpc_finish":
self._storage.tpc_finish(t)
self._storage.load(oid, '')
else:
self._storage.tpc_abort(t)
dowork(method_name)
# Make sure the server is still responsive
self._dostore()
def _dosetup1(self, storage, trans, tid):
storage.tpc_begin(trans, tid)
def _dowork1(self, method_name):
for store, trans in self._storages:
oid = store.new_oid()
store.store(oid, ZERO, zodb_pickle(MinPO("c")), '', trans)
store.tpc_vote(trans)
if method_name == "tpc_finish":
store.tpc_finish(trans)
else:
store.tpc_abort(trans)
def _dosetup2(self, storage, trans, tid):
self._threads = []
t = WorkerThread(storage, trans)
self._threads.append(t)
t.start()
def _dowork2(self, method_name):
for t in self._threads:
t.join()
def _duplicate_client(self):
"Open another ClientStorage to the same server."
# XXX argh it's hard to find the actual address
# The rpc mgr addr attribute is a list. Each element in the
# list is a socket domain (AF_INET, AF_UNIX, etc.) and an
# address.
addr = self._storage._rpc_mgr.addr[0][1]
new = ZEO.ClientStorage.ClientStorage(addr, wait=1)
new.registerDB(DummyDB(), None)
return new
def _get_timestamp(self):
t = time.time()
t = apply(TimeStamp,(time.gmtime(t)[:5]+(t%60,)))
return `t`
##############################################################################
#
# Copyright (c) 2002 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE
#
##############################################################################
"""Compromising positions involving threads."""
import threading
from ZODB.Transaction import Transaction
from ZODB.tests.StorageTestBase import zodb_pickle, MinPO
import ZEO.ClientStorage
from ZEO.Exceptions import Disconnected
ZERO = '\0'*8
class BasicThread(threading.Thread):
def __init__(self, storage, doNextEvent, threadStartedEvent):
self.storage = storage
self.trans = Transaction()
self.doNextEvent = doNextEvent
self.threadStartedEvent = threadStartedEvent
self.gotValueError = 0
self.gotDisconnected = 0
threading.Thread.__init__(self)
class GetsThroughVoteThread(BasicThread):
# This thread gets partially through a transaction before it turns
# execution over to another thread. We're trying to establish that a
# tpc_finish() after a storage has been closed by another thread will get
# a ClientStorageError error.
#
# This class gets does a tpc_begin(), store(), tpc_vote() and is waiting
# to do the tpc_finish() when the other thread closes the storage.
def run(self):
self.storage.tpc_begin(self.trans)
oid = self.storage.new_oid()
self.storage.store(oid, ZERO, zodb_pickle(MinPO("c")), '', self.trans)
self.storage.tpc_vote(self.trans)
self.threadStartedEvent.set()
self.doNextEvent.wait(10)
try:
self.storage.tpc_finish(self.trans)
except ZEO.ClientStorage.ClientStorageError:
self.gotValueError = 1
self.storage.tpc_abort(self.trans)
class GetsThroughBeginThread(BasicThread):
# This class is like the above except that it is intended to be run when
# another thread is already in a tpc_begin(). Thus, this thread will
# block in the tpc_begin until another thread closes the storage. When
# that happens, this one will get disconnected too.
def run(self):
try:
self.storage.tpc_begin(self.trans)
except ZEO.ClientStorage.ClientStorageError:
self.gotValueError = 1
class AbortsAfterBeginFailsThread(BasicThread):
# This class is identical to GetsThroughBeginThread except that it
# attempts to tpc_abort() after the tpc_begin() fails. That will raise a
# ClientDisconnected exception which implies that we don't have the lock,
# and that's what we really want to test (but it's difficult given the
# threading module's API).
def run(self):
try:
self.storage.tpc_begin(self.trans)
except ZEO.ClientStorage.ClientStorageError:
self.gotValueError = 1
try:
self.storage.tpc_abort(self.trans)
except Disconnected:
self.gotDisconnected = 1
class ThreadTests:
# Thread 1 should start a transaction, but not get all the way through it.
# Main thread should close the connection. Thread 1 should then get
# disconnected.
def checkDisconnectedOnThread2Close(self):
doNextEvent = threading.Event()
threadStartedEvent = threading.Event()
thread1 = GetsThroughVoteThread(self._storage,
doNextEvent, threadStartedEvent)
thread1.start()
threadStartedEvent.wait(10)
self._storage.close()
doNextEvent.set()
thread1.join()
self.assertEqual(thread1.gotValueError, 1)
# Thread 1 should start a transaction, but not get all the way through
# it. While thread 1 is in the middle of the transaction, a second thread
# should start a transaction, and it will block in the tcp_begin() --
# because thread 1 has acquired the lock in its tpc_begin(). Now the main
# thread closes the storage and both sub-threads should get disconnected.
def checkSecondBeginFails(self):
doNextEvent = threading.Event()
threadStartedEvent = threading.Event()
thread1 = GetsThroughVoteThread(self._storage,
doNextEvent, threadStartedEvent)
thread2 = GetsThroughBeginThread(self._storage,
doNextEvent, threadStartedEvent)
thread1.start()
threadStartedEvent.wait(1)
thread2.start()
self._storage.close()
doNextEvent.set()
thread1.join()
thread2.join()
self.assertEqual(thread1.gotValueError, 1)
self.assertEqual(thread2.gotValueError, 1)
def checkThatFailedBeginDoesNotHaveLock(self):
doNextEvent = threading.Event()
threadStartedEvent = threading.Event()
thread1 = GetsThroughVoteThread(self._storage,
doNextEvent, threadStartedEvent)
thread2 = AbortsAfterBeginFailsThread(self._storage,
doNextEvent, threadStartedEvent)
thread1.start()
threadStartedEvent.wait(1)
thread2.start()
self._storage.close()
doNextEvent.set()
thread1.join()
thread2.join()
self.assertEqual(thread1.gotValueError, 1)
self.assertEqual(thread2.gotValueError, 1)
self.assertEqual(thread2.gotDisconnected, 1)
...@@ -15,14 +15,17 @@ ...@@ -15,14 +15,17 @@
import asyncore import asyncore
import os import os
import profile
import random import random
import socket import socket
import sys import sys
import traceback
import types import types
import ZEO.ClientStorage, ZEO.StorageServer import ZEO.ClientStorage
# Change value of PROFILE to enable server-side profiling
PROFILE = 0 PROFILE = 0
if PROFILE:
import hotshot
def get_port(): def get_port():
"""Return a port that is not in use. """Return a port that is not in use.
...@@ -47,21 +50,23 @@ def get_port(): ...@@ -47,21 +50,23 @@ def get_port():
if os.name == "nt": if os.name == "nt":
def start_zeo_server(storage_name, args, port=None): def start_zeo_server(storage_name, args, addr=None):
"""Start a ZEO server in a separate process. """Start a ZEO server in a separate process.
Returns the ZEO port, the test server port, and the pid. Returns the ZEO port, the test server port, and the pid.
""" """
import ZEO.tests.winserver import ZEO.tests.winserver
if port is None: if addr is None:
port = get_port() port = get_port()
else:
port = addr[1]
script = ZEO.tests.winserver.__file__ script = ZEO.tests.winserver.__file__
if script.endswith('.pyc'): if script.endswith('.pyc'):
script = script[:-1] script = script[:-1]
args = (sys.executable, script, str(port), storage_name) + args args = (sys.executable, script, str(port), storage_name) + args
d = os.environ.copy() d = os.environ.copy()
d['PYTHONPATH'] = os.pathsep.join(sys.path) d['PYTHONPATH'] = os.pathsep.join(sys.path)
pid = os.spawnve(os.P_NOWAIT, sys.executable, args, os.environ) pid = os.spawnve(os.P_NOWAIT, sys.executable, args, d)
return ('localhost', port), ('localhost', port + 1), pid return ('localhost', port), ('localhost', port + 1), pid
else: else:
...@@ -79,9 +84,11 @@ else: ...@@ -79,9 +84,11 @@ else:
buf = self.recv(4) buf = self.recv(4)
if buf: if buf:
assert buf == "done" assert buf == "done"
server.close_server()
asyncore.socket_map.clear() asyncore.socket_map.clear()
def handle_close(self): def handle_close(self):
server.close_server()
asyncore.socket_map.clear() asyncore.socket_map.clear()
class ZEOClientExit: class ZEOClientExit:
...@@ -90,38 +97,56 @@ else: ...@@ -90,38 +97,56 @@ else:
self.pipe = pipe self.pipe = pipe
def close(self): def close(self):
try:
os.write(self.pipe, "done") os.write(self.pipe, "done")
os.close(self.pipe) os.close(self.pipe)
except os.error:
pass
def start_zeo_server(storage, addr): def start_zeo_server(storage_name, args, addr):
assert isinstance(args, types.TupleType)
rd, wr = os.pipe() rd, wr = os.pipe()
pid = os.fork() pid = os.fork()
if pid == 0: if pid == 0:
import ZEO.zrpc.log
reload(ZEO.zrpc.log)
try:
if PROFILE: if PROFILE:
p = profile.Profile() p = hotshot.Profile("stats.s.%d" % os.getpid())
p.runctx("run_server(storage, addr, rd, wr)", globals(), p.runctx("run_server(storage, addr, rd, wr)",
locals()) globals(), locals())
p.dump_stats("stats.s.%d" % os.getpid()) p.close()
else: else:
run_server(storage, addr, rd, wr) run_server(addr, rd, wr, storage_name, args)
except:
print "Exception in ZEO server process"
traceback.print_exc()
os._exit(0) os._exit(0)
else: else:
os.close(rd) os.close(rd)
return pid, ZEOClientExit(wr) return pid, ZEOClientExit(wr)
def run_server(storage, addr, rd, wr): def load_storage(name, args):
package = __import__("ZODB." + name)
mod = getattr(package, name)
klass = getattr(mod, name)
return klass(*args)
def run_server(addr, rd, wr, storage_name, args):
# in the child, run the storage server # in the child, run the storage server
global server
os.close(wr) os.close(wr)
ZEOServerExit(rd) ZEOServerExit(rd)
serv = ZEO.StorageServer.StorageServer(addr, {'1':storage}) import ZEO.StorageServer, ZEO.zrpc.server
asyncore.loop() storage = load_storage(storage_name, args)
os.close(rd) server = ZEO.StorageServer.StorageServer(addr, {'1':storage})
ZEO.zrpc.server.loop()
storage.close() storage.close()
if isinstance(addr, types.StringType): if isinstance(addr, types.StringType):
os.unlink(addr) os.unlink(addr)
def start_zeo(storage, cache=None, cleanup=None, domain="AF_INET", def start_zeo(storage_name, args, cache=None, cleanup=None,
storage_id="1", cache_size=20000000): domain="AF_INET", storage_id="1", cache_size=20000000):
"""Setup ZEO client-server for storage. """Setup ZEO client-server for storage.
Returns a ClientStorage instance and a ZEOClientExit instance. Returns a ClientStorage instance and a ZEOClientExit instance.
...@@ -137,10 +162,10 @@ else: ...@@ -137,10 +162,10 @@ else:
else: else:
raise ValueError, "bad domain: %s" % domain raise ValueError, "bad domain: %s" % domain
pid, exit = start_zeo_server(storage, addr) pid, exit = start_zeo_server(storage_name, args, addr)
s = ZEO.ClientStorage.ClientStorage(addr, storage_id, s = ZEO.ClientStorage.ClientStorage(addr, storage_id,
debug=1, client=cache, client=cache,
cache_size=cache_size, cache_size=cache_size,
min_disconnect_poll=0.5) min_disconnect_poll=0.5,
wait=1)
return s, exit, pid return s, exit, pid
...@@ -69,6 +69,7 @@ def start_server(addr): ...@@ -69,6 +69,7 @@ def start_server(addr):
def start_client(addr, client_func=None): def start_client(addr, client_func=None):
pid = os.fork() pid = os.fork()
if pid == 0: if pid == 0:
try:
import ZEO.ClientStorage import ZEO.ClientStorage
if VERBOSE: if VERBOSE:
print "Client process started:", os.getpid() print "Client process started:", os.getpid()
...@@ -78,6 +79,7 @@ def start_client(addr, client_func=None): ...@@ -78,6 +79,7 @@ def start_client(addr, client_func=None):
else: else:
client_func(cli) client_func(cli)
cli.close() cli.close()
finally:
os._exit(0) os._exit(0)
else: else:
return pid return pid
......
...@@ -103,8 +103,13 @@ def start_child(zaddr): ...@@ -103,8 +103,13 @@ def start_child(zaddr):
pid = os.fork() pid = os.fork()
if pid != 0: if pid != 0:
return pid return pid
try:
_start_child(zaddr)
finally:
os._exit(0)
storage = ClientStorage(zaddr, debug=1, min_disconnect_poll=0.5) def _start_child(zaddr):
storage = ClientStorage(zaddr, debug=1, min_disconnect_poll=0.5, wait=1)
db = ZODB.DB(storage, pool_size=NUM_CONNECTIONS) db = ZODB.DB(storage, pool_size=NUM_CONNECTIONS)
setup(db.open()) setup(db.open())
conns = [] conns = []
...@@ -129,7 +134,5 @@ def start_child(zaddr): ...@@ -129,7 +134,5 @@ def start_child(zaddr):
c.__count += 1 c.__count += 1
work(c) work(c)
os._exit(0)
if __name__ == "__main__": if __name__ == "__main__":
main() main()
##############################################################################
#
# Copyright (c) 2001, 2002 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE
#
##############################################################################
import random
import unittest
from ZEO.TransactionBuffer import TransactionBuffer
def random_string(size):
"""Return a random string of size size."""
l = [chr(random.randrange(256)) for i in range(size)]
return "".join(l)
def new_store_data():
"""Return arbitrary data to use as argument to store() method."""
return random_string(8), '', random_string(random.randrange(1000))
def new_invalidate_data():
"""Return arbitrary data to use as argument to invalidate() method."""
return random_string(8), ''
class TransBufTests(unittest.TestCase):
def checkTypicalUsage(self):
tbuf = TransactionBuffer()
tbuf.store(*new_store_data())
tbuf.invalidate(*new_invalidate_data())
tbuf.begin_iterate()
while 1:
o = tbuf.next()
if o is None:
break
tbuf.clear()
def doUpdates(self, tbuf):
data = []
for i in range(10):
d = new_store_data()
tbuf.store(*d)
data.append(d)
d = new_invalidate_data()
tbuf.invalidate(*d)
data.append(d)
tbuf.begin_iterate()
for i in range(len(data)):
x = tbuf.next()
if x[2] is None:
# the tbuf add a dummy None to invalidates
x = x[:2]
self.assertEqual(x, data[i])
def checkOrderPreserved(self):
tbuf = TransactionBuffer()
self.doUpdates(tbuf)
def checkReusable(self):
tbuf = TransactionBuffer()
self.doUpdates(tbuf)
tbuf.clear()
self.doUpdates(tbuf)
tbuf.clear()
self.doUpdates(tbuf)
def test_suite():
return unittest.makeSuite(TransBufTests, 'check')
...@@ -16,9 +16,11 @@ ...@@ -16,9 +16,11 @@
import asyncore import asyncore
import os import os
import random import random
import select
import socket import socket
import sys import sys
import tempfile import tempfile
import thread
import time import time
import types import types
import unittest import unittest
...@@ -26,22 +28,20 @@ import unittest ...@@ -26,22 +28,20 @@ import unittest
import ZEO.ClientStorage, ZEO.StorageServer import ZEO.ClientStorage, ZEO.StorageServer
import ThreadedAsync, ZEO.trigger import ThreadedAsync, ZEO.trigger
from ZODB.FileStorage import FileStorage from ZODB.FileStorage import FileStorage
from ZODB.TimeStamp import TimeStamp
from ZODB.Transaction import Transaction from ZODB.Transaction import Transaction
import thread from ZODB.tests.StorageTestBase import zodb_pickle, MinPO
import zLOG
from ZEO.tests import forker, Cache from ZEO.tests import forker, Cache, CommitLockTests, ThreadTests
from ZEO.smac import Disconnected from ZEO.smac import Disconnected
# Sorry Jim...
from ZODB.tests import StorageTestBase, BasicStorage, VersionStorage, \ from ZODB.tests import StorageTestBase, BasicStorage, VersionStorage, \
TransactionalUndoStorage, TransactionalUndoVersionStorage, \ TransactionalUndoStorage, TransactionalUndoVersionStorage, \
PackableStorage, Synchronization, ConflictResolution PackableStorage, Synchronization, ConflictResolution, RevisionStorage, \
MTStorage, ReadOnlyStorage
from ZODB.tests.MinPO import MinPO from ZODB.tests.MinPO import MinPO
from ZODB.tests.StorageTestBase import zodb_unpickle from ZODB.tests.StorageTestBase import zodb_unpickle
ZERO = '\0'*8
class DummyDB: class DummyDB:
def invalidate(self, *args): def invalidate(self, *args):
pass pass
...@@ -56,93 +56,22 @@ class PackWaitWrapper: ...@@ -56,93 +56,22 @@ class PackWaitWrapper:
def pack(self, t, f): def pack(self, t, f):
self.storage.pack(t, f, wait=1) self.storage.pack(t, f, wait=1)
class ZEOTestBase(StorageTestBase.StorageTestBase): class GenericTests(StorageTestBase.StorageTestBase,
"""Version of the storage test class that supports ZEO.
For ZEO, we don't always get the serialno/exception for a
particular store as the return value from the store. But we
will get no later than the return value from vote.
"""
def _dostore(self, oid=None, revid=None, data=None, version=None,
already_pickled=0, user=None, description=None):
"""Do a complete storage transaction.
The defaults are:
- oid=None, ask the storage for a new oid
- revid=None, use a revid of ZERO
- data=None, pickle up some arbitrary data (the integer 7)
- version=None, use the empty string version
Returns the object's new revision id.
"""
if oid is None:
oid = self._storage.new_oid()
if revid is None:
revid = ZERO
if data is None:
data = MinPO(7)
if not already_pickled:
data = StorageTestBase.zodb_pickle(data)
if version is None:
version = ''
# Begin the transaction
t = Transaction()
if user is not None:
t.user = user
if description is not None:
t.description = description
self._storage.tpc_begin(t)
# Store an object
r1 = self._storage.store(oid, revid, data, version, t)
s1 = self._get_serial(r1)
# Finish the transaction
r2 = self._storage.tpc_vote(t)
s2 = self._get_serial(r2)
self._storage.tpc_finish(t)
# s1, s2 can be None or dict
assert not (s1 and s2)
return s1 and s1[oid] or s2 and s2[oid]
def _get_serial(self, r):
"""Return oid -> serialno dict from sequence of ZEO replies."""
d = {}
if r is None:
return None
if type(r) == types.StringType:
raise RuntimeError, "unexpected ZEO response: no oid"
else:
for oid, serial in r:
if isinstance(serial, Exception):
raise serial
d[oid] = serial
return d
# Some of the ZEO tests depend on the version of FileStorage available
# for the tests. If we run these tests using Zope 2.3, FileStorage
# doesn't support TransactionalUndo.
if hasattr(FileStorage, 'supportsTransactionalUndo'):
# XXX Assume that a FileStorage that supports transactional undo
# also supports conflict resolution.
class VersionDependentTests(
TransactionalUndoStorage.TransactionalUndoStorage, TransactionalUndoStorage.TransactionalUndoStorage,
TransactionalUndoVersionStorage.TransactionalUndoVersionStorage, TransactionalUndoVersionStorage.TransactionalUndoVersionStorage,
ConflictResolution.ConflictResolvingStorage, ConflictResolution.ConflictResolvingStorage,
ConflictResolution.ConflictResolvingTransUndoStorage): ConflictResolution.ConflictResolvingTransUndoStorage,
pass
else:
class VersionDependentTests:
pass
class GenericTests(ZEOTestBase,
VersionDependentTests,
Cache.StorageWithCache, Cache.StorageWithCache,
Cache.TransUndoStorageWithCache, Cache.TransUndoStorageWithCache,
BasicStorage.BasicStorage, BasicStorage.BasicStorage,
VersionStorage.VersionStorage, VersionStorage.VersionStorage,
RevisionStorage.RevisionStorage,
PackableStorage.PackableStorage, PackableStorage.PackableStorage,
Synchronization.SynchronizedStorage, Synchronization.SynchronizedStorage,
MTStorage.MTStorage,
ReadOnlyStorage.ReadOnlyStorage,
CommitLockTests.CommitLockTests,
ThreadTests.ThreadTests,
): ):
"""An abstract base class for ZEO tests """An abstract base class for ZEO tests
...@@ -152,94 +81,61 @@ class GenericTests(ZEOTestBase, ...@@ -152,94 +81,61 @@ class GenericTests(ZEOTestBase,
returns a specific storage, e.g. FileStorage. returns a specific storage, e.g. FileStorage.
""" """
__super_setUp = StorageTestBase.StorageTestBase.setUp
__super_tearDown = StorageTestBase.StorageTestBase.tearDown
def setUp(self): def setUp(self):
"""Start a ZEO server using a Unix domain socket zLOG.LOG("testZEO", zLOG.INFO, "setUp() %s" % self.id())
The ZEO server uses the storage object returned by the
getStorage() method.
"""
self.__super_setUp()
self.running = 1 self.running = 1
client, exit, pid = forker.start_zeo(self.getStorage()) client, exit, pid = forker.start_zeo(*self.getStorage())
self._pid = pid self._pids = [pid]
self._server = exit self._servers = [exit]
self._storage = PackWaitWrapper(client) self._storage = PackWaitWrapper(client)
client.registerDB(DummyDB(), None) client.registerDB(DummyDB(), None)
def tearDown(self): def tearDown(self):
"""Try to cause the tests to halt"""
self.running = 0 self.running = 0
self._storage.status()
self._storage.close() self._storage.close()
self._server.close() for server in self._servers:
os.waitpid(self._pid, 0) server.close()
for pid in self._pids:
os.waitpid(pid, 0)
self.delStorage() self.delStorage()
self.__super_tearDown()
def checkTwoArgBegin(self): def open(self, read_only=0):
# XXX ZEO doesn't support 2-arg begin # XXX Needed to support ReadOnlyStorage tests. Ought to be a
pass # cleaner way.
# Is this the only way to get the address?
addr = self._storage._rpc_mgr.addr[0][1]
self._storage.close()
self._storage = ZEO.ClientStorage.ClientStorage(addr, read_only=1,
wait=1)
def checkLargeUpdate(self): def checkLargeUpdate(self):
obj = MinPO("X" * (10 * 128 * 1024)) obj = MinPO("X" * (10 * 128 * 1024))
self._dostore(data=obj) self._dostore(data=obj)
def checkCommitLockOnCommit(self): def checkZEOInvalidation(self):
self._checkCommitLock("tpc_finish") addr = self._storage._rpc_mgr.addr[0][1]
storage2 = ZEO.ClientStorage.ClientStorage(addr, wait=1,
def checkCommitLockOnAbort(self): min_disconnect_poll=0.1)
self._checkCommitLock("tpc_abort") try:
def _checkCommitLock(self, method_name):
# check the commit lock when a client attemps a transaction,
# but fails/exits before finishing the commit.
# Start on transaction normally.
t = Transaction()
self._storage.tpc_begin(t)
# Start a second transaction on a different connection without
# blocking the test thread.
self._storages = []
for i in range(3):
storage2 = self._duplicate_client()
t2 = Transaction()
tid = self._get_timestamp()
storage2._call.sendMessage('tpc_begin_sync',
tid, t2.user, t2.description,
t2._extension)
if i == 0:
storage2.close()
else:
self._storages.append((storage2, t2))
oid = self._storage.new_oid() oid = self._storage.new_oid()
self._storage.store(oid, None, '', '', t) ob = MinPO('first')
self._storage.tpc_vote(t) revid1 = self._dostore(oid, data=ob)
self._storage.status() data, serial = storage2.load(oid, '')
self._storage.tpc_finish(t) self.assertEqual(zodb_unpickle(data), MinPO('first'))
self.assertEqual(serial, revid1)
for store, trans in self._storages: revid2 = self._dostore(oid, data=MinPO('second'), revid=revid1)
store.tpc_abort(trans) for n in range(3):
store.close() # Let the server and client talk for a moment.
# Is there a better way to do this?
# Make sure the server is still responsive asyncore.poll(0.1)
self._dostore() data, serial = storage2.load(oid, '')
self.assertEqual(zodb_unpickle(data), MinPO('second'),
def _duplicate_client(self): 'Invalidation message was not sent!')
"Open another ClientStorage to the same server." self.assertEqual(serial, revid2)
addr = self._storage._connection finally:
new = ZEO.ClientStorage.ClientStorage(addr) storage2.close()
new.registerDB(DummyDB(), None)
return new
def _get_timestamp(self):
t = time.time()
t = apply(TimeStamp,(time.gmtime(t)[:5]+(t%60,)))
return `t`
class ZEOFileStorageTests(GenericTests): class ZEOFileStorageTests(GenericTests):
__super_setUp = GenericTests.setUp __super_setUp = GenericTests.setUp
...@@ -249,11 +145,12 @@ class ZEOFileStorageTests(GenericTests): ...@@ -249,11 +145,12 @@ class ZEOFileStorageTests(GenericTests):
self.__super_setUp() self.__super_setUp()
def getStorage(self): def getStorage(self):
return FileStorage(self.__fs_base, create=1) self.__fs_base = tempfile.mktemp()
return 'FileStorage', (self.__fs_base, '1')
def delStorage(self): def delStorage(self):
# file storage appears to create four files # file storage appears to create four files
for ext in '', '.index', '.lock', '.tmp': for ext in '', '.index', '.lock', '.tmp', '.old':
path = self.__fs_base + ext path = self.__fs_base + ext
try: try:
os.remove(path) os.remove(path)
...@@ -267,17 +164,14 @@ class WindowsGenericTests(GenericTests): ...@@ -267,17 +164,14 @@ class WindowsGenericTests(GenericTests):
can't be created in the parent process and passed to the child. can't be created in the parent process and passed to the child.
All the work has to be done in the server's process. All the work has to be done in the server's process.
""" """
__super_setUp = StorageTestBase.StorageTestBase.setUp
__super_tearDown = StorageTestBase.StorageTestBase.tearDown
def setUp(self): def setUp(self):
self.__super_setUp()
args = self.getStorageInfo() args = self.getStorageInfo()
name = args[0] name = args[0]
args = args[1:] args = args[1]
zeo_addr, self.test_addr, self.test_pid = \ zeo_addr, self.test_addr, self.test_pid = \
forker.start_zeo_server(name, args) forker.start_zeo_server(name, args)
storage = ZEO.ClientStorage.ClientStorage(zeo_addr, debug=1, storage = ZEO.ClientStorage.ClientStorage(zeo_addr, wait=1,
min_disconnect_poll=0.1) min_disconnect_poll=0.1)
self._storage = PackWaitWrapper(storage) self._storage = PackWaitWrapper(storage)
storage.registerDB(DummyDB(), None) storage.registerDB(DummyDB(), None)
...@@ -288,16 +182,14 @@ class WindowsGenericTests(GenericTests): ...@@ -288,16 +182,14 @@ class WindowsGenericTests(GenericTests):
s.connect(self.test_addr) s.connect(self.test_addr)
s.close() s.close()
# the connection should cause the storage server to die # the connection should cause the storage server to die
## os.waitpid(self.test_pid, 0)
time.sleep(0.5) time.sleep(0.5)
self.delStorage() self.delStorage()
self.__super_tearDown()
class WindowsZEOFileStorageTests(WindowsGenericTests): class WindowsZEOFileStorageTests(WindowsGenericTests):
def getStorageInfo(self): def getStorageInfo(self):
self.__fs_base = tempfile.mktemp() self.__fs_base = tempfile.mktemp()
return 'FileStorage', self.__fs_base, '1' return 'FileStorage', (self.__fs_base, '1') # create=1
def delStorage(self): def delStorage(self):
# file storage appears to create four files # file storage appears to create four files
...@@ -308,7 +200,7 @@ class WindowsZEOFileStorageTests(WindowsGenericTests): ...@@ -308,7 +200,7 @@ class WindowsZEOFileStorageTests(WindowsGenericTests):
except os.error: except os.error:
pass pass
class ConnectionTests(ZEOTestBase): class ConnectionTests(StorageTestBase.StorageTestBase):
"""Tests that explicitly manage the server process. """Tests that explicitly manage the server process.
To test the cache or re-connection, these test cases explicit To test the cache or re-connection, these test cases explicit
...@@ -322,53 +214,124 @@ class ConnectionTests(ZEOTestBase): ...@@ -322,53 +214,124 @@ class ConnectionTests(ZEOTestBase):
ports.append(random.randrange(25000, 30000)) ports.append(random.randrange(25000, 30000))
del i del i
def openClientStorage(self, cache='', cache_size=200000, wait=1): def setUp(self):
# defined by subclasses """Start a ZEO server using a Unix domain socket
pass
def shutdownServer(self): The ZEO server uses the storage object returned by the
# defined by subclasses getStorage() method.
pass """
self.running = 1
self.file = tempfile.mktemp()
self.addr = []
self._pids = []
self._servers = []
self._newAddr()
self._startServer()
def _newAddr(self):
self.addr.append(self._getAddr())
def _getAddr(self):
return 'localhost', self.ports.pop()
def openClientStorage(self, cache='', cache_size=200000, wait=1):
base = ZEO.ClientStorage.ClientStorage(self.addr,
client=cache,
cache_size=cache_size,
wait=wait,
min_disconnect_poll=0.1)
storage = PackWaitWrapper(base)
storage.registerDB(DummyDB(), None)
return storage
def tearDown(self): def tearDown(self):
"""Try to cause the tests to halt""" """Try to cause the tests to halt"""
if getattr(self, '_storage', None) is not None:
self._storage.close()
self.shutdownServer() self.shutdownServer()
# file storage appears to create four files # file storage appears to create four files
for i in range(len(self.addr)):
for ext in '', '.index', '.lock', '.tmp': for ext in '', '.index', '.lock', '.tmp':
path = self.file + ext path = "%s.%s%s" % (self.file, i, ext)
if os.path.exists(path): if os.path.exists(path):
try:
os.unlink(path) os.unlink(path)
except os.error:
pass
for i in 0, 1: for i in 0, 1:
path = "c1-test-%d.zec" % i path = "c1-test-%d.zec" % i
if os.path.exists(path): if os.path.exists(path):
try:
os.unlink(path) os.unlink(path)
except os.error:
pass
self.__super_tearDown() self.__super_tearDown()
def checkMultipleAddresses(self):
for i in range(4):
self._newAddr()
self._storage = self.openClientStorage('test', 100000, wait=1)
oid = self._storage.new_oid()
obj = MinPO(12)
revid1 = self._dostore(oid, data=obj)
self._storage.close()
def checkMultipleServers(self):
# XXX crude test at first -- just start two servers and do a
# commit at each one.
self._newAddr()
self._storage = self.openClientStorage('test', 100000, wait=1)
self._dostore()
self.shutdownServer(index=0)
self._startServer(index=1)
# If we can still store after shutting down one of the
# servers, we must be reconnecting to the other server.
for i in range(10):
try:
self._dostore()
break
except Disconnected:
time.sleep(0.5)
def checkDisconnectionError(self):
# Make sure we get a Disconnected when we try to read an
# object when we're not connected to a storage server and the
# object is not in the cache.
self.shutdownServer()
self._storage = self.openClientStorage('test', 1000, wait=0)
self.assertRaises(Disconnected, self._storage.load, 'fredwash', '')
def checkBasicPersistence(self): def checkBasicPersistence(self):
"""Verify cached data persists across client storage instances. # Verify cached data persists across client storage instances.
To verify that the cache is being used, the test closes the # To verify that the cache is being used, the test closes the
server and then starts a new client with the server down. # server and then starts a new client with the server down.
""" # When the server is down, a load() gets the data from its cache.
self._storage = self.openClientStorage('test', 100000, 1)
self._storage = self.openClientStorage('test', 100000, wait=1)
oid = self._storage.new_oid() oid = self._storage.new_oid()
obj = MinPO(12) obj = MinPO(12)
revid1 = self._dostore(oid, data=obj) revid1 = self._dostore(oid, data=obj)
self._storage.close() self._storage.close()
self.shutdownServer() self.shutdownServer()
self._storage = self.openClientStorage('test', 100000, 0) self._storage = self.openClientStorage('test', 100000, wait=0)
data, revid2 = self._storage.load(oid, '') data, revid2 = self._storage.load(oid, '')
assert zodb_unpickle(data) == MinPO(12) self.assertEqual(zodb_unpickle(data), MinPO(12))
assert revid1 == revid2 self.assertEqual(revid1, revid2)
self._storage.close() self._storage.close()
def checkRollover(self): def checkRollover(self):
"""Check that the cache works when the files are swapped. # Check that the cache works when the files are swapped.
In this case, only one object fits in a cache file. When the # In this case, only one object fits in a cache file. When the
cache files swap, the first object is effectively uncached. # cache files swap, the first object is effectively uncached.
"""
self._storage = self.openClientStorage('test', 1000, 1) self._storage = self.openClientStorage('test', 1000, wait=1)
oid1 = self._storage.new_oid() oid1 = self._storage.new_oid()
obj1 = MinPO("1" * 500) obj1 = MinPO("1" * 500)
revid1 = self._dostore(oid1, data=obj1) revid1 = self._dostore(oid1, data=obj1)
...@@ -377,123 +340,88 @@ class ConnectionTests(ZEOTestBase): ...@@ -377,123 +340,88 @@ class ConnectionTests(ZEOTestBase):
revid2 = self._dostore(oid2, data=obj2) revid2 = self._dostore(oid2, data=obj2)
self._storage.close() self._storage.close()
self.shutdownServer() self.shutdownServer()
self._storage = self.openClientStorage('test', 1000, 0) self._storage = self.openClientStorage('test', 1000, wait=0)
self._storage.load(oid1, '')
self._storage.load(oid2, '') self._storage.load(oid2, '')
self.assertRaises(Disconnected, self._storage.load, oid1, '')
def checkReconnection(self): def checkReconnection(self):
"""Check that the client reconnects when a server restarts.""" # Check that the client reconnects when a server restarts.
# XXX Seem to get occasional errors that look like this:
# File ZEO/zrpc2.py, line 217, in handle_request
# File ZEO/StorageServer.py, line 325, in storea
# File ZEO/StorageServer.py, line 209, in _check_tid
# StorageTransactionError: (None, <tid>)
# could system reconnect and continue old transaction?
from ZEO.ClientStorage import ClientDisconnected
self._storage = self.openClientStorage() self._storage = self.openClientStorage()
oid = self._storage.new_oid() oid = self._storage.new_oid()
obj = MinPO(12) obj = MinPO(12)
revid1 = self._dostore(oid, data=obj) revid1 = self._dostore(oid, data=obj)
zLOG.LOG("checkReconnection", zLOG.INFO,
"About to shutdown server")
self.shutdownServer() self.shutdownServer()
self.running = 1 self.running = 1
zLOG.LOG("checkReconnection", zLOG.INFO,
"About to restart server")
self._startServer(create=0) self._startServer(create=0)
oid = self._storage.new_oid() oid = self._storage.new_oid()
obj = MinPO(12) obj = MinPO(12)
while 1: while 1:
try: try:
revid1 = self._dostore(oid, data=obj) revid1 = self._dostore(oid, data=obj)
except (ClientDisconnected, thread.error, socket.error), err:
get_transaction().abort()
time.sleep(0.1)
else:
break break
except (Disconnected, select.error, thread.error, socket.error), \
err:
zLOG.LOG("checkReconnection", zLOG.INFO,
"Error after server restart; retrying.",
error=sys.exc_info())
get_transaction().abort()
time.sleep(0.1) # XXX how long to sleep
# XXX This is a bloody pain. We're placing a heavy burden # XXX This is a bloody pain. We're placing a heavy burden
# on users to catch a plethora of exceptions in order to # on users to catch a plethora of exceptions in order to
# write robust code. Need to think about implementing # write robust code. Need to think about implementing
# John Heintz's suggestion to make sure all exceptions # John Heintz's suggestion to make sure all exceptions
# inherit from POSException. # inherit from POSException.
zLOG.LOG("checkReconnection", zLOG.INFO, "finished")
class UnixConnectionTests(ConnectionTests): class UnixConnectionTests(ConnectionTests):
__super_setUp = StorageTestBase.StorageTestBase.setUp
def setUp(self):
"""Start a ZEO server using a Unix domain socket
The ZEO server uses the storage object returned by the
getStorage() method.
"""
self.running = 1
self.file = tempfile.mktemp()
self.addr = '', self.ports.pop()
self._startServer()
self.__super_setUp()
def _startServer(self, create=1):
fs = FileStorage(self.file, create=create)
self._pid, self._server = forker.start_zeo_server(fs, self.addr)
def openClientStorage(self, cache='', cache_size=200000, wait=1): def _startServer(self, create=1, index=0):
base = ZEO.ClientStorage.ClientStorage(self.addr, path = "%s.%d" % (self.file, index)
client=cache, addr = self.addr[index]
cache_size=cache_size, pid, server = forker.start_zeo_server('FileStorage',
wait_for_server_on_startup=wait) (path, create), addr)
storage = PackWaitWrapper(base) self._pids.append(pid)
storage.registerDB(DummyDB(), None) self._servers.append(server)
return storage
def shutdownServer(self): def shutdownServer(self, index=0):
if self.running: if self.running:
self.running = 0 self.running = 0
self._server.close() self._servers[index].close()
os.waitpid(self._pid, 0) try:
os.waitpid(self._pids[index], 0)
except os.error:
pass
class WindowsConnectionTests(ConnectionTests): class WindowsConnectionTests(ConnectionTests):
__super_setUp = StorageTestBase.StorageTestBase.setUp
def setUp(self):
self.file = tempfile.mktemp()
self._startServer()
self.__super_setUp()
def _startServer(self, create=1): def _startServer(self, create=1, index=0):
if create == 0: path = "%s.%d" % (self.file, index)
port = self.addr[1] addr = self.addr[index]
else: _addr, test_addr, test_pid = forker.start_zeo_server('FileStorage',
port = None (path, str(create)), addr)
self.addr, self.test_a, pid = forker.start_zeo_server('FileStorage', self._pids.append(test_pid)
(self.file, self._servers.append(test_addr)
str(create)),
port)
self.running = 1
def openClientStorage(self, cache='', cache_size=200000, wait=1):
base = ZEO.ClientStorage.ClientStorage(self.addr,
client=cache,
cache_size=cache_size,
debug=1,
wait_for_server_on_startup=wait)
storage = PackWaitWrapper(base)
storage.registerDB(DummyDB(), None)
return storage
def shutdownServer(self): def shutdownServer(self, index=0):
if self.running: if self.running:
self.running = 0 self.running = 0
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(self.test_a) s.connect(self._servers[index])
s.close() s.close()
time.sleep(1.0) # XXX waitpid() isn't available until Python 2.3
time.sleep(0.5)
def tearDown(self):
self.shutdownServer()
def get_methods(klass):
l = [klass]
meth = {}
while l:
klass = l.pop(0)
for base in klass.__bases__:
l.append(base)
for k, v in klass.__dict__.items():
if callable(v):
meth[k] = 1
return meth.keys()
if os.name == "posix": if os.name == "posix":
test_classes = ZEOFileStorageTests, UnixConnectionTests test_classes = ZEOFileStorageTests, UnixConnectionTests
...@@ -502,36 +430,12 @@ elif os.name == "nt": ...@@ -502,36 +430,12 @@ elif os.name == "nt":
else: else:
raise RuntimeError, "unsupported os: %s" % os.name raise RuntimeError, "unsupported os: %s" % os.name
def makeTestSuite(testname=''): def test_suite():
suite = unittest.TestSuite() suite = unittest.TestSuite()
name = 'check' + testname
lname = len(name)
for klass in test_classes: for klass in test_classes:
for meth in get_methods(klass): sub = unittest.makeSuite(klass, 'check')
if meth[:lname] == name: suite.addTest(sub)
suite.addTest(klass(meth))
return suite return suite
def test_suite():
return makeTestSuite()
def main():
import sys, getopt
name_of_test = ''
opts, args = getopt.getopt(sys.argv[1:], 'n:')
for flag, val in opts:
if flag == '-n':
name_of_test = val
if args:
print "Did not expect arguments. Got %s" % args
return 0
tests = makeTestSuite(name_of_test)
runner = unittest.TextTestRunner()
runner.run(tests)
if __name__ == "__main__": if __name__ == "__main__":
main() unittest.main(defaultTest='test_suite')
...@@ -11,12 +11,8 @@ ...@@ -11,12 +11,8 @@
# FOR A PARTICULAR PURPOSE # FOR A PARTICULAR PURPOSE
# #
############################################################################## ##############################################################################
# This module is a simplified version of the select_trigger module
# from Sam Rushing's Medusa server.
import asyncore import asyncore
import errno
import os import os
import socket import socket
import string import string
...@@ -24,7 +20,7 @@ import thread ...@@ -24,7 +20,7 @@ import thread
if os.name == 'posix': if os.name == 'posix':
class trigger(asyncore.file_dispatcher): class trigger (asyncore.file_dispatcher):
"Wake up a call to select() running in the main thread" "Wake up a call to select() running in the main thread"
...@@ -56,46 +52,50 @@ if os.name == 'posix': ...@@ -56,46 +52,50 @@ if os.name == 'posix':
# new data onto a channel's outgoing data queue at the same time that # new data onto a channel's outgoing data queue at the same time that
# the main thread is trying to remove some] # the main thread is trying to remove some]
def __init__(self): def __init__ (self):
r, w = self._fds = os.pipe() r, w = self._fds = os.pipe()
self.trigger = w self.trigger = w
asyncore.file_dispatcher.__init__(self, r) asyncore.file_dispatcher.__init__ (self, r)
self.lock = thread.allocate_lock() self.lock = thread.allocate_lock()
self.thunks = [] self.thunks = []
self._closed = None
# Override the asyncore close() method, because it seems that
# it would only close the r file descriptor and not w. The
# constructor calls file_dispactcher.__init__ and passes r,
# which would get stored in a file_wrapper and get closed by
# the default close. But that would leave w open...
def __del__(self): def close(self):
os.close(self._fds[0]) if self._closed is None:
os.close(self._fds[1]) self._closed = 1
self.del_channel()
for fd in self._fds:
os.close(fd)
def __repr__(self): def __repr__ (self):
return '<select-trigger(pipe) at %x>' % id(self) return '<select-trigger (pipe) at %x>' % id(self)
def readable(self): def readable (self):
return 1 return 1
def writable(self): def writable (self):
return 0 return 0
def handle_connect(self): def handle_connect (self):
pass pass
def pull_trigger(self, thunk=None): def pull_trigger (self, thunk=None):
# print 'PULL_TRIGGER: ', len(self.thunks)
if thunk: if thunk:
try: try:
self.lock.acquire() self.lock.acquire()
self.thunks.append(thunk) self.thunks.append (thunk)
finally: finally:
self.lock.release() self.lock.release()
os.write(self.trigger, 'x') os.write (self.trigger, 'x')
def handle_read(self): def handle_read (self):
try: self.recv (8192)
self.recv(8192)
except os.error, err:
if err[0] == errno.EAGAIN: # resource temporarily unavailable
return
raise
try: try:
self.lock.acquire() self.lock.acquire()
for thunk in self.thunks: for thunk in self.thunks:
...@@ -104,7 +104,7 @@ if os.name == 'posix': ...@@ -104,7 +104,7 @@ if os.name == 'posix':
except: except:
nil, t, v, tbinfo = asyncore.compact_traceback() nil, t, v, tbinfo = asyncore.compact_traceback()
print ('exception in trigger thunk:' print ('exception in trigger thunk:'
'(%s:%s %s)' % (t, v, tbinfo)) ' (%s:%s %s)' % (t, v, tbinfo))
self.thunks = [] self.thunks = []
finally: finally:
self.lock.release() self.lock.release()
...@@ -116,13 +116,13 @@ else: ...@@ -116,13 +116,13 @@ else:
# win32-safe version # win32-safe version
class trigger(asyncore.dispatcher): class trigger (asyncore.dispatcher):
address = ('127.9.9.9', 19999) address = ('127.9.9.9', 19999)
def __init__(self): def __init__ (self):
a = socket.socket(socket.AF_INET, socket.SOCK_STREAM) a = socket.socket (socket.AF_INET, socket.SOCK_STREAM)
w = socket.socket(socket.AF_INET, socket.SOCK_STREAM) w = socket.socket (socket.AF_INET, socket.SOCK_STREAM)
# set TCP_NODELAY to true to avoid buffering # set TCP_NODELAY to true to avoid buffering
w.setsockopt(socket.IPPROTO_TCP, 1, 1) w.setsockopt(socket.IPPROTO_TCP, 1, 1)
...@@ -140,50 +140,45 @@ else: ...@@ -140,50 +140,45 @@ else:
raise 'Bind Error', 'Cannot bind trigger!' raise 'Bind Error', 'Cannot bind trigger!'
port=port - 1 port=port - 1
a.listen(1) a.listen (1)
w.setblocking(0) w.setblocking (0)
try: try:
w.connect(self.address) w.connect (self.address)
except: except:
pass pass
r, addr = a.accept() r, addr = a.accept()
a.close() a.close()
w.setblocking(1) w.setblocking (1)
self.trigger = w self.trigger = w
asyncore.dispatcher.__init__(self, r) asyncore.dispatcher.__init__ (self, r)
self.lock = thread.allocate_lock() self.lock = thread.allocate_lock()
self.thunks = [] self.thunks = []
self._trigger_connected = 0 self._trigger_connected = 0
def __repr__(self): def __repr__ (self):
return '<select-trigger (loopback) at %x>' % id(self) return '<select-trigger (loopback) at %x>' % id(self)
def readable(self): def readable (self):
return 1 return 1
def writable(self): def writable (self):
return 0 return 0
def handle_connect(self): def handle_connect (self):
pass pass
def pull_trigger(self, thunk=None): def pull_trigger (self, thunk=None):
if thunk: if thunk:
try: try:
self.lock.acquire() self.lock.acquire()
self.thunks.append(thunk) self.thunks.append (thunk)
finally: finally:
self.lock.release() self.lock.release()
self.trigger.send('x') self.trigger.send ('x')
def handle_read(self): def handle_read (self):
try: self.recv (8192)
self.recv(8192)
except os.error, err:
if err[0] == errno.EAGAIN: # resource temporarily unavailable
return
raise
try: try:
self.lock.acquire() self.lock.acquire()
for thunk in self.thunks: for thunk in self.thunks:
......
...@@ -14,11 +14,14 @@ ...@@ -14,11 +14,14 @@
"""Sized message async connections """Sized message async connections
""" """
__version__ = "$Revision: 1.16 $"[11:-2] __version__ = "$Revision: 1.17 $"[11:-2]
import asyncore, struct
from Exceptions import Disconnected
from zLOG import LOG, TRACE, ERROR, INFO, BLATHER
from types import StringType
import asyncore, string, struct, zLOG, sys, Acquisition
import socket, errno import socket, errno
from logger import zLogger
# Use the dictionary to make sure we get the minimum number of errno # Use the dictionary to make sure we get the minimum number of errno
# entries. We expect that EWOULDBLOCK == EAGAIN on most systems -- # entries. We expect that EWOULDBLOCK == EAGAIN on most systems --
...@@ -38,81 +41,103 @@ tmp_dict = {errno.EAGAIN: 0, ...@@ -38,81 +41,103 @@ tmp_dict = {errno.EAGAIN: 0,
expected_socket_write_errors = tuple(tmp_dict.keys()) expected_socket_write_errors = tuple(tmp_dict.keys())
del tmp_dict del tmp_dict
class SizedMessageAsyncConnection(Acquisition.Explicit, asyncore.dispatcher): class SizedMessageAsyncConnection(asyncore.dispatcher):
__super_init = asyncore.dispatcher.__init__
__super_close = asyncore.dispatcher.close
__closed = 1 # Marker indicating that we're closed
__append=None # Marker indicating that we're closed socket = None # to outwit Sam's getattr
socket=None # to outwit Sam's getattr READ_SIZE = 8096
def __init__(self, sock, addr, map=None, debug=None): def __init__(self, sock, addr, map=None, debug=None):
SizedMessageAsyncConnection.inheritedAttribute( self.addr = addr
'__init__')(self, sock, map) if debug is not None:
self.addr=addr
if debug is None and __debug__:
self._debug = zLogger("smac")
else:
self._debug = debug self._debug = debug
self.__state=None elif not hasattr(self, '_debug'):
self.__inp=None self._debug = __debug__ and 'smac'
self.__inpl=0 self.__state = None
self.__l=4 self.__inp = None # None, a single String, or a list
self.__output=output=[] self.__input_len = 0
self.__append=output.append self.__msg_size = 4
self.__pop=output.pop self.__output = []
self.__closed = None
def handle_read(self, self.__super_init(sock, map)
join=string.join, StringType=type(''), _type=type,
_None=None): # XXX avoid expensive getattr calls? Can't remember exactly what
# this comment was supposed to mean, but it has something to do
# with the way asyncore uses getattr and uses if sock:
def __nonzero__(self):
return 1
def handle_read(self):
# Use a single __inp buffer and integer indexes to make this
# fast.
try: try:
d=self.recv(8096) d=self.recv(8096)
except socket.error, err: except socket.error, err:
if err[0] in expected_socket_read_errors: if err[0] in expected_socket_read_errors:
return return
raise raise
if not d: return if not d:
return
input_len = self.__input_len + len(d)
msg_size = self.__msg_size
state = self.__state
inp=self.__inp inp = self.__inp
if inp is _None: if msg_size > input_len:
inp=d if inp is None:
elif _type(inp) is StringType: self.__inp = d
inp=[inp,d] elif type(self.__inp) is StringType:
self.__inp = [self.__inp, d]
else:
self.__inp.append(d)
self.__input_len = input_len
return # keep waiting for more input
# load all previous input and d into single string inp
if isinstance(inp, StringType):
inp = inp + d
elif inp is None:
inp = d
else: else:
inp.append(d) inp.append(d)
inp = "".join(inp)
inpl=self.__inpl+len(d) offset = 0
l=self.__l while (offset + msg_size) <= input_len:
msg = inp[offset:offset + msg_size]
while 1: offset = offset + msg_size
if state is None:
if l <= inpl:
# Woo hoo, we have enough data
if _type(inp) is not StringType: inp=join(inp,'')
d=inp[:l]
inp=inp[l:]
inpl=inpl-l
if self.__state is _None:
# waiting for message # waiting for message
l=struct.unpack(">i",d)[0] msg_size = struct.unpack(">i", msg)[0]
self.__state=1 state = 1
else:
l=4
self.__state=_None
self.message_input(d)
else: else:
break # not enough data msg_size = 4
state = None
self.message_input(msg)
self.__l=l self.__state = state
self.__inp=inp self.__msg_size = msg_size
self.__inpl=inpl self.__inp = inp[offset:]
self.__input_len = input_len - offset
def readable(self): return 1 def readable(self):
def writable(self): return not not self.__output return 1
def writable(self):
if len(self.__output) == 0:
return 0
else:
return 1
def handle_write(self): def handle_write(self):
output=self.__output output = self.__output
while output: while output:
v=output[0] v = output[0]
try: try:
n=self.send(v) n=self.send(v)
except socket.error, err: except socket.error, err:
...@@ -120,37 +145,33 @@ class SizedMessageAsyncConnection(Acquisition.Explicit, asyncore.dispatcher): ...@@ -120,37 +145,33 @@ class SizedMessageAsyncConnection(Acquisition.Explicit, asyncore.dispatcher):
break # we couldn't write anything break # we couldn't write anything
raise raise
if n < len(v): if n < len(v):
output[0]=v[n:] output[0] = v[n:]
break # we can't write any more break # we can't write any more
else: else:
del output[0] del output[0]
#break # waaa
def handle_close(self): def handle_close(self):
self.close() self.close()
def message_output(self, message, def message_output(self, message):
pack=struct.pack, len=len): if __debug__:
if self._debug is not None: if self._debug:
if len(message) > 40: if len(message) > 40:
m = message[:40]+' ...' m = message[:40]+' ...'
else: else:
m = message m = message
self._debug.trace('message_output %s' % `m`) LOG(self._debug, TRACE, 'message_output %s' % `m`)
append=self.__append
if append is None:
raise Disconnected("This action is temporarily unavailable.<p>")
append(pack(">i",len(message))+message) if self.__closed is not None:
raise Disconnected, (
"This action is temporarily unavailable."
"<p>"
)
# do two separate appends to avoid copying the message string
self.__output.append(struct.pack(">i", len(message)))
self.__output.append(message)
def close(self): def close(self):
if self.__append is not None: if self.__closed is None:
self.__append=None self.__closed = 1
SizedMessageAsyncConnection.inheritedAttribute('close')(self) self.__super_close()
class Disconnected(Exception):
"""The client has become disconnected from the server
"""
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment