Skip to content

Z
ZEO
Commit c499ea7d authored by Jim Fulton's avatar Jim Fulton Committed by GitHub
Browse files
Options

Merge pull request #74 from zopefoundation/ZEO5 

Zeo 5
parents bc92bdd7 787cb615
Show whitespace changes
Inline Side-by-side
Showing with 9542 additions and 5342 deletions
.travis.yml
View file @ c499ea7d
......@@ -5,20 +5,31 @@ matrix:
- os: linux
python: 2.7
- os: linux
python: 3.3
python: 3.4
- os: linux
python: 3.5
- os: linux
python: 3.4
env: ZEO_MTACCEPTOR=1
- os: linux
python: 3.5
env: ZEO_MTACCEPTOR=1
- os: linux
python: 2.7
env: ZEO4_SERVER=1
- os: linux
python: 3.5
env: ZEO4_SERVER=1
- os: linux
python: pypy
python: 3.5
env: BUILOUT_OPTIONS=extra=,uvloop
install:
- pip install zc.buildout
- buildout
- buildout $BUILOUT_OPTIONS
cache:
directories:
- eggs
script:
- bin/test -v1 -j99
- bin/test -v1j99
notifications:
email: false
CHANGES.rst
View file @ c499ea7d
Changelog
=========
4.3.0 (2016-08-02)
------------------
- Added a ``ClientStorage`` ``server-sync`` configuration option and
``server_sync`` constructor argument to force a server round trip at
the beginning of transactions to wait for any outstanding
invalidations at the start of the transaction to be delivered.
- Refuse to work with ZODB 5.
- When creating an ad hoc server, a log file isn't created by
default. You must pass a ``log`` option specifying a log file name.
Some recent cleanups in the ZODB commit protocol are incompatible with ZEO 4.
- The ZEO server register method now returns the storage last
transaction, allowing the client to avoid an extra round trip during
cache verification.
- Fix ZEO cache tracing on Python 3.
- Client disconnect errors are now transient errors. When
applications retry jobs that raise transient errors, jobs (e.g. web
requests) with disconnect errors will be retried. Together with
blocking synchronous ZEO server calls for a limited time while
disconnected, this change should allow brief disconnections due to
server restart to avoid generating client-visible errors (e.g. 500
web responses).
4.2.1 (2016-06-30)
------------------
- Fixed bugs in using the ZEO 5 client with ZEO 4 servers.
5.0.0a2 (2016-07-30)
--------------------
- Added the ability to pass credentials when creating client storages.
This is experimental in that passing credentials will cause
connections to an ordinary ZEO server to fail, but it facilitates
experimentation with custom ZEO servers. Doing this with custom ZEO
clients would have been awkward due to the many levels of
composition involved.
In the future, we expect to support server security plugins that
consume credentials for authentication (typically over SSL).
Note that credentials are opaque to ZEO. They can be any object with
a true value. The client mearly passes them to the server, which
will someday pass them to a plugin.
5.0.0a1 (2016-07-21)
--------------------
- Added a ClientStorage prefetch method to prefetch oids.
When oids are prefetched, requests are made at once, but the caller
doesn't block waiting for the results. Rather, then the caller
later tries to fetch data for one of the object ids, it's either
delivered right away from the ZEO cache, if the prefetch for the
object id has completed, or the caller blocks until the inflight
prefetch completes. (No new request is made.)
- Fixed: SSL clients of servers with signed certs didn't load default
certs and were unable to connect.
5.0.0a0 (2016-07-08)
--------------------
This is a major ZEO revision, which replaces the ZEO network protocol
implementation.
New features:
- SSL support
- Optional client-side conflict resolution.
- Lots of mostly internal clean ups.
Dropped features:
- The ZEO authentication protocol.
This will be replaced by new authentication mechanims leveraging SSL.
- The ZEO monitor server.
- Full cache verification.
- Client suppprt for servers older than ZODB 3.9
- Fix bug connecting to ``localhost`` on Windows. (#8).
- Server support for clients older than ZEO 4.2.0
4.2.0 (2016-06-15)
------------------
......
README.rst
View file @ c499ea7d
===
ZEO
===
ZEO provides a client-server storage implementation for ZODB.
==========================
ZEO HOWTO
==========================
.. contents::
Usage
=====
Introduction
============
ZEO is a client-server system for sharing a single storage among many
clients. When you use ZEO, the storage is opened in the ZEO server
ZEO (Zope Enterprise Objects) is a client-server system for sharing a
single storage among many clients. When you use ZEO, a lower-level
storage, typically a file storage, is opened in the ZEO server
process. Client programs connect to this process using a ZEO
ClientStorage. ZEO provides a consistent view of the database to all
clients. The ZEO client and server communicate using a custom RPC
clients. The ZEO client and server communicate using a custom
protocol layered on top of TCP.
Options
-------
There are several configuration options that affect the behavior of a
ZEO server. This section describes how a few of these features
working. Subsequent sections describe how to configure every option.
There are several features that affect the behavior of
ZEO. This section describes how a few of these features
work. Subsequent sections describe how to configure every option.
Client cache
~~~~~~~~~~~~
------------
Each ZEO client keeps an on-disk cache of recently used objects to
avoid fetching those objects from the server each time they are
Each ZEO client keeps an on-disk cache of recently used data records
to avoid fetching those records from the server each time they are
requested. It is usually faster to read the objects from disk than it
is to fetch them over the network. The cache can also provide
read-only copies of objects during server outages.
......@@ -41,44 +37,29 @@ The client cache size is configured when the ClientStorage is created.
The default size is 20MB, but the right size depends entirely on the
particular database. Setting the cache size too small can hurt
performance, but in most cases making it too big just wastes disk
space. The document "Client cache tracing" describes how to collect a
cache trace that can be used to determine a good cache size.
space.
ZEO uses invalidations for cache consistency. Every time an object is
modified, the server sends a message to each client informing it of
the change. The client will discard the object from its cache when it
receives an invalidation. These invalidations are often batched.
receives an invalidation. (It's actually a little more complicated,
but we won't get into that here.)
Each time a client connects to a server, it must verify that its cache
contents are still valid. (It did not receive any invalidation
messages while it was disconnected.) There are several mechanisms
used to perform cache verification. In the worst case, the client
sends the server a list of all objects in its cache along with their
timestamps; the server sends back an invalidation message for each
stale object. The cost of verification is one drawback to making the
cache too large.
Note that every time a client crashes or disconnects, it must verify
its cache. Every time a server crashes, all of its clients must
verify their caches.
The cache verification process is optimized in two ways to eliminate
costs when restarting clients and servers. Each client keeps the
timestamp of the last invalidation message it has seen. When it
connects to the server, it checks to see if any invalidation messages
were sent after that timestamp. If not, then the cache is up-to-date
and no further verification occurs. The other optimization is the
invalidation queue, described below.
messages while it was disconnected.) This involves asking the server
to replay invalidations it missed. If it's been disconnected too long,
it discards its cache.
Invalidation queue
~~~~~~~~~~~~~~~~~~
------------------
The ZEO server keeps a queue of recent invalidation messages in
memory. When a client connects to the server, it sends the timestamp
of the most recent invalidation message it has received. If that
message is still in the invalidation queue, then the server sends the
client all the missing invalidations. This is often cheaper than
perform full cache verification.
client all the missing invalidations.
The default size of the invalidation queue is 100. If the
invalidation queue is larger, it will be more likely that a client
......@@ -88,8 +69,16 @@ the message. Invalidation messages tend to be small, perhaps a few
hundred bytes each on average; it depends on the number of objects
modified by a transaction.
You can also provide an invalidation age when configuring the
server. In this case, if the invalidation queue is too small, but a
client has been disconnected for a time interval that is less than the
invalidation age, then invalidations are replayed by iterating over
the lower-level storage on the server. If the age is too high, and
clients are disconneced for a long time, then this can put a lot of
load on the server.
Transaction timeouts
~~~~~~~~~~~~~~~~~~~~
--------------------
A ZEO server can be configured to timeout a transaction if it takes
too long to complete. Only a single transaction can commit at a time;
......@@ -119,78 +108,101 @@ If it did, it is possible that some storages committed the client
changes and others did not.
Connection management
~~~~~~~~~~~~~~~~~~~~~
---------------------
A ZEO client manages its connection to the ZEO server. If it loses
the connection, it attempts to reconnect. While
it is disconnected, it can satisfy some reads by using its cache.
The client can be configured to wait for a connection when it is created
or to return immediately and provide data from its persistent cache.
It usually simplifies programming to have the client wait for a
connection on startup.
When the client is disconnected, it polls periodically to see if the
server is available. The rate at which it polls is configurable.
The client can be configured with multiple server addresses. In this
case, it assumes that each server has identical content and will use
any server that is available. It is possible to configure the client
to accept a read-only connection to one of these servers if no
read-write connection is available. If it has a read-only connection,
it will continue to poll for a read-write connection. This feature
supports the Zope Replication Services product,
http://www.zope.com/Products/ZopeProducts/ZRS. In general, it could
be used to with a system that arranges to provide hot backups of
servers in the case of failure.
it will continue to poll for a read-write connection.
If a single address resolves to multiple IPv4 or IPv6 addresses,
the client will connect to an arbitrary of these addresses.
Authentication
~~~~~~~~~~~~~~
SSL
---
ZEO supports optional authentication of client and server using a
password scheme similar to HTTP digest authentication (RFC 2069). It
is a simple challenge-response protocol that does not send passwords
in the clear, but does not offer strong security. The RFC discusses
many of the limitations of this kind of protocol. Note that this
feature provides authentication only. It does not provide encryption
or confidentiality.
ZEO supports the use of SSL connections between servers and clients,
including certificate authentication. We're still understanding use
cases for this, so details of operation may change.
The challenge-response also produces a session key that is used to
generate message authentication codes for each ZEO message. This
should prevent session hijacking.
Installing software
===================
Guard the password database as if it contained plaintext passwords.
It stores the hash of a username and password. This does not expose
the plaintext password, but it is sensitive nonetheless. An attacker
with the hash can impersonate the real user. This is a limitation of
the simple digest scheme.
ZEO is installed like any other Python package using pip, buildout, or
pther Python packaging tools.
The authentication framework allows third-party developers to provide
new authentication modules.
Running the server
==================
Installing software
-------------------
Typically, the ZEO server is run using the ``runzeo`` script that's
installed as part of a ZEO installation. The ``runzeo`` script
accepts command line options, the most important of which is the
``-C`` (``--configuration``) option. ZEO servers are best configured
via configuration files. The ``runzeo`` script also accepts some
command-line arguments for ad-hoc configurations, but there's an
easier way to run an ad-hoc server described below. For more on
configuraing a ZEO server see `Server configuration`_ below.
ZEO is installed like other Python packages using pip, easy_install,
buildout, etc.
Server quick-start/ad-hoc operation
-----------------------------------
Configuring server
------------------
You can quickly start a ZEO server from a Python prompt::
import ZEO
address, stop = ZEO.server()
This runs a ZEO server on a dynamic address and using an in-memory
storage.
We can then create a ZEO client connection using the address
returned::
connection = ZEO.connection(addr)
This is a ZODB connection for a database opened on a client storage
instance created on the fly. This is a shorthand for::
db = ZEO.DB(addr)
connection = db.open()
Which is a short-hand for::
client_storage = ZEO.client(addr)
import ZODB
db = ZODB.db(client_storage)
connection = db.open()
If you exit the Python process, the storage exits as well, as it's run
in an in-process thread.
You shut down the server more cleanly by calling the stop function
returned by the ``ZEO.server`` function.
To have data stored persistently, you can specify a file-storage path
name using a ``path`` parameter. If you want blob support, you can
specify a blob-file directory using the ``blob_dir`` directory.
You can also supply a port to listen on, full storage configuration
and ZEO server configuration options to the ``ZEO.server``
function. See it's documentation string for more information.
Server configuration
--------------------
The script ``runzeo`` runs the ZEO server. The server can be
The script runzeo.py runs the ZEO server. The server can be
configured using command-line arguments or a config file. This
document only describes the config file. Run runzeo.py
-h to see the list of command-line arguments.
The ``runzeo`` script imports the ZEO package. ZEO must either be
installed in Python's site-packages directory or be in a directory on
PYTHONPATH.
The configuration file specifies the underlying storage the server
uses, the address it binds, and a few other optional parameters.
uses, the address it binds to, and a few other optional parameters.
An example is::
<zeo>
......@@ -202,27 +214,37 @@ An example is::
</filestorage>
<eventlog>
level INFO
<logfile>
path /var/tmp/zeo.log
format %(asctime)s %(message)s
</logfile>
</eventlog>
This file configures a server to use a FileStorage from
``/var/tmp/Data.fs``. The server listens on port 8090 of
zeo.example.com. The ZEO server writes its log file to
/var/tmp/zeo.log and uses a custom format for each line. Assuming the
example configuration it stored in zeo.config, you can run a server by
The format is similar to the Apache configuration format. Individual
settings have a name, 1 or more spaces and a value, as in::
address zeo.example.com:8090
Settings are grouped into hierarchical sections.
The example above configures a server to use a file storage from
``/var/tmp/Data.fs``. The server listens on port ``8090`` of
``zeo.example.com``. The ZEO server writes its log file to
``/var/tmp/zeo.log`` and uses a custom format for each line. Assuming the
example configuration it stored in ``zeo.config``, you can run a server by
typing::
python runzeo -C zeo.config
runzeo -C zeo.config
A configuration file consists of a <zeo> section and a storage
section, where the storage section can use any of the valid ZODB
storage types. It may also contain an eventlog configuration. See
the document "Configuring a ZODB database" for more information about
configuring storages and eventlogs.
ZODB documentation for information on configuring storages. See
`Configuring event logs <logs.rst>`_ for information on configuring
server logs.
An **easy way to get started** with configuration is to run an add-hoc
server and copy the generated configuration.
The zeo section must list the address. All the other keys are
optional.
......@@ -248,99 +270,77 @@ read-only
invalidation-queue-size
The storage server keeps a queue of the objects modified by the
last N transactions, where N == invalidation_queue_size. This
queue is used to speed client cache verification when a client
queue is used to support client cache verification when a client
disconnects for a short period of time.
transaction-timeout
The maximum amount of time to wait for a transaction to commit
after acquiring the storage lock, specified in seconds. If the
transaction takes too long, the client connection will be closed
and the transaction aborted.
authentication-protocol
The name of the protocol used for authentication. The
only protocol provided with ZEO is "digest," but extensions
may provide other protocols.
authentication-database
The path of the database containing authentication credentials.
authentication-realm
The authentication realm of the server. Some authentication
schemes use a realm to identify the logic set of usernames
that are accepted by this server.
Configuring clients
-------------------
The ZEO client can also be configured using ZConfig. The ZODB.config
module provides several function for opening a storage based on its
configuration.
invalidation-age
The maximum age of a client for which quick-verification
invalidations will be provided by iterating over the served
storage. This option should only be used if the served storage
supports efficient iteration from a starting point near the
end of the transaction history (e.g. end of file).
- ZODB.config.storageFromString()
- ZODB.config.storageFromFile()
- ZODB.config.storageFromURL()
The ZEO client configuration requires the server address be
specified. Everything else is optional. An example configuration is::
transaction-timeout
The maximum amount of time, in seconds, to wait for a
transaction to commit after acquiring the storage lock,
specified in seconds. If the transaction takes too long, the
client connection will be closed and the transaction aborted.
<zeoclient>
server zeo.example.com:8090
</zeoclient>
This defaults to 30 seconds.
The other configuration options are listed below.
client-conflict-resolution
Flag indicating that clients should perform conflict
resolution. This option defaults to false.
cache-size
The maximum size of the client cache, in bytes.
Server SSL configuration
~~~~~~~~~~~~~~~~~~~~~~~~
name
The storage name. If unspecified, the address of the server
will be used as the name.
A server can optionally support SSL. Do do so, include a `ssl`
subsection of the ZEO section, as in::
client
Enables persistent cache files. The string passed here is
used to construct the cache filenames. If it is not
specified, the client creates a temporary cache that will
only be used by the current object.
<zeo>
address zeo.example.com:8090
<ssl>
certificate server_certificate.pem
key server_certificate_key.pem
</ssl>
</zeo>
var
The directory where persistent cache files are stored. By
default cache files, if they are persistent, are stored in
the current directory.
<filestorage>
path /var/tmp/Data.fs
</filestorage>
min-disconnect-poll
The minimum delay in seconds between attempts to connect to
the server, in seconds. Defaults to 5 seconds.
<eventlog>
<logfile>
path /var/tmp/zeo.log
format %(asctime)s %(message)s
</logfile>
</eventlog>
max-disconnect-poll
The maximum delay in seconds between attempts to connect to
the server, in seconds. Defaults to 300 seconds.
The ``ssl`` section has settings:
wait
A boolean indicating whether the constructor should wait
for the client to connect to the server and verify the cache
before returning. The default is true.
certificate
The path to an SSL certificate file for the server. (required)
read-only
A flag indicating whether this should be a read-only storage,
defaulting to false (i.e. writing is allowed by default).
key
The path to the SSL key file for the server certificate (if not
included in certificate file).
read-only-fallback
A flag indicating whether a read-only remote storage should be
acceptable as a fallback when no writable storages are
available. Defaults to false. At most one of read_only and
read_only_fallback should be true.
password-function
The dotted name if an importable function that, when imported, returns
the password needed to unlock the key (if the key requires a password.)
realm
The authentication realm of the server. Some authentication
schemes use a realm to identify the logic set of usernames
that are accepted by this server.
authenticate
The path to a file or directory containing client certificates
to authenticate. ((See the ``cafile`` and ``capath``
parameters in the Python documentation for
``ssl.SSLContext.load_verify_locations``.)
A ZEO client can also be created by calling the ClientStorage
constructor explicitly. For example::
If this setting is used. then certificate authentication is
used to authenticate clients. A client must be configured
with one of the certificates supplied using this setting.
from ZEO.ClientStorage import ClientStorage
storage = ClientStorage(("zeo.example.com", 8090))
This option assumes that you're using self-signed certificates.
Running the ZEO server as a daemon
----------------------------------
......@@ -351,134 +351,317 @@ provides two tools for running daemons: zdrun.py and zdctl.py. You can
find zdaemon and it's documentation at
http://pypi.python.org/pypi/zdaemon.
Note that ``runzeo`` makes no attempt to implemnt a well behaved
daemon. It expects that functionality to be provided by a wrapper like
zdaemon or supervisord.
Rotating log files
~~~~~~~~~~~~~~~~~~
------------------
ZEO will re-initialize its logging subsystem when it receives a
``runzeo`` will re-initialize its logging subsystem when it receives a
SIGUSR2 signal. If you are using the standard event logger, you
should first rename the log file and then send the signal to the
server. The server will continue writing to the renamed log file
until it receives the signal. After it receives the signal, the
server will create a new file with the old name and write to it.
Tools
-----
ZEO Clients
===========
To use a ZEO server, you need to connect to it using a ZEO client
storage. You create client storages either using a Python API or
using a ZODB storage configuration in a ZODB storage configuration
section.
Python API for creating a ZEO client storage
--------------------------------------------
To create a client storage from Python, use the ``ZEO.client``
function::
import ZEO
client = ZEO.client(8200)
In the example above, we created a client that connected to a storage
listening on port 8200 on local host. The first argument is an
address, or list of addresses to connect to. There are many additinal
options, decumented below that should be given as keyword arguments.
Addresses can be:
- A host/port tuple
- An integer, which implies that the host is '127.0.0.1'
- A unix domain socket file name.
Options:
cache_size
The cache size in bytes. This defaults to a 20MB.
cache
The ZEO cache to be used. This can be a file name, which will
cause a persisetnt standard persistent ZEO cache to be used and
stored in the given name. This can also be an object that
implements ``ZEO.interfaces.ICache``.
If not specified, then a non-persistent cache will be used.
blob_dir
The name of a directory to hold/cache blob data downloaded from the
server. This must be provided if blobs are to be used. (Of
course, the server storage must be configured to use blobs as
well.)
shared_blob_dir
A client can use a network files system (or a local directory if
the server runs on the same machine) to share a blob directory with
the server. This allows downloading of blobs (except via a
distributed file system) to be avoided.
blob_cache_size
The size of the blob cache in bytes. IF unset, then blobs will
accumulate. If set, then blobs are removed when the total size
exceeds this amount. Blobs accessed least recently are removed
first.
blob_cache_size_check
The total size of data to be downloaded to trigger blob cache size
reduction. The defaukt is 10 (percent). This controls how often to
remove blobs from the cache.
There are a few scripts that may help running a ZEO server. The
zeopack script connects to a server and packs the storage. It can
be run as a cron job. The zeopasswd.py script
manages a ZEO servers password database.
ssl
An ``ssl.SSLContext`` object used to make SSL connections.
Diagnosing problems
-------------------
ssl_server_hostname
Host name to use for SSL host name checks.
If an exception occurs on the server, the server will log a traceback
and send an exception to the client. The traceback on the client will
show a ZEO protocol library as the source of the error. If you need
to diagnose the problem, you will have to look in the server log for
the rest of the traceback.
If using SSL and if host name checking is enabled in the given SSL
context then use this as the value to check. If an address is a
host/port pair, then this defaults to the host in the address.
Compatibility
=============
read_only
Set to true for a read-only connection.
ZEO 4.0.0 requires Python 2.6 or later.
If false (the default), then request a read/write connection.
Note --
When using ZEO and upgrading from Python 2.4, you need to upgrade
clients and servers at the same time, or upgrade clients first and
then servers. Clients running Python 2.5 or 2.6 will work with
servers running Python 2.4. Clients running Python 2.4 won't work
properly with servers running Python 2.5 or later due to changes in
the way Python implements exceptions.
This option is ignored if ``read_only_fallback`` is set to a true value.
For a long time ZEO has been distributes with ZODB. ZEO 4 is
is now maintained as a separate project.
read_only_fallback
Set to true, then prefer a read/write connection, but be willing to
use a read-only connection. This defaults to a false value.
ZEO clients from ZODB 3.2 on can talk to ZEO 4.0 servers.
ZEO 4.0 clients talk to ZODB 3.8, 3.9, and 3.10 and ZEO 4.0 servers.
If ``read_only_fallback`` is set, then ``read_only`` is ignored.
Note --
ZEO 4.0 servers don't support undo for clients older than ZODB 3.10.
server_sync
Flag, false by default, indicating whether the ``sync`` method
should make a server request. The ``sync`` method is called at the
start of explcitly begin transactions. Making a server requests assures
that any invalidations outstanding at the beginning of a
transaction are processed.
Testing for downloaders
=======================
Setting this to True is important when application activity is
spread over multiple ZEO clients. The classic example of this is
when a web browser makes a request to an application server (ZEO
client) that makes a change and then makes a request to another
application server that depends on the change.
You can run the tests with::
Setting this to True makes transactions a little slower because of
the added server round trip. For transactions that don't otherwise
need to access the storage server, the impact can be significant.
python setup.py test
wait_timeout
How long to wait for an initial connection, defaulting to 30
seconds. If an initial connection can't be made within this time
limit, then creation of the client storage will fail with a
``ZEO.Exceptions.ClientDisconnected`` exception.
however, there's an issue with getting the dependencies installed
propely in a single run. If the first run fails installing
dependencies, try running the above command a second time.
After the initial connection, if the client is disconnected:
Testing for Developers
======================
- In-flight server requests will fail with a
``ZEO.Exceptions.ClientDisconnected`` exception.
The ZEO checkouts are `buildouts <http://www.python.org/pypi/zc.buildout>`_.
When working from a ZODB checkout, first run the bootstrap.py script
to initialize the buildout:
- New requests will block for up to ``wait_timeout`` waiting for a
connection to be established before failing with a
``ZEO.Exceptions.ClientDisconnected`` exception.
client_label
A short string to display in *server* logs for an event relating to
this client. This can be helpful when debugging.
disconnect_poll
The delay in seconds between attempts to connect to the
server, in seconds. Defaults to 1 second.
Configuration strings/files
---------------------------
ZODB databases and storages can be configured using configuration
files, or strings (extracted from configuration files). They use the
same syntax as the server configuration files described above, but
with different sections and options.
An application that used ZODB might configure it's database using a
string like::
<zodb>
cache-size-bytes 1000MB
<filestorage>
path /var/lib/Data.fs
</filestorage>
</zodb>
In this example, we configured a ZODB database with a object cache
size of 1GB. Inside the database, we configured a file storage. The
``filestorage`` section provided file-storage parameters. We saw a
similar section in the storage-server configuration example in `Server
configuration`_.
To configure a client storage, you use a ``clientstorage`` section,
but first you have to import it's definition, because ZEO isn't built
into ZODB. Here's an example::
<zodb>
cache-size-bytes 1000MB
%import ZEO
<clientstorage>
server 8200
</clientstorage>
</zodb>
In this example, we defined a client storage that connected to a
server on port 8200.
The following settings are supported:
cache-size
The cache size in bytes, KB or MB. This defaults to a 20MB.
Optional ``KB`` or ``MB`` suffixes can (and usually are) used to
specify units other than bytes.
cache-path
The file path of a persistent cache file
blob-dir
The name of a directory to hold/cache blob data downloaded from the
server. This must be provided if blobs are to be used. (Of
course, the server storage must be configured to use blobs as
well.)
shared-blob-dir
A client can use a network files system (or a local directory if
the server runs on the same machine) to share a blob directory with
the server. This allows downloading of blobs (except via a
distributed file system) to be avoided.
blob-cache-size
The size of the blob cache in bytes. IF unset, then blobs will
accumulate. If set, then blobs are removed when the total size
exceeds this amount. Blobs accessed least recently are removed
first.
blob-cache-size-check
The total size of data to be downloaded to trigger blob cache size
reduction. The defaukt is 10 (percent). This controls how often to
remove blobs from the cache.
read-only
Set to true for a read-only connection.
If false (the default), then request a read/write connection.
This option is ignored if ``read_only_fallback`` is set to a true value.
read-only-fallback
Set to true, then prefer a read/write connection, but be willing to
use a read-only connection. This defaults to a false value.
% python bootstrap.py
If ``read_only_fallback`` is set, then ``read_only`` is ignored.
and then use the buildout script to build ZODB and gather the dependencies:
server-sync
Sets thr ``server_sync`` option described above.
% bin/buildout
wait_timeout
How long to wait for an initial connection, defaulting to 30
seconds. If an initial connection can't be made within this time
limit, then creation of the client storage will fail with a
``ZEO.Exceptions.ClientDisconnected`` exception.
This creates a test script:
After the initial connection, if the client is disconnected:
% bin/test -v
- In-flight server requests will fail with a
``ZEO.Exceptions.ClientDisconnected`` exception.
This command will run all the tests, printing a single dot for each
test. When it finishes, it will print a test summary. The exact
number of tests can vary depending on platform and available
third-party libraries.::
- New requests will block for up to ``wait_timeout`` waiting for a
connection to be established before failing with a
``ZEO.Exceptions.ClientDisconnected`` exception.
Ran 1182 tests in 241.269s
client_label
A short string to display in *server* logs for an event relating to
this client. This can be helpful when debugging.
OK
disconnect_poll
The delay in seconds between attempts to connect to the
server, in seconds. Defaults to 1 second.
The test script has many more options. Use the ``-h`` or ``--help``
options to see a file list of options. The default test suite omits
several tests that depend on third-party software or that take a long
time to run. To run all the available tests use the ``--all`` option.
Running all the tests takes much longer.::
Client SSL configuration
~~~~~~~~~~~~~~~~~~~~~~~~
Ran 1561 tests in 1461.557s
An ``ssl`` subsection can be used to enable and configure SSL, as in::
OK
%import ZEO
More information
================
<clientstorage>
server zeo.example.com8200
<ssl>
</ssl>
</clientstorage>
For more information on ZEO, see http://zodb.org
In the example above, SSL is enabled in it's simplest form:
There is a Mailman mailing list in place to discuss all issues related
to ZODB, including ZEO. You can send questions to
- The cient expects the server to have a signed certificate, which the
client validates.
zodb-dev@zope.org
- The server server host name ``zeo.example.com`` is checked against
the server's certificate.
or subscribe at
A number of settings can be provided to configure SSL:
http://lists.zope.org/mailman/listinfo/zodb-dev
certificate
The path to an SSL certificate file for the client. This is
needed to allow the server to authenticate the client.
and view its archives at
key
The path to the SSL key file for the client certificate (if not
included in the certificate file).
http://lists.zope.org/pipermail/zodb-dev
password-function
A dotted name if an importable function that, when imported, returns
the password needed to unlock the key (if the key requires a password.)
Note that Zope mailing lists have a subscriber-only posting policy.
authenticate
The path to a file or directory containing server certificates
to authenticate. ((See the ``cafile`` and ``capath``
parameters in the Python documentation for
``ssl.SSLContext.load_verify_locations``.)
Bugs and Patches
================
If this setting is used. then certificate authentication is
used to authenticate the server. The server must be configuted
with one of the certificates supplied using this setting.
Bug reports and patches should be added to the Launchpad:
check-hostname
This is a boolean setting that defaults to true. Verify the
host name in the server certificate is as expected.
https://launchpad.net/zodb
server-hostname
The expected server host name. This defaults to the host name
used in the server address. This option must be used when
``check-hostname`` is true and when a server address has no host
name (localhost, or unix domain socket) or when there is more
than one seerver and server hostnames differ.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
End:
Using this setting implies a true value for the ``check-hostname`` setting.
asyncio-todo.rst 0 → 100644
View file @ c499ea7d
ZEO networking implemention based on asyncio to-dos
===================================================
First iteration, client only
----------------------------
- socketless tests for protocol and adapters
- Disconnect/reconnect strategy
- Integration with ClientStorage
Second iteration, server
------------------------
TBD after client release.
buildout.cfg
View file @ c499ea7d
......@@ -4,6 +4,7 @@ parts =
test
scripts
versions = versions
extra =
[versions]
......@@ -11,7 +12,7 @@ versions = versions
[test]
recipe = zc.recipe.testrunner
eggs =
ZEO [test]
ZEO [test${buildout:extra}]
initialization =
import os, tempfile
try: os.mkdir('tmp')
......@@ -21,6 +22,5 @@ defaults = ['--all']
[scripts]
recipe = zc.recipe.egg
eggs =
ZEO [test]
eggs = ${test:eggs}
interpreter = py
doc/zeo.txt deleted 100644 → 0
View file @ bc92bdd7
==========================
Running a ZEO Server HOWTO
==========================
Introduction
------------
ZEO (Zope Enterprise Objects) is a client-server system for sharing a
single storage among many clients. Normally, a ZODB storage can only
be used by a single process. When you use ZEO, the storage is opened
in the ZEO server process. Client programs connect to this process
using a ZEO ClientStorage. ZEO provides a consistent view of the
database to all clients. The ZEO client and server communicate using
a custom RPC protocol layered on top of TCP.
There are several configuration options that affect the behavior of a
ZEO server. This section describes how a few of these features
working. Subsequent sections describe how to configure every option.
Client cache
~~~~~~~~~~~~
Each ZEO client keeps an on-disk cache of recently used objects to
avoid fetching those objects from the server each time they are
requested. It is usually faster to read the objects from disk than it
is to fetch them over the network. The cache can also provide
read-only copies of objects during server outages.
The cache may be persistent or transient. If the cache is persistent,
then the cache files are retained for use after process restarts. A
non-persistent cache uses temporary files that are removed when the
client storage is closed.
The client cache size is configured when the ClientStorage is created.
The default size is 20MB, but the right size depends entirely on the
particular database. Setting the cache size too small can hurt
performance, but in most cases making it too big just wastes disk
space. The document "Client cache tracing" describes how to collect a
cache trace that can be used to determine a good cache size.
ZEO uses invalidations for cache consistency. Every time an object is
modified, the server sends a message to each client informing it of
the change. The client will discard the object from its cache when it
receives an invalidation. These invalidations are often batched.
Each time a client connects to a server, it must verify that its cache
contents are still valid. (It did not receive any invalidation
messages while it was disconnected.) There are several mechanisms
used to perform cache verification. In the worst case, the client
sends the server a list of all objects in its cache along with their
timestamps; the server sends back an invalidation message for each
stale object. The cost of verification is one drawback to making the
cache too large.
Note that every time a client crashes or disconnects, it must verify
its cache. Every time a server crashes, all of its clients must
verify their caches.
The cache verification process is optimized in two ways to eliminate
costs when restarting clients and servers. Each client keeps the
timestamp of the last invalidation message it has seen. When it
connects to the server, it checks to see if any invalidation messages
were sent after that timestamp. If not, then the cache is up-to-date
and no further verification occurs. The other optimization is the
invalidation queue, described below.
Invalidation queue
~~~~~~~~~~~~~~~~~~
The ZEO server keeps a queue of recent invalidation messages in
memory. When a client connects to the server, it sends the timestamp
of the most recent invalidation message it has received. If that
message is still in the invalidation queue, then the server sends the
client all the missing invalidations. This is often cheaper than
perform full cache verification.
The default size of the invalidation queue is 100. If the
invalidation queue is larger, it will be more likely that a client
that reconnects will be able to verify its cache using the queue. On
the other hand, a large queue uses more memory on the server to store
the message. Invalidation messages tend to be small, perhaps a few
hundred bytes each on average; it depends on the number of objects
modified by a transaction.
Transaction timeouts
~~~~~~~~~~~~~~~~~~~~
A ZEO server can be configured to timeout a transaction if it takes
too long to complete. Only a single transaction can commit at a time;
so if one transaction takes too long, all other clients will be
delayed waiting for it. In the extreme, a client can hang during the
commit process. If the client hangs, the server will be unable to
commit other transactions until it restarts. A well-behaved client
will not hang, but the server can be configured with a transaction
timeout to guard against bugs that cause a client to hang.
If any transaction exceeds the timeout threshold, the client's
connection to the server will be closed and the transaction aborted.
Once the transaction is aborted, the server can start processing other
client's requests. Most transactions should take very little time to
commit. The timer begins for a transaction after all the data has
been sent to the server. At this point, the cost of commit should be
dominated by the cost of writing data to disk; it should be unusual
for a commit to take longer than 1 second. A transaction timeout of
30 seconds should tolerate heavy load and slow communications between
client and server, while guarding against hung servers.
When a transaction times out, the client can be left in an awkward
position. If the timeout occurs during the second phase of the two
phase commit, the client will log a panic message. This should only
cause problems if the client transaction involved multiple storages.
If it did, it is possible that some storages committed the client
changes and others did not.
Connection management
~~~~~~~~~~~~~~~~~~~~~
A ZEO client manages its connection to the ZEO server. If it loses
the connection, it attempts to reconnect. While
it is disconnected, it can satisfy some reads by using its cache.
The client can be configured to wait for a connection when it is created
or to return immediately and provide data from its persistent cache.
It usually simplifies programming to have the client wait for a
connection on startup.
When the client is disconnected, it polls periodically to see if the
server is available. The rate at which it polls is configurable.
The client can be configured with multiple server addresses. In this
case, it assumes that each server has identical content and will use
any server that is available. It is possible to configure the client
to accept a read-only connection to one of these servers if no
read-write connection is available. If it has a read-only connection,
it will continue to poll for a read-write connection. This feature
supports the Zope Replication Services product,
http://www.zope.com/Products/ZopeProducts/ZRS. In general, it could
be used to with a system that arranges to provide hot backups of
servers in the case of failure.
If a single address resolves to multiple IPv4 or IPv6 addresses,
the client will connect to an arbitrary of these addresses.
Authentication
~~~~~~~~~~~~~~
ZEO supports optional authentication of client and server using a
password scheme similar to HTTP digest authentication (RFC 2069). It
is a simple challenge-response protocol that does not send passwords
in the clear, but does not offer strong security. The RFC discusses
many of the limitations of this kind of protocol. Note that this
feature provides authentication only. It does not provide encryption
or confidentiality.
The challenge-response also produces a session key that is used to
generate message authentication codes for each ZEO message. This
should prevent session hijacking.
Guard the password database as if it contained plaintext passwords.
It stores the hash of a username and password. This does not expose
the plaintext password, but it is sensitive nonetheless. An attacker
with the hash can impersonate the real user. This is a limitation of
the simple digest scheme.
The authentication framework allows third-party developers to provide
new authentication modules.
Installing software
-------------------
ZEO is distributed as part of the ZODB3 package and with Zope,
starting with Zope 2.7. You can download it from
http://pypi.python.org/pypi/ZODB3.
Configuring server
------------------
The script runzeo.py runs the ZEO server. The server can be
configured using command-line arguments or a config file. This
document only describes the config file. Run runzeo.py
-h to see the list of command-line arguments.
The runzeo.py script imports the ZEO package. ZEO must either be
installed in Python's site-packages directory or be in a directory on
PYTHONPATH.
The configuration file specifies the underlying storage the server
uses, the address it binds, and a few other optional parameters.
An example is::
<zeo>
address zeo.example.com:8090
monitor-address zeo.example.com:8091
</zeo>
<filestorage 1>
path /var/tmp/Data.fs
</filestorage>
<eventlog>
<logfile>
path /var/tmp/zeo.log
format %(asctime)s %(message)s
</logfile>
</eventlog>
This file configures a server to use a FileStorage from
/var/tmp/Data.fs. The server listens on port 8090 of zeo.example.com.
It also starts a monitor server that lists in port 8091. The ZEO
server writes its log file to /var/tmp/zeo.log and uses a custom
format for each line. Assuming the example configuration it stored in
zeo.config, you can run a server by typing::
python /usr/local/bin/runzeo.py -C zeo.config
A configuration file consists of a <zeo> section and a storage
section, where the storage section can use any of the valid ZODB
storage types. It may also contain an eventlog configuration. See
the document "Configuring a ZODB database" for more information about
configuring storages and eventlogs.
The zeo section must list the address. All the other keys are
optional.
address
The address at which the server should listen. This can be in
the form 'host:port' to signify a TCP/IP connection or a
pathname string to signify a Unix domain socket connection (at
least one '/' is required). A hostname may be a DNS name or a
dotted IP address. If the hostname is omitted, the platform's
default behavior is used when binding the listening socket (''
is passed to socket.bind() as the hostname portion of the
address).
read-only
Flag indicating whether the server should operate in read-only
mode. Defaults to false. Note that even if the server is
operating in writable mode, individual storages may still be
read-only. But if the server is in read-only mode, no write
operations are allowed, even if the storages are writable. Note
that pack() is considered a read-only operation.
invalidation-queue-size
The storage server keeps a queue of the objects modified by the
last N transactions, where N == invalidation_queue_size. This
queue is used to speed client cache verification when a client
disconnects for a short period of time.
monitor-address
The address at which the monitor server should listen. If
specified, a monitor server is started. The monitor server
provides server statistics in a simple text format. This can
be in the form 'host:port' to signify a TCP/IP connection or a
pathname string to signify a Unix domain socket connection (at
least one '/' is required). A hostname may be a DNS name or a
dotted IP address. If the hostname is omitted, the platform's
default behavior is used when binding the listening socket (''
is passed to socket.bind() as the hostname portion of the
address).
transaction-timeout
The maximum amount of time to wait for a transaction to commit
after acquiring the storage lock, specified in seconds. If the
transaction takes too long, the client connection will be closed
and the transaction aborted.
authentication-protocol
The name of the protocol used for authentication. The
only protocol provided with ZEO is "digest," but extensions
may provide other protocols.
authentication-database
The path of the database containing authentication credentials.
authentication-realm
The authentication realm of the server. Some authentication
schemes use a realm to identify the logic set of usernames
that are accepted by this server.
Configuring clients
-------------------
The ZEO client can also be configured using ZConfig. The ZODB.config
module provides several function for opening a storage based on its
configuration.
- ZODB.config.storageFromString()
- ZODB.config.storageFromFile()
- ZODB.config.storageFromURL()
The ZEO client configuration requires the server address be
specified. Everything else is optional. An example configuration is::
<zeoclient>
server zeo.example.com:8090
</zeoclient>
The other configuration options are listed below.
storage
The name of the storage that the client wants to use. If the
ZEO server serves more than one storage, the client selects
the storage it wants to use by name. The default name is '1',
which is also the default name for the ZEO server.
cache-size
The maximum size of the client cache, in bytes.
name
The storage name. If unspecified, the address of the server
will be used as the name.
client
Enables persistent cache files. The string passed here is
used to construct the cache filenames. If it is not
specified, the client creates a temporary cache that will
only be used by the current object.
var
The directory where persistent cache files are stored. By
default cache files, if they are persistent, are stored in
the current directory.
min-disconnect-poll
The minimum delay in seconds between attempts to connect to
the server, in seconds. Defaults to 5 seconds.
max-disconnect-poll
The maximum delay in seconds between attempts to connect to
the server, in seconds. Defaults to 300 seconds.
wait
A boolean indicating whether the constructor should wait
for the client to connect to the server and verify the cache
before returning. The default is true.
read-only
A flag indicating whether this should be a read-only storage,
defaulting to false (i.e. writing is allowed by default).
read-only-fallback
A flag indicating whether a read-only remote storage should be
acceptable as a fallback when no writable storages are
available. Defaults to false. At most one of read_only and
read_only_fallback should be true.
realm
The authentication realm of the server. Some authentication
schemes use a realm to identify the logic set of usernames
that are accepted by this server.
A ZEO client can also be created by calling the ClientStorage
constructor explicitly. For example::
from ZEO.ClientStorage import ClientStorage
storage = ClientStorage(("zeo.example.com", 8090))
Running the ZEO server as a daemon
----------------------------------
In an operational setting, you will want to run the ZEO server a
daemon process that is restarted when it dies. The zdaemon package
provides two tools for running daemons: zdrun.py and zdctl.py. You can
find zdaemon and it's documentation at
http://pypi.python.org/pypi/zdaemon.
Rotating log files
~~~~~~~~~~~~~~~~~~
ZEO will re-initialize its logging subsystem when it receives a
SIGUSR2 signal. If you are using the standard event logger, you
should first rename the log file and then send the signal to the
server. The server will continue writing to the renamed log file
until it receives the signal. After it receives the signal, the
server will create a new file with the old name and write to it.
Tools
-----
There are a few scripts that may help running a ZEO server. The
zeopack.py script connects to a server and packs the storage. It can
be run as a cron job. The zeoup.py script attempts to connect to a
ZEO server and verify that is is functioning. The zeopasswd.py script
manages a ZEO servers password database.
Diagnosing problems
-------------------
If an exception occurs on the server, the server will log a traceback
and send an exception to the client. The traceback on the client will
show a ZEO protocol library as the source of the error. If you need
to diagnose the problem, you will have to look in the server log for
the rest of the traceback.
setup.py
View file @ c499ea7d
......@@ -11,31 +11,44 @@
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
version = '4.3.0'
version = '5.0.0a2'
from setuptools import setup, find_packages
import os
import sys
if sys.version_info < (2, 7):
print("This version of ZEO requires Python 2.7 or higher")
if sys.version_info < (2, 7, 9):
print("This version of ZEO requires Python 2.7.9 or higher")
sys.exit(0)
if (3, 0) < sys.version_info < (3, 3):
print("This version of ZEO requires Python 3.3 or higher")
if (3, 0) < sys.version_info < (3, 4):
print("This version of ZEO requires Python 3.4 or higher")
sys.exit(0)
install_requires = [
'ZODB >= 5.0.0a5',
'six',
'transaction >= 1.6.0',
'persistent >= 4.1.0',
'zc.lockfile',
'ZConfig',
'zdaemon',
'zope.interface',
]
tests_require = ['zope.testing', 'manuel', 'random2', 'mock']
classifiers = """\
if sys.version_info[:2] < (3, ):
install_requires.extend(('futures', 'trollius'))
classifiers = """
Intended Audience :: Developers
License :: OSI Approved :: Zope Public License
Programming Language :: Python
Programming Language :: Python :: 2
Programming Language :: Python :: 2.7
Programming Language :: Python :: 3
Programming Language :: Python :: 3.3
Programming Language :: Python :: 3.4
Programming Language :: Python :: 3.5
Programming Language :: Python :: Implementation :: CPython
Programming Language :: Python :: Implementation :: PyPy
Topic :: Database
......@@ -43,7 +56,7 @@ Topic :: Software Development :: Libraries :: Python Modules
Operating System :: Microsoft :: Windows
Operating System :: Unix
Framework :: ZODB
"""
""".strip().split('\n')
def _modname(path, base, name=''):
if path == base:
......@@ -96,8 +109,6 @@ def alltests():
_unittests_only(suite, mod.test_suite())
return suite
tests_require = ['zope.testing', 'manuel', 'random2']
long_description = (
open('README.rst').read()
+ '\n' +
......@@ -114,20 +125,11 @@ setup(name="ZEO",
package_dir = {'': 'src'},
license = "ZPL 2.1",
platforms = ["any"],
classifiers = filter(None, classifiers.split("\n")),
classifiers = classifiers,
test_suite="__main__.alltests", # to support "setup.py test"
tests_require = tests_require,
extras_require = dict(test=tests_require),
install_requires = [
'ZODB >= 4.2.0b1, <4.999',
'six',
'transaction',
'persistent >= 4.1.0',
'zc.lockfile',
'ZConfig',
'zdaemon',
'zope.interface',
],
extras_require = dict(test=tests_require, uvloop=['uvloop >=0.5.1']),
install_requires = install_requires,
zip_safe = False,
entry_points = """
[console_scripts]
......
src/ZEO/ClientStorage.py
View file @ c499ea7d
......@@ -18,48 +18,38 @@ Public contents of this module:
ClientStorage -- the main class, implementing the Storage API
"""
import BTrees.IOBTree
import gc
import logging
import os
import re
import socket
import stat
import sys
import tempfile
import threading
import time
import weakref
from binascii import hexlify
import BTrees.OOBTree
import zc.lockfile
import ZEO.interfaces
import ZODB
import ZODB.BaseStorage
import ZODB.ConflictResolution
import ZODB.interfaces
import ZODB.event
import zope.interface
import six
from persistent.TimeStamp import TimeStamp
from ZEO._compat import Pickler, Unpickler, get_ident, PY3
from ZEO.auth import get_module
from ZEO.cache import ClientCache
from ZEO.Exceptions import ClientStorageError, ClientDisconnected, AuthError
from ZEO import ServerStub
from ZEO._compat import get_ident
from ZEO.Exceptions import ClientDisconnected
from ZEO.TransactionBuffer import TransactionBuffer
from ZEO.zrpc.client import ConnectionManager
from ZODB import POSException
from ZODB import utils
logger = logging.getLogger(__name__)
# max signed 64-bit value ~ infinity :) Signed cuz LBTree and TimeStamp
m64 = b'\x7f\xff\xff\xff\xff\xff\xff\xff'
import ZEO.asyncio.client
import ZEO.cache
try:
from ZODB.ConflictResolution import ResolvedSerial
except ImportError:
ResolvedSerial = 'rs'
logger = logging.getLogger(__name__)
def tid2time(tid):
return str(TimeStamp(tid))
......@@ -78,25 +68,10 @@ def get_timestamp(prev_ts=None):
t = t.laterThan(prev_ts)
return t
class DisconnectedServerStub:
"""Internal helper class used as a faux RPC stub when disconnected.
This raises ClientDisconnected on all attribute accesses.
This is a singleton class -- there should be only one instance,
the global disconnected_stub, so it can be tested by identity.
"""
def __getattr__(self, attr):
raise ClientDisconnected()
# Singleton instance of DisconnectedServerStub
disconnected_stub = DisconnectedServerStub()
MB = 1024**2
class ClientStorage(object):
@zope.interface.implementer(ZODB.interfaces.IMultiCommitStorage)
class ClientStorage(ZODB.ConflictResolution.ConflictResolvingStorage):
"""A storage class that is a network client to a remote storage.
This is a faithful implementation of the Storage API.
......@@ -106,28 +81,26 @@ class ClientStorage(object):
"""
# ClientStorage does not declare any interfaces here. Interfaces are
# declared according to the server's storage once a connection is
# established.
# Classes we instantiate. A subclass might override.
TransactionBufferClass = TransactionBuffer
ClientCacheClass = ClientCache
ConnectionManagerClass = ConnectionManager
StorageServerStubClass = ServerStub.stub
def __init__(self, addr, storage='1', cache_size=20 * MB,
name='', client=None, var=None,
min_disconnect_poll=1, max_disconnect_poll=30,
wait_for_server_on_startup=None, # deprecated alias for wait
wait=None, wait_timeout=None,
name='', wait_timeout=None,
disconnect_poll=None,
read_only=0, read_only_fallback=0,
drop_cache_rather_verify=False,
username='', password='', realm=None,
blob_dir=None, shared_blob_dir=False,
blob_cache_size=None, blob_cache_size_check=10,
client_label=None,
cache=None,
ssl = None, ssl_server_hostname=None,
# Mostly ignored backward-compatability options
client=None, var=None,
min_disconnect_poll=1, max_disconnect_poll=None,
wait=True,
drop_cache_rather_verify=True,
credentials=None,
server_sync=False,
# The ZODB-define ZConfig support may ball these:
username=None, password=None, realm=None,
# For tests:
_client_factory=ZEO.asyncio.client.ClientThread,
):
"""ClientStorage constructor.
......@@ -145,50 +118,27 @@ class ClientStorage(object):
address. Required.
storage
The storage name, defaulting to '1'. The name must
The server storage name, defaulting to '1'. The name must
match one of the storage names supported by the server(s)
specified by the addr argument. The storage name is
displayed in the Zope control panel.
specified by the addr argument.
cache_size
The disk cache size, defaulting to 20 megabytes.
This is passed to the ClientCache constructor.
name
The storage name, defaulting to ''. If this is false,
str(addr) is used as the storage name.
The storage name, defaulting to a combination of the
address and the server storage name. This is used to
construct the response to getName()
client
A name used to construct persistent cache filenames.
Defaults to None, in which case the cache is not persistent.
See ClientCache for more info.
var
When client is not None, this specifies the directory
where the persistent cache files are created. It defaults
to None, in which case the current directory is used.
min_disconnect_poll
The minimum delay in seconds between
attempts to connect to the server, in seconds. Defaults
to 1 second.
max_disconnect_poll
The maximum delay in seconds between
attempts to connect to the server, in seconds. Defaults
to 300 seconds.
wait_for_server_on_startup
A backwards compatible alias for
the wait argument.
wait
A flag indicating whether to wait until a connection
with a server is made, defaulting to true.
cache
A cache object or a name, relative to the current working
directory, used to construct persistent cache filenames.
Defaults to None, in which case the cache is not
persistent. See ClientCache for more info.
wait_timeout
Maximum time to wait for a connection before
giving up. Only meaningful if wait is True.
Maximum time to wait for results, including connecting.
read_only
A flag indicating whether this should be a
......@@ -202,21 +152,6 @@ class ClientStorage(object):
most one of read_only and read_only_fallback should be
true.
username
string with username to be used when authenticating.
These only need to be provided if you are connecting to an
authenticated server storage.
password
string with plaintext password to be used when authenticated.
realm
not documented.
drop_cache_rather_verify
a flag indicating that the cache should be dropped rather
than expensively verified.
blob_dir
directory path for blob data. 'blob data' is data that
is retrieved via the loadBlob API.
......@@ -224,7 +159,7 @@ class ClientStorage(object):
shared_blob_dir
Flag whether the blob_dir is a server-shared filesystem
that should be used instead of transferring blob data over
zrpc.
ZEO protocol.
blob_cache_size
Maximum size of the ZEO blob cache, in bytes. If not set, then
......@@ -248,11 +183,19 @@ class ClientStorage(object):
"""
assert not username or password or realm
if isinstance(addr, int):
addr = '127.0.0.1', addr
addr = ('127.0.0.1', addr)
self.__name__ = name or str(addr) # Standard convention for storages
if isinstance(addr, str):
addr = [addr]
elif (isinstance(addr, tuple) and len(addr) == 2 and
isinstance(addr[0], str) and isinstance(addr[1], int)):
addr = [addr]
logger.info(
"%s %s (pid=%d) created %s/%s for storage: %r",
self.__name__,
......@@ -263,121 +206,29 @@ class ClientStorage(object):
storage,
)
self._drop_cache_rather_verify = drop_cache_rather_verify
# wait defaults to True, but wait_for_server_on_startup overrides
# if not None
if wait_for_server_on_startup is not None:
if wait is not None and wait != wait_for_server_on_startup:
logger.warning(
"%s ClientStorage(): conflicting values for wait and "
"wait_for_server_on_startup; wait prevails",
self.__name__)
else:
logger.info(
"%s ClientStorage(): wait_for_server_on_startup "
"is deprecated; please use wait instead",
self.__name__)
wait = wait_for_server_on_startup
elif wait is None:
wait = 1
self._addr = addr # For tests
# A ZEO client can run in disconnected mode, using data from
# its cache, or in connected mode. Several instance variables
# are related to whether the client is connected.
# _server: All method calls are invoked through the server
# stub. When not connect, set to disconnected_stub an
# object that raises ClientDisconnected errors.
# _ready: A threading Event that is set only if _server
# is set to a real stub.
# _connection: The current zrpc connection or None.
# _connection is set as soon as a connection is established,
# but _server is set only after cache verification has finished
# and clients can safely use the server. _pending_server holds
# a server stub while it is being verified.
self._server = disconnected_stub
self._connection = None
self._pending_server = None
self._ready = threading.Event()
# _is_read_only stores the constructor argument
self._is_read_only = read_only
self._storage = storage
self._read_only_fallback = read_only_fallback
self._username = username
self._password = password
self._realm = realm
self._addr = addr # For tests
self._iterators = weakref.WeakValueDictionary()
self._iterator_ids = set()
# Flag tracking disconnections in the middle of a transaction. This
# is reset in tpc_begin() and set in notifyDisconnected().
self._midtxn_disconnect = 0
self._storage = storage
# _server_addr is used by sortKey()
self._server_addr = None
self._client_label = client_label
self._pickler = self._tfile = None
self._info = {'length': 0, 'size': 0, 'name': 'ZEO Client',
'supportsUndo': 0, 'interfaces': ()}
self._tbuf = self.TransactionBufferClass()
self._db = None
self._ltid = None # the last committed transaction
# _serials: stores (oid, serialno) as returned by server
# _seriald: _check_serials() moves from _serials to _seriald,
# which maps oid to serialno
# TODO: If serial number matches transaction id, then there is
# no need to have all this extra infrastructure for handling
# serial numbers. The vote call can just return the tid.
# If there is a conflict error, we can't have a special method
# called just to propagate the error.
self._serials = []
self._seriald = {}
# A ClientStorage only allows one thread to commit at a time.
# Mutual exclusion is achieved using _tpc_cond, which
# protects _transaction. A thread that wants to assign to
# self._transaction must acquire _tpc_cond first. A thread
# that decides it's done with a transaction (whether via success
# or failure) must set _transaction to None and do
# _tpc_cond.notify() before releasing _tpc_cond.
self._tpc_cond = threading.Condition()
self._transaction = None
# Prevent multiple new_oid calls from going out. The _oids
# variable should only be modified while holding the
# _oid_lock.
self._oid_lock = threading.Lock()
self._oids = [] # Object ids retrieved from new_oids()
# load() and tpc_finish() must be serialized to guarantee
# that cache modifications from each occur atomically.
# It also prevents multiple load calls occuring simultaneously,
# which simplifies the cache logic.
self._load_lock = threading.Lock()
# _load_oid and _load_status are protected by _lock
self._load_oid = None
self._load_status = None
# Can't read data in one thread while writing data
# (tpc_finish) in another thread. In general, the lock
# must prevent access to the cache while _update_cache
# is executing.
self._lock = threading.Lock()
self._oids = [] # List of pre-fetched oids from server
cache = self._cache = open_cache(
cache, var, client, storage, cache_size)
# XXX need to check for POSIX-ness here
self.blob_dir = blob_dir
......@@ -399,15 +250,6 @@ class ClientStorage(object):
else:
self.fshelper = None
if client is not None:
dir = var or os.getcwd()
cache_path = os.path.join(dir, "%s-%s.zec" % (client, storage))
else:
cache_path = None
self._cache = self.ClientCacheClass(cache_path, size=cache_size)
self._blob_cache_size = blob_cache_size
self._blob_data_bytes_loaded = 0
if blob_cache_size is not None:
......@@ -416,65 +258,49 @@ class ClientStorage(object):
blob_cache_size * blob_cache_size_check // 100)
self._check_blob_size()
self._rpc_mgr = self.ConnectionManagerClass(addr, self,
tmin=min_disconnect_poll,
tmax=max_disconnect_poll)
self.server_sync = server_sync
self._server = _client_factory(
addr, self, cache, storage,
ZEO.asyncio.client.Fallback if read_only_fallback else read_only,
wait_timeout or 30,
ssl = ssl, ssl_server_hostname=ssl_server_hostname,
credentials=credentials,
)
self._call = self._server.call
self._async = self._server.async
self._async_iter = self._server.async_iter
self._wait = self._server.wait
self._commit_lock = threading.Lock()
if wait:
self._wait(wait_timeout)
else:
# attempt_connect() will make an attempt that doesn't block
# "too long," for a very vague notion of too long. If that
# doesn't succeed, call connect() to start a thread.
if not self._rpc_mgr.attempt_connect():
self._rpc_mgr.connect()
try:
self._wait()
except Exception:
# No point in keeping the server going of the storage
# creation fails
self._server.close()
raise
def new_addr(self, addr):
self._addr = addr
self._rpc_mgr.new_addrs(addr)
self._server.new_addrs(self._normalize_addr(addr))
def _wait(self, timeout=None):
if timeout is not None:
deadline = time.time() + timeout
logger.debug("%s Setting deadline to %f", self.__name__, deadline)
else:
deadline = None
# Wait for a connection to be established.
self._rpc_mgr.connect(sync=1)
# When a synchronous connect() call returns, there is
# a valid _connection object but cache validation may
# still be going on. This code must wait until validation
# finishes, but if the connection isn't a zrpc async
# connection it also needs to poll for input.
while 1:
self._ready.wait(30)
if self._ready.isSet():
break
if timeout and time.time() > deadline:
logger.warning("%s Timed out waiting for connection",
self.__name__)
break
logger.info("%s Waiting for cache verification to finish",
self.__name__)
def _normalize_addr(self, addr):
if isinstance(addr, int):
addr = ('127.0.0.1', addr)
if isinstance(addr, str):
addr = [addr]
elif (isinstance(addr, tuple) and len(addr) == 2 and
isinstance(addr[0], str) and isinstance(addr[1], int)):
addr = [addr]
return addr
def close(self):
"Storage API: finalize the storage, releasing external resources."
_rpc_mgr = self._rpc_mgr
self._rpc_mgr = None
if _rpc_mgr is None:
return # already closed
if self._connection is not None:
self._connection.register_object(None) # Don't call me!
self._connection = None
_rpc_mgr.close()
self._tbuf.close()
if self._cache is not None:
self._cache.close()
self._cache = None
if self._tfile is not None:
self._tfile.close()
self._server.close()
if self._check_blob_size_thread is not None:
self._check_blob_size_thread.join()
......@@ -509,157 +335,41 @@ class ClientStorage(object):
The storage isn't really ready to use until after this call.
"""
super(ClientStorage, self).registerDB(db)
self._db = db
def is_connected(self, test=False):
"""Return whether the storage is currently connected to a server."""
# This function is used by clients, so we only report that a
# connection exists when the connection is ready to use.
if test:
try:
self._server.lastTransaction()
except Exception:
pass
return self._ready.isSet()
return self._server.is_connected()
def sync(self):
# The separate async thread should keep us up to date
pass
def doAuth(self, protocol, stub):
if not (self._username and self._password):
raise AuthError("empty username or password")
module = get_module(protocol)
if not module:
logger.error("%s %s: no such an auth protocol: %s",
self.__name__, self.__class__.__name__, protocol)
return
storage_class, client, db_class = module
if not client:
logger.error(
"%s %s: %s isn't a valid protocol, must have a Client class",
self.__name__, self.__class__.__name__, protocol)
raise AuthError("invalid protocol")
c = client(stub)
# Initiate authentication, returns boolean specifying whether OK
return c.start(self._username, self._realm, self._password)
def testConnection(self, conn):
"""Internal: test the given connection.
This returns: 1 if the connection is an optimal match,
0 if it is a suboptimal but acceptable match.
It can also raise DisconnectedError or ReadOnlyError.
This is called by ZEO.zrpc.ConnectionManager to decide which
connection to use in case there are multiple, and some are
read-only and others are read-write.
This works by calling register() on the server. In read-only
mode, register() is called with the read_only flag set. In
writable mode and in read-only fallback mode, register() is
called with the read_only flag cleared. In read-only fallback
mode only, if the register() call raises ReadOnlyError, it is
retried with the read-only flag set, and if this succeeds,
this is deemed a suboptimal match. In all other cases, a
succeeding register() call is deemed an optimal match, and any
exception raised by register() is passed through.
"""
logger.info("%s Testing connection %r", self.__name__, conn)
# TODO: Should we check the protocol version here?
conn._is_read_only = self._is_read_only
stub = self.StorageServerStubClass(conn)
auth = stub.getAuthProtocol()
logger.info("%s Server authentication protocol %r", self.__name__, auth)
if auth:
skey = self.doAuth(auth, stub)
if skey:
logger.info("%s Client authentication successful",
self.__name__)
conn.setSessionKey(skey)
else:
logger.info("%s Authentication failed",
self.__name__)
raise AuthError("Authentication failed")
try:
stub.register(str(self._storage), self._is_read_only)
return 1
except POSException.ReadOnlyError:
if not self._read_only_fallback:
raise
logger.info("%s Got ReadOnlyError; trying again with read_only=1",
self.__name__)
stub.register(str(self._storage), read_only=1)
conn._is_read_only = True
return 0
def notifyConnected(self, conn):
"""Internal: start using the given connection.
This is called by ConnectionManager after it has decided which
connection should be used.
"""
if self._cache is None:
# the storage was closed, but the connect thread called
# this method before it was stopped.
return
if self._connection is not None:
# If we are upgrading from a read-only fallback connection,
# we must close the old connection to prevent it from being
# used while the cache is verified against the new connection.
self._connection.register_object(None) # Don't call me!
self._connection.close()
self._connection = None
self._ready.clear()
reconnect = 1
else:
reconnect = 0
self.set_server_addr(conn.get_addr())
self._connection = conn
_connection_generation = 0
def notify_connected(self, conn, info):
reconnected = self._connection_generation
self.set_server_addr(conn.get_peername())
self.protocol_version = conn.protocol_version
self._is_read_only = conn.is_read_only()
# invalidate our db cache
if self._db is not None:
self._db.invalidateCache()
if reconnect:
logger.info("%s Reconnected to storage: %s",
self.__name__, self._server_addr)
else:
logger.info("%s Connected to storage: %s",
self.__name__, self._server_addr)
stub = self.StorageServerStubClass(conn)
if self._client_label and conn.peer_protocol_version >= b"Z310":
stub.set_client_label(self._client_label)
if conn.peer_protocol_version < b"Z3101":
logger.warning("Old server doesn't suppport "
"checkCurrentSerialInTransaction")
self.checkCurrentSerialInTransaction = lambda *args: None
logger.info("%s %s to storage: %s",
self.__name__,
'Reconnected' if self._connection_generation
else 'Connected',
self._server_addr)
self._oids = []
self.verify_cache(stub)
self._connection_generation += 1
# It's important to call get_info after calling verify_cache.
# If we end up doing a full-verification, we need to wait till
# it's done. By doing a synchonous call, we are guarenteed
# that the verification will be done because operations are
# handled in order.
self._info.update(stub.get_info())
if self._client_label:
conn.call_async_from_same_thread(
'set_client_label', self._client_label)
self._handle_extensions()
self._info.update(info)
for iface in (
ZODB.interfaces.IStorageRestoreable,
......@@ -673,13 +383,13 @@ class ClientStorage(object):
'interfaces', ()):
zope.interface.alsoProvides(self, iface)
def _handle_extensions(self):
for name in self.getExtensionMethods().keys():
if not hasattr(self, name):
def mklambda(mname):
return (lambda *args, **kw:
self._server.rpc.call(mname, *args, **kw))
setattr(self, name, mklambda(name))
if self.protocol_version >= b'Z5':
self.ping = lambda : self._call('ping')
else:
self.ping = lambda : self._call('lastTransaction')
if self.server_sync:
self.sync = self.ping
def set_server_addr(self, addr):
# Normalize server address and convert to string
......@@ -702,6 +412,8 @@ class ClientStorage(object):
self._server_addr = str((canonical, addr[1]))
def sortKey(self):
# XXX sortKey should be explicit, possibly based on database name.
# If the client isn't connected to anything, it can't have a
# valid sortKey(). Raise an error to stop the transaction early.
if self._server_addr is None:
......@@ -709,15 +421,7 @@ class ClientStorage(object):
else:
return '%s:%s' % (self._storage, self._server_addr)
### Is there a race condition between notifyConnected and
### notifyDisconnected? In Particular, what if we get
### notifyDisconnected in the middle of notifyConnected?
### The danger is that we'll proceed as if we were connected
### without worrying if we were, but this would happen any way if
### notifyDisconnected had to get the instance lock. There's
### nothing to gain by getting the instance lock.
def notifyDisconnected(self):
def notify_disconnected(self):
"""Internal: notify that the server connection was terminated.
This is called by ConnectionManager when the connection is
......@@ -726,11 +430,9 @@ class ClientStorage(object):
"""
logger.info("%s Disconnected from storage: %r",
self.__name__, self._server_addr)
self._connection = None
self._ready.clear()
self._server = disconnected_stub
self._midtxn_disconnect = 1
self._iterator_gc(True)
self._connection_generation += 1
self._is_read_only = self._server.is_read_only()
def __len__(self):
"""Return the size of the storage."""
......@@ -755,117 +457,86 @@ class ClientStorage(object):
"""Storage API: an approximate size of the database, in bytes."""
return self._info['size']
def getExtensionMethods(self):
"""getExtensionMethods
This returns a dictionary whose keys are names of extra methods
provided by this storage. Storage proxies (such as ZEO) should
call this method to determine the extra methods that they need
to proxy in addition to the standard storage methods.
Dictionary values should be None; this will be a handy place
for extra marshalling information, should we need it
"""
return self._info.get('extensionMethods', {})
def supportsUndo(self):
"""Storage API: return whether we support undo."""
return self._info['supportsUndo']
def isReadOnly(self):
"""Storage API: return whether we are in read-only mode."""
if self._is_read_only:
return True
else:
# If the client is configured for a read-write connection
# but has a read-only fallback connection, conn._is_read_only
# will be True. If self._connection is None, we'll behave as
# read_only
try:
return self._connection._is_read_only
except AttributeError:
return True
def is_read_only(self):
"""Storage API: return whether we are in read-only mode.
"""
return self._is_read_only or self._server.is_read_only()
def _check_trans(self, trans):
isReadOnly = is_read_only
def _check_trans(self, trans, meth):
"""Internal helper to check a transaction argument for sanity."""
if self._is_read_only:
raise POSException.ReadOnlyError()
if self._transaction is not trans:
raise POSException.StorageTransactionError(self._transaction,
trans)
try:
buf = trans.data(self)
except KeyError:
buf = None
if buf is None:
raise POSException.StorageTransactionError(
"Transaction not committing", meth, trans)
if buf.connection_generation != self._connection_generation:
# We were disconneected, so this one is poisoned
raise ClientDisconnected(meth, 'on a disconnected transaction')
return buf
def history(self, oid, size=1):
"""Storage API: return a sequence of HistoryEntry objects.
"""
return self._server.history(oid, size)
return self._call('history', oid, size)
def record_iternext(self, next=None):
"""Storage API: get the next database record.
This is part of the conversion-support API.
"""
return self._server.record_iternext(next)
return self._call('record_iternext', next)
def getTid(self, oid):
"""Storage API: return current serial number for oid."""
return self._server.getTid(oid)
# XXX deprecated: but ZODB tests use this. They shouldn't
return self._call('getTid', oid)
def loadSerial(self, oid, serial):
"""Storage API: load a historical revision of an object."""
return self._server.loadSerial(oid, serial)
return self._call('loadSerial', oid, serial)
def load(self, oid, version=''):
"""Storage API: return the data for a given object.
This returns the pickle data and serial number for the object
specified by the given object id, if they exist;
otherwise a KeyError is raised.
"""
result = self.loadBefore(oid, m64)
result = self.loadBefore(oid, utils.maxtid)
if result is None:
raise POSException.POSKeyError(oid)
return result[:2]
def loadBefore(self, oid, tid):
"""Load the object data written before a transaction id
"""
with self._lock: # for atomic processing of invalidations
result = self._cache.loadBefore(oid, tid)
if result:
return result
if self._server is None:
raise ClientDisconnected()
with self._load_lock:
with self._lock:
self._load_oid = oid
self._load_status = 1
result = self._server.loadBefore(oid, tid)
return self._server.load_before(oid, tid)
with self._lock: # for atomic processing of invalidations
if result and self._load_status:
data, tid, end = result
self._cache.store(oid, tid, end, data)
self._load_oid = None
return result
def prefetch(self, oids, tid):
self._server.prefetch(oids, tid)
def new_oid(self):
"""Storage API: return a new object identifier."""
"""Storage API: return a new object identifier.
"""
if self._is_read_only:
raise POSException.ReadOnlyError()
# avoid multiple oid requests to server at the same time
self._oid_lock.acquire()
while 1:
try:
if not self._oids:
self._oids = self._server.new_oids()
self._oids.reverse()
return self._oids.pop()
finally:
self._oid_lock.release()
except IndexError:
pass # We ran out. We need to get some more.
self._oids[:0] = reversed(self._call('new_oids'))
def pack(self, t=None, referencesf=None, wait=1, days=0):
"""Storage API: pack the storage.
......@@ -886,40 +557,25 @@ class ClientStorage(object):
if t is None:
t = time.time()
t = t - (days * 86400)
return self._server.pack(t, wait)
def _check_serials(self):
"""Internal helper to move data from _serials to _seriald."""
# serials are always going to be the same, the only
# question is whether an exception has been raised.
if self._serials:
l = len(self._serials)
r = self._serials[:l]
del self._serials[:l]
for oid, s in r:
if isinstance(s, Exception):
self._cache.invalidate(oid, None)
raise s
self._seriald[oid] = s
return r
return self._call('pack', t, wait)
def store(self, oid, serial, data, version, txn):
"""Storage API: store data for an object."""
assert not version
self._check_trans(txn)
self._server.storea(oid, serial, data, id(txn))
self._tbuf.store(oid, data)
return self._check_serials()
tbuf = self._check_trans(txn, 'store')
self._async('storea', oid, serial, data, id(txn))
tbuf.store(oid, data)
def checkCurrentSerialInTransaction(self, oid, serial, transaction):
self._check_trans(transaction)
self._server.checkCurrentSerialInTransaction(oid, serial,
id(transaction))
self._check_trans(transaction, 'checkCurrentSerialInTransaction')
self._async(
'checkCurrentSerialInTransaction', oid, serial, id(transaction))
def storeBlob(self, oid, serial, data, blobfilename, version, txn):
"""Storage API: store a blob object."""
assert not version
tbuf = self._check_trans(txn, 'storeBlob')
# Grab the file right away. That way, if we don't have enough
# room for a copy, we'll know now rather than in tpc_finish.
......@@ -937,11 +593,29 @@ class ClientStorage(object):
serials = self.store(oid, serial, data, '', txn)
if self.shared_blob_dir:
self._server.storeBlobShared(
self._async(
'storeBlobShared',
oid, serial, data, os.path.basename(target), id(txn))
else:
self._server.storeBlob(oid, serial, data, target, txn)
self._tbuf.storeBlob(oid, target)
# Store a blob to the server. We don't want to real all of
# the data into memory, so we use a message iterator. This
# allows us to read the blob data as needed.
def store():
yield ('storeBlobStart', ())
f = open(target, 'rb')
while 1:
chunk = f.read(59000)
if not chunk:
break
yield ('storeBlobChunk', (chunk, ))
f.close()
yield ('storeBlobEnd', (oid, serial, data, id(txn)))
self._async_iter(store())
tbuf.storeBlob(oid, target)
return serials
def receiveBlobStart(self, oid, serial):
......@@ -970,9 +644,9 @@ class ClientStorage(object):
os.chmod(blob_filename, stat.S_IREAD)
def deleteObject(self, oid, serial, txn):
self._check_trans(txn)
self._server.deleteObject(oid, serial, id(txn))
self._tbuf.store(oid, None)
tbuf = self._check_trans(txn, 'deleteObject')
self._async('deleteObject', oid, serial, id(txn))
tbuf.store(oid, None)
def loadBlob(self, oid, serial):
# Load a blob. If it isn't present and we have a shared blob
......@@ -1017,7 +691,7 @@ class ClientStorage(object):
# returns, it will have been sent. (The recieving will
# have been handled by the asyncore thread.)
self._server.sendBlob(oid, serial)
self._call('sendBlob', oid, serial)
if os.path.exists(blob_filename):
return _accessed(blob_filename)
......@@ -1047,7 +721,7 @@ class ClientStorage(object):
# We're using a server shared cache. If the file isn't
# here, it's not anywhere.
raise POSException.POSKeyError("No blob file", oid, serial)
self._server.sendBlob(oid, serial)
self._call('sendBlob', oid, serial)
if not os.path.exists(blob_filename):
raise POSException.POSKeyError("No blob file", oid, serial)
......@@ -1064,12 +738,57 @@ class ClientStorage(object):
return self.fshelper.temp_dir
def tpc_vote(self, txn):
"""Storage API: vote on a transaction."""
if txn is not self._transaction:
raise POSException.StorageTransactionError(
"tpc_vote called with wrong transaction")
self._server.vote(id(txn))
return self._check_serials()
"""Storage API: vote on a transaction.
"""
tbuf = self._check_trans(txn, 'tpc_vote')
try:
conflicts = True
vote_attempts = 0
while conflicts and vote_attempts < 9: # 9? Mainly avoid inf. loop
conflicts = False
for oid in self._call('vote', id(txn)) or ():
if isinstance(oid, dict):
# Conflict, let's try to resolve it
conflicts = True
conflict = oid
oid = conflict['oid']
committed, read = conflict['serials']
data = self.tryToResolveConflict(
oid, committed, read, conflict['data'])
self._async('storea', oid, committed, data, id(txn))
tbuf.resolve(oid, data)
else:
tbuf.server_resolve(oid)
vote_attempts += 1
except POSException.StorageTransactionError:
# Hm, we got disconnected and reconnected bwtween
# _check_trans and voting. Let's chack the transaction again:
self._check_trans(txn, 'tpc_vote')
raise
except POSException.ConflictError as err:
oid = getattr(err, 'oid', None)
if oid is not None:
# This is a band-aid to help recover from a situation
# that shouldn't happen. A Client somehow misses some
# invalidations and has out of date data in its
# cache. We need some whay to invalidate the cache
# entry without invalidations. So, if we see a
# (unresolved) conflict error, we assume that the
# cache entry is bad and invalidate it.
self._cache.invalidate(oid, None)
raise
if tbuf.exception:
raise tbuf.exception
if tbuf.server_resolved or tbuf.client_resolved:
return list(tbuf.server_resolved) + list(tbuf.client_resolved)
else:
return None
def tpc_transaction(self):
return self._transaction
......@@ -1078,137 +797,102 @@ class ClientStorage(object):
"""Storage API: begin a transaction."""
if self._is_read_only:
raise POSException.ReadOnlyError()
self._tpc_cond.acquire()
try:
self._midtxn_disconnect = 0
while self._transaction is not None:
# It is allowable for a client to call two tpc_begins in a
# row with the same transaction, and the second of these
# must be ignored.
if self._transaction == txn:
tbuf = txn.data(self)
except AttributeError:
# Gaaaa. This is a recovery transaction. Work around this
# until we can think of something better. XXX
tb = {}
txn.data = tb.__getitem__
txn.set_data = tb.__setitem__
except KeyError:
pass
else:
if tbuf is not None:
raise POSException.StorageTransactionError(
"Duplicate tpc_begin calls for same transaction")
self._tpc_cond.wait(30)
self._transaction = txn
finally:
self._tpc_cond.release()
txn.set_data(self, TransactionBuffer(self._connection_generation))
# XXX we'd like to allow multiple transactions at a time at some point,
# but for now, due to server limitations, TCBOO.
self._commit_lock.acquire()
self._tbuf = txn.data(self)
try:
self._server.tpc_begin(id(txn), txn.user, txn.description,
txn._extension, tid, status)
except:
# Client may have disconnected during the tpc_begin().
if self._server is not disconnected_stub:
self.end_transaction()
self._async(
'tpc_begin', id(txn),
txn.user, txn.description, txn._extension, tid, status)
except ClientDisconnected:
self.tpc_end(txn)
raise
self._tbuf.clear()
self._seriald.clear()
del self._serials[:]
def end_transaction(self):
"""Internal helper to end a transaction."""
# the right way to set self._transaction to None
# calls notify() on _tpc_cond in case there are waiting threads
self._tpc_cond.acquire()
try:
self._transaction = None
self._tpc_cond.notify()
finally:
self._tpc_cond.release()
def tpc_end(self, txn):
tbuf = txn.data(self)
if tbuf is not None:
tbuf.close()
txn.set_data(self, None)
self._commit_lock.release()
def lastTransaction(self):
with self._lock:
return self._cache.getLastTid()
def tpc_abort(self, txn):
"""Storage API: abort a transaction."""
if txn is not self._transaction:
def tpc_abort(self, txn, timeout=None):
"""Storage API: abort a transaction.
(The timeout keyword argument is for tests to wat longer than
they normally would.)
"""
try:
tbuf = txn.data(self)
except KeyError:
return
try:
# Caution: Are there any exceptions that should prevent an
# abort from occurring? It seems wrong to swallow them
# all, yet you want to be sure that other abort logic is
# executed regardless.
try:
self._server.tpc_abort(id(txn))
# It's tempting to make an asynchronous call here, but
# it's useful for it to be synchronous because, if we
# failed due to a disconnect, synchronous calls will
# wait a little while in hopes of reconnecting. If
# we're able to reconnect and retry the transaction,
# ten it might succeed!
self._call('tpc_abort', id(txn), timeout=timeout)
except ClientDisconnected:
logger.debug("%s ClientDisconnected in tpc_abort() ignored",
self.__name__)
finally:
self._tbuf.clear()
self._seriald.clear()
del self._serials[:]
self._iterator_gc()
self.end_transaction()
self.tpc_end(txn)
def tpc_finish(self, txn, f=None):
def tpc_finish(self, txn, f=lambda tid: None):
"""Storage API: finish a transaction."""
if txn is not self._transaction:
raise POSException.StorageTransactionError(
"tpc_finish called with wrong transaction")
self._load_lock.acquire()
try:
if self._midtxn_disconnect:
raise ClientDisconnected(
'Calling tpc_finish() on a disconnected transaction')
tbuf = self._check_trans(txn, 'tpc_finish')
finished = 0
try:
self._lock.acquire() # for atomic processing of invalidations
try:
tid = self._server.tpc_finish(id(txn))
finished = 1
self._update_cache(tid)
if f is not None:
f(tid)
finally:
self._lock.release()
r = self._check_serials()
assert r is None or len(r) == 0, "unhandled serialnos: %s" % r
except:
if finished:
# The server successfully committed. If we get a failure
# here, our own state will be in question, so reconnect.
self._connection.close()
raise
self.end_transaction()
tid = self._server.tpc_finish(id(txn), tbuf, f)
finally:
self._load_lock.release()
self.tpc_end(txn)
self._iterator_gc()
def _update_cache(self, tid):
"""Internal helper to handle objects modified by a transaction.
self._update_blob_cache(tbuf, tid)
This iterates over the objects in the transaction buffer and
update or invalidate the cache.
return tid
def _update_blob_cache(self, tbuf, tid):
"""Internal helper move blobs updated by a transaction to the cache.
"""
# Must be called with _lock already acquired.
# Not sure why _update_cache() would be called on a closed storage.
if self._cache is None:
return
for oid, _ in six.iteritems(self._seriald):
self._cache.invalidate(oid, tid)
for oid, data in self._tbuf:
# If data is None, we just invalidate.
if data is not None:
s = self._seriald[oid]
if s != ResolvedSerial:
assert s == tid, (s, tid)
self._cache.store(oid, s, None, data)
else:
# object deletion
self._cache.invalidate(oid, tid)
if self.fshelper is not None:
blobs = self._tbuf.blobs
blobs = tbuf.blobs
had_blobs = False
while blobs:
oid, blobfilename = blobs.pop()
......@@ -1228,9 +912,6 @@ class ClientStorage(object):
if had_blobs:
self._check_blob_size(self._blob_data_bytes_loaded)
self._cache.setLastTid(tid)
self._tbuf.clear()
def undo(self, trans_id, txn):
"""Storage API: undo a transaction.
......@@ -1241,12 +922,12 @@ class ClientStorage(object):
a storage.
"""
self._check_trans(txn)
self._server.undoa(trans_id, id(txn))
self._check_trans(txn, 'undo')
self._async('undoa', trans_id, id(txn))
def undoInfo(self, first=0, last=-20, specification=None):
"""Storage API: return undo information."""
return self._server.undoInfo(first, last, specification)
return self._call('undoInfo', first, last, specification)
def undoLog(self, first=0, last=-20, filter=None):
"""Storage API: return a sequence of TransactionDescription objects.
......@@ -1259,7 +940,7 @@ class ClientStorage(object):
"""
if filter is not None:
return []
return self._server.undoLog(first, last)
return self._call('undoLog', first, last)
# Recovery support
......@@ -1274,229 +955,36 @@ class ClientStorage(object):
def restore(self, oid, serial, data, version, prev_txn, transaction):
"""Write data already committed in a separate database."""
assert not version
self._check_trans(transaction)
self._server.restorea(oid, serial, data, prev_txn, id(transaction))
# Don't update the transaction buffer, because current data are
# unaffected.
return self._check_serials()
self._check_trans(transaction, 'restore')
self._async('restorea', oid, serial, data, prev_txn, id(transaction))
# Below are methods invoked by the StorageServer
def serialnos(self, args):
"""Server callback to pass a list of changed (oid, serial) pairs."""
self._serials.extend(args)
"""Server callback to pass a list of changed (oid, serial) pairs.
"""
self._tbuf.serialnos(args)
def info(self, dict):
"""Server callback to update the info dictionary."""
self._info.update(dict)
def verify_cache(self, server):
"""Internal routine called to verify the cache.
The return value (indicating which path we took) is used by
the test suite.
"""
self._pending_server = server
# setup tempfile to hold zeoVerify results and interim
# invalidation results
self._tfile = tempfile.TemporaryFile(suffix=".inv")
if PY3:
self._pickler = Pickler(self._tfile, 3)
else:
self._pickler = Pickler(self._tfile, 1)
self._pickler.fast = 1 # Don't use the memo
if self._connection.peer_protocol_version < b'Z309':
client = ClientStorage308Adapter(self)
else:
client = self
# allow incoming invalidations:
self._connection.register_object(client)
# If verify_cache() finishes the cache verification process,
# it should set self._server. If it goes through full cache
# verification, then endVerify() should self._server.
server_tid = server.lastTransaction()
if not self._cache:
logger.info("%s No verification necessary -- empty cache",
self.__name__)
if server_tid != utils.z64:
self._cache.setLastTid(server_tid)
self.finish_verification()
return "empty cache"
cache_tid = self._cache.getLastTid()
if cache_tid != utils.z64:
if server_tid == cache_tid:
logger.info(
"%s No verification necessary"
" (cache_tid up-to-date %r)",
self.__name__, server_tid)
self.finish_verification()
return "no verification"
elif server_tid < cache_tid:
message = ("%s Client has seen newer transactions than server!"
% self.__name__)
logger.critical(message)
raise ClientStorageError(message)
# log some hints about last transaction
logger.info("%s last inval tid: %r %s\n",
self.__name__, cache_tid,
tid2time(cache_tid))
logger.info("%s last transaction: %r %s",
self.__name__, server_tid,
server_tid and tid2time(server_tid))
pair = server.getInvalidations(cache_tid)
if pair is not None:
logger.info("%s Recovering %d invalidations",
self.__name__, len(pair[1]))
self.finish_verification(pair)
return "quick verification"
elif server_tid != utils.z64:
# Hm, to have gotten here, the cache is non-empty, but
# it has no last tid. This doesn't seem like good situation.
# We'll have to verify the cache, if we're willing.
self._cache.setLastTid(server_tid)
ZODB.event.notify(ZEO.interfaces.StaleCache(self))
# From this point on, we do not have complete information about
# the missed transactions. The reason is that cache
# verification only checks objects in the client cache and
# there may be objects in the object caches that aren't in the
# client cach that would need verification too. We avoid that
# problem by just invalidating the objects in the object caches.
def invalidateCache(self):
if self._db is not None:
self._db.invalidateCache()
if self._cache and self._drop_cache_rather_verify:
logger.critical("%s dropping stale cache", self.__name__)
self._cache.clear()
if server_tid:
self._cache.setLastTid(server_tid)
self.finish_verification()
return "cache dropped"
logger.info("%s Verifying cache", self.__name__)
for oid, tid in self._cache.contents():
server.verify(oid, tid)
server.endZeoVerify()
with self._lock:
if server_tid > self._cache.getLastTid():
# We verified the cache, and got no new invalidations
# while doing so. The records in the cache are valid,
# in that invalid current records were invalidated,
# but the last tid is wrong. Let's fix it:
self._cache.setLastTid(server_tid)
return "full verification"
def invalidateVerify(self, oid):
"""Server callback to invalidate an oid pair.
This is called as part of cache validation.
"""
# Invalidation as result of verify_cache().
# Queue an invalidate for the end the verification procedure.
if self._pickler is None:
# This should never happen.
logger.error("%s invalidateVerify with no _pickler", self.__name__)
return
self._pickler.dump((None, [oid]))
def endVerify(self):
"""Server callback to signal end of cache validation."""
logger.info("%s endVerify finishing", self.__name__)
self.finish_verification()
logger.info("%s endVerify finished", self.__name__)
def finish_verification(self, catch_up=None):
self._lock.acquire()
try:
if catch_up:
# process catch-up invalidations
self._process_invalidations(*catch_up)
if self._pickler is None:
return
# write end-of-data marker
self._pickler.dump((None, None))
self._pickler = None
self._tfile.seek(0)
unpickler = Unpickler(self._tfile)
min_tid = self._cache.getLastTid()
while 1:
tid, oids = unpickler.load()
logger.debug('pickled inval %r %r', tid, min_tid)
if oids is None:
break
if ((tid is None)
or (min_tid is None)
or (tid > min_tid)
):
self._process_invalidations(tid, oids)
self._tfile.close()
self._tfile = None
finally:
self._lock.release()
self._server = self._pending_server
self._ready.set()
self._pending_server = None
def invalidateTransaction(self, tid, oids):
"""Server callback: Invalidate objects modified by tid."""
self._lock.acquire()
try:
if self._pickler is not None:
logger.debug(
"%s Transactional invalidation during cache verification",
self.__name__)
self._pickler.dump((tid, oids))
else:
self._process_invalidations(tid, oids)
finally:
self._lock.release()
def _process_invalidations(self, tid, oids):
for oid in oids:
if oid == self._load_oid:
self._load_status = 0
self._cache.invalidate(oid, tid)
if self._db is not None:
self._db.invalidate(tid, oids)
self._cache.setLastTid(tid)
# The following are for compatibility with protocol version 2.0.0
def invalidateTrans(self, oids):
return self.invalidateTransaction(None, oids)
invalidate = invalidateVerify
end = endVerify
Invalidate = invalidateTrans
# IStorageIteration
def iterator(self, start=None, stop=None):
"""Return an IStorageTransactionInformation iterator."""
# iids are "iterator IDs" that can be used to query an iterator whose
# status is held on the server.
iid = self._server.iterator_start(start, stop)
iid = self._call('iterator_start', start, stop)
return self._setup_iterator(TransactionIterator, iid)
def _setup_iterator(self, factory, iid, *args):
......@@ -1533,10 +1021,11 @@ class ClientStorage(object):
# objects waiting to be cleaned up. So there's never any risk
# of confusing TransactionIterator objects that are in use.
iids = self._iterator_ids - set(self._iterators)
self._iterators._last_gc = time.time() # let tests know we've been called
# let tests know we've been called:
self._iterators._last_gc = time.time()
if iids:
try:
self._server.iterator_gc(list(iids))
self._async('iterator_gc', list(iids))
except ClientDisconnected:
# If we get disconnected, all of the iterators on the
# server are thrown away. We should clear ours too:
......@@ -1544,8 +1033,7 @@ class ClientStorage(object):
self._iterator_ids -= iids
def server_status(self):
return self._server.server_status()
return self._call('server_status')
class TransactionIterator(object):
......@@ -1564,7 +1052,7 @@ class TransactionIterator(object):
if self._iid < 0:
raise ClientDisconnected("Disconnected iterator")
tx_data = self._storage._server.iterator_next(self._iid)
tx_data = self._storage._call('iterator_next', self._iid)
if tx_data is None:
# The iterator is exhausted, and the server has already
# disposed it.
......@@ -1594,8 +1082,8 @@ class ClientStorageTransactionInformation(ZODB.BaseStorage.TransactionRecord):
self.extension = extension
def __iter__(self):
riid = self._storage._server.iterator_record_start(self._txiter._iid,
self.tid)
riid = self._storage._call('iterator_record_start',
self._txiter._iid, self.tid)
return self._storage._setup_iterator(RecordIterator, riid)
......@@ -1614,7 +1102,7 @@ class RecordIterator(object):
# We finished iteration once already and the server can't know
# about the iteration anymore.
raise StopIteration()
item = self._storage._server.iterator_record_next(self._riid)
item = self._storage._call('iterator_record_next', self._riid)
if item is None:
# The iterator is exhausted, and the server has already
# disposed it.
......@@ -1625,21 +1113,6 @@ class RecordIterator(object):
next = __next__
class ClientStorage308Adapter:
def __init__(self, client):
self.client = client
def invalidateTransaction(self, tid, args):
self.client.invalidateTransaction(tid, [arg[0] for arg in args])
def invalidateVerify(self, arg):
self.client.invalidateVerify(arg[0])
def __getattr__(self, name):
return getattr(self.client, name)
class BlobCacheLayout(object):
size = 997
......@@ -1782,3 +1255,18 @@ def _lock_blob(path):
raise
else:
break
def open_cache(cache, var, client, storage, cache_size):
if isinstance(cache, (None.__class__, str)):
from ZEO.cache import ClientCache
if cache is None:
if client:
cache = os.path.join(var or os.getcwd(),
"%s-%s.zec" % (client, storage))
else:
# ephemeral cache
return ClientCache(None, cache_size)
cache = ClientCache(cache, cache_size)
return cache
src/ZEO/Exceptions.py
View file @ c499ea7d
......@@ -13,16 +13,31 @@
##############################################################################
"""Exceptions for ZEO."""
import transaction.interfaces
from ZODB.POSException import StorageError
class ClientStorageError(StorageError):
"""An error occurred in the ZEO Client Storage."""
"""An error occurred in the ZEO Client Storage.
"""
class UnrecognizedResult(ClientStorageError):
"""A server call returned an unrecognized result."""
"""A server call returned an unrecognized result.
"""
class ClientDisconnected(ClientStorageError):
"""The database storage is disconnected from the storage."""
class ClientDisconnected(ClientStorageError,
transaction.interfaces.TransientError):
"""The database storage is disconnected from the storage.
"""
class AuthError(StorageError):
"""The client provided invalid authentication credentials."""
"""The client provided invalid authentication credentials.
"""
class ProtocolError(ClientStorageError):
"""A client contacted a server with an incomparible protocol
"""
class ServerException(ClientStorageError):
"""
"""
src/ZEO/ServerStub.py deleted 100644 → 0
View file @ bc92bdd7
##############################################################################
#
# Copyright (c) 2001, 2002, 2003 Zope Foundation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE
#
##############################################################################
"""RPC stubs for interface exported by StorageServer."""
import time
from ZODB.utils import z64
##
# ZEO storage server.
# <p>
# Remote method calls can be synchronous or asynchronous. If the call
# is synchronous, the client thread blocks until the call returns. A
# single client can only have one synchronous request outstanding. If
# several threads share a single client, threads other than the caller
# will block only if the attempt to make another synchronous call.
# An asynchronous call does not cause the client thread to block. An
# exception raised by an asynchronous method is logged on the server,
# but is not returned to the client.
class StorageServer:
"""An RPC stub class for the interface exported by ClientStorage.
This is the interface presented by the StorageServer to the
ClientStorage; i.e. the ClientStorage calls these methods and they
are executed in the StorageServer.
See the StorageServer module for documentation on these methods,
with the exception of _update(), which is documented here.
"""
def __init__(self, rpc):
"""Constructor.
The argument is a connection: an instance of the
zrpc.connection.Connection class.
"""
self.rpc = rpc
def extensionMethod(self, name):
return ExtensionMethodWrapper(self.rpc, name).call
##
# Register current connection with a storage and a mode.
# In effect, it is like an open call.
# @param storage_name a string naming the storage. This argument
# is primarily for backwards compatibility with servers
# that supported multiple storages.
# @param read_only boolean
# @exception ValueError unknown storage_name or already registered
# @exception ReadOnlyError storage is read-only and a read-write
# connectio was requested
def register(self, storage_name, read_only):
self.rpc.call('register', storage_name, read_only)
##
# Return dictionary of meta-data about the storage.
# @defreturn dict
def get_info(self):
return self.rpc.call('get_info')
##
# Check whether the server requires authentication. Returns
# the name of the protocol.
# @defreturn string
def getAuthProtocol(self):
return self.rpc.call('getAuthProtocol')
##
# Return id of the last committed transaction
# @defreturn string
def lastTransaction(self):
# Not in protocol version 2.0.0; see __init__()
return self.rpc.call('lastTransaction') or z64
##
# Return invalidations for all transactions after tid.
# @param tid transaction id
# @defreturn 2-tuple, (tid, list)
# @return tuple containing the last committed transaction
# and a list of oids that were invalidated. Returns
# None and an empty list if the server does not have
# the list of oids available.
def getInvalidations(self, tid):
# Not in protocol version 2.0.0; see __init__()
return self.rpc.call('getInvalidations', tid)
##
# Check whether a serial number is current for oid.
# If the serial number is not current, the
# server will make an asynchronous invalidateVerify() call.
# @param oid object id
# @param s serial number
# @defreturn async
def zeoVerify(self, oid, s):
self.rpc.callAsync('zeoVerify', oid, s)
##
# Check whether current serial number is valid for oid.
# If the serial number is not current, the server will make an
# asynchronous invalidateVerify() call.
# @param oid object id
# @param serial client's current serial number
# @defreturn async
def verify(self, oid, serial):
self.rpc.callAsync('verify', oid, serial)
##
# Signal to the server that cache verification is done.
# @defreturn async
def endZeoVerify(self):
self.rpc.callAsync('endZeoVerify')
##
# Generate a new set of oids.
# @param n number of new oids to return
# @defreturn list
# @return list of oids
def new_oids(self, n=None):
if n is None:
return self.rpc.call('new_oids')
else:
return self.rpc.call('new_oids', n)
##
# Pack the storage.
# @param t pack time
# @param wait optional, boolean. If true, the call will not
# return until the pack is complete.
def pack(self, t, wait=None):
if wait is None:
self.rpc.call('pack', t)
else:
self.rpc.call('pack', t, wait)
##
# Return current data for oid.
# @param oid object id
# @defreturn 2-tuple
# @return 2-tuple, current non-version data, serial number
# @exception KeyError if oid is not found
def zeoLoad(self, oid):
return self.rpc.call('zeoLoad', oid)[:2]
##
# Return current data for oid, and the tid of the
# transaction that wrote the most recent revision.
# @param oid object id
# @defreturn 2-tuple
# @return data, transaction id
# @exception KeyError if oid is not found
def loadEx(self, oid):
return self.rpc.call("loadEx", oid)
##
# Return non-current data along with transaction ids that identify
# the lifetime of the specific revision.
# @param oid object id
# @param tid a transaction id that provides an upper bound on
# the lifetime of the revision. That is, loadBefore
# returns the revision that was current before tid committed.
# @defreturn 4-tuple
# @return data, serial numbr, start transaction id, end transaction id
def loadBefore(self, oid, tid):
return self.rpc.call("loadBefore", oid, tid)
##
# Storage new revision of oid.
# @param oid object id
# @param serial serial number that this transaction read
# @param data new data record for oid
# @param id id of current transaction
# @defreturn async
def storea(self, oid, serial, data, id):
self.rpc.callAsync('storea', oid, serial, data, id)
def checkCurrentSerialInTransaction(self, oid, serial, id):
self.rpc.callAsync('checkCurrentSerialInTransaction', oid, serial, id)
def restorea(self, oid, serial, data, prev_txn, id):
self.rpc.callAsync('restorea', oid, serial, data, prev_txn, id)
def storeBlob(self, oid, serial, data, blobfilename, txn):
# Store a blob to the server. We don't want to real all of
# the data into memory, so we use a message iterator. This
# allows us to read the blob data as needed.
def store():
yield ('storeBlobStart', ())
f = open(blobfilename, 'rb')
while 1:
chunk = f.read(59000)
if not chunk:
break
yield ('storeBlobChunk', (chunk, ))
f.close()
yield ('storeBlobEnd', (oid, serial, data, id(txn)))
self.rpc.callAsyncIterator(store())
def storeBlobShared(self, oid, serial, data, filename, id):
self.rpc.callAsync('storeBlobShared', oid, serial, data, filename, id)
def deleteObject(self, oid, serial, id):
self.rpc.callAsync('deleteObject', oid, serial, id)
##
# Start two-phase commit for a transaction
# @param id id used by client to identify current transaction. The
# only purpose of this argument is to distinguish among multiple
# threads using a single ClientStorage.
# @param user name of user committing transaction (can be "")
# @param description string containing transaction metadata (can be "")
# @param ext dictionary of extended metadata (?)
# @param tid optional explicit tid to pass to underlying storage
# @param status optional status character, e.g "p" for pack
# @defreturn async
def tpc_begin(self, id, user, descr, ext, tid, status):
self.rpc.callAsync('tpc_begin', id, user, descr, ext, tid, status)
def vote(self, trans_id):
return self.rpc.call('vote', trans_id)
def tpc_finish(self, id):
return self.rpc.call('tpc_finish', id)
def tpc_abort(self, id):
self.rpc.call('tpc_abort', id)
def history(self, oid, length=None):
if length is None:
return self.rpc.call('history', oid)
else:
return self.rpc.call('history', oid, length)
def record_iternext(self, next):
return self.rpc.call('record_iternext', next)
def sendBlob(self, oid, serial):
return self.rpc.call('sendBlob', oid, serial)
def getTid(self, oid):
return self.rpc.call('getTid', oid)
def loadSerial(self, oid, serial):
return self.rpc.call('loadSerial', oid, serial)
def new_oid(self):
return self.rpc.call('new_oid')
def undoa(self, trans_id, trans):
self.rpc.callAsync('undoa', trans_id, trans)
def undoLog(self, first, last):
return self.rpc.call('undoLog', first, last)
def undoInfo(self, first, last, spec):
return self.rpc.call('undoInfo', first, last, spec)
def iterator_start(self, start, stop):
return self.rpc.call('iterator_start', start, stop)
def iterator_next(self, iid):
return self.rpc.call('iterator_next', iid)
def iterator_record_start(self, txn_iid, tid):
return self.rpc.call('iterator_record_start', txn_iid, tid)
def iterator_record_next(self, iid):
return self.rpc.call('iterator_record_next', iid)
def iterator_gc(self, iids):
return self.rpc.callAsync('iterator_gc', iids)
def server_status(self):
return self.rpc.call("server_status")
def set_client_label(self, label):
return self.rpc.callAsync('set_client_label', label)
class StorageServer308(StorageServer):
def __init__(self, rpc):
if rpc.peer_protocol_version == b'Z200':
self.lastTransaction = lambda: z64
self.getInvalidations = lambda tid: None
self.getAuthProtocol = lambda: None
StorageServer.__init__(self, rpc)
def history(self, oid, length=None):
if length is None:
return self.rpc.call('history', oid, '')
else:
return self.rpc.call('history', oid, '', length)
def getInvalidations(self, tid):
# Not in protocol version 2.0.0; see __init__()
result = self.rpc.call('getInvalidations', tid)
if result is not None:
result = result[0], [oid for (oid, version) in result[1]]
return result
def verify(self, oid, serial):
self.rpc.callAsync('verify', oid, '', serial)
def loadEx(self, oid):
return self.rpc.call("loadEx", oid, '')[:2]
def storea(self, oid, serial, data, id):
self.rpc.callAsync('storea', oid, serial, data, '', id)
def storeBlob(self, oid, serial, data, blobfilename, txn):
# Store a blob to the server. We don't want to real all of
# the data into memory, so we use a message iterator. This
# allows us to read the blob data as needed.
def store():
yield ('storeBlobStart', ())
f = open(blobfilename, 'rb')
while 1:
chunk = f.read(59000)
if not chunk:
break
yield ('storeBlobChunk', (chunk, ))
f.close()
yield ('storeBlobEnd', (oid, serial, data, '', id(txn)))
self.rpc.callAsyncIterator(store())
def storeBlobShared(self, oid, serial, data, filename, id):
self.rpc.callAsync('storeBlobShared', oid, serial, data, filename,
'', id)
def zeoVerify(self, oid, s):
self.rpc.callAsync('zeoVerify', oid, s, None)
def iterator_start(self, start, stop):
raise NotImplementedError
def iterator_next(self, iid):
raise NotImplementedError
def iterator_record_start(self, txn_iid, tid):
raise NotImplementedError
def iterator_record_next(self, iid):
raise NotImplementedError
def iterator_gc(self, iids):
raise NotImplementedError
def stub(client, connection):
start = time.time()
# Wait until we know what version the other side is using.
while connection.peer_protocol_version is None:
if time.time()-start > 10:
raise ValueError("Timeout waiting for protocol handshake")
time.sleep(0.1)
if connection.peer_protocol_version < b'Z309':
return StorageServer308(connection)
return StorageServer(connection)
class ExtensionMethodWrapper:
def __init__(self, rpc, name):
self.rpc = rpc
self.name = name
def call(self, *a, **kwa):
return self.rpc.call(self.name, *a, **kwa)
src/ZEO/StorageServer.py
View file @ c499ea7d
......@@ -19,18 +19,18 @@ file storage or Berkeley storage.
TODO: Need some basic access control-- a declaration of the methods
exported for invocation by the server.
"""
import asyncore
import codecs
import itertools
import logging
import os
import socket
import sys
import tempfile
import threading
import time
import transaction
import warnings
from .zrpc.error import DisconnectedError
import ZEO.asyncio.server
import ZODB.blob
import ZODB.event
import ZODB.serialize
......@@ -40,15 +40,18 @@ import six
from ZEO._compat import Pickler, Unpickler, PY3, BytesIO
from ZEO.Exceptions import AuthError
from .monitor import StorageStats, StatsServer
from .zrpc.connection import ManagedServerConnection, Delay, MTDelay, Result
from .zrpc.server import Dispatcher
from ZODB.ConflictResolution import ResolvedSerial
from ZEO.monitor import StorageStats
from ZEO.asyncio.server import Delay, MTDelay, Result
from ZODB.loglevels import BLATHER
from ZODB.POSException import StorageError, StorageTransactionError
from ZODB.POSException import TransactionError, ReadOnlyError, ConflictError
from ZODB.serialize import referencesf
from ZODB.utils import oid_repr, p64, u64, z64
from ZODB.utils import oid_repr, p64, u64, z64, Lock, RLock
if os.environ.get("ZEO_MTACCEPTOR"): # mainly for tests
from .asyncio.mtacceptor import Acceptor
else:
from .asyncio.server import Acceptor
logger = logging.getLogger('ZEO.StorageServer')
......@@ -62,61 +65,49 @@ def log(message, level=logging.INFO, label='', exc_info=False):
class StorageServerError(StorageError):
"""Error reported when an unpicklable exception is raised."""
registered_methods = set(( 'get_info', 'lastTransaction',
'getInvalidations', 'new_oids', 'pack', 'loadBefore', 'storea',
'checkCurrentSerialInTransaction', 'restorea', 'storeBlobStart',
'storeBlobChunk', 'storeBlobEnd', 'storeBlobShared',
'deleteObject', 'tpc_begin', 'vote', 'tpc_finish', 'tpc_abort',
'history', 'record_iternext', 'sendBlob', 'getTid', 'loadSerial',
'new_oid', 'undoa', 'undoLog', 'undoInfo', 'iterator_start',
'iterator_next', 'iterator_record_start', 'iterator_record_next',
'iterator_gc', 'server_status', 'set_client_label', 'ping'))
class ZEOStorage:
"""Proxy to underlying storage for a single remote client."""
# A list of extension methods. A subclass with extra methods
# should override.
extensions = []
connected = connection = stats = storage = storage_id = transaction = None
blob_tempfile = None
log_label = 'unconnected'
locked = False # Don't have storage lock
verifying = 0
def __init__(self, server, read_only=0, auth_realm=None):
def __init__(self, server, read_only=0):
self.server = server
self.client_conflict_resolution = server.client_conflict_resolution
# timeout and stats will be initialized in register()
self.stats = None
self.connection = None
self.client = None
self.storage = None
self.storage_id = "uninitialized"
self.transaction = None
self.read_only = read_only
self.log_label = 'unconnected'
self.locked = False # Don't have storage lock
self.verifying = 0
self.store_failed = 0
self.authenticated = 0
self.auth_realm = auth_realm
self.blob_tempfile = None
# The authentication protocol may define extra methods.
self._extensions = {}
for func in self.extensions:
self._extensions[func.__name__] = None
self._iterators = {}
self._iterator_ids = itertools.count()
# Stores the last item that was handed out for a
# transaction iterator.
self._txn_iterators_last = {}
def _finish_auth(self, authenticated):
if not self.auth_realm:
return 1
self.authenticated = authenticated
return authenticated
def set_database(self, database):
self.database = database
def notifyConnected(self, conn):
def notify_connected(self, conn):
self.connection = conn
assert conn.peer_protocol_version is not None
if conn.peer_protocol_version < b'Z309':
self.client = ClientStub308(conn)
conn.register_object(ZEOStorage308Adapter(self))
else:
self.client = ClientStub(conn)
self.call_soon_threadsafe = conn.call_soon_threadsafe
self.connected = True
assert conn.protocol_version is not None
self.log_label = _addr_label(conn.addr)
self.async = conn.async
self.async_threadsafe = conn.async_threadsafe
def notifyDisconnected(self):
def notify_disconnected(self):
# When this storage closes, we must ensure that it aborts
# any pending transaction.
if self.transaction is not None:
......@@ -126,7 +117,8 @@ class ZEOStorage:
else:
self.log("disconnected")
self.connection = None
self.connected = False
self.server.close_conn(self)
def __repr__(self):
tid = self.transaction and repr(self.transaction.id)
......@@ -153,23 +145,13 @@ class ZEOStorage:
if not info['supportsUndo']:
self.undoLog = self.undoInfo = lambda *a,**k: ()
# XXX deprecated: but ZODB tests use getTid. They shouldn't
self.getTid = storage.getTid
self.load = storage.load
self.loadSerial = storage.loadSerial
record_iternext = getattr(storage, 'record_iternext', None)
if record_iternext is not None:
self.record_iternext = record_iternext
try:
fn = storage.getExtensionMethods
except AttributeError:
pass # no extension methods
else:
d = fn()
self._extensions.update(d)
for name in d:
assert not hasattr(self, name)
setattr(self, name, getattr(storage, name))
self.lastTransaction = storage.lastTransaction
try:
......@@ -185,6 +167,8 @@ class ZEOStorage:
else:
raise
self.connection.methods = registered_methods
def history(self,tid,size=1):
# This caters for storages which still accept
# a version parameter.
......@@ -212,15 +196,6 @@ class ZEOStorage:
return 0
return 1
def getAuthProtocol(self):
"""Return string specifying name of authentication module to use.
The module name should be auth_%s where %s is auth_protocol."""
protocol = self.server.auth_protocol
if not protocol or protocol == 'none':
return None
return protocol
def register(self, storage_id, read_only):
"""Select the storage that this client will use
......@@ -228,9 +203,6 @@ class ZEOStorage:
For authenticated storages this method will be called by the client
immediately after authentication is finished.
"""
if self.auth_realm and not self.authenticated:
raise AuthError("Client was never authenticated with server!")
if self.storage is not None:
self.log("duplicate register() call")
raise ValueError("duplicate register() call")
......@@ -248,13 +220,15 @@ class ZEOStorage:
self.storage = storage
self.setup_delegation()
self.stats = self.server.register_connection(storage_id, self)
self.lock_manager = self.server.lock_managers[storage_id]
return self.lastTransaction()
def get_info(self):
storage = self.storage
supportsUndo = (getattr(storage, 'supportsUndo', lambda : False)()
and self.connection.peer_protocol_version >= b'Z310')
and self.connection.protocol_version >= b'Z310')
# Communicate the backend storage interfaces to the client
storage_provides = zope.interface.providedBy(storage)
......@@ -266,7 +240,6 @@ class ZEOStorage:
'size': storage.getSize(),
'name': storage.getName(),
'supportsUndo': supportsUndo,
'extensionMethods': self.getExtensionMethods(),
'supports_record_iternext': hasattr(self, 'record_iternext'),
'interfaces': tuple(interfaces),
}
......@@ -276,13 +249,6 @@ class ZEOStorage:
'size': self.storage.getSize(),
}
def getExtensionMethods(self):
return self._extensions
def loadEx(self, oid):
self.stats.loads += 1
return self.storage.load(oid, '')
def loadBefore(self, oid, tid):
self.stats.loads += 1
return self.storage.loadBefore(oid, tid)
......@@ -295,37 +261,6 @@ class ZEOStorage:
% (len(invlist), u64(invtid)))
return invtid, invlist
def verify(self, oid, tid):
try:
t = self.getTid(oid)
except KeyError:
self.client.invalidateVerify(oid)
else:
if tid != t:
self.client.invalidateVerify(oid)
def zeoVerify(self, oid, s):
if not self.verifying:
self.verifying = 1
self.stats.verifying_clients += 1
try:
os = self.getTid(oid)
except KeyError:
self.client.invalidateVerify((oid, ''))
# It's not clear what we should do now. The KeyError
# could be caused by an object uncreation, in which case
# invalidation is right. It could be an application bug
# that left a dangling reference, in which case it's bad.
else:
if s != os:
self.client.invalidateVerify((oid, ''))
def endZeoVerify(self):
if self.verifying:
self.stats.verifying_clients -= 1
self.verifying = 0
self.client.endVerify()
def pack(self, time, wait=1):
# Yes, you can pack a read-only server or storage!
if wait:
......@@ -343,8 +278,7 @@ class ZEOStorage:
self.storage.pack(time, referencesf)
self.log("pack(time=%s) complete" % repr(time))
# Broadcast new size statistics
self.server.invalidate(0, self.storage_id, None,
(), self.get_size_info())
self.server.broadcast_info(self.storage_id, self.get_size_info())
def new_oids(self, n=100):
"""Return a sequence of n new oids, where n defaults to 100"""
......@@ -381,12 +315,12 @@ class ZEOStorage:
t._extension = ext
self.serials = []
self.conflicts = {}
self.invalidated = []
self.txnlog = CommitLog()
self.blob_log = []
self.tid = tid
self.status = status
self.store_failed = 0
self.stats.active_txns += 1
# Assign the transaction attribute last. This is so we don't
......@@ -406,6 +340,7 @@ class ZEOStorage:
self.stats.commits += 1
self.storage.tpc_finish(self.transaction, self._invalidate)
self.async('info', self.get_size_info())
# Note that the tid is still current because we still hold the
# commit lock. We'll relinquish it in _clear_transaction.
tid = self.storage.lastTransaction()
......@@ -413,9 +348,7 @@ class ZEOStorage:
return Result(tid, self._clear_transaction)
def _invalidate(self, tid):
if self.invalidated:
self.server.invalidate(self, self.storage_id, tid,
self.invalidated, self.get_size_info())
self.server.invalidate(self, self.storage_id, tid, self.invalidated)
def tpc_abort(self, tid):
if not self._check_tid(tid):
......@@ -426,11 +359,7 @@ class ZEOStorage:
def _clear_transaction(self):
# Common code at end of tpc_finish() and tpc_abort()
if self.locked:
self.server.unlock_storage(self)
self.locked = 0
if self.transaction is not None:
self.server.stop_waiting(self)
self.lock_manager.release(self)
self.transaction = None
self.stats.active_txns -= 1
if self.txnlog is not None:
......@@ -442,23 +371,14 @@ class ZEOStorage:
def vote(self, tid):
self._check_tid(tid, exc=StorageTransactionError)
if self.locked or self.server.already_waiting(self):
raise StorageTransactionError(
'Already voting (%s)' % (self.locked and 'locked' or 'waiting')
)
return self._try_to_vote()
return self.lock_manager.lock(self, self._vote)
def _try_to_vote(self, delay=None):
if self.connection is None:
def _vote(self, delay=None):
# Called from client thread
if not self.connected:
return # We're disconnected
if delay is not None and delay.sent:
# as a consequence of the unlocking strategy, _try_to_vote
# may be called multiple times for delayed
# transactions. The first call will mark the delay as
# sent. We should skip if the delay was already sent.
return
self.locked, delay = self.server.lock_storage(self, delay)
if self.locked:
try:
self.log(
"Preparing to commit transaction: %d objects, %d bytes"
......@@ -472,48 +392,48 @@ class ZEOStorage:
self.storage.tpc_begin(self.transaction)
for op, args in self.txnlog:
if not getattr(self, op)(*args):
break
getattr(self, op)(*args)
# Blob support
while self.blob_log and not self.store_failed:
while self.blob_log:
oid, oldserial, data, blobfilename = self.blob_log.pop()
self._store(oid, oldserial, data, blobfilename)
if not self.store_failed:
# Only call tpc_vote of no store call failed,
# otherwise the serialnos() call will deliver an
# exception that will be handled by the client in
# its tpc_vote() method.
if not self.conflicts:
try:
serials = self.storage.tpc_vote(self.transaction)
except ConflictError as err:
if (self.client_conflict_resolution and
err.oid and err.serials and err.data
):
self.conflicts[err.oid] = dict(
oid=err.oid, serials=err.serials, data=err.data)
else:
raise
else:
if serials:
self.serials.extend(serials)
self.client.serialnos(self.serials)
if self.conflicts:
self.storage.tpc_abort(self.transaction)
return list(self.conflicts.values())
else:
self.locked = True # signal to lock manager to hold lock
return self.serials
except Exception:
except Exception as err:
self.storage.tpc_abort(self.transaction)
self._clear_transaction()
if delay is not None:
delay.error(sys.exc_info())
else:
raise
else:
if delay is not None:
delay.reply(None)
else:
return None
else:
return delay
if isinstance(err, ConflictError):
self.stats.conflicts += 1
self.log("conflict error %s" % err, BLATHER)
def _unlock_callback(self, delay):
connection = self.connection
if connection is None:
self.server.stop_waiting(self)
else:
connection.call_from_thread(self._try_to_vote, delay)
if not isinstance(err, TransactionError):
logger.exception("While voting")
raise
# The public methods of the ZEO client API do not do the real work.
# They defer work until after the storage lock has been acquired.
......@@ -575,7 +495,19 @@ class ZEOStorage:
self.blob_log.append((oid, serial, data, filename))
def sendBlob(self, oid, serial):
self.client.storeBlob(oid, serial, self.storage.loadBlob(oid, serial))
blobfilename = self.storage.loadBlob(oid, serial)
def store():
yield ('receiveBlobStart', (oid, serial))
with open(blobfilename, 'rb') as f:
while 1:
chunk = f.read(59000)
if not chunk:
break
yield ('receiveBlobChunk', (oid, serial, chunk, ))
yield ('receiveBlobStop', (oid, serial))
self.connection.call_async_iter(store())
def undo(*a, **k):
raise NotImplementedError
......@@ -584,120 +516,41 @@ class ZEOStorage:
self._check_tid(tid, exc=StorageTransactionError)
self.txnlog.undo(trans_id)
def _op_error(self, oid, err, op):
self.store_failed = 1
if isinstance(err, ConflictError):
self.stats.conflicts += 1
self.log("conflict error oid=%s msg=%s" %
(oid_repr(oid), str(err)), BLATHER)
if not isinstance(err, TransactionError):
# Unexpected errors are logged and passed to the client
self.log("%s error: %s, %s" % ((op,)+ sys.exc_info()[:2]),
logging.ERROR, exc_info=True)
err = self._marshal_error(err)
# The exception is reported back as newserial for this oid
self.serials.append((oid, err))
def _delete(self, oid, serial):
err = None
try:
self.storage.deleteObject(oid, serial, self.transaction)
except (SystemExit, KeyboardInterrupt):
raise
except Exception as e:
err = e
self._op_error(oid, err, 'delete')
return err is None
def _checkread(self, oid, serial):
err = None
try:
self.storage.checkCurrentSerialInTransaction(
oid, serial, self.transaction)
except (SystemExit, KeyboardInterrupt):
raise
except Exception as e:
err = e
self._op_error(oid, err, 'checkCurrentSerialInTransaction')
return err is None
def _store(self, oid, serial, data, blobfile=None):
err = None
try:
if blobfile is None:
newserial = self.storage.store(
oid, serial, data, '', self.transaction)
self.storage.store(oid, serial, data, '', self.transaction)
else:
newserial = self.storage.storeBlob(
self.storage.storeBlob(
oid, serial, data, blobfile, '', self.transaction)
except (SystemExit, KeyboardInterrupt):
except ConflictError as err:
if self.client_conflict_resolution and err.serials:
self.conflicts[oid] = dict(
oid=oid, serials=err.serials, data=data)
else:
raise
except Exception as error:
self._op_error(oid, error, 'store')
err = error
else:
if oid in self.conflicts:
del self.conflicts[oid]
if serial != b"\0\0\0\0\0\0\0\0":
self.invalidated.append(oid)
if isinstance(newserial, bytes):
newserial = [(oid, newserial)]
for oid, s in newserial or ():
if s == ResolvedSerial:
self.stats.conflicts_resolved += 1
self.log("conflict resolved oid=%s"
% oid_repr(oid), BLATHER)
self.serials.append((oid, s))
return err is None
def _restore(self, oid, serial, data, prev_txn):
err = None
try:
self.storage.restore(oid, serial, data, '', prev_txn,
self.transaction)
except (SystemExit, KeyboardInterrupt):
raise
except Exception as err:
self._op_error(oid, err, 'restore')
return err is None
def _undo(self, trans_id):
err = None
try:
tid, oids = self.storage.undo(trans_id, self.transaction)
except (SystemExit, KeyboardInterrupt):
raise
except Exception as e:
err = e
self._op_error(z64, err, 'undo')
else:
self.invalidated.extend(oids)
self.serials.extend((oid, ResolvedSerial) for oid in oids)
return err is None
def _marshal_error(self, error):
# Try to pickle the exception. If it can't be pickled,
# the RPC response would fail, so use something that can be pickled.
if PY3:
pickler = Pickler(BytesIO(), 3)
else:
# The pure-python version requires at least one argument (PyPy)
pickler = Pickler(0)
pickler.fast = 1
try:
pickler.dump(error)
except:
msg = "Couldn't pickle storage exception: %s" % repr(error)
self.log(msg, logging.ERROR)
error = StorageServerError(msg)
return error
self.serials.extend(oids)
# IStorageIteration support
......@@ -760,7 +613,21 @@ class ZEOStorage:
def set_client_label(self, label):
self.log_label = str(label)+' '+_addr_label(self.connection.addr)
def ruok(self):
return self.server.ruok()
def ping(self):
pass
class StorageServerDB:
"""Adapter from StorageServerDB to ZODB.interfaces.IStorageWrapper
This is used in a ZEO fan-out situation, where a storage server
calls registerDB on a ClientStorage.
Note that this is called from the Client-storage's IO thread, so
always a separate thread from the storge-server connections.
"""
def __init__(self, server, storage_id):
self.server = server
......@@ -788,21 +655,14 @@ class StorageServer:
ZEOStorage instance only handles a single storage.
"""
# Classes we instantiate. A subclass might override.
from .zrpc.server import Dispatcher as DispatcherClass
ZEOStorageClass = ZEOStorage
ManagedServerConnectionClass = ManagedServerConnection
def __init__(self, addr, storages,
read_only=0,
invalidation_queue_size=100,
invalidation_age=None,
transaction_timeout=None,
monitor_address=None,
auth_protocol=None,
auth_database=None,
auth_realm=None,
ssl=None,
client_conflict_resolution=False,
Acceptor=Acceptor,
):
"""StorageServer constructor.
......@@ -847,29 +707,8 @@ class StorageServer:
a transaction to commit after acquiring the storage lock.
If the transaction takes too long, the client connection
will be closed and the transaction aborted.
monitor_address -- The address at which the monitor server
should listen. If specified, a monitor server is started.
The monitor server provides server statistics in a simple
text format.
auth_protocol -- The name of the authentication protocol to use.
Examples are "digest" and "srp".
auth_database -- The name of the password database filename.
It should be in a format compatible with the authentication
protocol used; for instance, "sha" and "srp" require different
formats.
Note that to implement an authentication protocol, a server
and client authentication mechanism must be implemented in a
auth_* module, which should be stored inside the "auth"
subdirectory. This module may also define a DatabaseClass
variable that should indicate what database should be used
by the authenticator.
"""
self.addr = addr
self.storages = storages
msg = ", ".join(
["%s:%s:%s" % (name, storage.isReadOnly() and "RO" or "RW",
......@@ -879,40 +718,32 @@ class StorageServer:
(self.__class__.__name__, read_only and "RO" or "RW", msg))
self._lock = threading.Lock()
self._commit_locks = {}
self._waiting = dict((name, []) for name in storages)
self._lock = Lock()
self.ssl = ssl # For dev convenience
self.read_only = read_only
self.auth_protocol = auth_protocol
self.auth_database = auth_database
self.auth_realm = auth_realm
self.database = None
if auth_protocol:
self._setup_auth(auth_protocol)
# A list, by server, of at most invalidation_queue_size invalidations.
# The list is kept in sorted order with the most recent
# invalidation at the front. The list never has more than
# self.invq_bound elements.
self.invq_bound = invalidation_queue_size
self.invq = {}
self.zeo_storages_by_storage_id = {} # {storage_id -> [ZEOStorage]}
self.lock_managers = {} # {storage_id -> LockManager}
self.stats = {} # {storage_id -> StorageStats}
for name, storage in storages.items():
self._setup_invq(name, storage)
storage.registerDB(StorageServerDB(self, name))
self.invalidation_age = invalidation_age
self.connections = {}
self.socket_map = {}
self.dispatcher = self.DispatcherClass(
addr, factory=self.new_connection, map=self.socket_map)
if len(self.addr) == 2 and self.addr[1] == 0 and self.addr[0]:
self.addr = self.dispatcher.socket.getsockname()
ZODB.event.notify(
Serving(self, address=self.dispatcher.socket.getsockname()))
self.stats = {}
self.timeouts = {}
for name in self.storages.keys():
self.connections[name] = []
self.stats[name] = StorageStats(self.connections[name])
if client_conflict_resolution:
# XXX this may go away later, when storages grow
# configuration for this.
storage.tryToResolveConflict = never_resolve_conflict
self.zeo_storages_by_storage_id[name] = []
self.stats[name] = stats = StorageStats(
self.zeo_storages_by_storage_id[name])
if transaction_timeout is None:
# An object with no-op methods
timeout = StubTimeoutThread()
......@@ -920,15 +751,22 @@ class StorageServer:
timeout = TimeoutThread(transaction_timeout)
timeout.setName("TimeoutThread for %s" % name)
timeout.start()
self.timeouts[name] = timeout
if monitor_address:
warnings.warn(
"The monitor server is deprecated. Use the server_status\n"
"ZEO method instead.",
DeprecationWarning)
self.monitor = StatsServer(monitor_address, self.stats)
self.lock_managers[name] = LockManager(name, stats, timeout)
self.invalidation_age = invalidation_age
self.client_conflict_resolution = client_conflict_resolution
if addr is not None:
self.acceptor = Acceptor(self, addr, ssl)
if isinstance(addr, tuple) and addr[0]:
self.addr = self.acceptor.addr
else:
self.monitor = None
self.addr = addr
self.loop = self.acceptor.loop
ZODB.event.notify(Serving(self, address=self.acceptor.addr))
def create_client_handler(self):
return ZEOStorage(self, self.read_only)
def _setup_invq(self, name, storage):
lastInvalidations = getattr(storage, 'lastInvalidations', None)
......@@ -944,72 +782,19 @@ class StorageServer:
self.invq[name] = list(lastInvalidations(self.invq_bound))
self.invq[name].reverse()
def _setup_auth(self, protocol):
# Can't be done in global scope, because of cyclic references
from .auth import get_module
name = self.__class__.__name__
module = get_module(protocol)
if not module:
log("%s: no such an auth protocol: %s" % (name, protocol))
return
storage_class, client, db_class = module
if not storage_class or not issubclass(storage_class, ZEOStorage):
log(("%s: %s isn't a valid protocol, must have a StorageClass" %
(name, protocol)))
self.auth_protocol = None
return
self.ZEOStorageClass = storage_class
log("%s: using auth protocol: %s" % (name, protocol))
# We create a Database instance here for use with the authenticator
# modules. Having one instance allows it to be shared between multiple
# storages, avoiding the need to bloat each with a new authenticator
# Database that would contain the same info, and also avoiding any
# possibly synchronization issues between them.
self.database = db_class(self.auth_database)
if self.database.realm != self.auth_realm:
raise ValueError("password database realm %r "
"does not match storage realm %r"
% (self.database.realm, self.auth_realm))
def new_connection(self, sock, addr):
"""Internal: factory to create a new connection.
This is called by the Dispatcher class in ZEO.zrpc.server
whenever accept() returns a socket for a new incoming
connection.
"""
if self.auth_protocol and self.database:
zstorage = self.ZEOStorageClass(self, self.read_only,
auth_realm=self.auth_realm)
zstorage.set_database(self.database)
else:
zstorage = self.ZEOStorageClass(self, self.read_only)
c = self.ManagedServerConnectionClass(sock, addr, zstorage, self)
log("new connection %s: %s" % (addr, repr(c)), logging.DEBUG)
return c
def register_connection(self, storage_id, conn):
"""Internal: register a connection with a particular storage.
def register_connection(self, storage_id, zeo_storage):
"""Internal: register a ZEOStorage with a particular storage.
This is called by ZEOStorage.register().
The dictionary self.connections maps each storage name to a
list of current connections for that storage; this information
is needed to handle invalidation. This function updates this
dictionary.
The dictionary self.zeo_storages_by_storage_id maps each
storage name to a list of current ZEOStorages for that
storage; this information is needed to handle invalidation.
This function updates this dictionary.
Returns the timeout and stats objects for the appropriate storage.
"""
self.connections[storage_id].append(conn)
self.zeo_storages_by_storage_id[storage_id].append(zeo_storage)
return self.stats[storage_id]
def _invalidateCache(self, storage_id):
......@@ -1020,24 +805,9 @@ class StorageServer:
and making them reconnect.
"""
# This method can be called from foreign threads. We have to
# This method is called from foreign threads. We have to
# worry about interaction with the main thread.
# 1. We modify self.invq which is read by get_invalidations
# below. This is why get_invalidations makes a copy of
# self.invq.
# 2. We access connections. There are two dangers:
#
# a. We miss a new connection. This is not a problem because
# if a client connects after we get the list of connections,
# then it will have to read the invalidation queue, which
# has already been reset.
#
# b. A connection is closes while we are iterating. This
# doesn't matter, bacause we can call should_close on a closed
# connection.
# Rebuild invq
self._setup_invq(storage_id, self.storages[storage_id])
......@@ -1045,74 +815,34 @@ class StorageServer:
# connections indirectoy by closing them. We don't care about
# later transactions since they will have to validate their
# caches anyway.
for p in self.connections[storage_id][:]:
try:
p.connection.should_close()
p.connection.trigger.pull_trigger()
except DisconnectedError:
pass
for zs in self.zeo_storages_by_storage_id[storage_id][:]:
zs.call_soon_threadsafe(zs.connection.close)
def invalidate(self, conn, storage_id, tid, invalidated=(), info=None):
"""Internal: broadcast info and invalidations to clients.
def invalidate(self, zeo_storage, storage_id, tid, invalidated):
"""Internal: broadcast invalidations to clients.
This is called from several ZEOStorage methods.
invalidated is a sequence of oids.
This can do three different things:
- If the invalidated argument is non-empty, it broadcasts
invalidateTransaction() messages to all clients of the given
storage except the current client (the conn argument).
- If the invalidated argument is empty and the info argument
is a non-empty dictionary, it broadcasts info() messages to
all clients of the given storage, including the current
client.
- If both the invalidated argument and the info argument are
non-empty, it broadcasts invalidateTransaction() messages to all
clients except the current, and sends an info() message to
the current client.
"""
# This method can be called from foreign threads. We have to
# worry about interaction with the main thread.
# 1. We modify self.invq which is read by get_invalidations
# below. This is why get_invalidations makes a copy of
# self.invq.
# 2. We access connections. There are two dangers:
#
# a. We miss a new connection. This is not a problem because
# we are called while the storage lock is held. A new
# connection that tries to read data won't read committed
# data without first recieving an invalidation. Also, if a
# client connects after getting the list of connections,
# then it will have to read the invalidation queue, which
# has been updated to reflect the invalidations.
#
# b. A connection is closes while we are iterating. We'll need
# to cactch and ignore Disconnected errors.
if invalidated:
invq = self.invq[storage_id]
if len(invq) >= self.invq_bound:
invq.pop()
invq.insert(0, (tid, invalidated))
for p in self.connections[storage_id]:
try:
if invalidated and p is not conn:
p.client.invalidateTransaction(tid, invalidated)
elif info is not None:
p.client.info(info)
except DisconnectedError:
pass
for zs in self.zeo_storages_by_storage_id[storage_id]:
if zs is not zeo_storage:
zs.async_threadsafe('invalidateTransaction', tid, invalidated)
def broadcast_info(self, storage_id, info):
"""Internal: broadcast info to clients.
"""
for zs in self.zeo_storages_by_storage_id[storage_id]:
zs.async_threadsafe('info', info)
def get_invalidations(self, storage_id, tid):
"""Return a tid and list of all objects invalidation since tid.
......@@ -1122,6 +852,12 @@ class StorageServer:
Returns None if it is unable to provide a complete list
of invalidations for tid. In this case, client should
do full cache verification.
XXX This API is stupid. It would be better to simply return a
list of oid-tid pairs. With this API, we can't really use the
tid returned and have to discard all versions for an OID. If
we used the max tid, then loadBefore results from the cache
might be incorrect.
"""
# We make a copy of invq because it might be modified by a
......@@ -1153,13 +889,6 @@ class StorageServer:
return latest_tid, list(oids)
def loop(self):
try:
asyncore.loop(map=self.socket_map)
except Exception:
if not self.__closed:
raise # Unexpected exc
__thread = None
def start_thread(self, daemon=True):
self.__thread = thread = threading.Thread(target=self.loop)
......@@ -1178,19 +907,18 @@ class StorageServer:
self.__closed = True
# Stop accepting connections
self.dispatcher.close()
if self.monitor is not None:
self.monitor.close()
self.acceptor.close()
ZODB.event.notify(Closed(self))
# Close open client connections
for sid, connections in self.connections.items():
for conn in connections[:]:
for sid, zeo_storages in self.zeo_storages_by_storage_id.items():
for zs in zeo_storages[:]:
try:
conn.connection.close()
except:
pass
logger.debug("Closing %s", zs.connection)
zs.call_soon_threadsafe(zs.connection.close)
except Exception:
logger.exception("closing connection %r", zs)
for name, storage in six.iteritems(self.storages):
logger.info("closing storage %r", name)
......@@ -1199,117 +927,21 @@ class StorageServer:
if self.__thread is not None:
self.__thread.join(join_timeout)
def close_conn(self, conn):
"""Internal: remove the given connection from self.connections.
def close_conn(self, zeo_storage):
"""Remove the given zeo_storage from self.zeo_storages_by_storage_id.
This is the inverse of register_connection().
"""
for cl in self.connections.values():
if conn.obj in cl:
cl.remove(conn.obj)
def lock_storage(self, zeostore, delay):
storage_id = zeostore.storage_id
waiting = self._waiting[storage_id]
with self._lock:
if storage_id in self._commit_locks:
# The lock is held by another zeostore
locked = self._commit_locks[storage_id]
assert locked is not zeostore, (storage_id, delay)
if locked.connection is None:
locked.log("Still locked after disconnected. Unlocking.",
logging.CRITICAL)
if locked.transaction:
locked.storage.tpc_abort(locked.transaction)
del self._commit_locks[storage_id]
# yuck: have to manipulate lock to appease with :(
self._lock.release()
try:
return self.lock_storage(zeostore, delay)
finally:
self._lock.acquire()
if delay is None:
# New request, queue it
assert not [i for i in waiting if i[0] is zeostore
], "already waiting"
delay = Delay()
waiting.append((zeostore, delay))
zeostore.log("(%r) queue lock: transactions waiting: %s"
% (storage_id, len(waiting)),
_level_for_waiting(waiting)
)
return False, delay
else:
self._commit_locks[storage_id] = zeostore
self.timeouts[storage_id].begin(zeostore)
self.stats[storage_id].lock_time = time.time()
if delay is not None:
# we were waiting, stop
waiting[:] = [i for i in waiting if i[0] is not zeostore]
zeostore.log("(%r) lock: transactions waiting: %s"
% (storage_id, len(waiting)),
_level_for_waiting(waiting)
)
return True, delay
def unlock_storage(self, zeostore):
storage_id = zeostore.storage_id
waiting = self._waiting[storage_id]
with self._lock:
assert self._commit_locks[storage_id] is zeostore
del self._commit_locks[storage_id]
self.timeouts[storage_id].end(zeostore)
self.stats[storage_id].lock_time = None
callbacks = waiting[:]
if callbacks:
assert not [i for i in waiting if i[0] is zeostore
], "waiting while unlocking"
zeostore.log("(%r) unlock: transactions waiting: %s"
% (storage_id, len(callbacks)),
_level_for_waiting(callbacks)
)
for zeostore, delay in callbacks:
try:
zeostore._unlock_callback(delay)
except (SystemExit, KeyboardInterrupt):
raise
except Exception:
logger.exception("Calling unlock callback")
def stop_waiting(self, zeostore):
storage_id = zeostore.storage_id
waiting = self._waiting[storage_id]
with self._lock:
new_waiting = [i for i in waiting if i[0] is not zeostore]
if len(new_waiting) == len(waiting):
return
waiting[:] = new_waiting
zeostore.log("(%r) dequeue lock: transactions waiting: %s"
% (storage_id, len(waiting)),
_level_for_waiting(waiting)
)
def already_waiting(self, zeostore):
storage_id = zeostore.storage_id
waiting = self._waiting[storage_id]
with self._lock:
return bool([i for i in waiting if i[0] is zeostore])
for zeo_storages in self.zeo_storages_by_storage_id.values():
if zeo_storage in zeo_storages:
zeo_storages.remove(zeo_storage)
def server_status(self, storage_id):
status = self.stats[storage_id].__dict__.copy()
status['connections'] = len(status['connections'])
status['waiting'] = len(self._waiting[storage_id])
status['timeout-thread-is-alive'] = self.timeouts[storage_id].isAlive()
lock_manager = self.lock_managers[storage_id]
status['waiting'] = len(lock_manager.waiting)
status['timeout-thread-is-alive'] = lock_manager.timeout.isAlive()
last_transaction = self.storages[storage_id].lastTransaction()
last_transaction_hex = codecs.encode(last_transaction, 'hex_codec')
if PY3:
......@@ -1322,14 +954,6 @@ class StorageServer:
return dict((storage_id, self.server_status(storage_id))
for storage_id in self.storages)
def _level_for_waiting(waiting):
if len(waiting) > 9:
return logging.CRITICAL
if len(waiting) > 3:
return logging.WARNING
else:
return logging.DEBUG
class StubTimeoutThread:
def begin(self, client):
......@@ -1358,7 +982,7 @@ class TimeoutThread(threading.Thread):
def begin(self, client):
# Called from the restart code the "main" thread, whenever the
# storage lock is being acquired. (Serialized by asyncore.)
# storage lock is being acquired.
with self._cond:
assert self._client is None
self._client = client
......@@ -1367,7 +991,7 @@ class TimeoutThread(threading.Thread):
def end(self, client):
# Called from the "main" thread whenever the storage lock is
# being released. (Serialized by asyncore.)
# being released.
with self._cond:
assert self._client is not None
assert self._client is client
......@@ -1390,7 +1014,7 @@ class TimeoutThread(threading.Thread):
client.log("Transaction timeout after %s seconds" %
self._timeout, logging.CRITICAL)
try:
client.connection.call_from_thread(client.connection.close)
client.call_soon_threadsafe(client.connection.close)
except:
client.log("Timeout failure", logging.CRITICAL,
exc_info=sys.exc_info())
......@@ -1413,7 +1037,7 @@ class SlowMethodThread(threading.Thread):
"""
# Some storage methods can take a long time to complete. If we
# run these methods via a standard asyncore read handler, they
# run these methods in response to an I/O event, they
# will block all other server activity until they complete. To
# avoid blocking, we spawn a separate thread, return an MTDelay()
# object, and have the thread reply() when it finishes.
......@@ -1436,145 +1060,6 @@ class SlowMethodThread(threading.Thread):
self.delay.reply(result)
class ClientStub:
def __init__(self, rpc):
self.rpc = rpc
def beginVerify(self):
self.rpc.callAsync('beginVerify')
def invalidateVerify(self, args):
self.rpc.callAsync('invalidateVerify', args)
def endVerify(self):
self.rpc.callAsync('endVerify')
def invalidateTransaction(self, tid, args):
# Note that this method is *always* called from a different
# thread than self.rpc's async thread. It is the only method
# for which this is true and requires special consideration!
# callAsyncNoSend is important here because:
# - callAsyncNoPoll isn't appropriate because
# the network thread may not wake up for a long time,
# delaying invalidations for too long. (This is demonstrateed
# by a test failure.)
# - callAsync isn't appropriate because (on the server) it tries
# to write to the socket. If self.rpc's network thread also
# tries to write at the ame time, we can run into problems
# because handle_write isn't thread safe.
self.rpc.callAsyncNoSend('invalidateTransaction', tid, args)
def serialnos(self, arg):
self.rpc.callAsyncNoPoll('serialnos', arg)
def info(self, arg):
self.rpc.callAsyncNoPoll('info', arg)
def storeBlob(self, oid, serial, blobfilename):
def store():
yield ('receiveBlobStart', (oid, serial))
f = open(blobfilename, 'rb')
while 1:
chunk = f.read(59000)
if not chunk:
break
yield ('receiveBlobChunk', (oid, serial, chunk, ))
f.close()
yield ('receiveBlobStop', (oid, serial))
self.rpc.callAsyncIterator(store())
class ClientStub308(ClientStub):
def invalidateTransaction(self, tid, args):
ClientStub.invalidateTransaction(
self, tid, [(arg, '') for arg in args])
def invalidateVerify(self, oid):
ClientStub.invalidateVerify(self, (oid, ''))
class ZEOStorage308Adapter:
def __init__(self, storage):
self.storage = storage
def __eq__(self, other):
return self is other or self.storage is other
def getSerial(self, oid):
return self.storage.loadEx(oid)[1] # Z200
def history(self, oid, version, size=1):
if version:
raise ValueError("Versions aren't supported.")
return self.storage.history(oid, size=size)
def getInvalidations(self, tid):
result = self.storage.getInvalidations(tid)
if result is not None:
result = result[0], [(oid, '') for oid in result[1]]
return result
def verify(self, oid, version, tid):
if version:
raise StorageServerError("Versions aren't supported.")
return self.storage.verify(oid, tid)
def loadEx(self, oid, version=''):
if version:
raise StorageServerError("Versions aren't supported.")
data, serial = self.storage.loadEx(oid)
return data, serial, ''
def storea(self, oid, serial, data, version, id):
if version:
raise StorageServerError("Versions aren't supported.")
self.storage.storea(oid, serial, data, id)
def storeBlobEnd(self, oid, serial, data, version, id):
if version:
raise StorageServerError("Versions aren't supported.")
self.storage.storeBlobEnd(oid, serial, data, id)
def storeBlobShared(self, oid, serial, data, filename, version, id):
if version:
raise StorageServerError("Versions aren't supported.")
self.storage.storeBlobShared(oid, serial, data, filename, id)
def getInfo(self):
result = self.storage.getInfo()
result['supportsVersions'] = False
return result
def zeoVerify(self, oid, s, sv=None):
if sv:
raise StorageServerError("Versions aren't supported.")
self.storage.zeoVerify(oid, s)
def modifiedInVersion(self, oid):
return ''
def versions(self):
return ()
def versionEmpty(self, version):
return True
def commitVersion(self, *a, **k):
raise NotImplementedError
abortVersion = commitVersion
def zeoLoad(self, oid): # Z200
p, s = self.storage.loadEx(oid)
return p, s, '', None, None
def __getattr__(self, name):
return getattr(self.storage, name)
def _addr_label(addr):
if isinstance(addr, six.binary_type):
return addr.decode('ascii')
......@@ -1637,3 +1122,136 @@ class Serving(ServerEvent):
class Closed(ServerEvent):
pass
def never_resolve_conflict(oid, committedSerial, oldSerial, newpickle,
committedData=b''):
raise ConflictError(oid=oid, serials=(committedSerial, oldSerial),
data=newpickle)
class LockManager(object):
def __init__(self, storage_id, stats, timeout):
self.storage_id = storage_id
self.stats = stats
self.timeout = timeout
self.locked = None
self.waiting = {} # {ZEOStorage -> (func, delay)}
self._lock = RLock()
def lock(self, zs, func):
"""Call the given function with the commit lock.
If we can get the lock right away, return the result of
calling the function.
If we can't get the lock right away, return a delay
The function must set ``locked`` on the zeo-storage to
indicate that the zeo-storage should be locked. Otherwise,
the lock isn't held pas the call.
"""
with self._lock:
if self._can_lock(zs):
self._locked(zs)
else:
if any(w for w in self.waiting if w is zs):
raise StorageTransactionError("Already voting (waiting)")
delay = Delay()
self.waiting[zs] = (func, delay)
self._log_waiting(
zs, "(%r) queue lock: transactions waiting: %s")
return delay
try:
result = func()
except Exception:
self.release(zs)
raise
else:
if not zs.locked:
self.release(zs)
return result
def _lock_waiting(self, zs):
waiting = None
with self._lock:
if self.locked is zs:
assert zs.locked
return
if self._can_lock(zs):
waiting = self.waiting.pop(zs, None)
if waiting:
self._locked(zs)
if waiting:
func, delay = waiting
try:
result = func()
except Exception:
delay.error(sys.exc_info())
self.release(zs)
else:
delay.reply(result)
if not zs.locked:
self.release(zs)
def release(self, zs):
with self._lock:
locked = self.locked
if locked is zs:
self._unlocked(zs)
for zs in list(self.waiting):
zs.call_soon_threadsafe(self._lock_waiting, zs)
else:
if self.waiting.pop(zs, None):
self._log_waiting(
zs, "(%r) dequeue lock: transactions waiting: %s")
def _log_waiting(self, zs, message):
l = len(self.waiting)
zs.log(message % (self.storage_id, l),
logging.CRITICAL if l > 9 else (
logging.WARNING if l > 3 else logging.DEBUG)
)
def _can_lock(self, zs):
locked = self.locked
if locked is zs:
raise StorageTransactionError("Already voting (locked)")
if locked is not None:
if not locked.connected:
locked.log("Still locked after disconnected. Unlocking.",
logging.CRITICAL)
if locked.transaction:
locked.storage.tpc_abort(locked.transaction)
self._unlocked(locked)
locked = None
# Note that locked.locked may not be true here, because
# .lock may be set in the lock callback, but may not have
# been set yet. This aspect of the API may need more
# thought. :/
return locked is None
def _locked(self, zs):
self.locked = zs
self.stats.lock_time = time.time()
self._log_waiting(zs, "(%r) lock: transactions waiting: %s")
self.timeout.begin(zs)
return True
def _unlocked(self, zs):
assert self.locked is zs
self.timeout.end(zs)
self.locked = self.stats.lock_time = None
zs.locked = False
self._log_waiting(zs, "(%r) unlock: transactions waiting: %s")
src/ZEO/TransactionBuffer.py
View file @ c499ea7d
......@@ -21,44 +21,22 @@ is used to store the data until a commit or abort.
# A faster implementation might store trans data in memory until it
# reaches a certain size.
from threading import Lock
import os
import tempfile
import ZODB.blob
from ZEO._compat import Pickler, Unpickler
class TransactionBuffer:
# Valid call sequences:
#
# ((store | invalidate)* begin_iterate next* clear)* close
#
# get_size can be called any time
# The TransactionBuffer is used by client storage to hold update
# data until the tpc_finish(). It is normally used by a single
# data until the tpc_finish(). It is only used by a single
# thread, because only one thread can be in the two-phase commit
# at one time.
# It is possible, however, for one thread to close the storage
# while another thread is in the two-phase commit. We must use
# a lock to guard against this race, because unpredictable things
# can happen in Python if one thread closes a file that another
# thread is reading. In a debug build, an assert() can fail.
# Caution: If an operation is performed on a closed TransactionBuffer,
# it has no effect and does not raise an exception. The only time
# this should occur is when a ClientStorage is closed in one
# thread while another thread is in its tpc_finish(). It's not
# clear what should happen in this case. If the tpc_finish()
# completes without error, the Connection using it could have
# inconsistent data. This should have minimal effect, though,
# because the Connection is connected to a closed storage.
def __init__(self):
def __init__(self, connection_generation):
self.connection_generation = connection_generation
self.file = tempfile.TemporaryFile(suffix=".tbuf")
self.lock = Lock()
self.closed = 0
self.count = 0
self.size = 0
self.blobs = []
......@@ -66,89 +44,65 @@ class TransactionBuffer:
# stored are builtin types -- strings or None.
self.pickler = Pickler(self.file, 1)
self.pickler.fast = 1
self.server_resolved = set() # {oid}
self.client_resolved = {} # {oid -> buffer_record_number}
self.exception = None
def close(self):
self.clear()
self.lock.acquire()
try:
self.closed = 1
try:
self.file.close()
except OSError:
pass
finally:
self.lock.release()
def store(self, oid, data):
"""Store oid, version, data for later retrieval"""
self.lock.acquire()
try:
if self.closed:
return
self.pickler.dump((oid, data))
self.count += 1
# Estimate per-record cache size
self.size = self.size + (data and len(data) or 0) + 31
finally:
self.lock.release()
def storeBlob(self, oid, blobfilename):
self.blobs.append((oid, blobfilename))
def invalidate(self, oid):
self.lock.acquire()
try:
if self.closed:
return
self.pickler.dump((oid, None))
self.count += 1
finally:
self.lock.release()
def clear(self):
"""Mark the buffer as empty"""
self.lock.acquire()
try:
if self.closed:
return
self.file.seek(0)
self.count = 0
self.size = 0
while self.blobs:
oid, blobfilename = self.blobs.pop()
if os.path.exists(blobfilename):
ZODB.blob.remove_committed(blobfilename)
finally:
self.lock.release()
def __iter__(self):
self.lock.acquire()
try:
if self.closed:
return
self.file.flush()
self.file.seek(0)
return TBIterator(self.file, self.count)
finally:
self.lock.release()
def resolve(self, oid, data):
"""Record client-resolved data
"""
self.store(oid, data)
self.client_resolved[oid] = self.count - 1
class TBIterator(object):
def server_resolve(self, oid):
self.server_resolved.add(oid)
def __init__(self, f, count):
self.file = f
self.count = count
self.unpickler = Unpickler(f)
def storeBlob(self, oid, blobfilename):
self.blobs.append((oid, blobfilename))
def __iter__(self):
return self
def __next__(self):
"""Return next tuple of data or None if EOF"""
if self.count == 0:
self.file.seek(0)
self.size = 0
raise StopIteration
oid_ver_data = self.unpickler.load()
self.count -= 1
return oid_ver_data
next = __next__
unpickler = Unpickler(self.file)
server_resolved = self.server_resolved
client_resolved = self.client_resolved
# Gaaaa, this is awkward. There can be entries in serials that
# aren't in the buffer, because undo. Entries can be repeated
# in the buffer, because ZODB. (Maybe this is a bug now, but
# it may be a feature later.
seen = set()
for i in range(self.count):
oid, data = unpickler.load()
if client_resolved.get(oid, i) == i:
seen.add(oid)
yield oid, data, oid in server_resolved
# We may have leftover oids because undo
for oid in server_resolved:
if oid not in seen:
yield oid, None, True
# Support ZEO4:
def serialnos(self, args):
for oid in args:
if isinstance(oid, bytes):
self.server_resolved.add(oid)
else:
oid, serial = oid
if isinstance(serial, Exception):
self.exception = serial
elif serial == b'rs':
self.server_resolved.add(oid)
src/ZEO/__init__.py
View file @ c499ea7d
......@@ -26,14 +26,24 @@ def client(*args, **kw):
return ZEO.ClientStorage.ClientStorage(*args, **kw)
def DB(*args, **kw):
s = client(*args, **kw)
try:
import ZODB
return ZODB.DB(client(*args, **kw))
return ZODB.DB(s)
except Exception:
s.close()
raise
def connection(*args, **kw):
return DB(*args, **kw).open_then_close_db_when_connection_closes()
db = DB(*args, **kw)
try:
return db.open_then_close_db_when_connection_closes()
except Exception:
db.close()
ra
def server(path=None, blob_dir=None, storage_conf=None, zeo_conf=None,
port=None):
port=0, threaded=True, **kw):
"""Convenience function to start a server for interactive exploration
This fuction starts a ZEO server, given a storage configuration or
......@@ -68,20 +78,13 @@ def server(path=None, blob_dir=None, storage_conf=None, zeo_conf=None,
port
If no ZEO configuration is supplied, the one will be computed
from the port. If no port is supplied, one will be chosedn
randomly.
dynamically.
"""
import os, ZEO.tests.forker
if storage_conf is None and path is None:
storage_conf = '<mappingstorage>\n</mappingstorage>'
if port is None and zeo_conf is None:
port = ZEO.tests.forker.get_port()
addr, admin, pid, config = ZEO.tests.forker.start_zeo_server(
return ZEO.tests.forker.start_zeo_server(
storage_conf, zeo_conf, port, keep=True, path=path,
blob_dir=blob_dir, suicide=False)
os.remove(config)
def stop_server():
ZEO.tests.forker.shutdown_zeo_server(admin)
os.waitpid(pid, 0)
return addr, stop_server
blob_dir=blob_dir, suicide=False, threaded=threaded, **kw)
src/ZEO/asyncio/README.rst 0 → 100644
View file @ c499ea7d
================================
asyncio-based networking for ZEO
================================
This package provides the networking interface for ZEO. It provides a
somewhat RPC-like API.
Notes
=====
Sending data immediately: ayncio vs asyncore
--------------------------------------------
The previous ZEO networking implementation used the ``asyncore`` library.
When writing with asyncore, writes were done only from the event loop.
This meant that when sending data, code would have to "wake up" the
event loop, typically after adding data to some sort of output buffer.
Asyncio takes an entirely different and saner approach. When an
application wants to send data, it writes to a transport. All
interactions with a transport (in a correct application) are from the
same thread, which is also the thread running any event loop.
Transports are always either idle or sending data. When idle, the
transport writes to the outout socket immediately. If not all data
isn't sent, then it buffers it and becomes sending. If a transport is
sending, then we know that the socket isn't ready for more data, so
``write`` can just buffer the data. There's no point in waking up the
event loop, because the socket will do so when it's ready for more
data.
An exception to the paragraph above occurs when operations cross
threads, as occures for most client operations and when a transaction
commits on the server and results have to be sent to other clients. In
these cases, a call_soon_threadsafe method is used which queues an
operation and has to wake up an event loop to process it.
Server threading
----------------
There are currently two server implementations, an implementation that
used a thread per client (and a thread to listen for connections),
``ZEO.asyncio.mtacceptor.Acceptor``, and an implementation that uses a
single networking thread, ``ZEO.asyncio.server.Acceptor``. The
implementation is selected by changing an import in
``ZEO.StorageServer``. The currently-used implementation is
``ZEO.asyncio.server.Acceptor``, although this sentance is likely to
rot, so check the import to be sure. (Maybe this should be configurable.)
ZEO switched to a multi-threaded implementation several years ago
because it was found to improve performance for large databases using
magnetic disks. Because client threads are always working on behalf of
a single client, there's not really an issue with making blocking
calls, such as executing slow I/O operations.
Initially, the asyncio-based implementation used a multi-threaded
server. A simple thread accepted connections and handed accepted
sockets to ``create_connection``. This became a problem when SSL was
added because ``create_connection`` sets up SSL conections as client
connections, and doesn't provide an option to create server
connections.
In response, I created an ``asyncio.Server``-based implementation.
This required using a single thread. This was a pretty trivial
change, however, it led to the tests becoming unstable to the point
that it was impossible to run all tests without some failing. One
test was broken due to a ``asyncio.Server`` `bug
<http://bugs.python.org/issue27386>`_. It's unclear whether the test
instability is due to ``asyncio.Server`` problems or due to latent
test (or ZEO) bugs, but even after beating the tests mostly into
submission, tests failures are more likely when using
``asyncio.Server``. Beatings will continue.
While fighting test failures using ``asyncio.Server``, the
multi-threaded implementation was updated to use a monkey patch to
allow it to create SSL server connections. Aside from the real risk of a
monkey patch, this works very well.
Both implementations seem to perform about the same.
src/ZEO/asyncio/base.py 0 → 100644
View file @ c499ea7d
from .._compat import PY3
if PY3:
import asyncio
else:
import trollius as asyncio
import logging
import socket
from struct import unpack
import sys
from .marshal import encoder
logger = logging.getLogger(__name__)
INET_FAMILIES = socket.AF_INET, socket.AF_INET6
class Protocol(asyncio.Protocol):
"""asyncio low-level ZEO base interface
"""
# All of the code in this class runs in a single dedicated
# thread. Thus, we can mostly avoid worrying about interleaved
# operations.
# One place where special care was required was in cache setup on
# connect. See finish connect below.
transport = protocol_version = None
def __init__(self, loop, addr):
self.loop = loop
self.addr = addr
self.input = [] # Input buffer when assembling messages
self.output = [] # Output buffer when paused
self.paused = [] # Paused indicator, mutable to avoid attr lookup
# Handle the first message, the protocol handshake, differently
self.message_received = self.first_message_received
def __repr__(self):
return self.name
closed = False
def close(self):
if not self.closed:
self.closed = True
if self.transport is not None:
self.transport.close()
def connection_made(self, transport):
logger.info("Connected %s", self)
if sys.version_info < (3, 6):
sock = transport.get_extra_info('socket')
if sock is not None and sock.family in INET_FAMILIES:
# See https://bugs.python.org/issue27456 :(
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, True)
self.transport = transport
paused = self.paused
output = self.output
append = output.append
writelines = transport.writelines
from struct import pack
def write(message):
if paused:
append(message)
else:
writelines((pack(">I", len(message)), message))
self._write = write
def writeit(data):
# Note, don't worry about combining messages. Iters
# will be used with blobs, in which case, the individual
# messages will be big to begin with.
data = iter(data)
for message in data:
writelines((pack(">I", len(message)), message))
if paused:
append(data)
break
self._writeit = writeit
got = 0
want = 4
getting_size = True
def data_received(self, data):
# Low-level input handler collects data into sized messages.
# Note that the logic below assume that when new data pushes
# us over what we want, we process it in one call until we
# need more, because we assume that excess data is all in the
# last item of self.input. This is why the exception handling
# in the while loop is critical. Without it, an exception
# might cause us to exit before processing all of the data we
# should, when then causes the logic to be broken in
# subsequent calls.
self.got += len(data)
self.input.append(data)
while self.got >= self.want:
try:
extra = self.got - self.want
if extra == 0:
collected = b''.join(self.input)
self.input = []
else:
input = self.input
self.input = [input[-1][-extra:]]
input[-1] = input[-1][:-extra]
collected = b''.join(input)
self.got = extra
if self.getting_size:
# we were recieving the message size
assert self.want == 4
self.want = unpack(">I", collected)[0]
self.getting_size = False
else:
self.want = 4
self.getting_size = True
self.message_received(collected)
except Exception:
logger.exception("data_received %s %s %s",
self.want, self.got, self.getting_size)
def first_message_received(self, protocol_version):
# Handler for first/handshake message, set up in __init__
del self.message_received # use default handler from here on
self.encode = encoder()
self.finish_connect(protocol_version)
def call_async(self, method, args):
self._write(self.encode(0, True, method, args))
def call_async_iter(self, it):
self._writeit(self.encode(0, True, method, args)
for method, args in it)
def pause_writing(self):
self.paused.append(1)
def resume_writing(self):
paused = self.paused
del paused[:]
output = self.output
writelines = self.transport.writelines
from struct import pack
while output and not paused:
message = output.pop(0)
if isinstance(message, bytes):
writelines((pack(">I", len(message)), message))
else:
data = message
for message in data:
writelines((pack(">I", len(message)), message))
if paused: # paused again. Put iter back.
output.insert(0, data)
break
def get_peername(self):
return self.transport.get_extra_info('peername')
src/ZEO/asyncio/client.py 0 → 100644
View file @ c499ea7d
from ZEO.Exceptions import ClientDisconnected, ServerException
import concurrent.futures
import functools
import logging
import random
import threading
import ZODB.event
import ZODB.POSException
import ZEO.Exceptions
import ZEO.interfaces
from . import base
from .compat import asyncio, new_event_loop
from .marshal import decode
logger = logging.getLogger(__name__)
Fallback = object()
local_random = random.Random() # use separate generator to facilitate tests
def future_generator(func):
"""Decorates a generator that generates futures
"""
@functools.wraps(func)
def call_generator(*args, **kw):
gen = func(*args, **kw)
try:
f = next(gen)
except StopIteration:
gen.close()
else:
def store(gen, future):
@future.add_done_callback
def _(future):
try:
try:
result = future.result()
except Exception as exc:
f = gen.throw(exc)
else:
f = gen.send(result)
except StopIteration:
gen.close()
else:
store(gen, f)
store(gen, f)
return call_generator
class Protocol(base.Protocol):
"""asyncio low-level ZEO client interface
"""
# All of the code in this class runs in a single dedicated
# thread. Thus, we can mostly avoid worrying about interleaved
# operations.
# One place where special care was required was in cache setup on
# connect. See finish connect below.
protocols = b'Z309', b'Z310', b'Z3101', b'Z4', b'Z5'
def __init__(self, loop,
addr, client, storage_key, read_only, connect_poll=1,
heartbeat_interval=60, ssl=None, ssl_server_hostname=None,
credentials=None):
"""Create a client interface
addr is either a host,port tuple or a string file name.
client is a ClientStorage. It must be thread safe.
cache is a ZEO.interfaces.IClientCache.
"""
super(Protocol, self).__init__(loop, addr)
self.storage_key = storage_key
self.read_only = read_only
self.name = "%s(%r, %r, %r)" % (
self.__class__.__name__, addr, storage_key, read_only)
self.client = client
self.connect_poll = connect_poll
self.heartbeat_interval = heartbeat_interval
self.futures = {} # { message_id -> future }
self.ssl = ssl
self.ssl_server_hostname = ssl_server_hostname
self.credentials = credentials
self.connect()
def close(self):
if not self.closed:
self.closed = True
self._connecting.cancel()
if self.transport is not None:
self.transport.close()
for future in self.pop_futures():
future.set_exception(ClientDisconnected("Closed"))
def pop_futures(self):
# Remove and return futures from self.futures. The caller
# will finalize them in some way and callbacks may modify
# self.futures.
futures = list(self.futures.values())
self.futures.clear()
return futures
def protocol_factory(self):
return self
def connect(self):
if isinstance(self.addr, tuple):
host, port = self.addr
cr = self.loop.create_connection(
self.protocol_factory, host or '127.0.0.1', port,
ssl=self.ssl, server_hostname=self.ssl_server_hostname)
else:
cr = self.loop.create_unix_connection(
self.protocol_factory, self.addr, ssl=self.ssl)
self._connecting = cr = asyncio.async(cr, loop=self.loop)
@cr.add_done_callback
def done_connecting(future):
if future.exception() is not None:
logger.info("Connection to %r failed, retrying, %s",
self.addr, future.exception())
# keep trying
if not self.closed:
self.loop.call_later(
self.connect_poll + local_random.random(),
self.connect,
)
def connection_made(self, transport):
super(Protocol, self).connection_made(transport)
self.heartbeat(write=False)
def connection_lost(self, exc):
logger.debug('connection_lost %r', exc)
self.heartbeat_handle.cancel()
if self.closed:
for f in self.pop_futures():
f.cancel()
else:
# We have to be careful processing the futures, because
# exception callbacks might modufy them.
for f in self.pop_futures():
f.set_exception(ClientDisconnected(exc or 'connection lost'))
self.closed = True
self.client.disconnected(self)
@future_generator
def finish_connect(self, protocol_version):
# The future implementation we use differs from
# asyncio.Future in that callbacks are called immediately,
# rather than using the loops call_soon. We want to avoid a
# race between invalidations and cache initialization. In
# particular, after getting a response from lastTransaction or
# getInvalidations, we want to make sure we set the cache's
# lastTid before processing (and possibly missing) subsequent
# invalidations.
self.protocol_version = min(protocol_version, self.protocols[-1])
if self.protocol_version not in self.protocols:
self.client.register_failed(
self, ZEO.Exceptions.ProtocolError(protocol_version))
return
self._write(self.protocol_version)
credentials = (self.credentials,) if self.credentials else ()
try:
try:
server_tid = yield self.fut(
'register', self.storage_key,
self.read_only if self.read_only is not Fallback else False,
*credentials)
except ZODB.POSException.ReadOnlyError:
if self.read_only is Fallback:
self.read_only = True
server_tid = yield self.fut(
'register', self.storage_key, True, *credentials)
else:
raise
else:
if self.read_only is Fallback:
self.read_only = False
except Exception as exc:
self.client.register_failed(self, exc)
else:
self.client.registered(self, server_tid)
exception_type_type = type(Exception)
def message_received(self, data):
msgid, async, name, args = decode(data)
if name == '.reply':
future = self.futures.pop(msgid)
if (async): # ZEO 5 exception
class_, args = args
factory = exc_factories.get(class_)
if factory:
exc = factory(class_, args)
if not isinstance(exc, unlogged_exceptions):
logger.error("%s from server: %s:%s",
self.name, class_, args)
else:
exc = ServerException(class_, args)
future.set_exception(exc)
elif (isinstance(args, tuple) and len(args) > 1 and
type(args[0]) == self.exception_type_type and
issubclass(args[0], Exception)
):
if not issubclass(args[0], unlogged_exceptions):
logger.error("%s from server: %s.%s:%s",
self.name,
args[0].__module__,
args[0].__name__,
args[1])
future.set_exception(args[1])
else:
future.set_result(args)
else:
assert async # clients only get async calls
if name in self.client_methods:
getattr(self.client, name)(*args)
else:
raise AttributeError(name)
message_id = 0
def call(self, future, method, args):
self.message_id += 1
self.futures[self.message_id] = future
self._write(self.encode(self.message_id, False, method, args))
return future
def fut(self, method, *args):
return self.call(Fut(), method, args)
def load_before(self, oid, tid):
# Special-case loadBefore, so we collapse outstanding requests
message_id = (oid, tid)
future = self.futures.get(message_id)
if future is None:
future = asyncio.Future(loop=self.loop)
self.futures[message_id] = future
self._write(
self.encode(message_id, False, 'loadBefore', (oid, tid)))
return future
# Methods called by the server.
# WARNING WARNING we can't call methods that call back to us
# syncronously, as that would lead to DEADLOCK!
client_methods = (
'invalidateTransaction', 'serialnos', 'info',
'receiveBlobStart', 'receiveBlobChunk', 'receiveBlobStop',
# plus: notify_connected, notify_disconnected
)
client_delegated = client_methods[2:]
def heartbeat(self, write=True):
if write:
self._write(b'(J\xff\xff\xff\xffK\x00U\x06.replyNt.')
self.heartbeat_handle = self.loop.call_later(
self.heartbeat_interval, self.heartbeat)
def create_Exception(class_, args):
return exc_classes[class_](*args)
def create_ConflictError(class_, args):
exc = exc_classes[class_](
message = args['message'],
oid = args['oid'],
serials = args['serials'],
)
exc.class_name = args.get('class_name')
return exc
def create_BTreesConflictError(class_, args):
return ZODB.POSException.BTreesConflictError(
p1 = args['p1'],
p2 = args['p2'],
p3 = args['p3'],
reason = args['reason'],
)
def create_MultipleUndoErrors(class_, args):
return ZODB.POSException.MultipleUndoErrors(args['_errs'])
exc_classes = {
'builtins.KeyError': KeyError,
'builtins.TypeError': TypeError,
'exceptions.KeyError': KeyError,
'exceptions.TypeError': TypeError,
'ZODB.POSException.ConflictError': ZODB.POSException.ConflictError,
'ZODB.POSException.POSKeyError': ZODB.POSException.POSKeyError,
'ZODB.POSException.ReadConflictError': ZODB.POSException.ReadConflictError,
'ZODB.POSException.ReadOnlyError': ZODB.POSException.ReadOnlyError,
'ZODB.POSException.StorageTransactionError':
ZODB.POSException.StorageTransactionError,
}
exc_factories = {
'builtins.KeyError': create_Exception,
'builtins.TypeError': create_Exception,
'exceptions.KeyError': create_Exception,
'exceptions.TypeError': create_Exception,
'ZODB.POSException.BTreesConflictError': create_BTreesConflictError,
'ZODB.POSException.ConflictError': create_ConflictError,
'ZODB.POSException.MultipleUndoErrors': create_MultipleUndoErrors,
'ZODB.POSException.POSKeyError': create_Exception,
'ZODB.POSException.ReadConflictError': create_ConflictError,
'ZODB.POSException.ReadOnlyError': create_Exception,
'ZODB.POSException.StorageTransactionError': create_Exception,
}
unlogged_exceptions = (ZODB.POSException.POSKeyError,
ZODB.POSException.ConflictError)
class Client(object):
"""asyncio low-level ZEO client interface
"""
# All of the code in this class runs in a single dedicated
# thread. Thus, we can mostly avoid worrying about interleaved
# operations.
# One place where special care was required was in cache setup on
# connect.
protocol = None
ready = None # Tri-value: None=Never connected, True=connected,
# False=Disconnected
def __init__(self, loop,
addrs, client, cache, storage_key, read_only, connect_poll,
register_failed_poll=9,
ssl=None, ssl_server_hostname=None, credentials=None):
"""Create a client interface
addr is either a host,port tuple or a string file name.
client is a ClientStorage. It must be thread safe.
cache is a ZEO.interfaces.IClientCache.
"""
self.loop = loop
self.addrs = addrs
self.storage_key = storage_key
self.read_only = read_only
self.connect_poll = connect_poll
self.register_failed_poll = register_failed_poll
self.client = client
self.ssl = ssl
self.ssl_server_hostname = ssl_server_hostname
self.credentials = credentials
for name in Protocol.client_delegated:
setattr(self, name, getattr(client, name))
self.cache = cache
self.protocols = ()
self.disconnected(None)
# Work around odd behavior of ZEO4 server. It may send
# invalidations for transactions later than the result of
# getInvalidations. While we support ZEO 4 servers, we'll
# need to keep an invalidation queue. :(
self.verify_invalidation_queue = []
def new_addrs(self, addrs):
self.addrs = addrs
if self.trying_to_connect():
self.disconnected(None)
def trying_to_connect(self):
"""Return whether we're trying to connect
Either because we're disconnected, or because we're connected
read-only, but want a writable connection if we can get one.
"""
return (not self.ready or
self.is_read_only() and self.read_only is Fallback)
closed = False
def close(self):
if not self.closed:
self.closed = True
self.ready = False
if self.protocol is not None:
self.protocol.close()
self.cache.close()
self._clear_protocols()
def _clear_protocols(self, protocol=None):
for p in self.protocols:
if p is not protocol:
p.close()
self.protocols = ()
def disconnected(self, protocol=None):
logger.debug('disconnected %r %r', self, protocol)
if protocol is None or protocol is self.protocol:
if protocol is self.protocol and protocol is not None:
self.client.notify_disconnected()
if self.ready:
self.ready = False
self.connected = concurrent.futures.Future()
self.protocol = None
self._clear_protocols()
if all(p.closed for p in self.protocols):
self.try_connecting()
def upgrade(self, protocol):
self.ready = False
self.connected = concurrent.futures.Future()
self.protocol.close()
self.protocol = protocol
self._clear_protocols(protocol)
def try_connecting(self):
logger.debug('try_connecting')
if not self.closed:
self.protocols = [
Protocol(self.loop, addr, self,
self.storage_key, self.read_only, self.connect_poll,
ssl=self.ssl,
ssl_server_hostname=self.ssl_server_hostname,
credentials=self.credentials,
)
for addr in self.addrs
]
def registered(self, protocol, server_tid):
if self.protocol is None:
self.protocol = protocol
if not (self.read_only is Fallback and protocol.read_only):
# We're happy with this protocol. Tell the others to
# stop trying.
self._clear_protocols(protocol)
self.verify(server_tid)
elif (self.read_only is Fallback and not protocol.read_only and
self.protocol.read_only):
self.upgrade(protocol)
self.verify(server_tid)
else:
protocol.close() # too late, we went home with another
def register_failed(self, protocol, exc):
# A protocol failed registration. That's weird. If they've all
# failed, we should try again in a bit.
if protocol is not self:
protocol.close()
logger.exception("Registration or cache validation failed, %s", exc)
if (self.protocol is None and not
any(not p.closed for p in self.protocols)
):
self.loop.call_later(
self.register_failed_poll + local_random.random(),
self.try_connecting)
verify_result = None # for tests
@future_generator
def verify(self, server_tid):
self.verify_invalidation_queue = [] # See comment in init :(
protocol = self.protocol
if server_tid is None:
server_tid = yield protocol.fut('lastTransaction')
try:
cache = self.cache
if cache:
cache_tid = cache.getLastTid()
if not cache_tid:
self.verify_result = "Non-empty cache w/o tid"
logger.error("Non-empty cache w/o tid -- clearing")
cache.clear()
self.client.invalidateCache()
elif cache_tid > server_tid:
self.verify_result = "Cache newer than server"
logger.critical(
'Client has seen newer transactions than server!')
raise AssertionError("Server behind client, %r < %r, %s",
server_tid, cache_tid, protocol)
elif cache_tid == server_tid:
self.verify_result = "Cache up to date"
else:
vdata = yield protocol.fut('getInvalidations', cache_tid)
if vdata:
self.verify_result = "quick verification"
server_tid, oids = vdata
for oid in oids:
cache.invalidate(oid, None)
self.client.invalidateTransaction(server_tid, oids)
else:
# cache is too old
self.verify_result = "cache too old, clearing"
try:
ZODB.event.notify(
ZEO.interfaces.StaleCache(self.client))
except Exception:
logger.exception("sending StaleCache event")
logger.critical(
"%s dropping stale cache",
getattr(self.client, '__name__', ''),
)
self.cache.clear()
self.client.invalidateCache()
else:
self.verify_result = "empty cache"
except Exception as exc:
del self.protocol
self.register_failed(protocol, exc)
else:
# The cache is validated and the last tid we got from the server.
# Set ready so we apply any invalidations that follow.
# We've been ignoring them up to this point.
self.cache.setLastTid(server_tid)
self.ready = True
# Gaaaa, ZEO 4 work around. See comment in __init__. :(
for tid, oids in self.verify_invalidation_queue:
if tid > server_tid:
self.invalidateTransaction(tid, oids)
self.verify_invalidation_queue = []
try:
info = yield protocol.fut('get_info')
except Exception as exc:
# This is weird. We were connected and verified our cache, but
# Now we errored getting info.
# XXX Need a test fpr this. The lone before is what we
# had, but it's wrong.
self.register_failed(self, exc)
else:
self.client.notify_connected(self, info)
self.connected.set_result(None)
def get_peername(self):
return self.protocol.get_peername()
def call_async_threadsafe(self, future, method, args):
if self.ready:
self.protocol.call_async(method, args)
future.set_result(None)
else:
future.set_exception(ClientDisconnected())
def call_async_from_same_thread(self, method, *args):
return self.protocol.call_async(method, args)
def call_async_iter_threadsafe(self, future, it):
if self.ready:
self.protocol.call_async_iter(it)
future.set_result(None)
else:
future.set_exception(ClientDisconnected())
def _when_ready(self, func, result_future, *args):
if self.ready is None:
# We started without waiting for a connection. (prob tests :( )
result_future.set_exception(ClientDisconnected("never connected"))
else:
@self.connected.add_done_callback
def done(future):
e = future.exception()
if e is not None:
future.set_exception(e)
else:
if self.ready:
func(result_future, *args)
else:
self._when_ready(func, result_future, *args)
def call_threadsafe(self, future, method, args):
if self.ready:
self.protocol.call(future, method, args)
else:
self._when_ready(self.call_threadsafe, future, method, args)
# Special methods because they update the cache.
@future_generator
def load_before_threadsafe(self, future, oid, tid):
data = self.cache.loadBefore(oid, tid)
if data is not None:
future.set_result(data)
elif self.ready:
try:
data = yield self.protocol.load_before(oid, tid)
except Exception as exc:
future.set_exception(exc)
else:
future.set_result(data)
if data:
data, start, end = data
self.cache.store(oid, start, end, data)
else:
self._when_ready(self.load_before_threadsafe, future, oid, tid)
@future_generator
def _prefetch(self, oid, tid):
try:
data = yield self.protocol.load_before(oid, tid)
if data:
data, start, end = data
self.cache.store(oid, start, end, data)
except Exception:
logger.exception("prefetch %r %r" % (oid, tid))
def prefetch(self, future, oids, tid):
if self.ready:
for oid in oids:
if self.cache.loadBefore(oid, tid) is None:
self._prefetch(oid, tid)
future.set_result(None)
else:
future.set_exception(ClientDisconnected())
@future_generator
def tpc_finish_threadsafe(self, future, tid, updates, f):
if self.ready:
try:
tid = yield self.protocol.fut('tpc_finish', tid)
cache = self.cache
for oid, data, resolved in updates:
cache.invalidate(oid, tid)
if data and not resolved:
cache.store(oid, tid, None, data)
cache.setLastTid(tid)
except Exception as exc:
future.set_exception(exc)
# At this point, our cache is in an inconsistent
# state. We need to reconnect in hopes of
# recovering to a consistent state.
self.protocol.close()
self.disconnected(self.protocol)
else:
f(tid)
future.set_result(tid)
else:
future.set_exception(ClientDisconnected())
def close_threadsafe(self, future):
self.close()
future.set_result(None)
def invalidateTransaction(self, tid, oids):
if self.ready:
for oid in oids:
self.cache.invalidate(oid, tid)
self.client.invalidateTransaction(tid, oids)
self.cache.setLastTid(tid)
else:
self.verify_invalidation_queue.append((tid, oids))
def serialnos(self, serials):
# Method called by ZEO4 storage servers.
# Before delegating, check for errors (likely ConflictErrors)
# and invalidate the oids they're associated with. In the
# past, this was done by the client, but now we control the
# cache and this is our last chance, as the client won't call
# back into us when there's an error.
for oid in serials:
if isinstance(oid, bytes):
self.cache.invalidate(oid, None)
else:
oid, serial = oid
if isinstance(serial, Exception) or serial == b'rs':
self.cache.invalidate(oid, None)
self.client.serialnos(serials)
@property
def protocol_version(self):
return self.protocol.protocol_version
def is_read_only(self):
try:
protocol = self.protocol
except AttributeError:
return self.read_only
else:
if protocol is None:
return self.read_only
else:
return protocol.read_only
class ClientRunner(object):
def set_options(self, addrs, wrapper, cache, storage_key, read_only,
timeout=30, disconnect_poll=1,
**kwargs):
self.__args = (addrs, wrapper, cache, storage_key, read_only,
disconnect_poll)
self.__kwargs = kwargs
self.timeout = timeout
def setup_delegation(self, loop):
self.loop = loop
self.client = Client(loop, *self.__args, **self.__kwargs)
self.call_threadsafe = self.client.call_threadsafe
self.call_async_threadsafe = self.client.call_async_threadsafe
from concurrent.futures import Future
call_soon_threadsafe = loop.call_soon_threadsafe
def call(meth, *args, **kw):
timeout = kw.pop('timeout', None)
assert not kw
result = Future()
call_soon_threadsafe(meth, result, *args)
return self.wait_for_result(result, timeout)
self.__call = call
def wait_for_result(self, future, timeout):
try:
return future.result(self.timeout if timeout is None else timeout)
except concurrent.futures.TimeoutError:
if not self.client.ready:
raise ClientDisconnected("timed out waiting for connection")
else:
raise
def call(self, method, *args, **kw):
return self.__call(self.call_threadsafe, method, args, **kw)
def call_future(self, method, *args):
# for tests
result = concurrent.futures.Future()
self.loop.call_soon_threadsafe(
self.call_threadsafe, result, method, args)
return result
def async(self, method, *args):
return self.__call(self.call_async_threadsafe, method, args)
def async_iter(self, it):
return self.__call(self.client.call_async_iter_threadsafe, it)
def prefetch(self, oids, tid):
return self.__call(self.client.prefetch, oids, tid)
def load_before(self, oid, tid):
return self.__call(self.client.load_before_threadsafe, oid, tid)
def tpc_finish(self, tid, updates, f):
return self.__call(self.client.tpc_finish_threadsafe, tid, updates, f)
def is_connected(self):
return self.client.ready
def is_read_only(self):
try:
protocol = self.client.protocol
except AttributeError:
return True
else:
if protocol is None:
return True
else:
return protocol.read_only
def close(self):
self.__call(self.client.close_threadsafe)
# Short circuit from now on. We're closed.
def call_closed(*a, **k):
raise ClientDisconnected('closed')
self.__call = call_closed
def apply_threadsafe(self, future, func, *args):
try:
future.set_result(func(*args))
except Exception as exc:
future.set_exception(exc)
def new_addrs(self, addrs):
# This usually doesn't have an immediate effect, since the
# addrs aren't used until the client disconnects.xs
self.__call(self.apply_threadsafe, self.client.new_addrs, addrs)
def wait(self, timeout=None):
if timeout is None:
timeout = self.timeout
self.wait_for_result(self.client.connected, timeout)
class ClientThread(ClientRunner):
"""Thread wrapper for client interface
A ClientProtocol is run in a dedicated thread.
Calls to it are made in a thread-safe fashion.
"""
def __init__(self, addrs, client, cache,
storage_key='1', read_only=False, timeout=30,
disconnect_poll=1, ssl=None, ssl_server_hostname=None,
credentials=None):
self.set_options(addrs, client, cache, storage_key, read_only,
timeout, disconnect_poll,
ssl=ssl, ssl_server_hostname=ssl_server_hostname,
credentials=credentials)
self.thread = threading.Thread(
target=self.run,
name="%s zeo client networking thread" % client.__name__,
)
self.thread.setDaemon(True)
self.started = threading.Event()
self.thread.start()
self.started.wait()
if self.exception:
raise self.exception
exception = None
def run(self):
loop = None
try:
loop = new_event_loop()
self.setup_delegation(loop)
self.started.set()
loop.run_forever()
except Exception as exc:
raise
logger.exception("Client thread")
self.exception = exc
finally:
if not self.closed:
self.closed = True
try:
if self.client.ready:
self.client.ready = False
self.client.client.notify_disconnected()
except AttributeError:
pass
logger.critical("Client loop stopped unexpectedly")
if loop is not None:
loop.close()
logger.debug('Stopping client thread')
closed = False
def close(self):
if not self.closed:
self.closed = True
super(ClientThread, self).close()
self.loop.call_soon_threadsafe(self.loop.stop)
self.thread.join(9)
if self.exception:
raise self.exception
class Fut(object):
"""Lightweight future that calls it's callback immediately rather than soon
"""
def add_done_callback(self, cb):
self.cb = cb
exc = None
def set_exception(self, exc):
self.exc = exc
self.cb(self)
def set_result(self, result):
self._result = result
self.cb(self)
def result(self):
if self.exc:
raise self.exc
else:
return self._result
src/ZEO/asyncio/compat.py 0 → 100644
View file @ c499ea7d
from .._compat import PY3
if PY3:
import asyncio
try:
from uvloop import new_event_loop
except ImportError:
from asyncio import new_event_loop
else:
import trollius as asyncio
from trollius import new_event_loop
src/ZEO/asyncio/marshal.py 0 → 100644
View file @ c499ea7d
##############################################################################
#
# Copyright (c) 2001, 2002 Zope Foundation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE
#
##############################################################################
"""Support for marshaling ZEO messages
Not to be confused with marshaling objects in ZODB.
We currently use pickle. In the future, we may use a
Python-independent format, or possibly a minimal pickle subset.
"""
import logging
from .._compat import Unpickler, Pickler, BytesIO, PY3, PYPY
from ..shortrepr import short_repr
logger = logging.getLogger(__name__)
def encoder():
"""Return a non-thread-safe encoder
"""
if PY3 or PYPY:
f = BytesIO()
getvalue = f.getvalue
seek = f.seek
truncate = f.truncate
pickler = Pickler(f, 3 if PY3 else 1)
pickler.fast = 1
dump = pickler.dump
def encode(*args):
seek(0)
truncate()
dump(args)
return getvalue()
else:
pickler = Pickler(1)
pickler.fast = 1
dump = pickler.dump
def encode(*args):
return dump(args, 2)
return encode
def encode(*args):
return encoder()(*args)
def decode(msg):
"""Decodes msg and returns its parts"""
unpickler = Unpickler(BytesIO(msg))
unpickler.find_global = find_global
try:
# PyPy, zodbpickle, the non-c-accelerated version
unpickler.find_class = find_global
except AttributeError:
pass
try:
return unpickler.load() # msgid, flags, name, args
except:
logger.error("can't decode message: %s" % short_repr(msg))
raise
def server_decode(msg):
"""Decodes msg and returns its parts"""
unpickler = Unpickler(BytesIO(msg))
unpickler.find_global = server_find_global
try:
# PyPy, zodbpickle, the non-c-accelerated version
unpickler.find_class = server_find_global
except AttributeError:
pass
try:
return unpickler.load() # msgid, flags, name, args
except:
logger.error("can't decode message: %s" % short_repr(msg))
raise
_globals = globals()
_silly = ('__doc__',)
exception_type_type = type(Exception)
def find_global(module, name):
"""Helper for message unpickler"""
try:
m = __import__(module, _globals, _globals, _silly)
except ImportError as msg:
raise ImportError("import error %s: %s" % (module, msg))
try:
r = getattr(m, name)
except AttributeError:
raise ImportError("module %s has no global %s" % (module, name))
safe = getattr(r, '__no_side_effects__', 0)
if safe:
return r
# TODO: is there a better way to do this?
if type(r) == exception_type_type and issubclass(r, Exception):
return r
raise ImportError("Unsafe global: %s.%s" % (module, name))
def server_find_global(module, name):
"""Helper for message unpickler"""
try:
if module != 'ZopeUndo.Prefix':
raise ImportError
m = __import__(module, _globals, _globals, _silly)
except ImportError as msg:
raise ImportError("import error %s: %s" % (module, msg))
try:
r = getattr(m, name)
except AttributeError:
raise ImportError("module %s has no global %s" % (module, name))
return r
src/ZEO/asyncio/mtacceptor.py 0 → 100644
View file @ c499ea7d
##############################################################################
#
# Copyright (c) 2001, 2002 Zope Foundation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE
#
##############################################################################
"""Multi-threaded server connectin acceptor
Each connection is run in it's own thread. Testing serveral years ago
suggsted that this was a win, but ZODB shootout and another
lower-level tests suggest otherwise. It's really unclear, which is
why we're keeping this around for now.
Asyncio doesn't let you accept connections in one thread and handle
them in another. To get around this, we have a listener implemented
using asyncore, but when we get a connection, we hand the socket to
asyncio. This worked well until we added SSL support. (Even then, it
worked on Mac OS X for some reason.)
SSL + non-blocking sockets requires special care, which asyncio
provides. Unfortunately, create_connection, assumes it's creating a
client connection. It would be easy to fix this,
http://bugs.python.org/issue27392, but it's hard to justify the fix to
get it accepted, so we won't bother for now. This currently uses a
horrible monley patch to work with SSL.
To use this module, replace::
from .asyncio.server import Acceptor
with::
from .asyncio.mtacceptor import Acceptor
in ZEO.StorageServer.
"""
import asyncore
import socket
import threading
import time
from .compat import asyncio, new_event_loop
from .server import ServerProtocol
# _has_dualstack: True if the dual-stack sockets are supported
try:
# Check whether IPv6 sockets can be created
s = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
except (socket.error, AttributeError):
_has_dualstack = False
else:
# Check whether enabling dualstack (disabling v6only) works
try:
s.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, False)
except (socket.error, AttributeError):
_has_dualstack = False
else:
_has_dualstack = True
s.close()
del s
import logging
logger = logging.getLogger(__name__)
class Acceptor(asyncore.dispatcher):
"""A server that accepts incoming RPC connections
And creates a separate thread for each.
"""
def __init__(self, storage_server, addr, ssl):
self.storage_server = storage_server
self.addr = addr
self.__socket_map = {}
asyncore.dispatcher.__init__(self, map=self.__socket_map)
self.ssl_context = ssl
self._open_socket()
def _open_socket(self):
addr = self.addr
if type(addr) == tuple:
if addr[0] == '' and _has_dualstack:
# Wildcard listen on all interfaces, both IPv4 and
# IPv6 if possible
self.create_socket(socket.AF_INET6, socket.SOCK_STREAM)
self.socket.setsockopt(
socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, False)
elif ':' in addr[0]:
self.create_socket(socket.AF_INET6, socket.SOCK_STREAM)
if _has_dualstack:
# On Linux, IPV6_V6ONLY is off by default.
# If the user explicitly asked for IPv6, don't bind to IPv4
self.socket.setsockopt(
socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, True)
else:
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
else:
self.create_socket(socket.AF_UNIX, socket.SOCK_STREAM)
self.set_reuse_addr()
for i in range(25):
try:
self.bind(addr)
except Exception as exc:
logger.info("bind on %s failed %s waiting", addr, i)
if i == 24:
raise
else:
time.sleep(5)
except:
logger.exception('binding')
raise
else:
break
if isinstance(addr, tuple) and addr[1] == 0:
self.addr = addr = self.socket.getsockname()[:2]
logger.info("listening on %s", str(addr))
self.listen(5)
def writable(self):
return 0
def readable(self):
return 1
def handle_accept(self):
try:
sock, addr = self.accept()
except socket.error as msg:
logger.info("accepted failed: %s", msg)
return
# We could short-circuit the attempt below in some edge cases
# and avoid a log message by checking for addr being None.
# Unfortunately, our test for the code below,
# quick_close_doesnt_kill_server, causes addr to be None and
# we'd have to write a test for the non-None case, which is
# *even* harder to provoke. :/ So we'll leave things as they
# are for now.
# It might be better to check whether the socket has been
# closed, but I don't see a way to do that. :(
# Drop flow-info from IPv6 addresses
if addr: # Sometimes None on Mac. See above.
addr = addr[:2]
try:
logger.debug("new connection %s" % (addr,))
def run():
loop = new_event_loop()
zs = self.storage_server.create_client_handler()
protocol = ServerProtocol(loop, self.addr, zs)
protocol.stop = loop.stop
if self.ssl_context is None:
cr = loop.create_connection((lambda : protocol), sock=sock)
else:
if hasattr(loop, 'connect_accepted_socket'):
cr = loop.connect_accepted_socket(
(lambda : protocol), sock, ssl=self.ssl_context)
else:
#######################################################
# XXX See http://bugs.python.org/issue27392 :(
_make_ssl_transport = loop._make_ssl_transport
def make_ssl_transport(*a, **kw):
kw['server_side'] = True
return _make_ssl_transport(*a, **kw)
loop._make_ssl_transport = make_ssl_transport
#
#######################################################
cr = loop.create_connection(
(lambda : protocol), sock=sock,
ssl=self.ssl_context,
server_hostname=''
)
asyncio.async(cr, loop=loop)
loop.run_forever()
loop.close()
thread = threading.Thread(target=run, name='zeo_client_hander')
thread.setDaemon(True)
thread.start()
except Exception:
if sock.fileno() in self.__socket_map:
del self.__socket_map[sock.fileno()]
logger.exception("Error in handle_accept")
else:
logger.info("connect from %s", repr(addr))
def loop(self, timeout=30.0):
try:
asyncore.loop(map=self.__socket_map, timeout=timeout)
except Exception:
if not self.__closed:
raise # Unexpected exc
logger.debug('acceptor %s loop stopped', self.addr)
__closed = False
def close(self):
if not self.__closed:
self.__closed = True
asyncore.dispatcher.close(self)
logger.debug("Closed accepter, %s", len(self.__socket_map))
src/ZEO/asyncio/server.py 0 → 100644
View file @ c499ea7d
import json
import logging
import os
import random
import threading
import ZODB.POSException
logger = logging.getLogger(__name__)
from ..shortrepr import short_repr
from . import base
from .compat import asyncio, new_event_loop
from .marshal import server_decode
class ServerProtocol(base.Protocol):
"""asyncio low-level ZEO server interface
"""
protocols = (b'Z5', )
name = 'server protocol'
methods = set(('register', ))
unlogged_exception_types = (
ZODB.POSException.POSKeyError,
)
def __init__(self, loop, addr, zeo_storage):
"""Create a server's client interface
"""
super(ServerProtocol, self).__init__(loop, addr)
self.zeo_storage = zeo_storage
closed = False
def close(self):
logger.debug("Closing server protocol")
if not self.closed:
self.closed = True
if self.transport is not None:
self.transport.close()
connected = None # for tests
def connection_made(self, transport):
self.connected = True
super(ServerProtocol, self).connection_made(transport)
self._write(best_protocol_version)
def connection_lost(self, exc):
self.connected = False
if exc:
logger.error("Disconnected %s:%s", exc.__class__.__name__, exc)
self.zeo_storage.notify_disconnected()
self.stop()
def stop(self):
pass # Might be replaced when running a thread per client
def finish_connect(self, protocol_version):
if protocol_version == b'ruok':
self._write(json.dumps(self.zeo_storage.ruok()).encode("ascii"))
self.close()
else:
if protocol_version in self.protocols:
logger.info("received handshake %r" %
str(protocol_version.decode('ascii')))
self.protocol_version = protocol_version
self.zeo_storage.notify_connected(self)
else:
logger.error("bad handshake %s" % short_repr(protocol_version))
self.close()
def call_soon_threadsafe(self, func, *args):
try:
self.loop.call_soon_threadsafe(func, *args)
except RuntimeError:
if self.connected:
logger.exception("call_soon_threadsafe failed while connected")
def message_received(self, message):
try:
message_id, async, name, args = server_decode(message)
except Exception:
logger.exception("Can't deserialize message")
self.close()
if message_id == -1:
return # keep-alive
if name not in self.methods:
logger.error('Invalid method, %r', name)
self.close()
try:
result = getattr(self.zeo_storage, name)(*args)
except Exception as exc:
if not isinstance(exc, self.unlogged_exception_types):
logger.exception(
"Bad %srequest, %r", 'async ' if async else '', name)
if async:
return self.close() # No way to recover/cry for help
else:
return self.send_error(message_id, exc)
if not async:
self.send_reply(message_id, result)
def send_reply(self, message_id, result, send_error=False, flag=0):
try:
result = self.encode(message_id, flag, '.reply', result)
except Exception:
if isinstance(result, Delay):
result.set_sender(message_id, self)
return
else:
logger.exception("Unpicklable response %r", result)
if not send_error:
self.send_error(
message_id,
ValueError("Couldn't pickle response"),
True)
self._write(result)
def send_reply_threadsafe(self, message_id, result):
self.loop.call_soon_threadsafe(self.reply, message_id, result)
def send_error(self, message_id, exc, send_error=False):
"""Abstracting here so we can make this cleaner in the future
"""
class_ = exc.__class__
class_ = "%s.%s" % (class_.__module__, class_.__name__)
args = class_, exc.__dict__ or exc.args
self.send_reply(message_id, args, send_error, 2)
def async(self, method, *args):
self.call_async(method, args)
def async_threadsafe(self, method, *args):
self.call_soon_threadsafe(self.call_async, method, args)
best_protocol_version = os.environ.get(
'ZEO_SERVER_PROTOCOL',
ServerProtocol.protocols[-1].decode('utf-8')).encode('utf-8')
assert best_protocol_version in ServerProtocol.protocols
def new_connection(loop, addr, socket, zeo_storage):
protocol = ServerProtocol(loop, addr, zeo_storage)
cr = loop.create_connection((lambda : protocol), sock=socket)
asyncio.async(cr, loop=loop)
class Delay(object):
"""Used to delay response to client for synchronous calls.
When a synchronous call is made and the original handler returns
without handling the call, it returns a Delay object that prevents
the mainloop from sending a response.
"""
msgid = protocol = sent = None
def set_sender(self, msgid, protocol):
self.msgid = msgid
self.protocol = protocol
def reply(self, obj):
self.sent = 'reply'
if self.protocol:
self.protocol.send_reply(self.msgid, obj)
def error(self, exc_info):
self.sent = 'error'
logger.error("Error raised in delayed method", exc_info=exc_info)
if self.protocol:
self.protocol.send_error(self.msgid, exc_info[1])
def __repr__(self):
return "%s[%s, %r, %r, %r]" % (
self.__class__.__name__, id(self),
self.msgid, self.protocol, self.sent)
def __reduce__(self):
raise TypeError("Can't pickle delays.")
class Result(Delay):
def __init__(self, *args):
self.args = args
def set_sender(self, msgid, protocol):
reply, callback = self.args
protocol.send_reply(msgid, reply)
callback()
class MTDelay(Delay):
def __init__(self):
self.ready = threading.Event()
def set_sender(self, *args):
Delay.set_sender(self, *args)
self.ready.set()
def reply(self, obj):
self.ready.wait()
self.protocol.call_soon_threadsafe(
self.protocol.send_reply, self.msgid, obj)
def error(self, exc_info):
self.ready.wait()
self.protocol.call_soon_threadsafe(Delay.error, self, exc_info)
class Acceptor(object):
def __init__(self, storage_server, addr, ssl):
self.storage_server = storage_server
self.addr = addr
self.ssl_context = ssl
self.event_loop = loop = new_event_loop()
if isinstance(addr, tuple):
cr = loop.create_server(self.factory, addr[0], addr[1],
reuse_address=True, ssl=ssl)
else:
cr = loop.create_unix_server(self.factory, addr, ssl=ssl)
f = asyncio.async(cr, loop=loop)
server = loop.run_until_complete(f)
self.server = server
if isinstance(addr, tuple) and addr[1] == 0:
addrs = [s.getsockname() for s in server.sockets]
addrs = [a for a in addrs if len(a) == len(addr)]
if addrs:
self.addr = addrs[0]
else:
self.addr = server.sockets[0].getsockname()[:len(addr)]
logger.info("listening on %s", str(addr))
def factory(self):
try:
logger.debug("Accepted connection")
zs = self.storage_server.create_client_handler()
protocol = ServerProtocol(self.event_loop, self.addr, zs)
except Exception:
logger.exception("Failure in protocol factory")
return protocol
def loop(self, timeout=None):
self.event_loop.run_forever()
self.event_loop.close()
closed = False
def close(self):
if not self.closed:
self.closed = True
self.event_loop.call_soon_threadsafe(self._close)
def _close(self):
loop = self.event_loop
self.server.close()
f = asyncio.async(self.server.wait_closed(), loop=loop)
@f.add_done_callback
def server_closed(f):
# stop the loop when the server closes:
loop.call_soon(loop.stop)
def timeout():
logger.warning("Timed out closing asyncio.Server")
loop.call_soon(loop.stop)
# But if the server doesn't close in a second, stop the loop anyway.
loop.call_later(1, timeout)
src/ZEO/asyncio/testing.py 0 → 100644
View file @ c499ea7d
from .._compat import PY3
if PY3:
import asyncio
else:
import trollius as asyncio
try:
ConnectionRefusedError
except NameError:
class ConnectionRefusedError(OSError):
pass
import pprint
class Loop(object):
protocol = transport = None
def __init__(self, addrs=(), debug=True):
self.addrs = addrs
self.get_debug = lambda : debug
self.connecting = {}
self.later = []
self.exceptions = []
def call_soon(self, func, *args):
func(*args)
def _connect(self, future, protocol_factory):
self.protocol = protocol = protocol_factory()
self.transport = transport = Transport(protocol)
protocol.connection_made(transport)
future.set_result((transport, protocol))
def connect_connecting(self, addr):
future, protocol_factory = self.connecting.pop(addr)
self._connect(future, protocol_factory)
def fail_connecting(self, addr):
future, protocol_factory = self.connecting.pop(addr)
if not future.cancelled():
future.set_exception(ConnectionRefusedError())
def create_connection(
self, protocol_factory, host=None, port=None, sock=None,
ssl=None, server_hostname=None
):
future = asyncio.Future(loop=self)
if sock is None:
addr = host, port
if addr in self.addrs:
self._connect(future, protocol_factory)
else:
self.connecting[addr] = future, protocol_factory
else:
self._connect(future, protocol_factory)
return future
def create_unix_connection(self, protocol_factory, path):
future = asyncio.Future(loop=self)
if path in self.addrs:
self._connect(future, protocol_factory)
else:
self.connecting[path] = future, protocol_factory
return future
def call_soon_threadsafe(self, func, *args):
func(*args)
return Handle()
def call_later(self, delay, func, *args):
handle = Handle()
self.later.append((delay, func, args, handle))
return handle
def call_exception_handler(self, context):
self.exceptions.append(context)
closed = False
def close(self):
self.closed = True
stopped = False
def stop(self):
self.stopped = True
class Handle(object):
cancelled = False
def cancel(self):
self.cancelled = True
class Transport(object):
capacity = 1 << 64
paused = False
extra = dict(peername='1.2.3.4', sockname=('127.0.0.1', 4200), socket=None)
def __init__(self, protocol):
self.data = []
self.protocol = protocol
def write(self, data):
self.data.append(data)
self.check_pause()
def writelines(self, lines):
self.data.extend(lines)
self.check_pause()
def check_pause(self):
if len(self.data) > self.capacity and not self.paused:
self.paused = True
self.protocol.pause_writing()
def pop(self, count=None):
if count:
r = self.data[:count]
del self.data[:count]
else:
r = self.data[:]
del self.data[:]
self.check_resume()
return r
def check_resume(self):
if len(self.data) < self.capacity and self.paused:
self.paused = False
self.protocol.resume_writing()
closed = False
def close(self):
self.closed = True
def get_extra_info(self, name):
return self.extra[name]
class AsyncRPC(object):
"""Adapt an asyncio API to an RPC to help hysterical tests
"""
def __init__(self, api):
self.api = api
def __getattr__(self, name):
return lambda *a, **kw: self.api.call(name, *a, **kw)
class ClientRunner(object):
def __init__(self, addr, client, cache, storage, read_only, timeout,
**kw):
self.addr = addr
self.client = client
self.cache = cache
self.storage = storage
self.read_only = read_only
self.timeout = timeout,
for name in kw:
self.__dict__[name] = kw[name]
def start(self, wait=True):
pass
def call(self, method, *args, **kw):
return getattr(self, method)(*args)
async = async_iter = call
def wait(self, timeout=None):
pass
src/ZEO/asyncio/tests.py 0 → 100644
View file @ c499ea7d
from .._compat import PY3
if PY3:
import asyncio
else:
import trollius as asyncio
from zope.testing import setupstack
from concurrent.futures import Future
import mock
from ZODB.POSException import ReadOnlyError
from ZODB.utils import maxtid
import collections
import logging
import struct
import unittest
from ..Exceptions import ClientDisconnected, ProtocolError
from .testing import Loop
from .client import ClientRunner, Fallback
from .server import new_connection, best_protocol_version
from .marshal import encoder, decode
class Base(object):
def setUp(self):
super(Base, self).setUp()
self.encode = encoder()
def unsized(self, data, unpickle=False):
result = []
while data:
size, message = data[:2]
data = data[2:]
self.assertEqual(struct.unpack(">I", size)[0], len(message))
if unpickle:
message = decode(message)
result.append(message)
if len(result) == 1:
result = result[0]
return result
def parse(self, data):
return self.unsized(data, True)
target = None
def send(self, method, *args, **kw):
target = kw.pop('target', self.target)
called = kw.pop('called', True)
no_output = kw.pop('no_output', True)
self.assertFalse(kw)
self.loop.protocol.data_received(
sized(self.encode(0, True, method, args)))
if target is not None:
target = getattr(target, method)
if called:
target.assert_called_with(*args)
target.reset_mock()
else:
self.assertFalse(target.called)
if no_output:
self.assertFalse(self.loop.transport.pop())
def pop(self, count=None, parse=True):
return self.unsized(self.loop.transport.pop(count), parse)
class ClientTests(Base, setupstack.TestCase, ClientRunner):
maxDiff = None
def tearDown(self):
self.client.close()
super(ClientTests, self)
def start(self,
addrs=(('127.0.0.1', 8200), ), loop_addrs=None,
read_only=False,
finish_start=False,
):
# To create a client, we need to specify an address, a client
# object and a cache.
wrapper = mock.Mock()
self.target = wrapper
cache = MemoryCache()
self.set_options(addrs, wrapper, cache, 'TEST', read_only, timeout=1)
# We can also provide an event loop. We'll use a testing loop
# so we don't have to actually make any network connection.
loop = Loop(addrs if loop_addrs is None else loop_addrs)
self.setup_delegation(loop)
self.assertFalse(wrapper.notify_disconnected.called)
protocol = loop.protocol
transport = loop.transport
if finish_start:
protocol.data_received(sized(b'Z3101'))
self.assertEqual(self.pop(2, False), b'Z3101')
self.respond(1, None)
self.respond(2, 'a'*8)
self.pop(4)
self.assertEqual(self.pop(), (3, False, 'get_info', ()))
self.respond(3, dict(length=42))
return (wrapper, cache, self.loop, self.client, protocol, transport)
def respond(self, message_id, result):
self.loop.protocol.data_received(
sized(self.encode(message_id, False, '.reply', result)))
def wait_for_result(self, future, timeout):
return future
def testClientBasics(self):
# Here, we'll go through the basic usage of the asyncio ZEO
# network client. The client is responsible for the core
# functionality of a ZEO client storage. The client storage
# is largely just a wrapper around the asyncio client.
wrapper, cache, loop, client, protocol, transport = self.start()
self.assertFalse(wrapper.notify_disconnected.called)
# The client isn't connected until the server sends it some data.
self.assertFalse(client.connected.done() or transport.data)
# The server sends the client it's protocol. In this case,
# it's a very high one. The client will send it's highest that
# it can use.
protocol.data_received(sized(b'Z99999'))
# The client sends back a handshake, and registers the
# storage, and requests the last transaction.
self.assertEqual(self.pop(2, False), b'Z5')
self.assertEqual(self.pop(), (1, False, 'register', ('TEST', False)))
# The client isn't connected until it initializes it's cache:
self.assertFalse(client.connected.done() or transport.data)
# If we try to make calls while the client is *initially*
# connecting, we get an error. This is because some dufus
# decided to create a client storage without waiting for it to
# connect.
f1 = self.call('foo', 1, 2)
self.assertTrue(isinstance(f1.exception(), ClientDisconnected))
# When the client is reconnecting, it's ready flag is set to False and
# it queues calls:
client.ready = False
f1 = self.call('foo', 1, 2)
self.assertFalse(f1.done())
# If we try to make an async call, we get an immediate error:
f2 = self.async('bar', 3, 4)
self.assert_(isinstance(f2.exception(), ClientDisconnected))
# The wrapper object (ClientStorage) hasn't been notified:
self.assertFalse(wrapper.notify_connected.called)
# Let's respond to the register call:
self.respond(1, None)
# The client requests the last transaction:
self.assertEqual(self.pop(), (2, False, 'lastTransaction', ()))
# We respond
self.respond(2, 'a'*8)
# After verification, the client requests info:
self.assertEqual(self.pop(), (3, False, 'get_info', ()))
self.respond(3, dict(length=42))
# Now we're connected, the cache was initialized, and the
# queued message has been sent:
self.assert_(client.connected.done())
self.assertEqual(cache.getLastTid(), 'a'*8)
self.assertEqual(self.pop(), (4, False, 'foo', (1, 2)))
# The wrapper object (ClientStorage) has been notified:
wrapper.notify_connected.assert_called_with(client, {'length': 42})
self.respond(4, 42)
self.assertEqual(f1.result(), 42)
# Now we can make async calls:
f2 = self.async('bar', 3, 4)
self.assert_(f2.done() and f2.exception() is None)
self.assertEqual(self.pop(), (0, True, 'bar', (3, 4)))
# Loading objects gets special handling to leverage the cache.
loaded = self.load_before(b'1'*8, maxtid)
# The data wasn't in the cache, so we made a server call:
self.assertEqual(
self.pop(),
((b'1'*8, maxtid), False, 'loadBefore', (b'1'*8, maxtid)))
# Note load_before uses the oid as the message id.
self.respond((b'1'*8, maxtid), (b'data', b'a'*8, None))
self.assertEqual(loaded.result(), (b'data', b'a'*8, None))
# If we make another request, it will be satisfied from the cache:
loaded = self.load_before(b'1'*8, maxtid)
self.assertEqual(loaded.result(), (b'data', b'a'*8, None))
self.assertFalse(transport.data)
# Let's send an invalidation:
self.send('invalidateTransaction', b'b'*8, [b'1'*8])
# Now, if we try to load current again, we'll make a server request.
loaded = self.load_before(b'1'*8, maxtid)
# Note that if we make another request for the same object,
# the requests will be collapsed:
loaded2 = self.load_before(b'1'*8, maxtid)
self.assertEqual(
self.pop(),
((b'1'*8, maxtid), False, 'loadBefore', (b'1'*8, maxtid)))
self.respond((b'1'*8, maxtid), (b'data2', b'b'*8, None))
self.assertEqual(loaded.result(), (b'data2', b'b'*8, None))
self.assertEqual(loaded2.result(), (b'data2', b'b'*8, None))
# Loading non-current data may also be satisfied from cache
loaded = self.load_before(b'1'*8, b'b'*8)
self.assertEqual(loaded.result(), (b'data', b'a'*8, b'b'*8))
self.assertFalse(transport.data)
loaded = self.load_before(b'1'*8, b'c'*8)
self.assertEqual(loaded.result(), (b'data2', b'b'*8, None))
self.assertFalse(transport.data)
loaded = self.load_before(b'1'*8, b'_'*8)
self.assertEqual(
self.pop(),
((b'1'*8, b'_'*8), False, 'loadBefore', (b'1'*8, b'_'*8)))
self.respond((b'1'*8, b'_'*8), (b'data0', b'^'*8, b'_'*8))
self.assertEqual(loaded.result(), (b'data0', b'^'*8, b'_'*8))
# When committing transactions, we need to update the cache
# with committed data. To do this, we pass a (oid, data, resolved)
# iteratable to tpc_finish_threadsafe.
tids = []
def finished_cb(tid):
tids.append(tid)
committed = self.tpc_finish(
b'd'*8,
[(b'2'*8, 'committed 2', False),
(b'1'*8, 'committed 3', True),
(b'4'*8, 'committed 4', False),
],
finished_cb)
self.assertFalse(committed.done() or
cache.load(b'2'*8) or
cache.load(b'4'*8))
self.assertEqual(cache.load(b'1'*8), (b'data2', b'b'*8))
self.assertEqual(self.pop(),
(5, False, 'tpc_finish', (b'd'*8,)))
self.respond(5, b'e'*8)
self.assertEqual(committed.result(), b'e'*8)
self.assertEqual(cache.load(b'1'*8), None)
self.assertEqual(cache.load(b'2'*8), ('committed 2', b'e'*8))
self.assertEqual(cache.load(b'4'*8), ('committed 4', b'e'*8))
self.assertEqual(tids.pop(), b'e'*8)
# If the protocol is disconnected, it will reconnect and will
# resolve outstanding requests with exceptions:
loaded = self.load_before(b'1'*8, maxtid)
f1 = self.call('foo', 1, 2)
self.assertFalse(loaded.done() or f1.done())
self.assertEqual(
self.pop(),
[((b'1'*8, maxtid), False, 'loadBefore', (b'1'*8, maxtid)),
(6, False, 'foo', (1, 2))],
)
exc = TypeError(43)
self.assertFalse(wrapper.notify_disconnected.called)
wrapper.notify_connected.reset_mock()
protocol.connection_lost(exc)
wrapper.notify_disconnected.assert_called_with()
self.assertTrue(isinstance(loaded.exception(), ClientDisconnected))
self.assertEqual(loaded.exception().args, (exc,))
self.assertTrue(isinstance(f1.exception(), ClientDisconnected))
self.assertEqual(f1.exception().args, (exc,))
# Because we reconnected, a new protocol and transport were created:
self.assert_(protocol is not loop.protocol)
self.assert_(transport is not loop.transport)
protocol = loop.protocol
transport = loop.transport
# and we have a new incomplete connect future:
self.assertFalse(client.connected.done() or transport.data)
# This time we'll send a lower protocol version. The client
# will send it back, because it's lower than the client's
# protocol:
protocol.data_received(sized(b'Z310'))
self.assertEqual(self.unsized(transport.pop(2)), b'Z310')
self.assertEqual(self.pop(), (1, False, 'register', ('TEST', False)))
self.assertFalse(wrapper.notify_connected.called)
# If the register response is a tid, then the client won't
# request lastTransaction
self.respond(1, b'e'*8)
self.assertEqual(self.pop(), (2, False, 'get_info', ()))
self.respond(2, dict(length=42))
# Because the server tid matches the cache tid, we're done connecting
wrapper.notify_connected.assert_called_with(client, {'length': 42})
self.assert_(client.connected.done() and not transport.data)
self.assertEqual(cache.getLastTid(), b'e'*8)
# Because we were able to update the cache, we didn't have to
# invalidate the database cache:
self.assertFalse(wrapper.invalidateTransaction.called)
# The close method closes the connection and cache:
client.close()
self.assert_(transport.closed and cache.closed)
# The client doesn't reconnect
self.assertEqual(loop.protocol, protocol)
self.assertEqual(loop.transport, transport)
def test_cache_behind(self):
wrapper, cache, loop, client, protocol, transport = self.start()
cache.setLastTid(b'a'*8)
cache.store(b'4'*8, b'a'*8, None, '4 data')
cache.store(b'2'*8, b'a'*8, None, '2 data')
self.assertFalse(client.connected.done() or transport.data)
protocol.data_received(sized(b'Z3101'))
self.assertEqual(self.unsized(transport.pop(2)), b'Z3101')
self.respond(1, None)
self.respond(2, b'e'*8)
self.pop(4)
# We have to verify the cache, so we're not done connecting:
self.assertFalse(client.connected.done())
self.assertEqual(self.pop(), (3, False, 'getInvalidations', (b'a'*8, )))
self.respond(3, (b'e'*8, [b'4'*8]))
self.assertEqual(self.pop(), (4, False, 'get_info', ()))
self.respond(4, dict(length=42))
# Now that verification is done, we're done connecting
self.assert_(client.connected.done() and not transport.data)
self.assertEqual(cache.getLastTid(), b'e'*8)
# And the cache has been updated:
self.assertEqual(cache.load(b'2'*8),
('2 data', b'a'*8)) # unchanged
self.assertEqual(cache.load(b'4'*8), None)
# Because we were able to update the cache, we didn't have to
# invalidate the database cache:
self.assertFalse(wrapper.invalidateCache.called)
def test_cache_way_behind(self):
wrapper, cache, loop, client, protocol, transport = self.start()
cache.setLastTid(b'a'*8)
cache.store(b'4'*8, b'a'*8, None, '4 data')
self.assertTrue(cache)
self.assertFalse(client.connected.done() or transport.data)
protocol.data_received(sized(b'Z3101'))
self.assertEqual(self.unsized(transport.pop(2)), b'Z3101')
self.respond(1, None)
self.respond(2, b'e'*8)
self.pop(4)
# We have to verify the cache, so we're not done connecting:
self.assertFalse(client.connected.done())
self.assertEqual(self.pop(), (3, False, 'getInvalidations', (b'a'*8, )))
# We respond None, indicating that we're too far out of date:
self.respond(3, None)
self.assertEqual(self.pop(), (4, False, 'get_info', ()))
self.respond(4, dict(length=42))
# Now that verification is done, we're done connecting
self.assert_(client.connected.done() and not transport.data)
self.assertEqual(cache.getLastTid(), b'e'*8)
# But the cache is now empty and we invalidated the database cache
self.assertFalse(cache)
wrapper.invalidateCache.assert_called_with()
def test_multiple_addresses(self):
# We can pass multiple addresses to client constructor
addrs = [('1.2.3.4', 8200), ('2.2.3.4', 8200)]
wrapper, cache, loop, client, protocol, transport = self.start(
addrs, ())
# We haven't connected yet
self.assert_(protocol is None and transport is None)
# There are 2 connection attempts outstanding:
self.assertEqual(sorted(loop.connecting), addrs)
# We cause the first one to fail:
loop.fail_connecting(addrs[0])
self.assertEqual(sorted(loop.connecting), addrs[1:])
# The failed connection is attempted in the future:
delay, func, args, _ = loop.later.pop(0)
self.assert_(1 <= delay <= 2)
func(*args)
self.assertEqual(sorted(loop.connecting), addrs)
# Let's connect the second address
loop.connect_connecting(addrs[1])
self.assertEqual(sorted(loop.connecting), addrs[:1])
protocol = loop.protocol
transport = loop.transport
protocol.data_received(sized(b'Z3101'))
self.assertEqual(self.unsized(transport.pop(2)), b'Z3101')
self.respond(1, None)
# Now, when the first connection fails, it won't be retried,
# because we're already connected.
# (first in later is heartbeat)
self.assertEqual(sorted(loop.later[1:]), [])
loop.fail_connecting(addrs[0])
self.assertEqual(sorted(loop.connecting), [])
self.assertEqual(sorted(loop.later[1:]), [])
def test_bad_server_tid(self):
# If in verification we get a server_tid behing the cache's, make sure
# we retry the connection later.
wrapper, cache, loop, client, protocol, transport = self.start()
cache.store(b'4'*8, b'a'*8, None, '4 data')
cache.setLastTid('b'*8)
protocol.data_received(sized(b'Z3101'))
self.assertEqual(self.unsized(transport.pop(2)), b'Z3101')
self.respond(1, None)
self.respond(2, 'a'*8)
self.pop()
self.assertFalse(client.connected.done() or transport.data)
delay, func, args, _ = loop.later.pop(1) # first in later is heartbeat
self.assert_(8 < delay < 10)
self.assertEqual(len(loop.later), 1) # first in later is heartbeat
func(*args) # connect again
self.assertFalse(protocol is loop.protocol)
self.assertFalse(transport is loop.transport)
protocol = loop.protocol
transport = loop.transport
protocol.data_received(sized(b'Z3101'))
self.assertEqual(self.unsized(transport.pop(2)), b'Z3101')
self.respond(1, None)
self.respond(2, 'b'*8)
self.pop(4)
self.assertEqual(self.pop(), (3, False, 'get_info', ()))
self.respond(3, dict(length=42))
self.assert_(client.connected.done() and not transport.data)
self.assert_(client.ready)
def test_readonly_fallback(self):
addrs = [('1.2.3.4', 8200), ('2.2.3.4', 8200)]
wrapper, cache, loop, client, protocol, transport = self.start(
addrs, (), read_only=Fallback)
self.assertTrue(self.is_read_only())
# We'll treat the first address as read-only and we'll let it connect:
loop.connect_connecting(addrs[0])
protocol, transport = loop.protocol, loop.transport
protocol.data_received(sized(b'Z3101'))
self.assertEqual(self.unsized(transport.pop(2)), b'Z3101')
# We see that the client tried a writable connection:
self.assertEqual(self.pop(),
(1, False, 'register', ('TEST', False)))
# We respond with a read-only exception:
self.respond(1, (ReadOnlyError, ReadOnlyError()))
self.assertTrue(self.is_read_only())
# The client tries for a read-only connection:
self.assertEqual(self.pop(), (2, False, 'register', ('TEST', True)))
# We respond with successfully:
self.respond(2, None)
self.pop(2)
self.respond(3, 'b'*8)
self.assertTrue(self.is_read_only())
# At this point, the client is ready and using the protocol,
# and the protocol is read-only:
self.assert_(client.ready)
self.assertEqual(client.protocol, protocol)
self.assertEqual(protocol.read_only, True)
connected = client.connected
# The client asks for info, and we respond:
self.assertEqual(self.pop(), (4, False, 'get_info', ()))
self.respond(4, dict(length=42))
self.assert_(connected.done())
# We connect the second address:
loop.connect_connecting(addrs[1])
loop.protocol.data_received(sized(b'Z3101'))
self.assertEqual(self.unsized(loop.transport.pop(2)), b'Z3101')
self.assertEqual(self.parse(loop.transport.pop()),
(1, False, 'register', ('TEST', False)))
self.assertTrue(self.is_read_only())
# We respond and the writable connection succeeds:
self.respond(1, None)
# at this point, a lastTransaction request is emitted:
self.assertEqual(self.parse(loop.transport.pop()),
(2, False, 'lastTransaction', ()))
self.assertFalse(self.is_read_only())
# Now, the original protocol is closed, and the client is
# no-longer ready:
self.assertFalse(client.ready)
self.assertFalse(client.protocol is protocol)
self.assertEqual(client.protocol, loop.protocol)
self.assertEqual(protocol.closed, True)
self.assert_(client.connected is not connected)
self.assertFalse(client.connected.done())
protocol, transport = loop.protocol, loop.transport
self.assertEqual(protocol.read_only, False)
# Now, we finish verification
self.respond(2, 'b'*8)
self.respond(3, dict(length=42))
self.assert_(client.ready)
self.assert_(client.connected.done())
def test_invalidations_while_verifying(self):
# While we're verifying, invalidations are ignored
wrapper, cache, loop, client, protocol, transport = self.start()
protocol.data_received(sized(b'Z3101'))
self.assertEqual(self.unsized(transport.pop(2)), b'Z3101')
self.respond(1, None)
self.pop(4)
self.send('invalidateTransaction', b'b'*8, [b'1'*8], called=False)
self.respond(2, b'a'*8)
self.send('invalidateTransaction', b'c'*8, [b'1'*8], no_output=False)
self.assertEqual(self.pop(), (3, False, 'get_info', ()))
# We'll disconnect:
protocol.connection_lost(Exception("lost"))
self.assert_(protocol is not loop.protocol)
self.assert_(transport is not loop.transport)
protocol = loop.protocol
transport = loop.transport
# Similarly, invalidations aren't processed while reconnecting:
protocol.data_received(sized(b'Z3101'))
self.assertEqual(self.unsized(transport.pop(2)), b'Z3101')
self.respond(1, None)
self.pop(4)
self.send('invalidateTransaction', b'd'*8, [b'1'*8], called=False)
self.respond(2, b'c'*8)
self.send('invalidateTransaction', b'e'*8, [b'1'*8], no_output=False)
self.assertEqual(self.pop(), (3, False, 'get_info', ()))
def test_flow_control(self):
# When sending a lot of data (blobs), we don't want to fill up
# memory behind a slow socket. Asycio's flow control helper
# seems a bit complicated. We'd rather pass an iterator that's
# consumed as we can.
wrapper, cache, loop, client, protocol, transport = self.start(
finish_start=True)
# Give the transport a small capacity:
transport.capacity = 2
self.async('foo')
self.async('bar')
self.async('baz')
self.async('splat')
# The first 2 were sent, but the remaining were queued.
self.assertEqual(self.pop(),
[(0, True, 'foo', ()), (0, True, 'bar', ())])
# But popping them allowed sending to resume:
self.assertEqual(self.pop(),
[(0, True, 'baz', ()), (0, True, 'splat', ())])
# This is especially handy with iterators:
self.async_iter((name, ()) for name in 'abcde')
self.assertEqual(self.pop(), [(0, True, 'a', ()), (0, True, 'b', ())])
self.assertEqual(self.pop(), [(0, True, 'c', ()), (0, True, 'd', ())])
self.assertEqual(self.pop(), (0, True, 'e', ()))
self.assertEqual(self.pop(), [])
def test_bad_protocol(self):
wrapper, cache, loop, client, protocol, transport = self.start()
with mock.patch("ZEO.asyncio.client.logger.error") as error:
self.assertFalse(error.called)
protocol.data_received(sized(b'Z200'))
self.assert_(isinstance(error.call_args[0][1], ProtocolError))
def test_get_peername(self):
wrapper, cache, loop, client, protocol, transport = self.start(
finish_start=True)
self.assertEqual(client.get_peername(), '1.2.3.4')
def test_call_async_from_same_thread(self):
# There are a few (1?) cases where we call into client storage
# where it needs to call back asyncronously. Because we're
# calling from the same thread, we don't need to use a futurte.
wrapper, cache, loop, client, protocol, transport = self.start(
finish_start=True)
client.call_async_from_same_thread('foo', 1)
self.assertEqual(self.pop(), (0, True, 'foo', (1, )))
def test_ClientDisconnected_on_call_timeout(self):
wrapper, cache, loop, client, protocol, transport = self.start()
self.wait_for_result = super(ClientTests, self).wait_for_result
self.assertRaises(ClientDisconnected, self.call, 'foo')
client.ready = False
self.assertRaises(ClientDisconnected, self.call, 'foo')
def test_errors_in_data_received(self):
# There was a bug in ZEO.async.client.Protocol.data_recieved
# that caused it to fail badly if errors were raised while
# handling data.
wrapper, cache, loop, client, protocol, transport =self.start(
finish_start=True)
wrapper.receiveBlobStart.side_effect = ValueError('test')
chunk = 'x' * 99999
try:
loop.protocol.data_received(
sized(
self.encode(0, True, 'receiveBlobStart', ('oid', 'serial'))
) +
sized(
self.encode(
0, True, 'receiveBlobChunk', ('oid', 'serial', chunk))
)
)
except ValueError:
pass
loop.protocol.data_received(sized(
self.encode(0, True, 'receiveBlobStop', ('oid', 'serial'))
))
wrapper.receiveBlobChunk.assert_called_with('oid', 'serial', chunk)
wrapper.receiveBlobStop.assert_called_with('oid', 'serial')
def test_heartbeat(self):
# Protocols run heartbeats on a configurable (sort of)
# heartbeat interval, which defaults to every 60 seconds.
wrapper, cache, loop, client, protocol, transport = self.start(
finish_start=True)
delay, func, args, handle = loop.later.pop()
self.assertEqual(
(delay, func, args, handle),
(60, protocol.heartbeat, (), protocol.heartbeat_handle),
)
self.assertFalse(loop.later or handle.cancelled)
# The heartbeat function sends heartbeat data and reschedules itself.
func()
self.assertEqual(self.pop(), (-1, 0, '.reply', None))
self.assertTrue(protocol.heartbeat_handle != handle)
delay, func, args, handle = loop.later.pop()
self.assertEqual(
(delay, func, args, handle),
(60, protocol.heartbeat, (), protocol.heartbeat_handle),
)
self.assertFalse(loop.later or handle.cancelled)
# The heartbeat is cancelled when the protocol connection is lost:
protocol.connection_lost(None)
self.assertTrue(handle.cancelled)
class MemoryCache(object):
def __init__(self):
# { oid -> [(start, end, data)] }
self.data = collections.defaultdict(list)
self.last_tid = None
clear = __init__
closed = False
def close(self):
self.closed = True
def __len__(self):
return len(self.data)
def load(self, oid):
revisions = self.data[oid]
if revisions:
start, end, data = revisions[-1]
if not end:
return data, start
return None
def store(self, oid, start_tid, end_tid, data):
assert start_tid is not None
revisions = self.data[oid]
data = (start_tid, end_tid, data)
if not revisions or data != revisions[-1]:
revisions.append(data)
revisions.sort()
def loadBefore(self, oid, tid):
for start, end, data in self.data[oid]:
if start < tid and (end is None or end >= tid):
return data, start, end
def invalidate(self, oid, tid):
revisions = self.data[oid]
if revisions:
if tid is None:
del revisions[:]
else:
start, end, data = revisions[-1]
if end is None:
revisions[-1] = start, tid, data
def getLastTid(self):
return self.last_tid
def setLastTid(self, tid):
self.last_tid = tid
class ServerTests(Base, setupstack.TestCase):
# The server side of things is pretty simple compared to the
# client, because it's the clien't job to make and keep
# connections. Servers are pretty passive.
def connect(self, finish=False):
protocol = server_protocol()
self.loop = protocol.loop
self.target = protocol.zeo_storage
if finish:
self.assertEqual(self.pop(parse=False), best_protocol_version)
protocol.data_received(sized(b'Z5'))
return protocol
message_id = 0
target = None
def call(self, meth, *args, **kw):
if kw:
expect = kw.pop('expect', self)
target = kw.pop('target', self.target)
self.assertFalse(kw)
if target is not None:
target = getattr(target, meth)
if expect is not self:
target.return_value = expect
self.message_id += 1
self.loop.protocol.data_received(
sized(self.encode(self.message_id, False, meth, args)))
if target is not None:
target.assert_called_once_with(*args)
target.reset_mock()
if expect is not self:
self.assertEqual(self.pop(),
(self.message_id, False, '.reply', expect))
def testServerBasics(self):
# A simple listening thread accepts connections. It creats
# asyncio connections by calling ZEO.asyncio.new_connection:
protocol = self.connect()
self.assertFalse(protocol.zeo_storage.notify_connected.called)
# The server sends it's protocol.
self.assertEqual(self.pop(parse=False), best_protocol_version)
# The client sends it's protocol:
protocol.data_received(sized(b'Z5'))
self.assertEqual(protocol.protocol_version, b'Z5')
protocol.zeo_storage.notify_connected.assert_called_once_with(protocol)
# The client registers:
self.call('register', False, expect=None)
# It does other things, like, send hearbeats:
protocol.data_received(sized(b'(J\xff\xff\xff\xffK\x00U\x06.replyNt.'))
# The client can make async calls:
self.send('register')
# Let's close the connection
self.assertFalse(protocol.zeo_storage.notify_disconnected.called)
protocol.connection_lost(None)
protocol.zeo_storage.notify_disconnected.assert_called_once_with()
def test_invalid_methods(self):
protocol = self.connect(True)
protocol.zeo_storage.notify_connected.assert_called_once_with(protocol)
# If we try to call a methid that isn't in the protocol's
# white list, it will disconnect:
self.assertFalse(protocol.loop.transport.closed)
self.call('foo', target=None)
self.assertTrue(protocol.loop.transport.closed)
def server_protocol(zeo_storage=None,
protocol_version=None,
addr=('1.2.3.4', '42'),
):
if zeo_storage is None:
zeo_storage = mock.Mock()
loop = Loop()
sock = () # anything not None
new_connection(loop, addr, sock, zeo_storage)
if protocol_version:
loop.protocol.data_received(sized(protocol_version))
return loop.protocol
def response(*data):
return sized(self.encode(*data))
def sized(message):
return struct.pack(">I", len(message)) + message
class Logging(object):
def __init__(self, level=logging.ERROR):
self.level = level
def __enter__(self):
self.handler = logging.StreamHandler()
logging.getLogger().addHandler(self.handler)
logging.getLogger().setLevel(self.level)
def __exit__(self, *args):
logging.getLogger().removeHandler(self.handler)
logging.getLogger().setLevel(logging.NOTSET)
def test_suite():
suite = unittest.TestSuite()
suite.addTest(unittest.makeSuite(ClientTests))
suite.addTest(unittest.makeSuite(ServerTests))
return suite
src/ZEO/cache.py
View file @ c499ea7d
......@@ -29,12 +29,11 @@ import BTrees.LOBTree
import logging
import os
import tempfile
import threading
import time
import ZODB.fsIndex
import zc.lockfile
from ZODB.utils import p64, u64, z64
from ZODB.utils import p64, u64, z64, RLock
import six
from ._compat import PYPY
......@@ -131,16 +130,6 @@ allocated_record_overhead = 43
# to the end of the file that the new object can't fit in one
# contiguous chunk, currentofs is reset to ZEC_HEADER_SIZE first.
def locked(func):
def _locked_wrapper(inst, *args, **kwargs):
inst._lock.acquire()
try:
return func(inst, *args, **kwargs)
finally:
inst._lock.release()
return _locked_wrapper
# Under PyPy, the available dict specializations perform significantly
# better (faster) than the pure-Python BTree implementation. They may
# use less memory too. And we don't require any of the special BTree features...
......@@ -193,6 +182,8 @@ class ClientCache(object):
# currentofs.
self.currentofs = ZEC_HEADER_SIZE
self._lock = RLock()
# self.f is the open file object.
# When we're not reusing an existing file, self.f is left None
# here -- the scan() method must be called then to open the file
......@@ -243,8 +234,6 @@ class ClientCache(object):
self._setup_trace(path)
self._lock = threading.RLock()
# Backward compatibility. Client code used to have to use the fc
# attr to get to the file cache to get cache stats.
@property
......@@ -252,6 +241,7 @@ class ClientCache(object):
return self
def clear(self):
with self._lock:
self.f.seek(ZEC_HEADER_SIZE)
self.f.truncate()
self._initfile(ZEC_HEADER_SIZE)
......@@ -463,8 +453,8 @@ class ClientCache(object):
# instance, and also written out near the start of the cache file. The
# new tid must be strictly greater than our current idea of the most
# recent tid.
@locked
def setLastTid(self, tid):
with self._lock:
if (not tid) or (tid == z64):
return
if (tid <= self.tid) and self._len:
......@@ -484,6 +474,7 @@ class ClientCache(object):
# @return a transaction id
# @defreturn string, or 8 nulls if no transaction is yet known
def getLastTid(self):
with self._lock:
return self.tid
##
......@@ -492,9 +483,8 @@ class ClientCache(object):
# @return (data record, serial number, tid), or None if the object is not
# in the cache
# @defreturn 3-tuple: (string, string, string)
@locked
def load(self, oid, before_tid=None):
with self._lock:
ofs = self.current.get(oid)
if ofs is None:
self._trace(0x20, oid)
......@@ -513,7 +503,8 @@ class ClientCache(object):
return None
data = read(ldata)
assert len(data) == ldata, (ofs, self.f.tell(), oid, len(data), ldata)
assert len(data) == ldata, (
ofs, self.f.tell(), oid, len(data), ldata)
# WARNING: The following assert changes the file position.
# We must not depend on this below or we'll fail in optimized mode.
......@@ -548,16 +539,15 @@ class ClientCache(object):
# @param tid id of transaction that wrote next revision of oid
# @return data record, serial number, start tid, and end tid
# @defreturn 4-tuple: (string, string, string, string)
@locked
def loadBefore(self, oid, before_tid):
with self._lock:
noncurrent_for_oid = self.noncurrent.get(u64(oid))
if noncurrent_for_oid is None:
result = self.load(oid, before_tid)
if result:
return result[0], result[1], None
else:
self._trace(0x24, oid, b"", before_tid)
self._trace(0x24, oid, "", before_tid)
return result
items = noncurrent_for_oid.items(None, u64(before_tid)-1)
......@@ -566,7 +556,7 @@ class ClientCache(object):
if result:
return result[0], result[1], None
else:
self._trace(0x24, oid, b"", before_tid)
self._trace(0x24, oid, "", before_tid)
return result
tid, ofs = items[-1]
......@@ -578,7 +568,8 @@ class ClientCache(object):
size, saved_oid, saved_tid, end_tid, lver, ldata = unpack(
">I8s8s8sHI", read(34))
assert saved_oid == oid, (ofs, self.f.tell(), oid, saved_oid)
assert saved_tid == p64(tid), (ofs, self.f.tell(), oid, saved_tid, tid)
assert saved_tid == p64(tid), (
ofs, self.f.tell(), oid, saved_tid, tid)
assert end_tid != z64, (ofs, self.f.tell(), oid)
assert lver == 0, "Versions aren't supported"
data = read(ldata)
......@@ -593,11 +584,11 @@ class ClientCache(object):
if result:
return result[0], result[1], None
else:
self._trace(0x24, oid, b"", before_tid)
self._trace(0x24, oid, "", before_tid)
return result
self._n_accesses += 1
self._trace(0x26, oid, b"", saved_tid)
self._trace(0x26, oid, "", saved_tid)
return data, saved_tid, end_tid
##
......@@ -608,9 +599,8 @@ class ClientCache(object):
# revision of oid. If end_tid is None, the data is
# current.
# @param data the actual data
@locked
def store(self, oid, start_tid, end_tid, data):
with self._lock:
seek = self.f.seek
if end_tid is None:
ofs = self.current.get(oid)
......@@ -621,14 +611,16 @@ class ClientCache(object):
assert status == b'a', (ofs, self.f.tell(), oid)
size, saved_oid, saved_tid, end_tid = unpack(
">I8s8s8s", read(28))
assert saved_oid == oid, (ofs, self.f.tell(), oid, saved_oid)
assert saved_oid == oid, (
ofs, self.f.tell(), oid, saved_oid)
assert end_tid == z64, (ofs, self.f.tell(), oid)
if saved_tid == start_tid:
return
raise ValueError("already have current data for oid")
else:
noncurrent_for_oid = self.noncurrent.get(u64(oid))
if noncurrent_for_oid and (u64(start_tid) in noncurrent_for_oid):
if noncurrent_for_oid and (
u64(start_tid) in noncurrent_for_oid):
return
size = allocated_record_overhead + len(data)
......@@ -715,8 +707,8 @@ class ClientCache(object):
# - oid object id
# - tid the id of the transaction that wrote a new revision of oid,
# or None to forget all cached info about oid.
@locked
def invalidate(self, oid, tid):
with self._lock:
ofs = self.current.get(oid)
if ofs is None:
# 0x10 == invalidate (miss)
......@@ -739,7 +731,8 @@ class ClientCache(object):
self._len -= 1
else:
if tid == saved_tid:
logger.warning("Ignoring invalidation with same tid as current")
logger.warning(
"Ignoring invalidation with same tid as current")
return
self.f.seek(ofs+21)
self.f.write(tid)
......@@ -756,19 +749,13 @@ class ClientCache(object):
seek = self.f.seek
read = self.f.read
for oid, ofs in six.iteritems(self.current):
self._lock.acquire()
try:
seek(ofs)
status = read(1)
assert status == b'a', (ofs, self.f.tell(), oid)
size, saved_oid, tid, end_tid = unpack(">I8s8s8s", read(28))
assert saved_oid == oid, (ofs, self.f.tell(), oid, saved_oid)
assert end_tid == z64, (ofs, self.f.tell(), oid)
result = oid, tid
finally:
self._lock.release()
yield result
yield oid, tid
def dump(self):
from ZODB.utils import oid_repr
......
src/ZEO/component.xml
View file @ c499ea7d
<component>
<sectiontype name="zeo">
<import package="ZODB"/>
<sectiontype name="ssl" datatype="ZEO.zconfig.client_ssl">
<key name="certificate" datatype="existing-dirpath" required="yes">
<description>
The content of a ZEO section describe operational parameters
of a ZEO server except for the storage(s) to be served.
The full path to an SSL certificate file.
</description>
</key>
<key name="address" datatype="socket-binding-address"
required="yes">
<key name="key" datatype="existing-dirpath" required="no">
<description>
The address at which the server should listen. This can be in
the form 'host:port' to signify a TCP/IP connection or a
pathname string to signify a Unix domain socket connection (at
least one '/' is required). A hostname may be a DNS name or a
dotted IP address. If the hostname is omitted, the platform's
default behavior is used when binding the listening socket (''
is passed to socket.bind() as the hostname portion of the
address).
The full path to an SSL key file for the client certificate.
</description>
</key>
<key name="read-only" datatype="boolean"
required="no"
default="false">
<key name="password-function" required="no">
<description>
Flag indicating whether the server should operate in read-only
mode. Defaults to false. Note that even if the server is
operating in writable mode, individual storages may still be
read-only. But if the server is in read-only mode, no write
operations are allowed, even if the storages are writable. Note
that pack() is considered a read-only operation.
Dotted name of importable function for retrieving a password
for the client certificate key.
</description>
</key>
<key name="invalidation-queue-size" datatype="integer"
required="no"
default="100">
<key name="authenticate" datatype="existing-dirpath" required="no">
<description>
The storage server keeps a queue of the objects modified by the
last N transactions, where N == invalidation_queue_size. This
queue is used to speed client cache verification when a client
disconnects for a short period of time.
Path to a file or directory containing server certificates to be
authenticated.
</description>
</key>
<key name="invalidation-age" datatype="float" required="no">
<key name="check-hostname" datatype="boolean" required="no" default="true">
<description>
The maximum age of a client for which quick-verification
invalidations will be provided by iterating over the served
storage. This option should only be used if the served storage
supports efficient iteration from a starting point near the
end of the transaction history (e.g. end of file).
Verify the host name in the server certificate is as expected.
</description>
</key>
<key name="monitor-address" datatype="socket-binding-address"
required="no">
<key name="server-hostname" required="no">
<description>
The address at which the monitor server should listen. If
specified, a monitor server is started. The monitor server
provides server statistics in a simple text format. This can
be in the form 'host:port' to signify a TCP/IP connection or a
pathname string to signify a Unix domain socket connection (at
least one '/' is required). A hostname may be a DNS name or a
dotted IP address. If the hostname is omitted, the platform's
default behavior is used when binding the listening socket (''
is passed to socket.bind() as the hostname portion of the
address).
Host name to use for SSL host name checks.
If ``check-hostname`` is true then use this as the
value to check. If an address is a host/port pair, then this
defaults to the host in the address.
</description>
</key>
<key name="transaction-timeout" datatype="integer"
required="no">
</sectiontype>
<sectiontype name="clientstorage" datatype="ZEO.zconfig.ClientStorageConfig"
implements="ZODB.storage">
<section type="ssl" name="*" attribute="ssl" />
<multikey name="server" datatype="socket-connection-address" required="yes"
/>
<key name="cache-size" datatype="byte-size" default="20MB">
<description>
The maximum amount of time to wait for a transaction to commit
after acquiring the storage lock, specified in seconds. If the
transaction takes too long, the client connection will be closed
and the transaction aborted.
The cache size in bytes, KB or MB. This defaults to a 20MB.
Optional ``KB`` or ``MB`` suffixes can (and usually are) used to
specify units other than bytes.
</description>
</key>
<key name="authentication-protocol" required="no">
<key name="cache-path" required="no">
<description>
The name of the protocol used for authentication. The
only protocol provided with ZEO is "digest," but extensions
may provide other protocols.
The file path of a persistent cache file
</description>
</key>
<key name="blob-dir" required="no">
<description>
Path name to the blob cache directory.
</description>
</key>
<key name="shared-blob-dir" required="no" default="no"
datatype="boolean">
<description>
Tells whether the cache is a shared writable directory
and that the ZEO protocol should not transfer the file
but only the filename when committing.
</description>
</key>
<key name="blob-cache-size" required="no" datatype="byte-size">
<description>
Maximum size of the ZEO blob cache, in bytes. If not set, then
the cache size isn't checked and the blob directory will
grow without bound.
This option is ignored if shared_blob_dir is true.
</description>
</key>
<key name="authentication-database" required="no">
<key name="blob-cache-size-check" required="no" datatype="integer">
<description>
The path of the database containing authentication credentials.
ZEO check size as percent of blob_cache_size. The ZEO
cache size will be checked when this many bytes have been
loaded into the cache. Defaults to 10% of the blob cache
size. This option is ignored if shared_blob_dir is true.
</description>
</key>
<key name="authentication-realm" required="no">
<key name="read-only" datatype="boolean" default="off">
<description>
The authentication realm of the server. Some authentication
schemes use a realm to identify the logical set of usernames
that are accepted by this server.
A flag indicating whether this should be a read-only storage,
defaulting to false (i.e. writing is allowed by default).
</description>
</key>
<key name="pid-filename" datatype="existing-dirpath"
required="no">
<key name="read-only-fallback" datatype="boolean" default="off">
<description>
A flag indicating whether a read-only remote storage should be
acceptable as a fallback when no writable storages are
available. Defaults to false. At most one of read_only and
read_only_fallback should be true.
</description>
</key>
<key name="server-sync" datatype="boolean" default="off">
<description>
A flag indicating whether calls to sync() should make a server
request, thus causing the storage to wait for any outstanding
invalidations. The sync method is called when transactions are
explicitly begun.
</description>
</key>
<key name="wait-timeout" datatype="integer" default="30">
<description>
How long to wait for an initial connection, defaulting to 30
seconds. If an initial connection can't be made within this time
limit, then creation of the client storage will fail with a
``ZEO.Exceptions.ClientDisconnected`` exception.
After the initial connection, if the client is disconnected:
- In-flight server requests will fail with a
``ZEO.Exceptions.ClientDisconnected`` exception.
- New requests will block for up to ``wait_timeout`` waiting for a
connection to be established before failing with a
``ZEO.Exceptions.ClientDisconnected`` exception.
</description>
</key>
<key name="client-label" required="no">
<description>
A label for the client in server logs
</description>
</key>
<!-- The following are undocumented, but not gone. :) -->
<key name="storage" default="1">
<description>
The full path to the file in which to write the ZEO server's Process ID
at startup. If omitted, $INSTANCE/var/ZEO.pid is used.
The name of the storage that the client wants to use. If the
ZEO server serves more than one storage, the client selects
the storage it wants to use by name. The default name is '1',
which is also the default name for the ZEO server.
</description>
<metadefault>$INSTANCE/var/ZEO.pid (or $clienthome/ZEO.pid)</metadefault>
</key>
<!-- DM 2006-06-12: added option -->
<key name="drop-cache-rather-verify" datatype="boolean"
required="no" default="false">
<key name="name" default="">
<description>
indicates that the cache should be dropped rather than
verified when the verification optimization is not
available (e.g. when the ZEO server restarted).
The storage name. If unspecified, the address of the server
will be used as the name.
</description>
</key>
......
src/ZEO/interfaces.py
View file @ c499ea7d
......@@ -21,16 +21,73 @@ class StaleCache(object):
def __init__(self, storage):
self.storage = storage
class IServeable(zope.interface.Interface):
"""Interface provided by storages that can be served by ZEO
class IClientCache(zope.interface.Interface):
"""Client cache interface.
Note that caches need to be thread safe.
"""
def close():
"""Close the cache
"""
def getTid(oid):
"""The last transaction to change an object
def load(oid):
"""Get current data for object
Returns data and serial, or None.
"""
Return the transaction id of the last transaction that committed a
change to an object with the given object id.
def __len__():
"""Retirn the number of items in the cache.
"""
def store(oid, start_tid, end_tid, data):
"""Store data for the object
The start_tid is the transaction that committed this data.
The end_tid is the tid of the next transaction that modified
the objects, or None if this is the current version.
"""
def loadBefore(oid, tid):
"""Load the data for the object last modified before the tid
Returns the data, and start and end tids.
"""
def invalidate(oid, tid):
"""Invalidate data for the object
If ``tid`` is None, forget all knowledge of `oid`. (``tid``
can be None only for invalidations generated by startup cache
verification.)
If ``tid`` isn't None, and we had current data for ``oid``,
stop believing we have current data, and mark the data we had
as being valid only up to `tid`. In all other cases, do
nothing.
"""
def getLastTid():
"""Get the last tid seen by the cache
This is the cached last tid we've seen from the server.
This method may be called from multiple threads. (It's assumed
to be trivial.)
"""
def setLastTid(tid):
"""Save the last tid sent by the server
"""
def clear():
"""Clear/empty the cache
"""
class IServeable(zope.interface.Interface):
"""Interface provided by storages that can be served by ZEO
"""
def tpc_transaction():
......
src/ZEO/monitor.py
View file @ c499ea7d
......@@ -57,7 +57,6 @@ class StorageStats:
self.commits = 0
self.aborts = 0
self.active_txns = 0
self.verifying_clients = 0
self.lock_time = None
self.conflicts = 0
self.conflicts_resolved = 0
......@@ -112,79 +111,3 @@ class StorageStats:
print("Stores:", self.stores, file=f)
print("Conflicts:", self.conflicts, file=f)
print("Conflicts resolved:", self.conflicts_resolved, file=f)
class StatsClient(asyncore.dispatcher):
def __init__(self, sock, addr):
asyncore.dispatcher.__init__(self, sock)
self.buf = []
self.closed = 0
def close(self):
self.closed = 1
# The socket is closed after all the data is written.
# See handle_write().
def write(self, s):
self.buf.append(s)
def writable(self):
return len(self.buf)
def readable(self):
return 0
def handle_write(self):
s = "".join(self.buf)
self.buf = []
n = self.socket.send(s.encode('ascii'))
if n < len(s):
self.buf.append(s[:n])
if self.closed and not self.buf:
asyncore.dispatcher.close(self)
class StatsServer(asyncore.dispatcher):
StatsConnectionClass = StatsClient
def __init__(self, addr, stats):
asyncore.dispatcher.__init__(self)
self.addr = addr
self.stats = stats
if type(self.addr) == tuple:
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
else:
self.create_socket(socket.AF_UNIX, socket.SOCK_STREAM)
self.set_reuse_addr()
logger = logging.getLogger('ZEO.monitor')
logger.info("listening on %s", repr(self.addr))
self.bind(self.addr)
self.listen(5)
def writable(self):
return 0
def readable(self):
return 1
def handle_accept(self):
try:
sock, addr = self.accept()
except socket.error:
return
f = self.StatsConnectionClass(sock, addr)
self.dump(f)
f.close()
def dump(self, f):
print("ZEO monitor server version %s" % zeo_version, file=f)
print(time.ctime(), file=f)
print(file=f)
L = sorted(self.stats.keys())
for k in L:
stats = self.stats[k]
print("Storage:", k, file=f)
stats.dump(f)
print(file=f)
src/ZEO/nagios.rst
View file @ c499ea7d
......@@ -10,7 +10,7 @@ ZEO includes a script that provides a nagios monitor plugin:
In it's simplest form, the script just checks if it can get status:
>>> import ZEO
>>> addr, stop = ZEO.server('test.fs')
>>> addr, stop = ZEO.server('test.fs', threaded=False)
>>> saddr = ':'.join(map(str, addr)) # (host, port) -> host:port
>>> nagios([saddr])
......@@ -39,7 +39,7 @@ The monitor will optionally output server metric data. There are 2
kinds of metrics it can output, level and rate metric. If we use the
-m/--output-metrics option, we'll just get rate metrics:
>>> addr, stop = ZEO.server('test.fs')
>>> addr, stop = ZEO.server('test.fs', threaded=False)
>>> saddr = ':'.join(map(str, addr)) # (host, port) -> host:port
>>> nagios([saddr, '-m'])
OK|active_txns=0
......@@ -115,7 +115,7 @@ profixes metrics with a storage id.
... </mappingstorage>
... <mappingstorage second>
... </mappingstorage>
... """)
... """, threaded=False)
>>> saddr = ':'.join(map(str, addr)) # (host, port) -> host:port
>>> nagios([saddr, '-m', '-sstatus'])
Empty storage u'first'|first:active_txns=0
......
src/ZEO/ordering.rst 0 → 100644
View file @ c499ea7d
==============================
Response ordering requirements
==============================
ZEO servers are logically concurrent because they serve multiple
clients in multiple threads. Because of this, we have to make sure
that information about object histories remain consistent.
An object history is a sequence of object revisions. Each revision has
a tid, which is essentially a time stamp.
We load objects using either ``load``, which returns the current
object. or ``loadBefore``, which returns the object before a specific time/tid.
When we cache revisions, we record the tid and the next/end tid, which
may be None. The end tid is important for choosing a revision for
``loadBefore``, as well as for determining whether a cached value is
current, for ``load``.
Because the client and server are multi-threaded, the client may see
data out of order. Let's consider some scenarios. In these
scenarios
Scenarios
=========
When considering ordering scenarioes, we'll consider 2 different
client behaviors, traditional (T) and loadBefore (B).
The *traditional* behaviors is that used in ZODB 4. It uses the storage
``load(oid)`` method to load objects if it hasn't seen an invalidation
for the object. If it has seen an invalidation, it uses
``loadBefore(oid, START)``, where ``START`` is the transaction time of
the first invalidation it's seen. If it hasn't seen an invalidation
*for an object*, it uses ``load(oid)`` and then checks again for an
invalidation. If it sees an invalidation, then it retries using
``loadBefore``. This approach **assumes that invalidations for a tid
are returned before loads for a tid**.
The *loadBefore* behavior, used in ZODB5, always determines
transaction start time, ``START`` at the beginning of a transaction by
calling the storage's ``sync`` method and then querying the storage's
``lastTransaction`` method (and adding 1). It loads objects
exclusively using ``loadBefore(oid, START)``.
Scenario 1, Invalidations seen after loads for transaction
----------------------------------------------------------
This scenario could occur because the commits are for a different
client, and a hypothetical; server doesn't block loads while
committing, or sends invalidations in a way that might delay them (but
not send them out of order).
T1
- client starts a transaction
- client load(O1) gets O1-T1
- client load(O2)
- Server commits O2-T2
- Server loads (O2-T2)
- Client gets O2-T2, updates the client cache, and completes load
- Client sees invalidation for O2-T2. If the
client is smart, it doesn't update the cache.
The transaction now has inconsistent data, because it should have
loaded whatever O2 was before T2. Because the invalidation came
in after O2 was loaded, the load was unaffected.
B1
- client starts a transaction. Sets START to T1+1
- client loadBefore(O1, T1+1) gets O1-T1, T1, None
- client loadBefore(O2, T1+1)
- Server commits O2-T2
- Server loadBefore(O2, T1+1) -> O2-T0-T2
(assuming that the revision of O2 before T2 was T0)
- Client gets O2-T0-T2, updates cache.
- Client sees invalidation for O2-T2. No update to the cache is
necessary.
In this scenario, loadBefore prevents reading incorrect data.
A variation on this scenario is that client sees invalidations
tpc_finish in another thread after loads for the same transaction.
Scenario 2, Client sees invalidations for later transaction before load result
------------------------------------------------------------------------------
T2
- client starts a transaction
- client load(O1) gets O1-T1
- client load(O2)
- Server loads (O2-T0)
- Server commits O2-T2
- Client sees invalidation for O2-T2. O2 isn't in the cache, so
nothing to do.
- Client gets O2-T0, updates the client cache, and completes load
The cache is now incorrect. It has O2-T0-None, meaning it thinks
O2-T0 is current.
The transaction is OK, because it got a consistent value for O2.
B2
- client starts a transaction. Sets START to T1+1
- client loadBefore(O1, T1+1) gets O1-T1, T1, None
- client loadBefore(O2, T1+1)
- Server loadBefore(O2, T1+1) -> O2-T0-None
- Server commits O2-T2
- Client sees invalidation for O2-T2. O2 isn't in the cache, so
nothing to do.
- Client gets O2-T0-None, and completes load
ZEO 4 doesn't cache loadBefore results with no ending transaction.
Assume ZEO 5 updates the client cache.
For ZEO 5, the cache is now incorrect. It has O2-T0-None, meaning
it thinks O2-T0 is current.
The transaction is OK, because it got a consistent value for O2.
In this case, ``loadBefore`` didn't prevent an invalid cache value.
Scenario 3, client sees invalidation after lastTransaction result
------------------------------------------------------------------
(This doesn't effect the traditional behavior.)
B3
- The client cache has a last tid of T1.
- ZODB calls sync() then calls lastTransaction. Is so configured,
ZEO calls lastTransaction on the server. This is mainly to make a
round trip to get in-flight invalidations. We don't necessarily
need to use the value. In fact, in protocol 5, we could just add a
sync method that just makes a round trip, but does nothing else.
- Server commits O1-T2, O2-T2.
- Server reads and returns T2. (It doesn't mater what it returns
- client sets START to T1+1, because lastTransaction is based on
what's in the cache, which is based on invalidations.
- Client loadBefore(O1, T2+1), finds O1-T1-None in cache and uses
it.
- Client gets invalidation for O1-T2. Updates cache to O1-T1-T2.
- Client loadBefore(O2, T1+1), gets O2-T1-None
This is OK, as long as the client doesn't do anything with the
lastTransaction result in ``sync``.
Implementation notes
===================
ZEO 4
-----
The ZEO 4 server sends data to the client in correct order with
respect to loads and invalidations (or tpc_finish results). This is a
consequence of the fact that invalidations are sent in a callback
called when the storage lock is held, blocking loads while committing,
and, fact that client requests, for a particular client, are
handled by a single thread on the server, and that all output for a
client goes through a thread-safe queue.
Invalidations are sent from different threads than clients. Outgoing
data is queued, however, using Python lists, which are protected by
the GIL. This means that the serialization provided though storage
locks is preserved by the way that server outputs are queued. **The
queueing mechanism is in part a consequence of the way asyncore, used
by ZEO4, works.
In ZEO 4 clients, invalidations and loads are handled by separate
threads. This means that even though data arive in order, they may not
be processed in order,
T1
The existing servers mitigate this by blocking loads while
committing. On the client, this is still a potential issue because loads
and invalidations are handled by separate threads, however, locks are
used on the client to assure that invalidations are processed before
blocked loads complete.
T2
Existing storage servers serialize commits (and thus sending of
invalidations) and loads. As with scenario T1, threading on the
client can cause load results and invalidations to be processed out
of order. To mitigate this, the client uses a load lock to track
when loads are invalidated while in flight and doesn't save to the
cache when they are. This is bad on multiple levels. It serializes
loads even when there are multiple threads. It may prevent writing
to the cache unnecessarily, if the invalidation is for a revision
before the one that was loaded.
B2
Here, we avoid incorrect returned values and incorrect cache at the
cost of caching nothing.
ZEO 4.2.0 addressed this by using the same locking strategy for
``loadBefore`` that was used for ``load``, thus mitigating B2 the
same way it mitigates T2.
ZEO 5
-----
In ZEO(/ZODB) 5, we want to get more concurrency, both on the client,
and on the server. On the client, cache invalidations and loads are
done by the same thread, which makes things a bit simpler. This let's
us get rid of the client load lock and prevents the scenarios above
with existing servers.
On the client, we'd like to stop serializing loads and commits. We'd
like commits (tpc_finish calls) to be in flight with loads (and with
other commits). In the current protocol, tpc_finish, load and
loadBefore are all synchronous calls that are handled by a single
thread on the server, so these calls end up being serialized on the
server anyway.
The server-side hndling of invalidations is a bit tricker in ZEO 5
because there isn't a thread-safe queue of outgoing messages in ZEO 5
as there was in ZEO 4. The natural approach in ZEO 5 would be to use
asyncio's ``call_soon_threadsafe`` to send invalidations in a client's
thread. This could easily cause invalidations to be sent after loads.
As shown above, this isn't a problem for ZODB 5, at least assuming
that invalidations arrive in order. This would be a problem for
ZODB 4. For this reason, we require ZODB 5 for ZEO 5.
Note that this approach can't cause invalidations to be sent early,
because they could only be sent by the thread that's busy loading, so
scenario 2 wouldn't happen.
B2
Because the server send invalidations by calling
``call_soon_threadsafe``, it's impoossible for invalidations to be
send while a load request is being handled.
The main server opportunity is allowing commits for separate oids to
happen concurrently. This wouldn't effect the invalidation/load
ordering though.
It would be nice not to block loads while making tpc_finish calls, but
storages do this anyway now, so there's nothing to be done about it
now. Storage locking requirements aren't well specified, and probably
should be rethought in light of ZODB5/loadBefore.
src/ZEO/runzeo.py
View file @ c499ea7d
......@@ -22,7 +22,6 @@ Options:
-f/--filename FILENAME -- filename for FileStorage
-t/--timeout TIMEOUT -- transaction timeout in seconds (default no timeout)
-h/--help -- print this usage message and exit
-m/--monitor ADDRESS -- address of monitor server ([HOST:]PORT or PATH)
--pid-file PATH -- relative path to output file containing this process's pid;
default $(INSTANCE_HOME)/var/ZEO.pid but only if envar
INSTANCE_HOME is defined
......@@ -71,9 +70,6 @@ class ZEOOptionsMixin:
def handle_address(self, arg):
self.family, self.address = parse_binding_address(arg)
def handle_monitor_address(self, arg):
self.monitor_family, self.monitor_address = parse_binding_address(arg)
def handle_filename(self, arg):
from ZODB.config import FileStorage # That's a FileStorage *opener*!
class FSConfig:
......@@ -101,21 +97,17 @@ class ZEOOptionsMixin:
self.add("address", "zeo.address.address",
required="no server address specified; use -a or -C")
self.add("read_only", "zeo.read_only", default=0)
self.add("client_conflict_resolution",
"zeo.client_conflict_resolution",
default=0)
self.add("invalidation_queue_size", "zeo.invalidation_queue_size",
default=100)
self.add("invalidation_age", "zeo.invalidation_age")
self.add("transaction_timeout", "zeo.transaction_timeout",
"t:", "timeout=", float)
self.add("monitor_address", "zeo.monitor_address.address",
"m:", "monitor=", self.handle_monitor_address)
self.add('auth_protocol', 'zeo.authentication_protocol',
None, 'auth-protocol=', default=None)
self.add('auth_database', 'zeo.authentication_database',
None, 'auth-database=')
self.add('auth_realm', 'zeo.authentication_realm',
None, 'auth-realm=')
self.add('pid_file', 'zeo.pid_filename',
None, 'pid-file=')
self.add("ssl", "zeo.ssl")
class ZEOOptions(ZDOptions, ZEOOptionsMixin):
......@@ -183,6 +175,7 @@ class ZEOServer:
self.options.address[1] is None):
self.options.address = self.options.address[0], 0
return
if self.can_connect(self.options.family, self.options.address):
self.options.usage("address %s already in use" %
repr(self.options.address))
......@@ -348,13 +341,11 @@ def create_server(storages, options):
options.address,
storages,
read_only = options.read_only,
client_conflict_resolution=options.client_conflict_resolution,
invalidation_queue_size = options.invalidation_queue_size,
invalidation_age = options.invalidation_age,
transaction_timeout = options.transaction_timeout,
monitor_address = options.monitor_address,
auth_protocol = options.auth_protocol,
auth_database = options.auth_database,
auth_realm = options.auth_realm,
ssl = options.ssl,
)
......@@ -392,5 +383,11 @@ def main(args=None):
s = ZEOServer(options)
s.main()
def run(args):
options = ZEOOptions()
options.realize(args)
s = ZEOServer(options)
s.run()
if __name__ == "__main__":
main()
src/ZEO/schema.xml
View file @ c499ea7d
......@@ -12,7 +12,7 @@
<import package="ZODB"/>
<!-- Use the ZEO server information structure. -->
<import package="ZEO"/>
<import package="ZEO" file="server.xml" />
<import package="ZConfig.components.logger"/>
......
src/ZEO/scripts/zeoserverlog.py
View file @ c499ea7d
......@@ -500,7 +500,8 @@ def days(f):
minute(f, 10, detail=0)
new_connection_idre = re.compile(r"new connection \('(\d+.\d+.\d+.\d+)', (\d+)\):")
new_connection_idre = re.compile(
r"new connection \('(\d+.\d+.\d+.\d+)', (\d+)\):")
def verify(f):
f, = f
......
src/ZEO/server.xml 0 → 100644
View file @ c499ea7d
<component>
<sectiontype name="ssl" datatype="ZEO.zconfig.server_ssl">
<key name="certificate" datatype="existing-dirpath" required="yes">
<description>
The full path to an SSL certificate file.
</description>
</key>
<key name="key" datatype="existing-dirpath" required="no">
<description>
The full path to an SSL key file for the server certificate.
</description>
</key>
<key name="password-function" required="no">
<description>
Dotted name of importable function for retrieving a password
for the client certificate key.
</description>
</key>
<key name="authenticate" required="yes">
<description>
Path to a file or directory containing client certificates to
be authenticated. This can also be - or SIGNED to require
signed client certificates.
</description>
</key>
</sectiontype>
<sectiontype name="zeo">
<section type="ssl" name="*" attribute="ssl" />
<description>
The content of a ZEO section describe operational parameters
of a ZEO server except for the storage(s) to be served.
</description>
<key name="address" datatype="socket-binding-address"
required="yes">
<description>
The address at which the server should listen. This can be in
the form 'host:port' to signify a TCP/IP connection or a
pathname string to signify a Unix domain socket connection (at
least one '/' is required). A hostname may be a DNS name or a
dotted IP address. If the hostname is omitted, the platform's
default behavior is used when binding the listening socket (''
is passed to socket.bind() as the hostname portion of the
address).
</description>
</key>
<key name="read-only" datatype="boolean"
required="no"
default="false">
<description>
Flag indicating whether the server should operate in read-only
mode. Defaults to false. Note that even if the server is
operating in writable mode, individual storages may still be
read-only. But if the server is in read-only mode, no write
operations are allowed, even if the storages are writable. Note
that pack() is considered a read-only operation.
</description>
</key>
<key name="invalidation-queue-size" datatype="integer"
required="no"
default="100">
<description>
The storage server keeps a queue of the objects modified by the
last N transactions, where N == invalidation_queue_size. This
queue is used to speed client cache verification when a client
disconnects for a short period of time.
</description>
</key>
<key name="invalidation-age" datatype="float" required="no">
<description>
The maximum age of a client for which quick-verification
invalidations will be provided by iterating over the served
storage. This option should only be used if the served storage
supports efficient iteration from a starting point near the
end of the transaction history (e.g. end of file).
</description>
</key>
<key name="transaction-timeout" datatype="integer"
required="no">
<description>
The maximum amount of time to wait for a transaction to commit
after acquiring the storage lock, specified in seconds. If the
transaction takes too long, the client connection will be closed
and the transaction aborted.
</description>
</key>
<key name="pid-filename" datatype="existing-dirpath"
required="no">
<description>
The full path to the file in which to write the ZEO server's Process ID
at startup. If omitted, $INSTANCE/var/ZEO.pid is used.
</description>
<metadefault>$INSTANCE/var/ZEO.pid (or $clienthome/ZEO.pid)</metadefault>
</key>
<key name="client-conflict-resolution" datatype="boolean"
required="no" default="false">
<description>
Flag indicating whether the server should return conflict
errors to the client, for resolution there.
</description>
</key>
</sectiontype>
</component>
src/ZEO/shortrepr.py 0 → 100644
View file @ c499ea7d
##############################################################################
#
# Copyright (c) 2001, 2002 Zope Foundation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE
#
##############################################################################
REPR_LIMIT = 60
def short_repr(obj):
"Return an object repr limited to REPR_LIMIT bytes."
# Some of the objects being repr'd are large strings. A lot of memory
# would be wasted to repr them and then truncate, so they are treated
# specially in this function.
# Also handle short repr of a tuple containing a long string.
# This strategy works well for arguments to StorageServer methods.
# The oid is usually first and will get included in its entirety.
# The pickle is near the beginning, too, and you can often fit the
# module name in the pickle.
if isinstance(obj, str):
if len(obj) > REPR_LIMIT:
r = repr(obj[:REPR_LIMIT])
else:
r = repr(obj)
if len(r) > REPR_LIMIT:
r = r[:REPR_LIMIT-4] + '...' + r[-1]
return r
elif isinstance(obj, (list, tuple)):
elts = []
size = 0
for elt in obj:
r = short_repr(elt)
elts.append(r)
size += len(r)
if size > REPR_LIMIT:
break
if isinstance(obj, tuple):
r = "(%s)" % (", ".join(elts))
else:
r = "[%s]" % (", ".join(elts))
else:
r = repr(obj)
if len(r) > REPR_LIMIT:
return r[:REPR_LIMIT] + '...'
else:
return r
src/ZEO/tests/Cache.py
View file @ c499ea7d
......@@ -44,8 +44,8 @@ class TransUndoStorageWithCache:
self._storage.tpc_finish(t)
assert len(oids) == 1
assert oids[0] == oid
[uoid] = oids
assert uoid == oid
data, revid = self._storage.load(oid, '')
obj = zodb_unpickle(data)
assert obj == MinPO(24)
src/ZEO/tests/CommitLockTests.py
View file @ c499ea7d
......@@ -30,6 +30,8 @@ class DummyDB:
def invalidate(self, *args, **kwargs):
pass
transform_record_data = untransform_record_data = lambda self, data: data
class WorkerThread(TestThread):
# run the entire test in a thread so that the blocking call for
......@@ -60,17 +62,11 @@ class WorkerThread(TestThread):
# coordinate the action of multiple threads that all call
# vote(). This method sends the vote call, then sets the
# event saying vote was called, then waits for the vote
# response. It digs deep into the implementation of the client.
# This method is a replacement for:
# self.ready.set()
# self.storage.tpc_vote(self.trans)
# response.
rpc = self.storage._server.rpc
msgid = rpc._deferred_call('vote', id(self.trans))
future = self.storage._server.call_future('vote', id(self.trans))
self.ready.set()
rpc._deferred_wait(msgid)
self.storage._check_serials()
future.result(9)
class CommitLockTests:
......@@ -133,7 +129,8 @@ class CommitLockTests:
# list is a socket domain (AF_INET, AF_UNIX, etc.) and an
# address.
addr = self._storage._addr
new = ZEO.ClientStorage.ClientStorage(addr, wait=1)
new = ZEO.ClientStorage.ClientStorage(
addr, wait=1, **self._client_options())
new.registerDB(DummyDB())
return new
......
src/ZEO/tests/ConnectionTests.py
View file @ c499ea7d
......@@ -11,68 +11,45 @@
# FOR A PARTICULAR PURPOSE
#
##############################################################################
import concurrent.futures
import contextlib
import os
import time
import socket
import asyncore
import threading
import logging
import ZEO.ServerStub
from ZEO.ClientStorage import ClientStorage
from ZEO.Exceptions import ClientDisconnected
from ZEO.zrpc.marshal import encode
from ZEO.asyncio.marshal import encode
from ZEO.tests import forker
from ZODB.DB import DB
from ZODB.POSException import ReadOnlyError, ConflictError
from ZODB.tests.StorageTestBase import StorageTestBase
from ZODB.tests.MinPO import MinPO
from ZODB.tests.StorageTestBase \
import zodb_pickle, zodb_unpickle, handle_all_serials, handle_serials
from ZODB.tests.StorageTestBase import zodb_pickle, zodb_unpickle
import ZODB.tests.util
import transaction
from transaction import Transaction
from . import testssl
logger = logging.getLogger('ZEO.tests.ConnectionTests')
ZERO = '\0'*8
class TestServerStub(ZEO.ServerStub.StorageServer):
__super_getInvalidations = ZEO.ServerStub.StorageServer.getInvalidations
def getInvalidations(self, tid):
# squirrel the results away for inspection by test case
self._last_invals = self.__super_getInvalidations(tid)
return self._last_invals
class TestClientStorage(ClientStorage):
test_connection = False
StorageServerStubClass = TestServerStub
connection_count_for_tests = 0
def notifyConnected(self, conn):
ClientStorage.notifyConnected(self, conn)
def notify_connected(self, conn, info):
ClientStorage.notify_connected(self, conn, info)
self.connection_count_for_tests += 1
def verify_cache(self, stub):
self.end_verify = threading.Event()
self.verify_result = ClientStorage.verify_cache(self, stub)
def endVerify(self):
ClientStorage.endVerify(self)
self.end_verify.set()
def testConnection(self, conn):
try:
return ClientStorage.testConnection(self, conn)
finally:
self.test_connection = True
self.verify_result = conn.verify_result
class DummyDB:
def invalidate(self, *args, **kwargs):
......@@ -81,6 +58,9 @@ class DummyDB:
def invalidateCache(self):
pass
transform_record_data = untransform_record_data = lambda self, data: data
class CommonSetupTearDown(StorageTestBase):
"""Common boilerplate"""
......@@ -89,7 +69,6 @@ class CommonSetupTearDown(StorageTestBase):
keep = 0
invq = None
timeout = None
monitor = 0
db_class = DummyDB
def setUp(self, before=None):
......@@ -102,41 +81,22 @@ class CommonSetupTearDown(StorageTestBase):
self.__super_setUp()
logging.info("setUp() %s", self.id())
self.file = 'storage_conf'
self.addr = []
self._pids = []
self._servers = []
self.conf_paths = []
self.caches = []
self._newAddr()
self.addr = [('127.0.0.1', 0)]
self.startServer()
# self._old_log_level = logging.getLogger().getEffectiveLevel()
# logging.getLogger().setLevel(logging.WARNING)
# self._log_handler = logging.StreamHandler()
# logging.getLogger().addHandler(self._log_handler)
def tearDown(self):
"""Try to cause the tests to halt"""
# logging.getLogger().setLevel(self._old_log_level)
# logging.getLogger().removeHandler(self._log_handler)
# logging.info("tearDown() %s" % self.id())
for p in self.conf_paths:
os.remove(p)
if getattr(self, '_storage', None) is not None:
self._storage.close()
if hasattr(self._storage, 'cleanup'):
logging.debug("cleanup storage %s" %
self._storage.__name__)
self._storage.cleanup()
for adminaddr in self._servers:
if adminaddr is not None:
forker.shutdown_zeo_server(adminaddr)
for pid in self._pids:
try:
os.waitpid(pid, 0)
except OSError:
pass # The subprocess module may already have waited
for stop in self._servers:
stop()
for c in self.caches:
for i in 0, 1:
......@@ -166,7 +126,7 @@ class CommonSetupTearDown(StorageTestBase):
self.addr.append(self._getAddr())
def _getAddr(self):
return 'localhost', forker.get_port(self)
return '127.0.0.1', forker.get_port(self)
def getConfig(self, path, create, read_only):
raise NotImplementedError
......@@ -188,18 +148,17 @@ class CommonSetupTearDown(StorageTestBase):
min_disconnect_poll=0.1,
read_only=read_only,
read_only_fallback=read_only_fallback,
username=username,
password=password,
realm=realm)
**self._client_options())
storage.registerDB(DummyDB())
return storage
def _client_options(self):
return {}
def getServerConfig(self, addr, ro_svr):
zconf = forker.ZEOConfig(addr)
zconf = forker.ZEOConfig(addr, log='server.log')
if ro_svr:
zconf.read_only = 1
if self.monitor:
zconf.monitor_address = ("", 42000)
if self.invq:
zconf.invalidation_queue_size = self.invq
if self.timeout:
......@@ -207,7 +166,7 @@ class CommonSetupTearDown(StorageTestBase):
return zconf
def startServer(self, create=1, index=0, read_only=0, ro_svr=0, keep=None,
path=None):
path=None, **kw):
addr = self.addr[index]
logging.info("startServer(create=%d, index=%d, read_only=%d) @ %s" %
(create, index, read_only, addr))
......@@ -217,48 +176,33 @@ class CommonSetupTearDown(StorageTestBase):
zconf = self.getServerConfig(addr, ro_svr)
if keep is None:
keep = self.keep
zeoport, adminaddr, pid, path = forker.start_zeo_server(
sconf, zconf, addr[1], keep)
self.conf_paths.append(path)
self._pids.append(pid)
self._servers.append(adminaddr)
zeoport, stop = forker.start_zeo_server(
sconf, zconf, addr[1], keep, **kw)
self._servers.append(stop)
if addr[1] == 0:
self.addr[index] = zeoport
def shutdownServer(self, index=0):
logging.info("shutdownServer(index=%d) @ %s" %
(index, self._servers[index]))
adminaddr = self._servers[index]
if adminaddr is not None:
forker.shutdown_zeo_server(adminaddr)
self._servers[index] = None
stop = self._servers[index]
if stop is not None:
stop()
self._servers[index] = lambda : None
def pollUp(self, timeout=30.0, storage=None):
if storage is None:
storage = self._storage
# Poll until we're connected.
now = time.time()
giveup = now + timeout
while not storage.is_connected():
asyncore.poll(0.1)
now = time.time()
if now > giveup:
self.fail("timed out waiting for storage to connect")
# When the socket map is empty, poll() returns immediately,
# and this is a pure busy-loop then. At least on some Linux
# flavors, that can starve the thread trying to connect,
# leading to grossly increased runtime (typical) or bogus
# "timed out" failures. A little sleep here cures both.
time.sleep(0.1)
storage.server_status()
def pollDown(self, timeout=30.0):
# Poll until we're disconnected.
now = time.time()
giveup = now + timeout
while self._storage.is_connected():
asyncore.poll(0.1)
now = time.time()
if now > giveup:
self.fail("timed out waiting for storage to disconnect")
# See pollUp() for why we sleep a little here.
time.sleep(0.1)
......@@ -334,6 +278,7 @@ class ConnectionTests(CommonSetupTearDown):
# object is not in the cache.
self.shutdownServer()
self._storage = self.openClientStorage('test', 1000, wait=0)
with short_timeout(self):
self.assertRaises(ClientDisconnected,
self._storage.load, b'fredwash', '')
self._storage.close()
......@@ -401,6 +346,7 @@ class ConnectionTests(CommonSetupTearDown):
self.assertEqual(expected2, self._storage.load(oid2, ''))
# But oid1 should have been purged, so that trying to load it will
# try to fetch it from the (non-existent) ZEO server.
with short_timeout(self):
self.assertRaises(ClientDisconnected, self._storage.load, oid1, '')
self._storage.close()
......@@ -465,7 +411,7 @@ class ConnectionTests(CommonSetupTearDown):
def checkBadMessage1(self):
# not even close to a real message
self._bad_message("salty")
self._bad_message(b"salty")
def checkBadMessage2(self):
# just like a real message, but with an unpicklable argument
......@@ -485,22 +431,36 @@ class ConnectionTests(CommonSetupTearDown):
self._storage = self.openClientStorage()
self._dostore()
# break into the internals to send a bogus message
zrpc_conn = self._storage._server.rpc
zrpc_conn.message_output(msg)
generation = self._storage._connection_generation
future = concurrent.futures.Future()
def write():
try:
self._dostore()
except ClientDisconnected:
pass
self._storage._server.client.protocol._write(msg)
except Exception as exc:
future.set_exception(exc)
else:
self._storage.close()
self.fail("Server did not disconnect after bogus message")
self._storage.close()
future.set_result(None)
self._storage = self.openClientStorage()
# break into the internals to send a bogus message
self._storage._server.loop.call_soon_threadsafe(write)
future.result()
# If we manage to call _dostore before the server disconnects
# us, we'll get a ClientDisconnected error. When we retry, it
# will succeed. It will succeed because:
# - _dostore calls tpc_abort
# - tpc_abort makes a synchronous call to the server to abort
# the transaction
# - when disconnected, synchronous calls are blocked for a little
# while while reconnecting (or they timeout of it takes too long).
try:
self._dostore()
except ClientDisconnected:
self._dostore()
self._storage.close()
self.assertTrue(self._storage._connection_generation > generation)
# Test case for multiple storages participating in a single
# transaction. This is not really a connection test, but it needs
......@@ -579,20 +539,35 @@ class ConnectionTests(CommonSetupTearDown):
self._storage = self.openClientStorage()
self._dostore()
self.shutdownServer()
self.assertRaises(ClientDisconnected, self._storage.load, b'\0'*8, '')
with short_timeout(self):
self.assertRaises(ClientDisconnected,
self._storage.load, b'\0'*8, '')
self.startServer()
# No matter how long we wait, the client won't reconnect:
time.sleep(2)
self.assertRaises(ClientDisconnected, self._storage.load, b'\0'*8, '')
with short_timeout(self):
self.assertRaises(ClientDisconnected,
self._storage.load, b'\0'*8, '')
class SSLConnectionTests(ConnectionTests):
def getServerConfig(self, addr, ro_svr):
return testssl.server_config.replace(
'127.0.0.1:0',
'{}: {}\nread-only {}'.format(
addr[0], addr[1], 'true' if ro_svr else 'false'))
def _client_options(self):
return {'ssl': testssl.client_ssl()}
class InvqTests(CommonSetupTearDown):
invq = 3
def checkQuickVerificationWith2Clients(self):
perstorage = self.openClientStorage(cache="test", cache_size=4000)
self.assertEqual(perstorage.verify_result, "empty cache")
self._storage = self.openClientStorage()
oid = self._storage.new_oid()
......@@ -622,8 +597,6 @@ class InvqTests(CommonSetupTearDown):
perstorage = self.openClientStorage(cache="test")
self.assertEqual(perstorage.verify_result, "quick verification")
self.assertEqual(perstorage._server._last_invals,
(revid, [oid]))
self.assertEqual(perstorage.load(oid, ''),
self._storage.load(oid, ''))
......@@ -655,13 +628,7 @@ class InvqTests(CommonSetupTearDown):
revid = self._dostore(oid, revid)
perstorage = self.openClientStorage(cache="test")
self.assertEqual(perstorage.verify_result, "full verification")
t = time.time() + 30
while not perstorage.end_verify.isSet():
perstorage.sync()
if time.time() > t:
self.fail("timed out waiting for endVerify")
self.assertEqual(perstorage.verify_result, "cache too old, clearing")
self.assertEqual(self._storage.load(oid, '')[1], revid)
self.assertEqual(perstorage.load(oid, ''),
self._storage.load(oid, ''))
......@@ -720,6 +687,7 @@ class ReconnectionTests(CommonSetupTearDown):
# Poll until the client disconnects
self.pollDown()
# Stores should fail now
with short_timeout(self):
self.assertRaises(ClientDisconnected, self._dostore)
# Restart the server
......@@ -769,6 +737,7 @@ class ReconnectionTests(CommonSetupTearDown):
# Poll until the client disconnects
self.pollDown()
# Stores should fail now
with short_timeout(self):
self.assertRaises(ClientDisconnected, self._dostore)
# Restart the server
......@@ -797,8 +766,10 @@ class ReconnectionTests(CommonSetupTearDown):
self._servers = []
# Poll until the client disconnects
self.pollDown()
# Stores should fail now
self.assertRaises(ClientDisconnected, self._dostore)
# Accesses should fail now
with short_timeout(self):
self.assertRaises(ClientDisconnected, self._storage.ping)
# Restart the server, this time read-write
self.startServer(create=0, keep=0)
......@@ -852,7 +823,7 @@ class ReconnectionTests(CommonSetupTearDown):
self.pollUp()
# There were no transactions committed, so no verification
# should be needed.
self.assertEqual(self._storage.verify_result, "no verification")
self.assertEqual(self._storage.verify_result, "Cache up to date")
def checkNoVerificationOnServerRestartWith2Clients(self):
perstorage = self.openClientStorage(cache="test")
......@@ -883,8 +854,8 @@ class ReconnectionTests(CommonSetupTearDown):
self.pollUp(storage=perstorage)
# There were no transactions committed, so no verification
# should be needed.
self.assertEqual(self._storage.verify_result, "no verification")
self.assertEqual(perstorage.verify_result, "no verification")
self.assertEqual(self._storage.verify_result, "Cache up to date")
self.assertEqual(perstorage.verify_result, "Cache up to date")
perstorage.close()
self._storage.close()
......@@ -898,10 +869,10 @@ class ReconnectionTests(CommonSetupTearDown):
data = zodb_pickle(MinPO(oid))
self._storage.store(oid, None, data, '', txn)
self.shutdownServer()
with short_timeout(self):
self.assertRaises(ClientDisconnected, self._storage.tpc_vote, txn)
self._storage.tpc_abort(txn)
self.startServer(create=0)
self._storage._wait()
self._storage.tpc_abort(txn)
self._dostore()
# This test is supposed to cover the following error, although
......@@ -985,15 +956,16 @@ class TimeoutTests(CommonSetupTearDown):
timeout = 1
def checkTimeout(self):
storage = self.openClientStorage()
self._storage = storage = self.openClientStorage()
txn = Transaction()
storage.tpc_begin(txn)
storage.tpc_vote(txn)
time.sleep(2)
with short_timeout(self):
self.assertRaises(ClientDisconnected, storage.tpc_finish, txn)
# Make sure it's logged as CRITICAL
for line in open("server-%s.log" % self.addr[0][1]):
for line in open("server.log"):
if (('Transaction timeout after' in line) and
('CRITICAL ZEO.StorageServer' in line)
):
......@@ -1048,90 +1020,6 @@ class TimeoutTests(CommonSetupTearDown):
# or the server.
self.assertRaises(KeyError, storage.load, oid, '')
def checkTimeoutProvokingConflicts(self):
self._storage = storage = self.openClientStorage()
# Assert that the zeo cache is empty.
self.assert_(not list(storage._cache.contents()))
# Create the object
oid = storage.new_oid()
obj = MinPO(7)
# We need to successfully commit an object now so we have something to
# conflict about.
t = Transaction()
storage.tpc_begin(t)
revid1a = storage.store(oid, ZERO, zodb_pickle(obj), '', t)
revid1b = storage.tpc_vote(t)
revid1 = handle_serials(oid, revid1a, revid1b)
storage.tpc_finish(t)
# Now do a store, sleeping before the finish so as to cause a timeout.
obj.value = 8
t = Transaction()
old_connection_count = storage.connection_count_for_tests
storage.tpc_begin(t)
revid2a = storage.store(oid, revid1, zodb_pickle(obj), '', t)
revid2b = storage.tpc_vote(t)
revid2 = handle_serials(oid, revid2a, revid2b)
# Now sleep long enough for the storage to time out.
# This used to sleep for 3 seconds, and sometimes (but very rarely)
# failed then. Now we try for a minute. It typically succeeds
# on the second time thru the loop, and, since self.timeout is 1,
# it's typically faster now (2/1.8 ~= 1.11 seconds sleeping instead
# of 3).
deadline = time.time() + 60 # wait up to a minute
while time.time() < deadline:
if (storage.is_connected() and
(storage.connection_count_for_tests == old_connection_count)
):
time.sleep(self.timeout / 1.8)
else:
break
self.assert_(
(not storage.is_connected())
or
(storage.connection_count_for_tests > old_connection_count)
)
storage._wait()
self.assert_(storage.is_connected())
# We expect finish to fail.
self.assertRaises(ClientDisconnected, storage.tpc_finish, t)
storage.tpc_abort(t)
# Now we think we've committed the second transaction, but we really
# haven't. A third one should produce a POSKeyError on the server,
# which manifests as a ConflictError on the client.
obj.value = 9
t = Transaction()
storage.tpc_begin(t)
storage.store(oid, revid2, zodb_pickle(obj), '', t)
self.assertRaises(ConflictError, storage.tpc_vote, t)
# Even aborting won't help.
storage.tpc_abort(t)
self.assertRaises(ZODB.POSException.StorageTransactionError,
storage.tpc_finish, t)
# Try again.
obj.value = 10
t = Transaction()
storage.tpc_begin(t)
storage.store(oid, revid2, zodb_pickle(obj), '', t)
# Even aborting won't help.
self.assertRaises(ConflictError, storage.tpc_vote, t)
# Abort this one and try a transaction that should succeed.
storage.tpc_abort(t)
# Now do a store.
obj.value = 11
t = Transaction()
storage.tpc_begin(t)
revid2a = storage.store(oid, revid1, zodb_pickle(obj), '', t)
revid2b = storage.tpc_vote(t)
revid2 = handle_serials(oid, revid2a, revid2b)
storage.tpc_finish(t)
# Now load the object and verify that it has a value of 11.
data, revid = storage.load(oid, '')
self.assertEqual(zodb_unpickle(data), MinPO(11))
self.assertEqual(revid, revid2)
class MSTThread(threading.Thread):
__super_init = threading.Thread.__init__
......@@ -1176,14 +1064,12 @@ class MSTThread(threading.Thread):
data = MinPO("%s.%s.t%d.o%d" % (tname, c.__name, i, j))
#print(data.value)
data = zodb_pickle(data)
s = c.store(oid, ZERO, data, '', t)
c.__serials.update(handle_all_serials(oid, s))
c.store(oid, ZERO, data, '', t)
# Vote on all servers and handle serials
for c in clients:
#print("%s.%s.%s vote" % (tname, c.__name, i))
s = c.tpc_vote(t)
c.__serials.update(handle_all_serials(None, s))
c.tpc_vote(t)
# Finish on all servers
for c in clients:
......@@ -1206,6 +1092,14 @@ class MSTThread(threading.Thread):
except:
pass
@contextlib.contextmanager
def short_timeout(self):
old = self._storage._server.timeout
self._storage._server.timeout = 1
yield
self._storage._server.timeout = old
# Run IPv6 tests if V6 sockets are supported
try:
socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
......
src/ZEO/tests/InvalidationTests.py
View file @ c499ea7d
......@@ -324,8 +324,8 @@ class InvalidationTests:
def checkConcurrentUpdates2Storages_emulated(self):
self._storage = storage1 = self.openClientStorage()
storage2 = self.openClientStorage()
db1 = DB(storage1)
storage2 = self.openClientStorage()
db2 = DB(storage2)
cn = db1.open()
......@@ -349,8 +349,8 @@ class InvalidationTests:
def checkConcurrentUpdates2Storages(self):
self._storage = storage1 = self.openClientStorage()
storage2 = self.openClientStorage()
db1 = DB(storage1)
storage2 = self.openClientStorage()
db2 = DB(storage2)
stop = threading.Event()
......
src/ZEO/tests/IterationTests.py
View file @ c499ea7d
......@@ -17,6 +17,8 @@ import transaction
import six
import gc
from ..asyncio.testing import AsyncRPC
class IterationTests:
def _assertIteratorIdsEmpty(self):
......@@ -52,7 +54,7 @@ class IterationTests:
def checkIteratorGCProtocol(self):
# Test garbage collection on protocol level.
server = self._storage._server
server = AsyncRPC(self._storage._server)
iid = server.iterator_start(None, None)
# None signals the end of iteration.
......@@ -79,7 +81,7 @@ class IterationTests:
self.assertEquals(0, len(self._storage._iterator_ids))
# The iterator has run through, so the server has already disposed it.
self.assertRaises(KeyError, self._storage._server.iterator_next, iid)
self.assertRaises(KeyError, self._storage._call, 'iterator_next', iid)
def checkIteratorGCSpanTransactions(self):
# Keep a hard reference to the iterator so it won't be automatically
......@@ -112,7 +114,7 @@ class IterationTests:
self._storage._iterators._last_gc = -1
self._dostore()
self._assertIteratorIdsEmpty()
self.assertRaises(KeyError, self._storage._server.iterator_next, iid)
self.assertRaises(KeyError, self._storage._call, 'iterator_next', iid)
def checkIteratorGCStorageTPCAborting(self):
# The odd little jig we do below arises from the fact that the
......@@ -129,7 +131,7 @@ class IterationTests:
self._storage.tpc_begin(t)
self._storage.tpc_abort(t)
self._assertIteratorIdsEmpty()
self.assertRaises(KeyError, self._storage._server.iterator_next, iid)
self.assertRaises(KeyError, self._storage._call, 'iterator_next', iid)
def checkIteratorGCStorageDisconnect(self):
......@@ -146,7 +148,7 @@ class IterationTests:
# Show that after disconnecting, the client side GCs the iterators
# as well. I'm calling this directly to avoid accidentally
# calling tpc_abort implicitly.
self._storage.notifyDisconnected()
self._storage.notify_disconnected()
self.assertEquals(0, len(self._storage._iterator_ids))
def checkIteratorParallel(self):
......
src/ZEO/tests/ThreadTests.py
View file @ c499ea7d
......@@ -17,7 +17,7 @@ import threading
import transaction
from ZODB.tests.StorageTestBase import zodb_pickle, MinPO
import ZEO.ClientStorage
import ZEO.Exceptions
ZERO = '\0'*8
......@@ -54,7 +54,7 @@ class GetsThroughVoteThread(BasicThread):
self.doNextEvent.wait(10)
try:
self.storage.tpc_finish(self.trans)
except ZEO.ClientStorage.ClientStorageError:
except ZEO.Exceptions.ClientStorageError:
self.gotValueError = 1
self.storage.tpc_abort(self.trans)
......@@ -67,7 +67,7 @@ class GetsThroughBeginThread(BasicThread):
def run(self):
try:
self.storage.tpc_begin(self.trans)
except ZEO.ClientStorage.ClientStorageError:
except ZEO.Exceptions.ClientStorageError:
self.gotValueError = 1
......
src/ZEO/tests/ZEO4/README.rst 0 → 100644
View file @ c499ea7d
======================
Copy of ZEO 4 server
======================
This copy was made by first converting the ZEO 4 server code to use
relative imports. The code was tested with ZEO 4 before copying. It
was unchanged aside from the relative imports.
The ZEO 4 server is used for tests if the ZEO4_SERVER environment
variable is set to a non-empty value.
src/ZEO/tests/ZEO4/StorageServer.py 0 → 100644
View file @ c499ea7d
##############################################################################
#
# Copyright (c) 2001, 2002, 2003 Zope Foundation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE
#
##############################################################################
"""The StorageServer class and the exception that it may raise.
This server acts as a front-end for one or more real storages, like
file storage or Berkeley storage.
TODO: Need some basic access control-- a declaration of the methods
exported for invocation by the server.
"""
import asyncore
import codecs
import itertools
import logging
import os
import sys
import tempfile
import threading
import time
import transaction
import warnings
from .zrpc.error import DisconnectedError
import ZODB.blob
import ZODB.event
import ZODB.serialize
import ZODB.TimeStamp
import zope.interface
import six
from ZEO._compat import Pickler, Unpickler, PY3, BytesIO
from ZEO.Exceptions import AuthError
from .monitor import StorageStats, StatsServer
from .zrpc.connection import ManagedServerConnection, Delay, MTDelay, Result
from .zrpc.server import Dispatcher
from ZODB.loglevels import BLATHER
from ZODB.POSException import StorageError, StorageTransactionError
from ZODB.POSException import TransactionError, ReadOnlyError, ConflictError
from ZODB.serialize import referencesf
from ZODB.utils import oid_repr, p64, u64, z64
ResolvedSerial = b'rs'
logger = logging.getLogger('ZEO.StorageServer')
def log(message, level=logging.INFO, label='', exc_info=False):
"""Internal helper to log a message."""
if label:
message = "(%s) %s" % (label, message)
logger.log(level, message, exc_info=exc_info)
class StorageServerError(StorageError):
"""Error reported when an unpicklable exception is raised."""
class ZEOStorage:
"""Proxy to underlying storage for a single remote client."""
# A list of extension methods. A subclass with extra methods
# should override.
extensions = []
def __init__(self, server, read_only=0, auth_realm=None):
self.server = server
# timeout and stats will be initialized in register()
self.stats = None
self.connection = None
self.client = None
self.storage = None
self.storage_id = "uninitialized"
self.transaction = None
self.read_only = read_only
self.log_label = 'unconnected'
self.locked = False # Don't have storage lock
self.verifying = 0
self.store_failed = 0
self.authenticated = 0
self.auth_realm = auth_realm
self.blob_tempfile = None
# The authentication protocol may define extra methods.
self._extensions = {}
for func in self.extensions:
self._extensions[func.__name__] = None
self._iterators = {}
self._iterator_ids = itertools.count()
# Stores the last item that was handed out for a
# transaction iterator.
self._txn_iterators_last = {}
def _finish_auth(self, authenticated):
if not self.auth_realm:
return 1
self.authenticated = authenticated
return authenticated
def set_database(self, database):
self.database = database
def notifyConnected(self, conn):
self.connection = conn
assert conn.peer_protocol_version is not None
if conn.peer_protocol_version < b'Z309':
self.client = ClientStub308(conn)
conn.register_object(ZEOStorage308Adapter(self))
else:
self.client = ClientStub(conn)
self.log_label = _addr_label(conn.addr)
def notifyDisconnected(self):
# When this storage closes, we must ensure that it aborts
# any pending transaction.
if self.transaction is not None:
self.log("disconnected during %s transaction"
% (self.locked and 'locked' or 'unlocked'))
self.tpc_abort(self.transaction.id)
else:
self.log("disconnected")
self.connection = None
def __repr__(self):
tid = self.transaction and repr(self.transaction.id)
if self.storage:
stid = (self.tpc_transaction() and
repr(self.tpc_transaction().id))
else:
stid = None
name = self.__class__.__name__
return "<%s %X trans=%s s_trans=%s>" % (name, id(self), tid, stid)
def log(self, msg, level=logging.INFO, exc_info=False):
log(msg, level=level, label=self.log_label, exc_info=exc_info)
def setup_delegation(self):
"""Delegate several methods to the storage
"""
# Called from register
storage = self.storage
info = self.get_info()
if not info['supportsUndo']:
self.undoLog = self.undoInfo = lambda *a,**k: ()
self.getTid = storage.getTid
self.load = storage.load
self.loadSerial = storage.loadSerial
record_iternext = getattr(storage, 'record_iternext', None)
if record_iternext is not None:
self.record_iternext = record_iternext
try:
fn = storage.getExtensionMethods
except AttributeError:
pass # no extension methods
else:
d = fn()
self._extensions.update(d)
for name in d:
assert not hasattr(self, name)
setattr(self, name, getattr(storage, name))
self.lastTransaction = storage.lastTransaction
try:
self.tpc_transaction = storage.tpc_transaction
except AttributeError:
if hasattr(storage, '_transaction'):
log("Storage %r doesn't have a tpc_transaction method.\n"
"See ZEO.interfaces.IServeable."
"Falling back to using _transaction attribute, which\n."
"is icky.",
logging.ERROR)
self.tpc_transaction = lambda : storage._transaction
else:
raise
def history(self,tid,size=1):
# This caters for storages which still accept
# a version parameter.
return self.storage.history(tid,size=size)
def _check_tid(self, tid, exc=None):
if self.read_only:
raise ReadOnlyError()
if self.transaction is None:
caller = sys._getframe().f_back.f_code.co_name
self.log("no current transaction: %s()" % caller,
level=logging.WARNING)
if exc is not None:
raise exc(None, tid)
else:
return 0
if self.transaction.id != tid:
caller = sys._getframe().f_back.f_code.co_name
self.log("%s(%s) invalid; current transaction = %s" %
(caller, repr(tid), repr(self.transaction.id)),
logging.WARNING)
if exc is not None:
raise exc(self.transaction.id, tid)
else:
return 0
return 1
def getAuthProtocol(self):
"""Return string specifying name of authentication module to use.
The module name should be auth_%s where %s is auth_protocol."""
protocol = self.server.auth_protocol
if not protocol or protocol == 'none':
return None
return protocol
def register(self, storage_id, read_only):
"""Select the storage that this client will use
This method must be the first one called by the client.
For authenticated storages this method will be called by the client
immediately after authentication is finished.
"""
if self.auth_realm and not self.authenticated:
raise AuthError("Client was never authenticated with server!")
if self.storage is not None:
self.log("duplicate register() call")
raise ValueError("duplicate register() call")
storage = self.server.storages.get(storage_id)
if storage is None:
self.log("unknown storage_id: %s" % storage_id)
raise ValueError("unknown storage: %s" % storage_id)
if not read_only and (self.read_only or storage.isReadOnly()):
raise ReadOnlyError()
self.read_only = self.read_only or read_only
self.storage_id = storage_id
self.storage = storage
self.setup_delegation()
self.stats = self.server.register_connection(storage_id, self)
def get_info(self):
storage = self.storage
supportsUndo = (getattr(storage, 'supportsUndo', lambda : False)()
and self.connection.peer_protocol_version >= b'Z310')
# Communicate the backend storage interfaces to the client
storage_provides = zope.interface.providedBy(storage)
interfaces = []
for candidate in storage_provides.__iro__:
interfaces.append((candidate.__module__, candidate.__name__))
return {'length': len(storage),
'size': storage.getSize(),
'name': storage.getName(),
'supportsUndo': supportsUndo,
'extensionMethods': self.getExtensionMethods(),
'supports_record_iternext': hasattr(self, 'record_iternext'),
'interfaces': tuple(interfaces),
}
def get_size_info(self):
return {'length': len(self.storage),
'size': self.storage.getSize(),
}
def getExtensionMethods(self):
return self._extensions
def loadEx(self, oid):
self.stats.loads += 1
return self.storage.load(oid, '')
def loadBefore(self, oid, tid):
self.stats.loads += 1
return self.storage.loadBefore(oid, tid)
def getInvalidations(self, tid):
invtid, invlist = self.server.get_invalidations(self.storage_id, tid)
if invtid is None:
return None
self.log("Return %d invalidations up to tid %s"
% (len(invlist), u64(invtid)))
return invtid, invlist
def verify(self, oid, tid):
try:
t = self.getTid(oid)
except KeyError:
self.client.invalidateVerify(oid)
else:
if tid != t:
self.client.invalidateVerify(oid)
def zeoVerify(self, oid, s):
if not self.verifying:
self.verifying = 1
self.stats.verifying_clients += 1
try:
os = self.getTid(oid)
except KeyError:
self.client.invalidateVerify((oid, ''))
# It's not clear what we should do now. The KeyError
# could be caused by an object uncreation, in which case
# invalidation is right. It could be an application bug
# that left a dangling reference, in which case it's bad.
else:
if s != os:
self.client.invalidateVerify((oid, ''))
def endZeoVerify(self):
if self.verifying:
self.stats.verifying_clients -= 1
self.verifying = 0
self.client.endVerify()
def pack(self, time, wait=1):
# Yes, you can pack a read-only server or storage!
if wait:
return run_in_thread(self._pack_impl, time)
else:
# If the client isn't waiting for a reply, start a thread
# and forget about it.
t = threading.Thread(target=self._pack_impl, args=(time,))
t.setName("zeo storage packing thread")
t.start()
return None
def _pack_impl(self, time):
self.log("pack(time=%s) started..." % repr(time))
self.storage.pack(time, referencesf)
self.log("pack(time=%s) complete" % repr(time))
# Broadcast new size statistics
self.server.invalidate(0, self.storage_id, None,
(), self.get_size_info())
def new_oids(self, n=100):
"""Return a sequence of n new oids, where n defaults to 100"""
n = min(n, 100)
if self.read_only:
raise ReadOnlyError()
if n <= 0:
n = 1
return [self.storage.new_oid() for i in range(n)]
# undoLog and undoInfo are potentially slow methods
def undoInfo(self, first, last, spec):
return run_in_thread(self.storage.undoInfo, first, last, spec)
def undoLog(self, first, last):
return run_in_thread(self.storage.undoLog, first, last)
def tpc_begin(self, id, user, description, ext, tid=None, status=" "):
if self.read_only:
raise ReadOnlyError()
if self.transaction is not None:
if self.transaction.id == id:
self.log("duplicate tpc_begin(%s)" % repr(id))
return
else:
raise StorageTransactionError("Multiple simultaneous tpc_begin"
" requests from one client.")
t = transaction.Transaction()
t.id = id
t.user = user
t.description = description
t._extension = ext
self.serials = []
self.invalidated = []
self.txnlog = CommitLog()
self.blob_log = []
self.tid = tid
self.status = status
self.store_failed = 0
self.stats.active_txns += 1
# Assign the transaction attribute last. This is so we don't
# think we've entered TPC until everything is set. Why?
# Because if we have an error after this, the server will
# think it is in TPC and the client will think it isn't. At
# that point, the client will keep trying to enter TPC and
# server won't let it. Errors *after* the tpc_begin call will
# cause the client to abort the transaction.
# (Also see https://bugs.launchpad.net/zodb/+bug/374737.)
self.transaction = t
def tpc_finish(self, id):
if not self._check_tid(id):
return
assert self.locked, "finished called wo lock"
self.stats.commits += 1
self.storage.tpc_finish(self.transaction, self._invalidate)
# Note that the tid is still current because we still hold the
# commit lock. We'll relinquish it in _clear_transaction.
tid = self.storage.lastTransaction()
# Return the tid, for cache invalidation optimization
return Result(tid, self._clear_transaction)
def _invalidate(self, tid):
if self.invalidated:
self.server.invalidate(self, self.storage_id, tid,
self.invalidated, self.get_size_info())
def tpc_abort(self, tid):
if not self._check_tid(tid):
return
self.stats.aborts += 1
self.storage.tpc_abort(self.transaction)
self._clear_transaction()
def _clear_transaction(self):
# Common code at end of tpc_finish() and tpc_abort()
if self.locked:
self.server.unlock_storage(self)
self.locked = 0
if self.transaction is not None:
self.server.stop_waiting(self)
self.transaction = None
self.stats.active_txns -= 1
if self.txnlog is not None:
self.txnlog.close()
self.txnlog = None
for oid, oldserial, data, blobfilename in self.blob_log:
ZODB.blob.remove_committed(blobfilename)
del self.blob_log
def vote(self, tid):
self._check_tid(tid, exc=StorageTransactionError)
if self.locked or self.server.already_waiting(self):
raise StorageTransactionError(
'Already voting (%s)' % (self.locked and 'locked' or 'waiting')
)
return self._try_to_vote()
def _try_to_vote(self, delay=None):
if self.connection is None:
return # We're disconnected
if delay is not None and delay.sent:
# as a consequence of the unlocking strategy, _try_to_vote
# may be called multiple times for delayed
# transactions. The first call will mark the delay as
# sent. We should skip if the delay was already sent.
return
self.locked, delay = self.server.lock_storage(self, delay)
if self.locked:
try:
self.log(
"Preparing to commit transaction: %d objects, %d bytes"
% (self.txnlog.stores, self.txnlog.size()),
level=BLATHER)
if (self.tid is not None) or (self.status != ' '):
self.storage.tpc_begin(self.transaction,
self.tid, self.status)
else:
self.storage.tpc_begin(self.transaction)
for op, args in self.txnlog:
if not getattr(self, op)(*args):
break
# Blob support
while self.blob_log and not self.store_failed:
oid, oldserial, data, blobfilename = self.blob_log.pop()
self._store(oid, oldserial, data, blobfilename)
if not self.store_failed:
# Only call tpc_vote of no store call failed,
# otherwise the serialnos() call will deliver an
# exception that will be handled by the client in
# its tpc_vote() method.
serials = self.storage.tpc_vote(self.transaction)
if serials:
self.serials.extend(serials)
self.client.serialnos(self.serials)
except Exception:
self.storage.tpc_abort(self.transaction)
self._clear_transaction()
if delay is not None:
delay.error(sys.exc_info())
else:
raise
else:
if delay is not None:
delay.reply(None)
else:
return None
else:
return delay
def _unlock_callback(self, delay):
connection = self.connection
if connection is None:
self.server.stop_waiting(self)
else:
connection.call_from_thread(self._try_to_vote, delay)
# The public methods of the ZEO client API do not do the real work.
# They defer work until after the storage lock has been acquired.
# Most of the real implementations are in methods beginning with
# an _.
def deleteObject(self, oid, serial, id):
self._check_tid(id, exc=StorageTransactionError)
self.stats.stores += 1
self.txnlog.delete(oid, serial)
def storea(self, oid, serial, data, id):
self._check_tid(id, exc=StorageTransactionError)
self.stats.stores += 1
self.txnlog.store(oid, serial, data)
def checkCurrentSerialInTransaction(self, oid, serial, id):
self._check_tid(id, exc=StorageTransactionError)
self.txnlog.checkread(oid, serial)
def restorea(self, oid, serial, data, prev_txn, id):
self._check_tid(id, exc=StorageTransactionError)
self.stats.stores += 1
self.txnlog.restore(oid, serial, data, prev_txn)
def storeBlobStart(self):
assert self.blob_tempfile is None
self.blob_tempfile = tempfile.mkstemp(
dir=self.storage.temporaryDirectory())
def storeBlobChunk(self, chunk):
os.write(self.blob_tempfile[0], chunk)
def storeBlobEnd(self, oid, serial, data, id):
self._check_tid(id, exc=StorageTransactionError)
assert self.txnlog is not None # effectively not allowed after undo
fd, tempname = self.blob_tempfile
self.blob_tempfile = None
os.close(fd)
self.blob_log.append((oid, serial, data, tempname))
def storeBlobShared(self, oid, serial, data, filename, id):
self._check_tid(id, exc=StorageTransactionError)
assert self.txnlog is not None # effectively not allowed after undo
# Reconstruct the full path from the filename in the OID directory
if (os.path.sep in filename
or not (filename.endswith('.tmp')
or filename[:-1].endswith('.tmp')
)
):
logger.critical(
"We're under attack! (bad filename to storeBlobShared, %r)",
filename)
raise ValueError(filename)
filename = os.path.join(self.storage.fshelper.getPathForOID(oid),
filename)
self.blob_log.append((oid, serial, data, filename))
def sendBlob(self, oid, serial):
self.client.storeBlob(oid, serial, self.storage.loadBlob(oid, serial))
def undo(*a, **k):
raise NotImplementedError
def undoa(self, trans_id, tid):
self._check_tid(tid, exc=StorageTransactionError)
self.txnlog.undo(trans_id)
def _op_error(self, oid, err, op):
self.store_failed = 1
if isinstance(err, ConflictError):
self.stats.conflicts += 1
self.log("conflict error oid=%s msg=%s" %
(oid_repr(oid), str(err)), BLATHER)
if not isinstance(err, TransactionError):
# Unexpected errors are logged and passed to the client
self.log("%s error: %s, %s" % ((op,)+ sys.exc_info()[:2]),
logging.ERROR, exc_info=True)
err = self._marshal_error(err)
# The exception is reported back as newserial for this oid
self.serials.append((oid, err))
def _delete(self, oid, serial):
err = None
try:
self.storage.deleteObject(oid, serial, self.transaction)
except (SystemExit, KeyboardInterrupt):
raise
except Exception as e:
err = e
self._op_error(oid, err, 'delete')
return err is None
def _checkread(self, oid, serial):
err = None
try:
self.storage.checkCurrentSerialInTransaction(
oid, serial, self.transaction)
except (SystemExit, KeyboardInterrupt):
raise
except Exception as e:
err = e
self._op_error(oid, err, 'checkCurrentSerialInTransaction')
return err is None
def _store(self, oid, serial, data, blobfile=None):
err = None
try:
if blobfile is None:
newserial = self.storage.store(
oid, serial, data, '', self.transaction)
else:
newserial = self.storage.storeBlob(
oid, serial, data, blobfile, '', self.transaction)
except (SystemExit, KeyboardInterrupt):
raise
except Exception as error:
self._op_error(oid, error, 'store')
err = error
else:
if serial != b"\0\0\0\0\0\0\0\0":
self.invalidated.append(oid)
if isinstance(newserial, bytes):
newserial = [(oid, newserial)]
for oid, s in newserial or ():
if s == ResolvedSerial:
self.stats.conflicts_resolved += 1
self.log("conflict resolved oid=%s"
% oid_repr(oid), BLATHER)
self.serials.append((oid, s))
return err is None
def _restore(self, oid, serial, data, prev_txn):
err = None
try:
self.storage.restore(oid, serial, data, '', prev_txn,
self.transaction)
except (SystemExit, KeyboardInterrupt):
raise
except Exception as err:
self._op_error(oid, err, 'restore')
return err is None
def _undo(self, trans_id):
err = None
try:
tid, oids = self.storage.undo(trans_id, self.transaction)
except (SystemExit, KeyboardInterrupt):
raise
except Exception as e:
err = e
self._op_error(z64, err, 'undo')
else:
self.invalidated.extend(oids)
self.serials.extend((oid, ResolvedSerial) for oid in oids)
return err is None
def _marshal_error(self, error):
# Try to pickle the exception. If it can't be pickled,
# the RPC response would fail, so use something that can be pickled.
if PY3:
pickler = Pickler(BytesIO(), 3)
else:
# The pure-python version requires at least one argument (PyPy)
pickler = Pickler(0)
pickler.fast = 1
try:
pickler.dump(error)
except:
msg = "Couldn't pickle storage exception: %s" % repr(error)
self.log(msg, logging.ERROR)
error = StorageServerError(msg)
return error
# IStorageIteration support
def iterator_start(self, start, stop):
iid = next(self._iterator_ids)
self._iterators[iid] = iter(self.storage.iterator(start, stop))
return iid
def iterator_next(self, iid):
iterator = self._iterators[iid]
try:
info = next(iterator)
except StopIteration:
del self._iterators[iid]
item = None
if iid in self._txn_iterators_last:
del self._txn_iterators_last[iid]
else:
item = (info.tid,
info.status,
info.user,
info.description,
info.extension)
# Keep a reference to the last iterator result to allow starting a
# record iterator off it.
self._txn_iterators_last[iid] = info
return item
def iterator_record_start(self, txn_iid, tid):
record_iid = next(self._iterator_ids)
txn_info = self._txn_iterators_last[txn_iid]
if txn_info.tid != tid:
raise Exception(
'Out-of-order request for record iterator for transaction %r'
% tid)
self._iterators[record_iid] = iter(txn_info)
return record_iid
def iterator_record_next(self, iid):
iterator = self._iterators[iid]
try:
info = next(iterator)
except StopIteration:
del self._iterators[iid]
item = None
else:
item = (info.oid,
info.tid,
info.data,
info.data_txn)
return item
def iterator_gc(self, iids):
for iid in iids:
self._iterators.pop(iid, None)
def server_status(self):
return self.server.server_status(self.storage_id)
def set_client_label(self, label):
self.log_label = str(label)+' '+_addr_label(self.connection.addr)
class StorageServerDB:
def __init__(self, server, storage_id):
self.server = server
self.storage_id = storage_id
self.references = ZODB.serialize.referencesf
def invalidate(self, tid, oids, version=''):
if version:
raise StorageServerError("Versions aren't supported.")
storage_id = self.storage_id
self.server.invalidate(None, storage_id, tid, oids)
def invalidateCache(self):
self.server._invalidateCache(self.storage_id)
transform_record_data = untransform_record_data = lambda self, data: data
class StorageServer:
"""The server side implementation of ZEO.
The StorageServer is the 'manager' for incoming connections. Each
connection is associated with its own ZEOStorage instance (defined
below). The StorageServer may handle multiple storages; each
ZEOStorage instance only handles a single storage.
"""
# Classes we instantiate. A subclass might override.
from .zrpc.server import Dispatcher as DispatcherClass
ZEOStorageClass = ZEOStorage
ManagedServerConnectionClass = ManagedServerConnection
def __init__(self, addr, storages,
read_only=0,
invalidation_queue_size=100,
invalidation_age=None,
transaction_timeout=None,
monitor_address=None,
auth_protocol=None,
auth_database=None,
auth_realm=None,
):
"""StorageServer constructor.
This is typically invoked from the start.py script.
Arguments (the first two are required and positional):
addr -- the address at which the server should listen. This
can be a tuple (host, port) to signify a TCP/IP connection
or a pathname string to signify a Unix domain socket
connection. A hostname may be a DNS name or a dotted IP
address.
storages -- a dictionary giving the storage(s) to handle. The
keys are the storage names, the values are the storage
instances, typically FileStorage or Berkeley storage
instances. By convention, storage names are typically
strings representing small integers starting at '1'.
read_only -- an optional flag saying whether the server should
operate in read-only mode. Defaults to false. Note that
even if the server is operating in writable mode,
individual storages may still be read-only. But if the
server is in read-only mode, no write operations are
allowed, even if the storages are writable. Note that
pack() is considered a read-only operation.
invalidation_queue_size -- The storage server keeps a queue
of the objects modified by the last N transactions, where
N == invalidation_queue_size. This queue is used to
speed client cache verification when a client disconnects
for a short period of time.
invalidation_age --
If the invalidation queue isn't big enough to support a
quick verification, but the last transaction seen by a
client is younger than the invalidation age, then
invalidations will be computed by iterating over
transactions later than the given transaction.
transaction_timeout -- The maximum amount of time to wait for
a transaction to commit after acquiring the storage lock.
If the transaction takes too long, the client connection
will be closed and the transaction aborted.
monitor_address -- The address at which the monitor server
should listen. If specified, a monitor server is started.
The monitor server provides server statistics in a simple
text format.
auth_protocol -- The name of the authentication protocol to use.
Examples are "digest" and "srp".
auth_database -- The name of the password database filename.
It should be in a format compatible with the authentication
protocol used; for instance, "sha" and "srp" require different
formats.
Note that to implement an authentication protocol, a server
and client authentication mechanism must be implemented in a
auth_* module, which should be stored inside the "auth"
subdirectory. This module may also define a DatabaseClass
variable that should indicate what database should be used
by the authenticator.
"""
self.addr = addr
self.storages = storages
msg = ", ".join(
["%s:%s:%s" % (name, storage.isReadOnly() and "RO" or "RW",
storage.getName())
for name, storage in storages.items()])
log("%s created %s with storages: %s" %
(self.__class__.__name__, read_only and "RO" or "RW", msg))
self._lock = threading.Lock()
self._commit_locks = {}
self._waiting = dict((name, []) for name in storages)
self.read_only = read_only
self.auth_protocol = auth_protocol
self.auth_database = auth_database
self.auth_realm = auth_realm
self.database = None
if auth_protocol:
self._setup_auth(auth_protocol)
# A list, by server, of at most invalidation_queue_size invalidations.
# The list is kept in sorted order with the most recent
# invalidation at the front. The list never has more than
# self.invq_bound elements.
self.invq_bound = invalidation_queue_size
self.invq = {}
for name, storage in storages.items():
self._setup_invq(name, storage)
storage.registerDB(StorageServerDB(self, name))
self.invalidation_age = invalidation_age
self.connections = {}
self.socket_map = {}
self.dispatcher = self.DispatcherClass(
addr, factory=self.new_connection, map=self.socket_map)
if len(self.addr) == 2 and self.addr[1] == 0 and self.addr[0]:
self.addr = self.dispatcher.socket.getsockname()
ZODB.event.notify(
Serving(self, address=self.dispatcher.socket.getsockname()))
self.stats = {}
self.timeouts = {}
for name in self.storages.keys():
self.connections[name] = []
self.stats[name] = StorageStats(self.connections[name])
if transaction_timeout is None:
# An object with no-op methods
timeout = StubTimeoutThread()
else:
timeout = TimeoutThread(transaction_timeout)
timeout.setName("TimeoutThread for %s" % name)
timeout.start()
self.timeouts[name] = timeout
if monitor_address:
warnings.warn(
"The monitor server is deprecated. Use the server_status\n"
"ZEO method instead.",
DeprecationWarning)
self.monitor = StatsServer(monitor_address, self.stats)
else:
self.monitor = None
def _setup_invq(self, name, storage):
lastInvalidations = getattr(storage, 'lastInvalidations', None)
if lastInvalidations is None:
# Using None below doesn't look right, but the first
# element in invq is never used. See get_invalidations.
# (If it was used, it would generate an error, which would
# be good. :) Doing this allows clients that were up to
# date when a server was restarted to pick up transactions
# it subsequently missed.
self.invq[name] = [(storage.lastTransaction() or z64, None)]
else:
self.invq[name] = list(lastInvalidations(self.invq_bound))
self.invq[name].reverse()
def _setup_auth(self, protocol):
# Can't be done in global scope, because of cyclic references
from .auth import get_module
name = self.__class__.__name__
module = get_module(protocol)
if not module:
log("%s: no such an auth protocol: %s" % (name, protocol))
return
storage_class, client, db_class = module
if not storage_class or not issubclass(storage_class, ZEOStorage):
log(("%s: %s isn't a valid protocol, must have a StorageClass" %
(name, protocol)))
self.auth_protocol = None
return
self.ZEOStorageClass = storage_class
log("%s: using auth protocol: %s" % (name, protocol))
# We create a Database instance here for use with the authenticator
# modules. Having one instance allows it to be shared between multiple
# storages, avoiding the need to bloat each with a new authenticator
# Database that would contain the same info, and also avoiding any
# possibly synchronization issues between them.
self.database = db_class(self.auth_database)
if self.database.realm != self.auth_realm:
raise ValueError("password database realm %r "
"does not match storage realm %r"
% (self.database.realm, self.auth_realm))
def new_connection(self, sock, addr):
"""Internal: factory to create a new connection.
This is called by the Dispatcher class in ZEO.zrpc.server
whenever accept() returns a socket for a new incoming
connection.
"""
if self.auth_protocol and self.database:
zstorage = self.ZEOStorageClass(self, self.read_only,
auth_realm=self.auth_realm)
zstorage.set_database(self.database)
else:
zstorage = self.ZEOStorageClass(self, self.read_only)
c = self.ManagedServerConnectionClass(sock, addr, zstorage, self)
log("new connection %s: %s" % (addr, repr(c)), logging.DEBUG)
return c
def register_connection(self, storage_id, conn):
"""Internal: register a connection with a particular storage.
This is called by ZEOStorage.register().
The dictionary self.connections maps each storage name to a
list of current connections for that storage; this information
is needed to handle invalidation. This function updates this
dictionary.
Returns the timeout and stats objects for the appropriate storage.
"""
self.connections[storage_id].append(conn)
return self.stats[storage_id]
def _invalidateCache(self, storage_id):
"""We need to invalidate any caches we have.
This basically means telling our clients to
invalidate/revalidate their caches. We do this by closing them
and making them reconnect.
"""
# This method can be called from foreign threads. We have to
# worry about interaction with the main thread.
# 1. We modify self.invq which is read by get_invalidations
# below. This is why get_invalidations makes a copy of
# self.invq.
# 2. We access connections. There are two dangers:
#
# a. We miss a new connection. This is not a problem because
# if a client connects after we get the list of connections,
# then it will have to read the invalidation queue, which
# has already been reset.
#
# b. A connection is closes while we are iterating. This
# doesn't matter, bacause we can call should_close on a closed
# connection.
# Rebuild invq
self._setup_invq(storage_id, self.storages[storage_id])
# Make a copy since we are going to be mutating the
# connections indirectoy by closing them. We don't care about
# later transactions since they will have to validate their
# caches anyway.
for p in self.connections[storage_id][:]:
try:
p.connection.should_close()
p.connection.trigger.pull_trigger()
except DisconnectedError:
pass
def invalidate(self, conn, storage_id, tid, invalidated=(), info=None):
"""Internal: broadcast info and invalidations to clients.
This is called from several ZEOStorage methods.
invalidated is a sequence of oids.
This can do three different things:
- If the invalidated argument is non-empty, it broadcasts
invalidateTransaction() messages to all clients of the given
storage except the current client (the conn argument).
- If the invalidated argument is empty and the info argument
is a non-empty dictionary, it broadcasts info() messages to
all clients of the given storage, including the current
client.
- If both the invalidated argument and the info argument are
non-empty, it broadcasts invalidateTransaction() messages to all
clients except the current, and sends an info() message to
the current client.
"""
# This method can be called from foreign threads. We have to
# worry about interaction with the main thread.
# 1. We modify self.invq which is read by get_invalidations
# below. This is why get_invalidations makes a copy of
# self.invq.
# 2. We access connections. There are two dangers:
#
# a. We miss a new connection. This is not a problem because
# we are called while the storage lock is held. A new
# connection that tries to read data won't read committed
# data without first recieving an invalidation. Also, if a
# client connects after getting the list of connections,
# then it will have to read the invalidation queue, which
# has been updated to reflect the invalidations.
#
# b. A connection is closes while we are iterating. We'll need
# to cactch and ignore Disconnected errors.
if invalidated:
invq = self.invq[storage_id]
if len(invq) >= self.invq_bound:
invq.pop()
invq.insert(0, (tid, invalidated))
for p in self.connections[storage_id]:
try:
if invalidated and p is not conn:
p.client.invalidateTransaction(tid, invalidated)
elif info is not None:
p.client.info(info)
except DisconnectedError:
pass
def get_invalidations(self, storage_id, tid):
"""Return a tid and list of all objects invalidation since tid.
The tid is the most recent transaction id seen by the client.
Returns None if it is unable to provide a complete list
of invalidations for tid. In this case, client should
do full cache verification.
"""
# We make a copy of invq because it might be modified by a
# foreign (other than main thread) calling invalidate above.
invq = self.invq[storage_id][:]
oids = set()
latest_tid = None
if invq and invq[-1][0] <= tid:
# We have needed data in the queue
for _tid, L in invq:
if _tid <= tid:
break
oids.update(L)
latest_tid = invq[0][0]
elif (self.invalidation_age and
(self.invalidation_age >
(time.time()-ZODB.TimeStamp.TimeStamp(tid).timeTime())
)
):
for t in self.storages[storage_id].iterator(p64(u64(tid)+1)):
for r in t:
oids.add(r.oid)
latest_tid = t.tid
elif not invq:
log("invq empty")
else:
log("tid to old for invq %s < %s" % (u64(tid), u64(invq[-1][0])))
return latest_tid, list(oids)
def loop(self, timeout=30):
try:
asyncore.loop(timeout, map=self.socket_map)
except Exception:
if not self.__closed:
raise # Unexpected exc
__thread = None
def start_thread(self, daemon=True):
self.__thread = thread = threading.Thread(target=self.loop)
thread.setName("StorageServer(%s)" % _addr_label(self.addr))
thread.setDaemon(daemon)
thread.start()
__closed = False
def close(self, join_timeout=1):
"""Close the dispatcher so that there are no new connections.
This is only called from the test suite, AFAICT.
"""
if self.__closed:
return
self.__closed = True
# Stop accepting connections
self.dispatcher.close()
if self.monitor is not None:
self.monitor.close()
ZODB.event.notify(Closed(self))
# Close open client connections
for sid, connections in self.connections.items():
for conn in connections[:]:
try:
conn.connection.close()
except:
pass
for name, storage in six.iteritems(self.storages):
logger.info("closing storage %r", name)
storage.close()
if self.__thread is not None:
self.__thread.join(join_timeout)
def close_conn(self, conn):
"""Internal: remove the given connection from self.connections.
This is the inverse of register_connection().
"""
for cl in self.connections.values():
if conn.obj in cl:
cl.remove(conn.obj)
def lock_storage(self, zeostore, delay):
storage_id = zeostore.storage_id
waiting = self._waiting[storage_id]
with self._lock:
if storage_id in self._commit_locks:
# The lock is held by another zeostore
locked = self._commit_locks[storage_id]
assert locked is not zeostore, (storage_id, delay)
if locked.connection is None:
locked.log("Still locked after disconnected. Unlocking.",
logging.CRITICAL)
if locked.transaction:
locked.storage.tpc_abort(locked.transaction)
del self._commit_locks[storage_id]
# yuck: have to manipulate lock to appease with :(
self._lock.release()
try:
return self.lock_storage(zeostore, delay)
finally:
self._lock.acquire()
if delay is None:
# New request, queue it
assert not [i for i in waiting if i[0] is zeostore
], "already waiting"
delay = Delay()
waiting.append((zeostore, delay))
zeostore.log("(%r) queue lock: transactions waiting: %s"
% (storage_id, len(waiting)),
_level_for_waiting(waiting)
)
return False, delay
else:
self._commit_locks[storage_id] = zeostore
self.timeouts[storage_id].begin(zeostore)
self.stats[storage_id].lock_time = time.time()
if delay is not None:
# we were waiting, stop
waiting[:] = [i for i in waiting if i[0] is not zeostore]
zeostore.log("(%r) lock: transactions waiting: %s"
% (storage_id, len(waiting)),
_level_for_waiting(waiting)
)
return True, delay
def unlock_storage(self, zeostore):
storage_id = zeostore.storage_id
waiting = self._waiting[storage_id]
with self._lock:
assert self._commit_locks[storage_id] is zeostore
del self._commit_locks[storage_id]
self.timeouts[storage_id].end(zeostore)
self.stats[storage_id].lock_time = None
callbacks = waiting[:]
if callbacks:
assert not [i for i in waiting if i[0] is zeostore
], "waiting while unlocking"
zeostore.log("(%r) unlock: transactions waiting: %s"
% (storage_id, len(callbacks)),
_level_for_waiting(callbacks)
)
for zeostore, delay in callbacks:
try:
zeostore._unlock_callback(delay)
except (SystemExit, KeyboardInterrupt):
raise
except Exception:
logger.exception("Calling unlock callback")
def stop_waiting(self, zeostore):
storage_id = zeostore.storage_id
waiting = self._waiting[storage_id]
with self._lock:
new_waiting = [i for i in waiting if i[0] is not zeostore]
if len(new_waiting) == len(waiting):
return
waiting[:] = new_waiting
zeostore.log("(%r) dequeue lock: transactions waiting: %s"
% (storage_id, len(waiting)),
_level_for_waiting(waiting)
)
def already_waiting(self, zeostore):
storage_id = zeostore.storage_id
waiting = self._waiting[storage_id]
with self._lock:
return bool([i for i in waiting if i[0] is zeostore])
def server_status(self, storage_id):
status = self.stats[storage_id].__dict__.copy()
status['connections'] = len(status['connections'])
status['waiting'] = len(self._waiting[storage_id])
status['timeout-thread-is-alive'] = self.timeouts[storage_id].isAlive()
last_transaction = self.storages[storage_id].lastTransaction()
last_transaction_hex = codecs.encode(last_transaction, 'hex_codec')
if PY3:
# doctests and maybe clients expect a str, not bytes
last_transaction_hex = str(last_transaction_hex, 'ascii')
status['last-transaction'] = last_transaction_hex
return status
def ruok(self):
return dict((storage_id, self.server_status(storage_id))
for storage_id in self.storages)
def _level_for_waiting(waiting):
if len(waiting) > 9:
return logging.CRITICAL
if len(waiting) > 3:
return logging.WARNING
else:
return logging.DEBUG
class StubTimeoutThread:
def begin(self, client):
pass
def end(self, client):
pass
isAlive = lambda self: 'stub'
class TimeoutThread(threading.Thread):
"""Monitors transaction progress and generates timeouts."""
# There is one TimeoutThread per storage, because there's one
# transaction lock per storage.
def __init__(self, timeout):
threading.Thread.__init__(self)
self.setName("TimeoutThread")
self.setDaemon(1)
self._timeout = timeout
self._client = None
self._deadline = None
self._cond = threading.Condition() # Protects _client and _deadline
def begin(self, client):
# Called from the restart code the "main" thread, whenever the
# storage lock is being acquired. (Serialized by asyncore.)
with self._cond:
assert self._client is None
self._client = client
self._deadline = time.time() + self._timeout
self._cond.notify()
def end(self, client):
# Called from the "main" thread whenever the storage lock is
# being released. (Serialized by asyncore.)
with self._cond:
assert self._client is not None
assert self._client is client
self._client = None
self._deadline = None
def run(self):
# Code running in the thread.
while 1:
with self._cond:
while self._deadline is None:
self._cond.wait()
howlong = self._deadline - time.time()
if howlong <= 0:
# Prevent reporting timeout more than once
self._deadline = None
client = self._client # For the howlong <= 0 branch below
if howlong <= 0:
client.log("Transaction timeout after %s seconds" %
self._timeout, logging.CRITICAL)
try:
client.connection.call_from_thread(client.connection.close)
except:
client.log("Timeout failure", logging.CRITICAL,
exc_info=sys.exc_info())
self.end(client)
else:
time.sleep(howlong)
def run_in_thread(method, *args):
t = SlowMethodThread(method, args)
t.start()
return t.delay
class SlowMethodThread(threading.Thread):
"""Thread to run potentially slow storage methods.
Clients can use the delay attribute to access the MTDelay object
used to send a zrpc response at the right time.
"""
# Some storage methods can take a long time to complete. If we
# run these methods via a standard asyncore read handler, they
# will block all other server activity until they complete. To
# avoid blocking, we spawn a separate thread, return an MTDelay()
# object, and have the thread reply() when it finishes.
def __init__(self, method, args):
threading.Thread.__init__(self)
self.setName("SlowMethodThread for %s" % method.__name__)
self._method = method
self._args = args
self.delay = MTDelay()
def run(self):
try:
result = self._method(*self._args)
except (SystemExit, KeyboardInterrupt):
raise
except Exception:
self.delay.error(sys.exc_info())
else:
self.delay.reply(result)
class ClientStub:
def __init__(self, rpc):
self.rpc = rpc
def beginVerify(self):
self.rpc.callAsync('beginVerify')
def invalidateVerify(self, args):
self.rpc.callAsync('invalidateVerify', args)
def endVerify(self):
self.rpc.callAsync('endVerify')
def invalidateTransaction(self, tid, args):
# Note that this method is *always* called from a different
# thread than self.rpc's async thread. It is the only method
# for which this is true and requires special consideration!
# callAsyncNoSend is important here because:
# - callAsyncNoPoll isn't appropriate because
# the network thread may not wake up for a long time,
# delaying invalidations for too long. (This is demonstrateed
# by a test failure.)
# - callAsync isn't appropriate because (on the server) it tries
# to write to the socket. If self.rpc's network thread also
# tries to write at the ame time, we can run into problems
# because handle_write isn't thread safe.
self.rpc.callAsyncNoSend('invalidateTransaction', tid, args)
def serialnos(self, arg):
self.rpc.callAsyncNoPoll('serialnos', arg)
def info(self, arg):
self.rpc.callAsyncNoPoll('info', arg)
def storeBlob(self, oid, serial, blobfilename):
def store():
yield ('receiveBlobStart', (oid, serial))
f = open(blobfilename, 'rb')
while 1:
chunk = f.read(59000)
if not chunk:
break
yield ('receiveBlobChunk', (oid, serial, chunk, ))
f.close()
yield ('receiveBlobStop', (oid, serial))
self.rpc.callAsyncIterator(store())
class ClientStub308(ClientStub):
def invalidateTransaction(self, tid, args):
ClientStub.invalidateTransaction(
self, tid, [(arg, '') for arg in args])
def invalidateVerify(self, oid):
ClientStub.invalidateVerify(self, (oid, ''))
class ZEOStorage308Adapter:
def __init__(self, storage):
self.storage = storage
def __eq__(self, other):
return self is other or self.storage is other
def getSerial(self, oid):
return self.storage.loadEx(oid)[1] # Z200
def history(self, oid, version, size=1):
if version:
raise ValueError("Versions aren't supported.")
return self.storage.history(oid, size=size)
def getInvalidations(self, tid):
result = self.storage.getInvalidations(tid)
if result is not None:
result = result[0], [(oid, '') for oid in result[1]]
return result
def verify(self, oid, version, tid):
if version:
raise StorageServerError("Versions aren't supported.")
return self.storage.verify(oid, tid)
def loadEx(self, oid, version=''):
if version:
raise StorageServerError("Versions aren't supported.")
data, serial = self.storage.loadEx(oid)
return data, serial, ''
def storea(self, oid, serial, data, version, id):
if version:
raise StorageServerError("Versions aren't supported.")
self.storage.storea(oid, serial, data, id)
def storeBlobEnd(self, oid, serial, data, version, id):
if version:
raise StorageServerError("Versions aren't supported.")
self.storage.storeBlobEnd(oid, serial, data, id)
def storeBlobShared(self, oid, serial, data, filename, version, id):
if version:
raise StorageServerError("Versions aren't supported.")
self.storage.storeBlobShared(oid, serial, data, filename, id)
def getInfo(self):
result = self.storage.getInfo()
result['supportsVersions'] = False
return result
def zeoVerify(self, oid, s, sv=None):
if sv:
raise StorageServerError("Versions aren't supported.")
self.storage.zeoVerify(oid, s)
def modifiedInVersion(self, oid):
return ''
def versions(self):
return ()
def versionEmpty(self, version):
return True
def commitVersion(self, *a, **k):
raise NotImplementedError
abortVersion = commitVersion
def zeoLoad(self, oid): # Z200
p, s = self.storage.loadEx(oid)
return p, s, '', None, None
def __getattr__(self, name):
return getattr(self.storage, name)
def _addr_label(addr):
if isinstance(addr, six.binary_type):
return addr.decode('ascii')
if isinstance(addr, six.string_types):
return addr
else:
host, port = addr
return str(host) + ":" + str(port)
class CommitLog:
def __init__(self):
self.file = tempfile.TemporaryFile(suffix=".comit-log")
self.pickler = Pickler(self.file, 1)
self.pickler.fast = 1
self.stores = 0
def size(self):
return self.file.tell()
def delete(self, oid, serial):
self.pickler.dump(('_delete', (oid, serial)))
self.stores += 1
def checkread(self, oid, serial):
self.pickler.dump(('_checkread', (oid, serial)))
self.stores += 1
def store(self, oid, serial, data):
self.pickler.dump(('_store', (oid, serial, data)))
self.stores += 1
def restore(self, oid, serial, data, prev_txn):
self.pickler.dump(('_restore', (oid, serial, data, prev_txn)))
self.stores += 1
def undo(self, transaction_id):
self.pickler.dump(('_undo', (transaction_id, )))
self.stores += 1
def __iter__(self):
self.file.seek(0)
unpickler = Unpickler(self.file)
for i in range(self.stores):
yield unpickler.load()
def close(self):
if self.file:
self.file.close()
self.file = None
class ServerEvent:
def __init__(self, server, **kw):
self.__dict__.update(kw)
self.server = server
class Serving(ServerEvent):
pass
class Closed(ServerEvent):
pass
src/ZEO/tests/ZEO4/component.xml 0 → 100644
View file @ c499ea7d
<component>
<sectiontype name="zeo">
<description>
The content of a ZEO section describe operational parameters
of a ZEO server except for the storage(s) to be served.
</description>
<key name="address" datatype="socket-binding-address"
required="yes">
<description>
The address at which the server should listen. This can be in
the form 'host:port' to signify a TCP/IP connection or a
pathname string to signify a Unix domain socket connection (at
least one '/' is required). A hostname may be a DNS name or a
dotted IP address. If the hostname is omitted, the platform's
default behavior is used when binding the listening socket (''
is passed to socket.bind() as the hostname portion of the
address).
</description>
</key>
<key name="read-only" datatype="boolean"
required="no"
default="false">
<description>
Flag indicating whether the server should operate in read-only
mode. Defaults to false. Note that even if the server is
operating in writable mode, individual storages may still be
read-only. But if the server is in read-only mode, no write
operations are allowed, even if the storages are writable. Note
that pack() is considered a read-only operation.
</description>
</key>
<key name="invalidation-queue-size" datatype="integer"
required="no"
default="100">
<description>
The storage server keeps a queue of the objects modified by the
last N transactions, where N == invalidation_queue_size. This
queue is used to speed client cache verification when a client
disconnects for a short period of time.
</description>
</key>
<key name="invalidation-age" datatype="float" required="no">
<description>
The maximum age of a client for which quick-verification
invalidations will be provided by iterating over the served
storage. This option should only be used if the served storage
supports efficient iteration from a starting point near the
end of the transaction history (e.g. end of file).
</description>
</key>
<key name="monitor-address" datatype="socket-binding-address"
required="no">
<description>
The address at which the monitor server should listen. If
specified, a monitor server is started. The monitor server
provides server statistics in a simple text format. This can
be in the form 'host:port' to signify a TCP/IP connection or a
pathname string to signify a Unix domain socket connection (at
least one '/' is required). A hostname may be a DNS name or a
dotted IP address. If the hostname is omitted, the platform's
default behavior is used when binding the listening socket (''
is passed to socket.bind() as the hostname portion of the
address).
</description>
</key>
<key name="transaction-timeout" datatype="integer"
required="no">
<description>
The maximum amount of time to wait for a transaction to commit
after acquiring the storage lock, specified in seconds. If the
transaction takes too long, the client connection will be closed
and the transaction aborted.
</description>
</key>
<key name="authentication-protocol" required="no">
<description>
The name of the protocol used for authentication. The
only protocol provided with ZEO is "digest," but extensions
may provide other protocols.
</description>
</key>
<key name="authentication-database" required="no">
<description>
The path of the database containing authentication credentials.
</description>
</key>
<key name="authentication-realm" required="no">
<description>
The authentication realm of the server. Some authentication
schemes use a realm to identify the logical set of usernames
that are accepted by this server.
</description>
</key>
<key name="pid-filename" datatype="existing-dirpath"
required="no">
<description>
The full path to the file in which to write the ZEO server's Process ID
at startup. If omitted, $INSTANCE/var/ZEO.pid is used.
</description>
<metadefault>$INSTANCE/var/ZEO.pid (or $clienthome/ZEO.pid)</metadefault>
</key>
<!-- DM 2006-06-12: added option -->
<key name="drop-cache-rather-verify" datatype="boolean"
required="no" default="false">
<description>
indicates that the cache should be dropped rather than
verified when the verification optimization is not
available (e.g. when the ZEO server restarted).
</description>
</key>
</sectiontype>
</component>
src/ZEO/tests/ZEO4/hash.py 0 → 100644
View file @ c499ea7d
##############################################################################
#
# Copyright (c) 2008 Zope Foundation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""In Python 2.6, the "sha" and "md5" modules have been deprecated
in favor of using hashlib for both. This class allows for compatibility
between versions."""
try:
import hashlib
sha1 = hashlib.sha1
new = sha1
except ImportError:
import sha
sha1 = sha.new
new = sha1
digest_size = sha.digest_size
src/ZEO/tests/ZEO4/monitor.py 0 → 100644
View file @ c499ea7d
##############################################################################
#
# Copyright (c) 2003 Zope Foundation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE
#
##############################################################################
"""Monitor behavior of ZEO server and record statistics.
"""
from __future__ import print_function
from __future__ import print_function
from __future__ import print_function
from __future__ import print_function
from __future__ import print_function
from __future__ import print_function
from __future__ import print_function
from __future__ import print_function
from __future__ import print_function
from __future__ import print_function
from __future__ import print_function
from __future__ import print_function
from __future__ import print_function
from __future__ import print_function
from __future__ import print_function
from __future__ import print_function
import asyncore
import socket
import time
import logging
zeo_version = 'unknown'
try:
import pkg_resources
except ImportError:
pass
else:
zeo_dist = pkg_resources.working_set.find(
pkg_resources.Requirement.parse('ZODB3')
)
if zeo_dist is not None:
zeo_version = zeo_dist.version
class StorageStats:
"""Per-storage usage statistics."""
def __init__(self, connections=None):
self.connections = connections
self.loads = 0
self.stores = 0
self.commits = 0
self.aborts = 0
self.active_txns = 0
self.verifying_clients = 0
self.lock_time = None
self.conflicts = 0
self.conflicts_resolved = 0
self.start = time.ctime()
@property
def clients(self):
return len(self.connections)
def parse(self, s):
# parse the dump format
lines = s.split("\n")
for line in lines:
field, value = line.split(":", 1)
if field == "Server started":
self.start = value
elif field == "Clients":
# Hack because we use this both on the server and on
# the client where there are no connections.
self.connections = [0] * int(value)
elif field == "Clients verifying":
self.verifying_clients = int(value)
elif field == "Active transactions":
self.active_txns = int(value)
elif field == "Commit lock held for":
# This assumes
self.lock_time = time.time() - int(value)
elif field == "Commits":
self.commits = int(value)
elif field == "Aborts":
self.aborts = int(value)
elif field == "Loads":
self.loads = int(value)
elif field == "Stores":
self.stores = int(value)
elif field == "Conflicts":
self.conflicts = int(value)
elif field == "Conflicts resolved":
self.conflicts_resolved = int(value)
def dump(self, f):
print("Server started:", self.start, file=f)
print("Clients:", self.clients, file=f)
print("Clients verifying:", self.verifying_clients, file=f)
print("Active transactions:", self.active_txns, file=f)
if self.lock_time:
howlong = time.time() - self.lock_time
print("Commit lock held for:", int(howlong), file=f)
print("Commits:", self.commits, file=f)
print("Aborts:", self.aborts, file=f)
print("Loads:", self.loads, file=f)
print("Stores:", self.stores, file=f)
print("Conflicts:", self.conflicts, file=f)
print("Conflicts resolved:", self.conflicts_resolved, file=f)
class StatsClient(asyncore.dispatcher):
def __init__(self, sock, addr):
asyncore.dispatcher.__init__(self, sock)
self.buf = []
self.closed = 0
def close(self):
self.closed = 1
# The socket is closed after all the data is written.
# See handle_write().
def write(self, s):
self.buf.append(s)
def writable(self):
return len(self.buf)
def readable(self):
return 0
def handle_write(self):
s = "".join(self.buf)
self.buf = []
n = self.socket.send(s.encode('ascii'))
if n < len(s):
self.buf.append(s[:n])
if self.closed and not self.buf:
asyncore.dispatcher.close(self)
class StatsServer(asyncore.dispatcher):
StatsConnectionClass = StatsClient
def __init__(self, addr, stats):
asyncore.dispatcher.__init__(self)
self.addr = addr
self.stats = stats
if type(self.addr) == tuple:
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
else:
self.create_socket(socket.AF_UNIX, socket.SOCK_STREAM)
self.set_reuse_addr()
logger = logging.getLogger('ZEO.monitor')
logger.info("listening on %s", repr(self.addr))
self.bind(self.addr)
self.listen(5)
def writable(self):
return 0
def readable(self):
return 1
def handle_accept(self):
try:
sock, addr = self.accept()
except socket.error:
return
f = self.StatsConnectionClass(sock, addr)
self.dump(f)
f.close()
def dump(self, f):
print("ZEO monitor server version %s" % zeo_version, file=f)
print(time.ctime(), file=f)
print(file=f)
L = sorted(self.stats.keys())
for k in L:
stats = self.stats[k]
print("Storage:", k, file=f)
stats.dump(f)
print(file=f)
src/ZEO/tests/ZEO4/runzeo.py 0 → 100644
View file @ c499ea7d
##############################################################################
#
# Copyright (c) 2001, 2002, 2003 Zope Foundation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE
#
##############################################################################
"""Start the ZEO storage server.
Usage: %s [-C URL] [-a ADDRESS] [-f FILENAME] [-h]
Options:
-C/--configuration URL -- configuration file or URL
-a/--address ADDRESS -- server address of the form PORT, HOST:PORT, or PATH
(a PATH must contain at least one "/")
-f/--filename FILENAME -- filename for FileStorage
-t/--timeout TIMEOUT -- transaction timeout in seconds (default no timeout)
-h/--help -- print this usage message and exit
-m/--monitor ADDRESS -- address of monitor server ([HOST:]PORT or PATH)
--pid-file PATH -- relative path to output file containing this process's pid;
default $(INSTANCE_HOME)/var/ZEO.pid but only if envar
INSTANCE_HOME is defined
Unless -C is specified, -a and -f are required.
"""
from __future__ import print_function
from __future__ import print_function
# The code here is designed to be reused by other, similar servers.
# For the forseeable future, it must work under Python 2.1 as well as
# 2.2 and above.
import asyncore
import os
import sys
import signal
import socket
import logging
import ZConfig.datatypes
from zdaemon.zdoptions import ZDOptions
logger = logging.getLogger('ZEO.runzeo')
_pid = str(os.getpid())
def log(msg, level=logging.INFO, exc_info=False):
"""Internal: generic logging function."""
message = "(%s) %s" % (_pid, msg)
logger.log(level, message, exc_info=exc_info)
def parse_binding_address(arg):
# Caution: Not part of the official ZConfig API.
obj = ZConfig.datatypes.SocketBindingAddress(arg)
return obj.family, obj.address
def windows_shutdown_handler():
# Called by the signal mechanism on Windows to perform shutdown.
import asyncore
asyncore.close_all()
class ZEOOptionsMixin:
storages = None
def handle_address(self, arg):
self.family, self.address = parse_binding_address(arg)
def handle_monitor_address(self, arg):
self.monitor_family, self.monitor_address = parse_binding_address(arg)
def handle_filename(self, arg):
from ZODB.config import FileStorage # That's a FileStorage *opener*!
class FSConfig:
def __init__(self, name, path):
self._name = name
self.path = path
self.stop = None
def getSectionName(self):
return self._name
if not self.storages:
self.storages = []
name = str(1 + len(self.storages))
conf = FileStorage(FSConfig(name, arg))
self.storages.append(conf)
testing_exit_immediately = False
def handle_test(self, *args):
self.testing_exit_immediately = True
def add_zeo_options(self):
self.add(None, None, None, "test", self.handle_test)
self.add(None, None, "a:", "address=", self.handle_address)
self.add(None, None, "f:", "filename=", self.handle_filename)
self.add("family", "zeo.address.family")
self.add("address", "zeo.address.address",
required="no server address specified; use -a or -C")
self.add("read_only", "zeo.read_only", default=0)
self.add("invalidation_queue_size", "zeo.invalidation_queue_size",
default=100)
self.add("invalidation_age", "zeo.invalidation_age")
self.add("transaction_timeout", "zeo.transaction_timeout",
"t:", "timeout=", float)
self.add("monitor_address", "zeo.monitor_address.address",
"m:", "monitor=", self.handle_monitor_address)
self.add('auth_protocol', 'zeo.authentication_protocol',
None, 'auth-protocol=', default=None)
self.add('auth_database', 'zeo.authentication_database',
None, 'auth-database=')
self.add('auth_realm', 'zeo.authentication_realm',
None, 'auth-realm=')
self.add('pid_file', 'zeo.pid_filename',
None, 'pid-file=')
class ZEOOptions(ZDOptions, ZEOOptionsMixin):
__doc__ = __doc__
logsectionname = "eventlog"
schemadir = os.path.dirname(__file__)
def __init__(self):
ZDOptions.__init__(self)
self.add_zeo_options()
self.add("storages", "storages",
required="no storages specified; use -f or -C")
def realize(self, *a, **k):
ZDOptions.realize(self, *a, **k)
nunnamed = [s for s in self.storages if s.name is None]
if nunnamed:
if len(nunnamed) > 1:
return self.usage("No more than one storage may be unnamed.")
if [s for s in self.storages if s.name == '1']:
return self.usage(
"Can't have an unnamed storage and a storage named 1.")
for s in self.storages:
if s.name is None:
s.name = '1'
break
class ZEOServer:
def __init__(self, options):
self.options = options
def main(self):
self.setup_default_logging()
self.check_socket()
self.clear_socket()
self.make_pidfile()
try:
self.open_storages()
self.setup_signals()
self.create_server()
self.loop_forever()
finally:
self.server.close()
self.clear_socket()
self.remove_pidfile()
def setup_default_logging(self):
if self.options.config_logger is not None:
return
# No log file is configured; default to stderr.
root = logging.getLogger()
root.setLevel(logging.INFO)
fmt = logging.Formatter(
"------\n%(asctime)s %(levelname)s %(name)s %(message)s",
"%Y-%m-%dT%H:%M:%S")
handler = logging.StreamHandler()
handler.setFormatter(fmt)
root.addHandler(handler)
def check_socket(self):
if (isinstance(self.options.address, tuple) and
self.options.address[1] is None):
self.options.address = self.options.address[0], 0
return
if self.can_connect(self.options.family, self.options.address):
self.options.usage("address %s already in use" %
repr(self.options.address))
def can_connect(self, family, address):
s = socket.socket(family, socket.SOCK_STREAM)
try:
s.connect(address)
except socket.error:
return 0
else:
s.close()
return 1
def clear_socket(self):
if isinstance(self.options.address, type("")):
try:
os.unlink(self.options.address)
except os.error:
pass
def open_storages(self):
self.storages = {}
for opener in self.options.storages:
log("opening storage %r using %s"
% (opener.name, opener.__class__.__name__))
self.storages[opener.name] = opener.open()
def setup_signals(self):
"""Set up signal handlers.
The signal handler for SIGFOO is a method handle_sigfoo().
If no handler method is defined for a signal, the signal
action is not changed from its initial value. The handler
method is called without additional arguments.
"""
if os.name != "posix":
if os.name == "nt":
self.setup_win32_signals()
return
if hasattr(signal, 'SIGXFSZ'):
signal.signal(signal.SIGXFSZ, signal.SIG_IGN) # Special case
init_signames()
for sig, name in signames.items():
method = getattr(self, "handle_" + name.lower(), None)
if method is not None:
def wrapper(sig_dummy, frame_dummy, method=method):
method()
signal.signal(sig, wrapper)
def setup_win32_signals(self):
# Borrow the Zope Signals package win32 support, if available.
# Signals does a check/log for the availability of pywin32.
try:
import Signals.Signals
except ImportError:
logger.debug("Signals package not found. "
"Windows-specific signal handler "
"will *not* be installed.")
return
SignalHandler = Signals.Signals.SignalHandler
if SignalHandler is not None: # may be None if no pywin32.
SignalHandler.registerHandler(signal.SIGTERM,
windows_shutdown_handler)
SignalHandler.registerHandler(signal.SIGINT,
windows_shutdown_handler)
SIGUSR2 = 12 # not in signal module on Windows.
SignalHandler.registerHandler(SIGUSR2, self.handle_sigusr2)
def create_server(self):
self.server = create_server(self.storages, self.options)
def loop_forever(self):
if self.options.testing_exit_immediately:
print("testing exit immediately")
else:
self.server.loop()
def handle_sigterm(self):
log("terminated by SIGTERM")
sys.exit(0)
def handle_sigint(self):
log("terminated by SIGINT")
sys.exit(0)
def handle_sighup(self):
log("restarted by SIGHUP")
sys.exit(1)
def handle_sigusr2(self):
# log rotation signal - do the same as Zope 2.7/2.8...
if self.options.config_logger is None or os.name not in ("posix", "nt"):
log("received SIGUSR2, but it was not handled!",
level=logging.WARNING)
return
loggers = [self.options.config_logger]
if os.name == "posix":
for l in loggers:
l.reopen()
log("Log files reopened successfully", level=logging.INFO)
else: # nt - same rotation code as in Zope's Signals/Signals.py
for l in loggers:
for f in l.handler_factories:
handler = f()
if hasattr(handler, 'rotate') and callable(handler.rotate):
handler.rotate()
log("Log files rotation complete", level=logging.INFO)
def _get_pidfile(self):
pidfile = self.options.pid_file
# 'pidfile' is marked as not required.
if not pidfile:
# Try to find a reasonable location if the pidfile is not
# set. If we are running in a Zope environment, we can
# safely assume INSTANCE_HOME.
instance_home = os.environ.get("INSTANCE_HOME")
if not instance_home:
# If all our attempts failed, just log a message and
# proceed.
logger.debug("'pidfile' option not set, and 'INSTANCE_HOME' "
"environment variable could not be found. "
"Cannot guess pidfile location.")
return
self.options.pid_file = os.path.join(instance_home,
"var", "ZEO.pid")
def make_pidfile(self):
if not self.options.read_only:
self._get_pidfile()
pidfile = self.options.pid_file
if pidfile is None:
return
pid = os.getpid()
try:
if os.path.exists(pidfile):
os.unlink(pidfile)
f = open(pidfile, 'w')
print(pid, file=f)
f.close()
log("created PID file '%s'" % pidfile)
except IOError:
logger.error("PID file '%s' cannot be opened" % pidfile)
def remove_pidfile(self):
if not self.options.read_only:
pidfile = self.options.pid_file
if pidfile is None:
return
try:
if os.path.exists(pidfile):
os.unlink(pidfile)
log("removed PID file '%s'" % pidfile)
except IOError:
logger.error("PID file '%s' could not be removed" % pidfile)
def create_server(storages, options):
from .StorageServer import StorageServer
return StorageServer(
options.address,
storages,
read_only = options.read_only,
invalidation_queue_size = options.invalidation_queue_size,
invalidation_age = options.invalidation_age,
transaction_timeout = options.transaction_timeout,
monitor_address = options.monitor_address,
auth_protocol = options.auth_protocol,
auth_database = options.auth_database,
auth_realm = options.auth_realm,
)
# Signal names
signames = None
def signame(sig):
"""Return a symbolic name for a signal.
Return "signal NNN" if there is no corresponding SIG name in the
signal module.
"""
if signames is None:
init_signames()
return signames.get(sig) or "signal %d" % sig
def init_signames():
global signames
signames = {}
for name, sig in signal.__dict__.items():
k_startswith = getattr(name, "startswith", None)
if k_startswith is None:
continue
if k_startswith("SIG") and not k_startswith("SIG_"):
signames[sig] = name
# Main program
def main(args=None):
options = ZEOOptions()
options.realize(args)
s = ZEOServer(options)
s.main()
if __name__ == "__main__":
main()
src/ZEO/tests/ZEO4/schema.xml 0 → 100644
View file @ c499ea7d
<schema>
<!-- note that zeoctl.xml is a closely related schema which should
match this schema, but should require the "runner" section -->
<description>
This schema describes the configuration of the ZEO storage server
process.
</description>
<!-- Use the storage types defined by ZODB. -->
<import package="ZODB"/>
<!-- Use the ZEO server information structure. -->
<import package="ZEO.tests.ZEO4"/>
<import package="ZConfig.components.logger"/>
<!-- runner control -->
<import package="zdaemon"/>
<section type="zeo" name="*" required="yes" attribute="zeo" />
<section type="runner" name="*" required="no" attribute="runner" />
<multisection name="*" type="ZODB.storage"
attribute="storages"
required="yes">
<description>
One or more storages that are provided by the ZEO server. The
section names are used as the storage names, and must be unique
within each ZEO storage server. Traditionally, these names
represent small integers starting at '1'.
</description>
</multisection>
<section name="*" type="eventlog" attribute="eventlog" required="no" />
</schema>
src/ZEO/tests/auth_plaintext.py deleted 100644 → 0
View file @ bc92bdd7
##############################################################################
#
# Copyright (c) 2003 Zope Foundation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE
#
##############################################################################
"""Implements plaintext password authentication. The password is stored in
an SHA hash in the Database. The client sends over the plaintext
password, and the SHA hashing is done on the server side.
This mechanism offers *no network security at all*; the only security
is provided by not storing plaintext passwords on disk.
"""
from ZEO.hash import sha1
from ZEO.StorageServer import ZEOStorage
from ZEO.auth import register_module
from ZEO.auth.base import Client, Database
def session_key(username, realm, password):
key = "%s:%s:%s" % (username, realm, password)
return sha1(key.encode('utf-8')).hexdigest().encode('ascii')
class StorageClass(ZEOStorage):
def auth(self, username, password):
try:
dbpw = self.database.get_password(username)
except LookupError:
return 0
password_dig = sha1(password.encode('utf-8')).hexdigest()
if dbpw == password_dig:
self.connection.setSessionKey(session_key(username,
self.database.realm,
password))
return self._finish_auth(dbpw == password_dig)
class PlaintextClient(Client):
extensions = ["auth"]
def start(self, username, realm, password):
if self.stub.auth(username, password):
return session_key(username, realm, password)
else:
return None
register_module("plaintext", StorageClass, PlaintextClient, Database)
src/ZEO/tests/client-config.test
View file @ c499ea7d
......@@ -18,7 +18,7 @@ The simplest client configuration specified a server address:
>>> storage.getName(), storage.__class__.__name__
... # doctest: +ELLIPSIS
("[('localhost', ...)] (connected)", 'ClientStorage')
("[('127.0.0.1', ...)] (connected)", 'ClientStorage')
>>> storage.blob_dir
>>> storage._storage
......@@ -26,16 +26,10 @@ The simplest client configuration specified a server address:
>>> storage._cache.maxsize
20971520
>>> storage._cache.path
>>> storage._rpc_mgr.tmin
5
>>> storage._rpc_mgr.tmax
300
>>> storage._is_read_only
False
>>> storage._read_only_fallback
False
>>> storage._drop_cache_rather_verify
False
>>> storage._blob_cache_size
>>> storage.close()
......@@ -48,8 +42,6 @@ The simplest client configuration specified a server address:
... cache-size 100
... name bob
... client cache
... min-disconnect-poll 1
... max-disconnect-poll 5
... read-only true
... drop-cache-rather-verify true
... blob-cache-size 1000MB
......@@ -72,16 +64,10 @@ The simplest client configuration specified a server address:
>>> storage._cache.path == os.path.abspath('cache-2.zec')
True
>>> storage._rpc_mgr.tmin
1
>>> storage._rpc_mgr.tmax
5
>>> storage._is_read_only
True
>>> storage._read_only_fallback
False
>>> storage._drop_cache_rather_verify
True
>>> storage._blob_cache_size
1048576000
......
src/ZEO/tests/client.pem 0 → 100644
View file @ c499ea7d
-----BEGIN CERTIFICATE-----
MIID/TCCAuWgAwIBAgIJAJWOSC4oLyp9MA0GCSqGSIb3DQEBBQUAMFwxCzAJBgNV
BAYTAlVTMQswCQYDVQQIEwJWQTENMAsGA1UEChMEWk9EQjERMA8GA1UEAxMIem9k
Yi5vcmcxHjAcBgkqhkiG9w0BCQEWD2NsaWVudEB6b2RiLm9yZzAeFw0xNjA2MjMx
NTA3MTNaFw0xNzA2MjMxNTA3MTNaMFwxCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJW
QTENMAsGA1UEChMEWk9EQjERMA8GA1UEAxMIem9kYi5vcmcxHjAcBgkqhkiG9w0B
CQEWD2NsaWVudEB6b2RiLm9yZzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBANqtLqtrzwDv0nakKiTX5ZtSHOaTmfwHzHsZJkcNf7kJUgGtE0oe3cIj4iAC
CCpPfUf9OfIA1NpmDsvPMqw80ho1o9g8QZWfW6QOTj2kbnanrRqMDmRvrWlzMQIe
+7EabYxbTjyduk76N2Sa4Tf/my6Yton3zyKtdjSJzoQ/SAKT+Nvjt5s47I4THEFY
gN7Njbg1FyihKwuwR64EUsyBanutKvXVT7gnB1V2cQhn+LW+NwgzsxKnptvyvBGr
Ago+uoxrHlIQu59xznS2vA3Ck2K3hIOnXpXYYGeRYKzplZLZnCfZ2uQheiO3ioEO
UCXICbPMxA7IEpTC75j1n9a3HKcCAwEAAaOBwTCBvjAdBgNVHQ4EFgQUCg1EhkHz
qYFOgbI/D6zR1tcwHBcwgY4GA1UdIwSBhjCBg4AUCg1EhkHzqYFOgbI/D6zR1tcw
HBehYKReMFwxCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJWQTENMAsGA1UEChMEWk9E
QjERMA8GA1UEAxMIem9kYi5vcmcxHjAcBgkqhkiG9w0BCQEWD2NsaWVudEB6b2Ri
Lm9yZ4IJAJWOSC4oLyp9MAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADggEB
AJk3TviNGmjzCuDzLajeLEB9Iy285Fl3D/3TvThSDECS3cUq+i8fQm0cjOYnE/rK
6Lpg6soeQFetoWQsNT2uvxyv3iZEOtBwNhQKaKTkzIRTmMx/aWM0zG0c3psepC6c
1Fgp0HAts2JKC4Ni7zHFBDb8YZi87IUHYNKuJKcUWKiEgeHu2zCI1Q43FMSKoaG8
XwYi1Mxw6TQQtjZrMnNPSeO7zySBuCw10bCZSMC5xvsqckfREifRT//4A0/COWYK
/p6TZMTaMjrK8fOaPpap314QnLf80P6oLEZ7wkghaNuyq8IzgATuxYVy21132MNB
qZIUS+iblQAZDSHnQJoehQQ=
-----END CERTIFICATE-----
src/ZEO/tests/client_key.pem 0 → 100644
View file @ c499ea7d
-----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEA2q0uq2vPAO/SdqQqJNflm1Ic5pOZ/AfMexkmRw1/uQlSAa0T
Sh7dwiPiIAIIKk99R/058gDU2mYOy88yrDzSGjWj2DxBlZ9bpA5OPaRudqetGowO
ZG+taXMxAh77sRptjFtOPJ26Tvo3ZJrhN/+bLpi2iffPIq12NInOhD9IApP42+O3
mzjsjhMcQViA3s2NuDUXKKErC7BHrgRSzIFqe60q9dVPuCcHVXZxCGf4tb43CDOz
Eqem2/K8EasCCj66jGseUhC7n3HOdLa8DcKTYreEg6deldhgZ5FgrOmVktmcJ9na
5CF6I7eKgQ5QJcgJs8zEDsgSlMLvmPWf1rccpwIDAQABAoIBAQDSju7hIG2x+Tou
AuSRlVEAvZAWdQlQJDJAVXcF83mIMfFEq+Jm/FGLHgIdz9cM5n07VBj3bNWHdb3J
gTjJn8audffNvjdoWoli7mNn92xl1A5aAYHaM65GWyRVZn/zh/7zpvcuZrF+Wm/7
7yXtRbGmrGUXdAV+3odzDz5LGKO91fTuM2nW0j+p7+q2Bzko7+rl9AVxaveco4pt
TtqXX2eOC3wTcNotBfJJD89/+/szg62K4CYCUAaetKMPcVrgQ4v0YHakOl8lJJxW
q7XiqhPyjeZp6h8e9dtVCkeZHa3xacuoftF2w3FslVEX/LOooAKf07PU1xxJezZN
trP11pgBAoGBAPoysqD9OW3C0WyIXm/Lx1Pncr/p/kKYlXNDWQRmizHmfLC4xrBI
n75mYSp7BJ7Gh2W7lW9lnyJ0uaqXdqOOswO/baTeSEvvhL+lLs4F1cQWemjTD4qy
KqCxCCbTf8gZPssiJXsXGmf6yWAdpjfYUxarxB0Ks6wHLBpQHctvVebJAoGBAN+/
W0+yAXssr7AWgKAyiHtqE/MiI8e0bDJWDjHxaPvnSVyDwVOARj/JKvw7jnhg8FKJ
1iWtl8ym2ktcAQygSl3HW4wggD59BbeXrUQr4nZi8wLMTpLxnSfITFrwXpwVShT4
8KJPR46W/Plkphm1fZn6Lr0uGJ12pV+iPAq/F+/vAoGALmRoKuHJXEjbfDxtBl3K
wAwSgvNoagDQ9WZvgxlghggu5rXcYaOVu0BQlAfre2VkhcCanOVC9KigJLmhDgLP
vsooEoIE9c+b1c1TOHBsiseAOx+nqhgPP2yUDl75Oqkzs4bJXGGUS+N8o43b3E8I
WRPQcXIijqtlyhtA6w/h5cECgYEA2M6DnGXQKZrTYr1rRc+xkGTpj9607P5XGS9p
8dsK74zd+VdyLYdOiuBTVrYfB2ZneJM3fqsHPLcxL3SnT6TCarySaOXVXremoo/G
xRgBCNY4w61VNe4JalMcKcJg6r12W3wdMCnCHNkRqFdu29qRKnLSd14DXBFrjY+W
vpMMjuECgYEAtNOm9WTCAlVbGTMplXcUJBWNwdOz+sHMi+PhP/fTNm0XH3fepFeQ
th7b/RYocTwAHGu8mArX2DgXAfDQAYMPFN8vmbh0XQsQV+iuja9MO2TagTyVSe3x
DBOTFCLg8oWtzwySZ6BKR/KsGI2ryRxyE1VojV0zNXRyZqDCm+G4Vs0=
-----END RSA PRIVATE KEY-----
src/ZEO/tests/drop_cache_rather_than_verify.txt
View file @ c499ea7d
......@@ -2,17 +2,14 @@ Avoiding cache verifification
=============================
For large databases it is common to also use very large ZEO cache
files. If a client has beed disconnected for too long, cache verification
might be necessary, but cache verification can be very hard on the
storage server.
files. If a client has beed disconnected for too long, the server
can't play back missing invalidations. In this case, the cache is
cleared. When this happens, a ZEO.interfaces.StaleCache event is
published, largely for backward compatibility.
When verification is needed, a ZEO.interfaces.StaleCache event is
published. Applications may handle this event to perform actions such
as exiting the process to avoid a cold restart.
ClientStorage provides an option to drop it's cache rather than doing
verification. When this option is used, and verification would be
necessary, after publishing the event, ClientStorage:
ClientStorage used to provide an option to drop it's cache rather than
doing verification. This is now the only behavior. Cache
verification is no longer supported.
- Invalidates all object caches
......@@ -23,12 +20,11 @@ necessary, after publishing the event, ClientStorage:
Here's an example that shows that this is actually what happens.
Start a server, create a cient to it and commit some data
Start a server, create a client to it and commit some data
>>> addr, admin = start_server(keep=1)
>>> import ZEO, transaction
>>> db = ZEO.DB(addr, drop_cache_rather_verify=True, client='cache',
... name='test')
>>> db = ZEO.DB(addr, client='cache', name='test')
>>> wait_connected(db.storage)
>>> conn = db.open()
>>> conn.root()[1] = conn.root().__class__()
......@@ -58,11 +54,12 @@ logging and event data:
>>> import logging, zope.testing.loggingsupport, ZODB.event
>>> handler = zope.testing.loggingsupport.InstalledHandler(
... 'ZEO.ClientStorage', level=logging.ERROR)
... 'ZEO', level=logging.ERROR)
>>> events = []
>>> def event_handler(e):
... if hasattr(e, 'storage'):
... events.append((
... len(e.storage._cache), str(handler), e.__class__.__name__))
... len(e.storage._server.client.cache), str(handler), e.__class__.__name__))
>>> old_notify = ZODB.event.notify
>>> ZODB.event.notify = event_handler
......@@ -75,6 +72,7 @@ Now, we'll restart the server on the original address:
>>> _, admin = start_server(zeo_conf=dict(invalidation_queue_size=1),
... addr=addr, keep=1)
>>> wait_connected(db.storage)
Now, let's verify our assertions above:
......@@ -105,8 +103,8 @@ Now, let's verify our assertions above:
- Logs a CRITICAL message.
>>> print(handler)
ZEO.ClientStorage CRITICAL
>>> print(handler) # doctest: +ELLIPSIS
ZEO... CRITICAL
test dropping stale cache
>>> handler.clear()
......@@ -156,8 +154,8 @@ in the database, which is why we get 1, rather than 0 objects in the cache.)
- Logs a CRITICAL message.
>>> print(handler)
ZEO.ClientStorage CRITICAL
>>> print(handler) # doctest: +ELLIPSIS
ZEO... CRITICAL
test dropping stale cache
>>> handler.clear()
......@@ -168,49 +166,6 @@ If we access the root object, it'll be loaded from the server:
>>> conn.root()[1].x
11
Finally, let's look at what happens without the
drop_cache_rather_verify option:
>>> db.close()
>>> db = ZEO.DB(addr, client='cache')
>>> wait_connected(db.storage)
>>> conn = db.open()
>>> conn.root()[1].x
11
>>> conn.root()[2] = conn.root().__class__()
>>> transaction.commit()
>>> len(db.storage._cache)
4
>>> stop_server(admin)
>>> addr2, admin = start_server(keep=1)
>>> db2 = ZEO.DB(addr2)
>>> wait_connected(db2.storage)
>>> conn2 = db2.open()
>>> for i in range(5):
... conn2.root()[1].x += 1
... transaction.commit()
>>> db2.close()
>>> stop_server(admin)
>>> _, admin = start_server(zeo_conf=dict(invalidation_queue_size=1),
... addr=addr)
>>> wait_connected(db.storage)
>>> for e in events:
... print(e)
(4, '', 'StaleCache')
>>> print(handler)
<BLANKLINE>
>>> len(db.storage._cache)
3
Here we see the cache wasn't dropped, although one of the records was
invalidated during verification.
.. Cleanup
>>> db.close()
......
src/ZEO/tests/dynamic_server_ports.test
View file @ c499ea7d
......@@ -14,67 +14,46 @@ ZODB.notify.
Now, let's start a server and verify that we get a serving event:
>>> import ZEO.StorageServer, ZODB.MappingStorage
>>> server = ZEO.StorageServer.StorageServer(
... ('127.0.0.1', 0), {'1': ZODB.MappingStorage.MappingStorage()})
>>> import ZEO
>>> addr, stop = ZEO.server()
>>> isinstance(last_event, ZEO.StorageServer.Serving)
True
>>> last_event.server is server
True
>>> last_event.address[0], last_event.address[1] > 0
('127.0.0.1', True)
If the host part pf the address passed to the constructor is not an
empty string. then the server addr attribute is the same as the
address attribute of the event:
>>> server.addr == last_event.address
>>> last_event.address == addr
True
Let's run the server in a thread, to make sure we can connect.
>>> server = last_event.server
>>> server.addr == addr
True
>>> server.start_thread()
Let's make sure we can connect.
>>> client = ZEO.client(last_event.address)
>>> client.is_connected()
True
>>> client = ZEO.client(last_event.address).close()
If we close the server, we'll get a closed event:
>>> server.close()
>>> stop()
>>> isinstance(last_event, ZEO.StorageServer.Closed)
True
>>> last_event.server is server
True
>>> wait_until(lambda : not client.is_connected(test=True))
>>> client.close()
If we pass an empty string as the host part of the server address, we
can't really assign a single address, so the server addr attribute is
left alone:
>>> server = ZEO.StorageServer.StorageServer(
... ('', 0), {'1': ZODB.MappingStorage.MappingStorage()})
>>> addr, stop = ZEO.server(port=('', 0))
>>> isinstance(last_event, ZEO.StorageServer.Serving)
True
>>> last_event.server is server
True
>>> last_event.address[1] > 0
True
If the host part pf the address passed to the constructor is not an
empty string. then the server addr attribute is the same as the
address attribute of the event:
>>> server.addr
>>> last_event.server.addr
('', 0)
>>> server.close()
>>> stop()
The runzeo module provides some process support, including getting the
server configuration via a ZConfig configuration file. To spell a
......
src/ZEO/tests/forker.py
View file @ c499ea7d
......@@ -13,63 +13,73 @@
##############################################################################
"""Library for forking storage server and connecting client storage"""
from __future__ import print_function
import gc
import os
import random
import sys
import time
import errno
import multiprocessing
import socket
import subprocess
import logging
import tempfile
import logging
import six
from six.moves.queue import Empty
import ZODB.tests.util
import zope.testing.setupstack
from ZEO._compat import BytesIO
from ZEO._compat import StringIO
logger = logging.getLogger('ZEO.tests.forker')
DEBUG = os.environ.get('ZEO_TEST_SERVER_DEBUG')
ZEO4_SERVER = os.environ.get('ZEO4_SERVER')
skip_if_testing_client_against_zeo4 = (
(lambda func: None)
if ZEO4_SERVER else
(lambda func: func)
)
class ZEOConfig:
"""Class to generate ZEO configuration file. """
def __init__(self, addr):
if isinstance(addr, str):
def __init__(self, addr, log=None, **options):
if log:
if isinstance(log, str):
self.logpath = log
elif isinstance(addr, str):
self.logpath = addr+'.log'
else:
self.logpath = 'server-%s.log' % addr[1]
self.logpath = 'server.log'
if not isinstance(addr, str):
addr = '%s:%s' % addr
self.log = log
self.address = addr
self.read_only = None
self.invalidation_queue_size = None
self.invalidation_age = None
self.monitor_address = None
self.transaction_timeout = None
self.authentication_protocol = None
self.authentication_database = None
self.authentication_realm = None
self.loglevel = 'INFO'
self.__dict__.update(options)
def dump(self, f):
print("<zeo>", file=f)
print("address " + self.address, file=f)
if self.read_only is not None:
print("read-only", self.read_only and "true" or "false", file=f)
if self.invalidation_queue_size is not None:
print("invalidation-queue-size", self.invalidation_queue_size, file=f)
if self.invalidation_age is not None:
print("invalidation-age", self.invalidation_age, file=f)
if self.monitor_address is not None:
print("monitor-address %s:%s" % self.monitor_address, file=f)
if self.transaction_timeout is not None:
print("transaction-timeout", self.transaction_timeout, file=f)
if self.authentication_protocol is not None:
print("authentication-protocol", self.authentication_protocol, file=f)
if self.authentication_database is not None:
print("authentication-database", self.authentication_database, file=f)
if self.authentication_realm is not None:
print("authentication-realm", self.authentication_realm, file=f)
for name in (
'invalidation_queue_size', 'invalidation_age',
'transaction_timeout', 'pid_filename',
'ssl_certificate', 'ssl_key', 'client_conflict_resolution',
):
v = getattr(self, name, None)
if v:
print(name.replace('_', '-'), v, file=f)
print("</zeo>", file=f)
if self.log:
print("""
<eventlog>
level %s
......@@ -80,9 +90,9 @@ class ZEOConfig:
""" % (self.loglevel, self.logpath), file=f)
def __str__(self):
f = BytesIO()
f = StringIO()
self.dump(f)
return f.getvalue().decode()
return f.getvalue()
def encode_format(fmt):
......@@ -93,10 +103,104 @@ def encode_format(fmt):
fmt = fmt.replace(*xform)
return fmt
def runner(config, qin, qout, timeout=None,
debug=False, name=None,
keep=False, protocol=None):
if debug or DEBUG:
debug_logging()
old_protocol = None
if protocol:
import ZEO.asyncio.server
old_protocol = ZEO.asyncio.server.best_protocol_version
ZEO.asyncio.server.best_protocol_version = protocol
old_protocols = ZEO.asyncio.server.ServerProtocol.protocols
ZEO.asyncio.server.ServerProtocol.protocols = tuple(sorted(
set(old_protocols) | set([protocol])
))
try:
import threading
if ZEO4_SERVER:
from .ZEO4 import runzeo
else:
from .. import runzeo
options = runzeo.ZEOOptions()
options.realize(['-C', config])
server = runzeo.ZEOServer(options)
globals()[(name if name else 'last') + '_server'] = server
server.open_storages()
server.clear_socket()
server.create_server()
logger.debug('SERVER CREATED')
if ZEO4_SERVER:
qout.put(server.server.addr)
else:
qout.put(server.server.acceptor.addr)
logger.debug('ADDRESS SENT')
thread = threading.Thread(
target=server.server.loop, kwargs=dict(timeout=.2),
name = None if name is None else name + '-server',
)
thread.setDaemon(True)
thread.start()
os.remove(config)
try:
qin.get(timeout=timeout) # wait for shutdown
except Empty:
pass
server.server.close()
thread.join(3)
if not keep:
# Try to cleanup storage files
for storage in server.server.storages.values():
try:
storage.cleanup()
except AttributeError:
pass
qout.put(thread.is_alive())
except Exception:
logger.exception("In server thread")
finally:
if old_protocol:
ZEO.asyncio.server.best_protocol_version = old_protocol
ZEO.asyncio.server.ServerProtocol.protocols = old_protocols
def stop_runner(thread, config, qin, qout, stop_timeout=19, pid=None):
qin.put('stop')
try:
dirty = qout.get(timeout=stop_timeout)
except Empty:
print("WARNING Couldn't stop server", file=sys.stderr)
if hasattr(thread, 'terminate'):
thread.terminate()
os.waitpid(thread.pid, 0)
else:
if dirty:
print("WARNING SERVER DIDN'T STOP CLEANLY", file=sys.stderr)
# The runner thread didn't stop. If it was a process,
# give it some time to exit
if hasattr(thread, 'pid') and thread.pid:
os.waitpid(thread.pid, 0)
thread.join(stop_timeout)
gc.collect()
def start_zeo_server(storage_conf=None, zeo_conf=None, port=None, keep=False,
path='Data.fs', protocol=None, blob_dir=None,
suicide=True, debug=False):
suicide=True, debug=False,
threaded=False, start_timeout=33, name=None, log=None,
):
"""Start a ZEO server in a separate process.
Takes two positional arguments a string containing the storage conf
......@@ -108,86 +212,63 @@ def start_zeo_server(storage_conf=None, zeo_conf=None, port=None, keep=False,
if not storage_conf:
storage_conf = '<filestorage>\npath %s\n</filestorage>' % path
if blob_dir:
storage_conf = '<blobstorage>\nblob-dir %s\n%s\n</blobstorage>' % (
blob_dir, storage_conf)
if zeo_conf is None or isinstance(zeo_conf, dict):
if port is None:
raise AssertionError("The port wasn't specified")
port = 0
if isinstance(port, int):
addr = 'localhost', port
adminaddr = 'localhost', port+1
addr = '127.0.0.1', port
else:
addr = port
adminaddr = port+'-test'
if zeo_conf is None or isinstance(zeo_conf, dict):
z = ZEOConfig(addr)
z = ZEOConfig(addr, log=log)
if zeo_conf:
z.__dict__.update(zeo_conf)
zeo_conf = z
zeo_conf = str(z)
# Store the config info in a temp file.
tmpfile = tempfile.mktemp(".conf", dir=os.getcwd())
fp = open(tmpfile, 'w')
zeo_conf.dump(fp)
fp.write(str(zeo_conf) + '\n\n')
fp.write(storage_conf)
fp.close()
# Find the zeoserver script
import ZEO.tests.zeoserver
script = ZEO.tests.zeoserver.__file__
if script.endswith('.pyc'):
script = script[:-1]
# Create a list of arguments, which we'll tuplify below
qa = _quote_arg
args = [qa(sys.executable), qa(script), '-C', qa(tmpfile)]
if keep:
args.append("-k")
if debug:
args.append("-d")
if not suicide:
args.append("-S")
if protocol:
args.extend(["-v", protocol])
d = os.environ.copy()
d['PYTHONPATH'] = os.pathsep.join(sys.path)
if sys.platform.startswith('win'):
pid = os.spawnve(os.P_NOWAIT, sys.executable, tuple(args), d)
if threaded:
from threading import Thread
from six.moves.queue import Queue
else:
pid = subprocess.Popen(args, env=d, close_fds=True).pid
# We need to wait until the server starts, but not forever. 150
# seconds is a somewhat arbitrary upper bound, but probably helps
# in an address already in use situation.
for i in range(1500):
time.sleep(0.1)
from multiprocessing import Process as Thread
Queue = ThreadlessQueue
qin = Queue()
qout = Queue()
thread = Thread(
target=runner,
args=[tmpfile, qin, qout, 999 if suicide else None],
kwargs=dict(debug=debug, name=name, protocol=protocol, keep=keep),
name = None if name is None else name + '-server-runner',
)
thread.daemon = True
thread.start()
try:
if isinstance(adminaddr, str) and not os.path.exists(adminaddr):
continue
logger.debug('connect %s', i)
if isinstance(adminaddr, str):
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
else:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(adminaddr)
ack = s.recv(1024)
s.close()
logging.debug('acked: %s' % ack)
break
except socket.error as e:
if e.args[0] not in (errno.ECONNREFUSED, errno.ECONNRESET):
addr = qout.get(timeout=start_timeout)
except Exception:
whine("SERVER FAILED TO START")
if thread.is_alive():
whine("Server thread/process is still running")
elif not threaded:
whine("Exit status", thread.exitcode)
raise
s.close()
else:
logging.debug('boo hoo')
raise RuntimeError("Failed to start server")
return addr, adminaddr, pid, tmpfile
def stop(stop_timeout=99):
stop_runner(thread, tmpfile, qin, qout, stop_timeout)
return addr, stop
if sys.platform[:3].lower() == "win":
def _quote_arg(s):
......@@ -196,42 +277,10 @@ else:
def _quote_arg(s):
return s
def shutdown_zeo_server(stop):
stop()
def shutdown_zeo_server(adminaddr):
# Do this in a loop to guard against the possibility that the
# client failed to connect to the adminaddr earlier. That really
# only requires two iterations, but do a third for pure
# superstition.
for i in range(3):
if isinstance(adminaddr, str):
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
else:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(.3)
try:
s.connect(adminaddr)
except socket.timeout:
# On FreeBSD 5.3 the connection just timed out
if i > 0:
break
raise
except socket.error as e:
if (e.args[0] == errno.ECONNREFUSED
or
# MAC OS X uses EINVAL when connecting to a port
# that isn't being listened on.
(sys.platform == 'darwin' and e.args[0] == errno.EINVAL)
) and i > 0:
break
raise
try:
ack = s.recv(1024)
except socket.error as e:
ack = 'no ack received'
logger.debug('shutdown_zeo_server(): acked: %s' % ack)
s.close()
def get_port(test=None):
def get_port(ignored=None):
"""Return a port that is not in use.
Checks if a port is in use by trying to connect to it. Assumes it
......@@ -242,23 +291,20 @@ def get_port(test=None):
Raises RuntimeError after 10 tries.
"""
if test is not None:
return get_port2(test)
for i in range(10):
port = random.randrange(20000, 30000)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
try:
s.connect(('localhost', port))
s.connect(('127.0.0.1', port))
except socket.error:
pass # Perhaps we should check value of error too.
else:
continue
try:
s1.connect(('localhost', port+1))
s1.connect(('127.0.0.1', port+1))
except socket.error:
pass # Perhaps we should check value of error too.
else:
......@@ -271,35 +317,11 @@ def get_port(test=None):
s1.close()
raise RuntimeError("Can't find port")
def get_port2(test):
for i in range(10):
while 1:
port = random.randrange(20000, 30000)
if port%3 == 0:
break
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
s.bind(('localhost', port+2))
except socket.error as e:
if e.args[0] != errno.EADDRINUSE:
raise
s.close()
continue
if not (can_connect(port) or can_connect(port+1)):
zope.testing.setupstack.register(test, s.close)
return port
s.close()
raise RuntimeError("Can't find port")
def can_connect(port):
c = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
try:
c.connect(('localhost', port))
c.connect(('127.0.0.1', port))
except socket.error:
return False # Perhaps we should check value of error too.
else:
......@@ -310,46 +332,47 @@ def can_connect(port):
def setUp(test):
ZODB.tests.util.setUp(test)
servers = {}
servers = []
def start_server(storage_conf=None, zeo_conf=None, port=None, keep=False,
addr=None, path='Data.fs', protocol=None, blob_dir=None,
suicide=True, debug=False):
suicide=True, debug=False, **kw):
"""Start a ZEO server.
Return the server and admin addresses.
"""
if port is None:
if addr is None:
port = get_port2(test)
port = 0
else:
port = addr[1]
elif addr is not None:
raise TypeError("Can't specify port and addr")
addr, adminaddr, pid, config_path = start_zeo_server(
storage_conf, zeo_conf, port, keep, path, protocol, blob_dir,
suicide, debug)
os.remove(config_path)
servers[adminaddr] = pid
return addr, adminaddr
addr, stop = start_zeo_server(
storage_conf=storage_conf,
zeo_conf=zeo_conf,
port=port,
keep=keep,
path=path,
protocol=protocol,
blob_dir=blob_dir,
suicide=suicide,
debug=debug,
**kw)
servers.append(stop)
return addr, stop
test.globs['start_server'] = start_server
def get_port():
return get_port2(test)
test.globs['get_port'] = get_port
def stop_server(adminaddr):
pid = servers.pop(adminaddr)
shutdown_zeo_server(adminaddr)
os.waitpid(pid, 0)
def stop_server(stop):
stop()
servers.remove(stop)
test.globs['stop_server'] = stop_server
def cleanup_servers():
for adminaddr in list(servers):
stop_server(adminaddr)
for stop in list(servers):
stop()
zope.testing.setupstack.register(test, cleanup_servers)
......@@ -387,3 +410,33 @@ def wait_connected(storage):
def wait_disconnected(storage):
wait_until("storage is disconnected",
lambda : not storage.is_connected())
def debug_logging(logger='ZEO', stream='stderr', level=logging.DEBUG):
handler = logging.StreamHandler(getattr(sys, stream))
logger = logging.getLogger(logger)
logger.addHandler(handler)
logger.setLevel(level)
def stop():
logger.removeHandler(handler)
logger.setLevel(logging.NOTSET)
return stop
def whine(*message):
print(*message, file=sys.stderr)
sys.stderr.flush()
class ThreadlessQueue(object):
def __init__(self):
self.cin, self.cout = multiprocessing.Pipe(False)
def put(self, v):
self.cout.send(v)
def get(self, timeout=None):
if self.cin.poll(timeout):
return self.cin.recv()
else:
raise Empty()
src/ZEO/tests/invalidation-age.txt
View file @ c499ea7d
......@@ -69,26 +69,17 @@ We'll open another client, and commit some transactions:
... transaction.commit()
>>> db.close()
If we reopen the first client, we'll do quick verification. We'll
turn on logging so we can see this:
>>> import logging, sys
>>> old_logging_level = logging.getLogger().getEffectiveLevel()
>>> logging.getLogger().setLevel(logging.INFO)
>>> handler = logging.StreamHandler(sys.stdout)
>>> logging.getLogger().addHandler(handler)
If we reopen the first client, we'll do quick verification.
>>> db = ZEO.DB(addr, client='test') # doctest: +ELLIPSIS
('localhost', ...
('localhost', ...) Recovering 2 invalidations
>>> logging.getLogger().removeHandler(handler)
>>> db._storage._server.client.verify_result
'quick verification'
>>> [v.x for v in db.open().root().values()]
[1, 1, 0, 0, 0, 0, 0, 0, 0]
Now, if we disconnect and commit more than 5 transactions, we'll see
that verification is necessary:
that we had to clear the cache:
>>> db.close()
>>> db = ZEO.DB(addr)
......@@ -99,14 +90,9 @@ that verification is necessary:
... transaction.commit()
>>> db.close()
>>> logging.getLogger().addHandler(handler)
>>> db = ZEO.DB(addr, client='test') # doctest: +ELLIPSIS
('localhost', ...
('localhost', ...) Verifying cache
('localhost', ...) endVerify finishing
('localhost', ...) endVerify finished
>>> logging.getLogger().removeHandler(handler)
>>> db = ZEO.DB(addr, client='test')
>>> db._storage._server.client.verify_result
'cache too old, clearing'
>>> [v.x for v in db.open().root().values()]
[2, 2, 2, 2, 2, 2, 2, 2, 2]
......@@ -128,16 +114,11 @@ do quick verification:
>>> db.close()
>>> logging.getLogger().addHandler(handler)
>>> db = ZEO.DB(addr, client='test') # doctest: +ELLIPSIS
('localhost', ...
('localhost', ...) Recovering 9 invalidations
>>> logging.getLogger().removeHandler(handler)
>>> db._storage._server.client.verify_result
'quick verification'
>>> [v.x for v in db.open().root().values()]
[3, 3, 3, 3, 3, 3, 3, 3, 3]
>>> db.close()
>>> logging.getLogger().setLevel(old_logging_level)
src/ZEO/tests/new_addr.test
View file @ c499ea7d
......@@ -2,12 +2,10 @@ You can change the address(es) of a client storaage.
We'll start by setting up a server and connecting to it:
>>> import ZEO, ZEO.StorageServer, ZODB.FileStorage, transaction
>>> server = ZEO.StorageServer.StorageServer(
... ('127.0.0.1', 0), {'1': ZODB.FileStorage.FileStorage('t.fs')})
>>> server.start_thread()
>>> import ZEO, transaction
>>> conn = ZEO.connection(server.addr, max_disconnect_poll=0.1)
>>> addr, stop = ZEO.server(path='test.fs', threaded=False)
>>> conn = ZEO.connection(addr)
>>> client = conn.db().storage
>>> client.is_connected()
True
......@@ -18,28 +16,25 @@ We'll start by setting up a server and connecting to it:
Now we'll close the server:
>>> server.close()
>>> stop()
And wait for the connectin to notice it's disconnected:
>>> wait_until(lambda : not client.is_connected())
Now, we'll restart the server and update the connection:
>>> server = ZEO.StorageServer.StorageServer(
... ('127.0.0.1', 0), {'1': ZODB.FileStorage.FileStorage('t.fs')})
>>> server.start_thread()
>>> client.new_addr(server.addr)
Now, we'll restart the server:
>>> addr, stop = ZEO.server(path='test.fs', threaded=False)
Update with another client:
>>> conn2 = ZEO.connection(server.addr)
>>> conn2 = ZEO.connection(addr)
>>> conn2.root.x += 1
>>> transaction.commit()
Wait for connect:
Update the connection and wait for connect:
>>> client.new_addr(addr)
>>> wait_until(lambda : client.is_connected())
>>> _ = transaction.begin()
>>> conn.root()
......@@ -49,4 +44,4 @@ Wait for connect:
>>> conn.close()
>>> conn2.close()
>>> server.close()
>>> stop()
src/ZEO/tests/protocols.test
View file @ c499ea7d
......@@ -5,7 +5,7 @@ A full test of all protocols isn't practical. But we'll do a limited
test that at least the current and previous protocols are supported in
both directions.
Let's start a Z308 server
Let's start a Z4 server
>>> storage_conf = '''
... <blobstorage>
......@@ -16,16 +16,16 @@ Let's start a Z308 server
... </blobstorage>
... '''
>>> addr, admin = start_server(
... storage_conf, dict(invalidation_queue_size=5), protocol=b'Z308')
>>> addr, stop = start_server(
... storage_conf, dict(invalidation_queue_size=5), protocol=b'Z4')
A current client should be able to connect to a old server:
>>> import ZEO, ZODB.blob, transaction
>>> db = ZEO.DB(addr, client='client', blob_dir='blobs')
>>> wait_connected(db.storage)
>>> db.storage._connection.peer_protocol_version
b'Z308'
>>> str(db.storage.protocol_version.decode('ascii'))
'Z4'
>>> conn = db.open()
>>> conn.root().x = 0
......@@ -85,112 +85,100 @@ A current client should be able to connect to a old server:
... f.read()
b'blob data 2'
Note that when taking to a 3.8 server, iteration won't work:
>>> db.storage.iterator()
Traceback (most recent call last):
...
NotImplementedError
>>> db2.close()
>>> db.close()
>>> stop_server(admin)
>>> stop_server(stop)
>>> import os, zope.testing.setupstack
>>> os.remove('client-1.zec')
>>> zope.testing.setupstack.rmtree('blobs')
>>> zope.testing.setupstack.rmtree('server-blobs')
And the other way around:
>>> addr, _ = start_server(storage_conf, dict(invalidation_queue_size=5))
Note that we'll have to pull some hijinks:
>>> import ZEO.zrpc.connection
>>> old_current_protocol = ZEO.zrpc.connection.Connection.current_protocol
>>> ZEO.zrpc.connection.Connection.current_protocol = b'Z308'
>>> db = ZEO.DB(addr, client='client', blob_dir='blobs')
>>> db.storage._connection.peer_protocol_version
b'Z308'
>>> wait_connected(db.storage)
>>> conn = db.open()
>>> conn.root().x = 0
>>> transaction.commit()
>>> len(db.history(conn.root()._p_oid, 99))
2
>>> conn.root()['blob1'] = ZODB.blob.Blob()
>>> with conn.root()['blob1'].open('w') as f:
... r = f.write(b'blob data 1')
>>> transaction.commit()
>>> db2 = ZEO.DB(addr, blob_dir='server-blobs', shared_blob_dir=True)
>>> wait_connected(db2.storage)
>>> conn2 = db2.open()
>>> for i in range(5):
... conn2.root().x += 1
... transaction.commit()
>>> conn2.root()['blob2'] = ZODB.blob.Blob()
>>> with conn2.root()['blob2'].open('w') as f:
... r = f.write(b'blob data 2')
>>> transaction.commit()
>>> @wait_until()
... def x_to_be_5():
... conn.sync()
... return conn.root().x == 5
>>> db.close()
>>> for i in range(2):
... conn2.root().x += 1
... transaction.commit()
>>> db = ZEO.DB(addr, client='client', blob_dir='blobs')
>>> wait_connected(db.storage)
>>> conn = db.open()
>>> conn.root().x
7
>>> db.close()
>>> for i in range(10):
... conn2.root().x += 1
... transaction.commit()
>>> db = ZEO.DB(addr, client='client', blob_dir='blobs')
>>> wait_connected(db.storage)
>>> conn = db.open()
>>> conn.root().x
17
>>> with conn.root()['blob1'].open() as f:
... f.read()
b'blob data 1'
>>> with conn.root()['blob2'].open() as f:
... f.read()
b'blob data 2'
Make some old protocol calls:
>>> db.storage._server.rpc.call('getSerial', conn.root()._p_oid
... ) == conn.root()._p_serial
True
>>> p, s, v, x, y = db.storage._server.rpc.call('zeoLoad',
... conn.root()._p_oid)
>>> (v, x, y) == ('', None, None)
True
>>> db.storage.load(conn.root()._p_oid) == (p, s)
True
>>> db2.close()
>>> db.close()
Undo the hijinks:
>>> ZEO.zrpc.connection.Connection.current_protocol = old_current_protocol
#############################################################################
# Note that the ZEO 5.0 server only supports clients that use the Z5 protocol
# And the other way around:
# >>> addr, _ = start_server(storage_conf, dict(invalidation_queue_size=5))
# Note that we'll have to pull some hijinks:
# >>> db = ZEO.DB(addr, client='client', blob_dir='blobs')
# >>> str(db.storage.protocol_version.decode('ascii'))
# 'Z4'
# >>> wait_connected(db.storage)
# >>> conn = db.open()
# >>> conn.root().x = 0
# >>> transaction.commit()
# >>> len(db.history(conn.root()._p_oid, 99))
# 2
# >>> db = ZEO.DB(addr, client='client', blob_dir='blobs')
# >>> db.storage.protocol_version
# b'Z4'
# >>> wait_connected(db.storage)
# >>> conn = db.open()
# >>> conn.root().x = 0
# >>> transaction.commit()
# >>> len(db.history(conn.root()._p_oid, 99))
# 2
# >>> conn.root()['blob1'] = ZODB.blob.Blob()
# >>> with conn.root()['blob1'].open('w') as f:
# ... r = f.write(b'blob data 1')
# >>> transaction.commit()
# >>> db2 = ZEO.DB(addr, blob_dir='server-blobs', shared_blob_dir=True)
# >>> wait_connected(db2.storage)
# >>> conn2 = db2.open()
# >>> for i in range(5):
# ... conn2.root().x += 1
# ... transaction.commit()
# >>> conn2.root()['blob2'] = ZODB.blob.Blob()
# >>> with conn2.root()['blob2'].open('w') as f:
# ... r = f.write(b'blob data 2')
# >>> transaction.commit()
# >>> @wait_until()
# ... def x_to_be_5():
# ... conn.sync()
# ... return conn.root().x == 5
# >>> db.close()
# >>> for i in range(2):
# ... conn2.root().x += 1
# ... transaction.commit()
# >>> db = ZEO.DB(addr, client='client', blob_dir='blobs')
# >>> wait_connected(db.storage)
# >>> conn = db.open()
# >>> conn.root().x
# 7
# >>> db.close()
# >>> for i in range(10):
# ... conn2.root().x += 1
# ... transaction.commit()
# >>> db = ZEO.DB(addr, client='client', blob_dir='blobs')
# >>> wait_connected(db.storage)
# >>> conn = db.open()
# >>> conn.root().x
# 17
# >>> with conn.root()['blob1'].open() as f:
# ... f.read()
# b'blob data 1'
# >>> with conn.root()['blob2'].open() as f:
# ... f.read()
# b'blob data 2'
# >>> db2.close()
# >>> db.close()
# Undo the hijinks:
# >>> ZEO.asyncio.client.Protocol.protocols = old_protocols
src/ZEO/tests/registerDB.test deleted 100644 → 0
View file @ bc92bdd7
Storage Servers should call registerDB on storages to propigate invalidations
=============================================================================
Storages servers propagate invalidations from their storages. Among
other things, this allows client storages to be used in storage
servers, allowing storage-server fan out, spreading read load over
multiple storage servers.
We'll create a Faux storage that has a registerDB method.
>>> class FauxStorage:
... invalidations = [('trans0', ['ob0']),
... ('trans1', ['ob0', 'ob1']),
... ]
... def registerDB(self, db):
... self.db = db
... def isReadOnly(self):
... return False
... def getName(self):
... return 'faux'
... def lastTransaction(self):
... return self.invq[0][0]
... def lastInvalidations(self, size):
... return list(self.invalidations)
We dont' want the storage server to try to bind to a socket. We'll
subclass it and give it a do-nothing dispatcher "class":
>>> import ZEO.StorageServer
>>> class StorageServer(ZEO.StorageServer.StorageServer):
... class DispatcherClass:
... __init__ = lambda *a, **kw: None
... class socket:
... getsockname = staticmethod(lambda : 'socket')
We'll create a storage instance and a storage server using it:
>>> storage = FauxStorage()
>>> server = StorageServer('addr', dict(t=storage))
Our storage now has a db attribute that provides IStorageDB. It's
references method is just the referencesf function from ZODB.Serialize
>>> import ZODB.serialize
>>> storage.db.references is ZODB.serialize.referencesf
True
To see the effects of the invalidation messages, we'll create a client
stub that implements the client invalidation calls:
>>> class Client:
... def __init__(self, name):
... self.name = name
... def invalidateTransaction(self, tid, invalidated):
... print('invalidateTransaction', tid, self.name)
... print(invalidated)
>>> class Connection:
... def __init__(self, mgr, obj):
... self.mgr = mgr
... self.obj = obj
... def should_close(self):
... print('closed', self.obj.name)
... self.mgr.close_conn(self)
... def poll(self):
... pass
...
... @property
... def trigger(self):
... return self
...
... def pull_trigger(self):
... pass
>>> class ZEOStorage:
... def __init__(self, server, name):
... self.name = name
... self.connection = Connection(server, self)
... self.client = Client(name)
Now, we'll register the client with the storage server:
>>> _ = server.register_connection('t', ZEOStorage(server, 1))
>>> _ = server.register_connection('t', ZEOStorage(server, 2))
Now, if we call invalidate, we'll see it propigate to the client:
>>> storage.db.invalidate('trans2', ['ob1', 'ob2'])
invalidateTransaction trans2 1
['ob1', 'ob2']
invalidateTransaction trans2 2
['ob1', 'ob2']
>>> storage.db.invalidate('trans3', ['ob1', 'ob2'])
invalidateTransaction trans3 1
['ob1', 'ob2']
invalidateTransaction trans3 2
['ob1', 'ob2']
The storage servers queue will reflect the invalidations:
>>> for tid, invalidated in server.invq['t']:
... print(repr(tid), invalidated)
'trans3' ['ob1', 'ob2']
'trans2' ['ob1', 'ob2']
'trans1' ['ob0', 'ob1']
'trans0' ['ob0']
If we call invalidateCache, the storage server will close each of it's
connections:
>>> storage.db.invalidateCache()
closed 1
closed 2
The connections will then reopen and revalidate their caches.
The servers's invalidation queue will get reset
>>> for tid, invalidated in server.invq['t']:
... print(repr(tid), invalidated)
'trans1' ['ob0', 'ob1']
'trans0' ['ob0']
src/ZEO/tests/server.pem 0 → 100644
View file @ c499ea7d
-----BEGIN CERTIFICATE-----
MIID/TCCAuWgAwIBAgIJAKpYWt7G+3R4MA0GCSqGSIb3DQEBBQUAMFwxCzAJBgNV
BAYTAlVTMQswCQYDVQQIEwJWQTENMAsGA1UEChMEWk9EQjERMA8GA1UEAxMIem9k
Yi5vcmcxHjAcBgkqhkiG9w0BCQEWD3NlcnZlckB6b2RiLm9yZzAeFw0xNjA2MjMx
NTA5MTZaFw0xNzA2MjMxNTA5MTZaMFwxCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJW
QTENMAsGA1UEChMEWk9EQjERMA8GA1UEAxMIem9kYi5vcmcxHjAcBgkqhkiG9w0B
CQEWD3NlcnZlckB6b2RiLm9yZzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBANqyMazB39wFKFh2nIspOw+e/7LPLQZpsd6BnYOxPbdlhb23Q930WqgW5qCA
aJjLqkmvQvZhElXdO2lOxGLR2/Cu71UgmYXkZbMKqPNtoYCPRPCCKm5EczAFXTy4
SUD40wAlndP7J8TjpIaKZjgdfy4GWnGQQAbXUR1eFZszQtU7pUvyjgXsY+BEgq4h
F4iIBarcf8k/6PTldTRLxEbRiTNZ4cdIEtTJL/LzSu5oBw8Z3J05aPg9DcWVl89P
XtemKxU8+Vm847xXJkuODkpSgOCqAPn959b/DGdn1fyx2CR0lSgC4n2rYE/oWxw0
hiyI1jeXTAfOtqvcg4XA5Cs3ivUCAwEAAaOBwTCBvjAdBgNVHQ4EFgQUoWgKmGKI
o2U9vFK48cyXuZrmPdgwgY4GA1UdIwSBhjCBg4AUoWgKmGKIo2U9vFK48cyXuZrm
PdihYKReMFwxCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJWQTENMAsGA1UEChMEWk9E
QjERMA8GA1UEAxMIem9kYi5vcmcxHjAcBgkqhkiG9w0BCQEWD3NlcnZlckB6b2Ri
Lm9yZ4IJAKpYWt7G+3R4MAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADggEB
ACiGzPx1445EdyO4Em3M6MwPQD8KVLvxsYE5HQMk0A2BTJOs2Vzf0NPL2hn947D/
wFe/ubkJXmzPG2CBCbNA//exWj8x2rR8256qJuWDwFx0WUFpRmEVFUGK+bpvgxQV
DlLxnWV8z6Tq9vINRjKT3mcBX4OpDgbEL4O92pbJ7kZNn4Z2+v6/lsWzg3Eo6LVY
fj902gQD/6hjVenD6J5Dqftj4x9nsKjdMz8n5Ok5E1J02ghiWjlUp1PNUKwc2Elw
oFnjPiacbko0fSnD9Zf6qBbACAYyBkHvBc1ZMebnGepZn3E6V91X5kZl84hGzgsb
C+2aGtAqSnvL4/DlIyss3hc=
-----END CERTIFICATE-----
src/ZEO/tests/server_key.pem 0 → 100644
View file @ c499ea7d
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEA2rIxrMHf3AUoWHaciyk7D57/ss8tBmmx3oGdg7E9t2WFvbdD
3fRaqBbmoIBomMuqSa9C9mESVd07aU7EYtHb8K7vVSCZheRlswqo822hgI9E8IIq
bkRzMAVdPLhJQPjTACWd0/snxOOkhopmOB1/LgZacZBABtdRHV4VmzNC1TulS/KO
Bexj4ESCriEXiIgFqtx/yT/o9OV1NEvERtGJM1nhx0gS1Mkv8vNK7mgHDxncnTlo
+D0NxZWXz09e16YrFTz5WbzjvFcmS44OSlKA4KoA+f3n1v8MZ2fV/LHYJHSVKALi
fatgT+hbHDSGLIjWN5dMB862q9yDhcDkKzeK9QIDAQABAoIBACtwA0/V/jm8SIQx
ouw9Fz8GDLGeVsoUSkDwq7GRjbmUj5jcAr3eH/eM/OfaOWxH353dEsbPBw5I79j9
zSH3nuDSTjUxUWz3rX9/WYloOBDJ5B6FLBpUvDBIkHlT/TDLe1VnI08Mbpy7vlz+
tkjlCvLATkyKIz14nOLhYhc+ekLRxQZrVRgHIPW13c0F61drkc4uCs/UMbYRzZiZ
nnMeQLghIUP13xMtMMkNx0P1ampvV18kCJWuvHUqOMnCaPnTGCnGfOUtq98sTdui
vBnleePmBzF5VGT7M3e3pr63EKyVPD/bx2dbItcxOahBTgDbMQR4AQ65NvD5+B0+
d9XbfgECgYEA7wLsMvFAmj5IrXYGYnaqy1LsMiZll9zv4QB2tAKLSIy5z0Cns54R
ttI6Ni94CfjBOKmR5IZXjgXZg5ydu+tBLwqNMJISziFSyTxzwNsAEPczUHefFhwq
zhAkIVsISy42AxpuyHlz5EBSJiUULhguUhmIDxCbQnmlSK6X9MxYO3UCgYEA6j2c
dfU4RC2tzjNKoKs52mUoWmwFsM5CKs0hDeRn2KV80nNKn/mRa0cZ36u/x0hN3fBm
wrwNiZWOX+ot5SfnePx0dOiTOxfWYeXtzVqF6KhPUK8K5424zL+N1uw941gvkXYR
Cc79AxDI/S5y/2ZR8aW4H91LdCcvPlE4Dp1JoYECgYAuE2gpYezMT1l/ZxNQBARk
8fVqrZBEOGld/NLlXOAw+kAPvi0WKVDM57YlH/2KHpRRMg9X+LYEQQhvoM+fnHiS
cvxI8sABUNc+yBKgiRd4Lc+MoaLfhkqSMvZkH8J3i88Jxhy5NQCsbeHoTJmZUTwM
w7NBBDiKFh1Q56ePn50ayQKBgQCA8guoT6Z6uZ6dDVU+nyOI2vjc1exICTMZdrSE
fkDAXVEaVMc2y17G7GwM2fIHlQDwdP9ModLd80td937uUAo3atn85W7vL88fM0C2
M+fVTJnk84cQMs8RPz2om4HyHcCJ1bHJcX2Ma3gJD8HUYJIpcS2rtNlthoiWSIWQ
XfuDgQKBgQCYW+x52XSaTjE0SviYmeNuO30bELgjxPyPLge407EAp1h6Sr9Za5ap
2aQsClsuzpHcHDoWmzO+dTr70Qyt/YoY5FN3uZRG9m7X/OH7+ceBEU/BtQ0Whem4
YNE5lpuw4eGsJQGkhRFcqKy66gId7Sw5b6CIaJqbhQCogkWt5JQuWQ==
-----END RSA PRIVATE KEY-----
src/ZEO/tests/serverpw.pem 0 → 100644
View file @ c499ea7d
-----BEGIN CERTIFICATE-----
MIID8DCCAtigAwIBAgIJALop9P9MBfzLMA0GCSqGSIb3DQEBBQUAMFgxCzAJBgNV
BAYTAlVTMQswCQYDVQQIEwJWQTENMAsGA1UEChMEWk9EQjERMA8GA1UEAxMIem9k
Yi5vcmcxGjAYBgkqhkiG9w0BCQEWC3B3QHpvZGIub3JnMB4XDTE2MDYyMzE1MTAz
MVoXDTE3MDYyMzE1MTAzMVowWDELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlZBMQ0w
CwYDVQQKEwRaT0RCMREwDwYDVQQDEwh6b2RiLm9yZzEaMBgGCSqGSIb3DQEJARYL
cHdAem9kYi5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDKw/iw
N1EPddU9QQQ+OnCJv9G3rbTOPt4zEbpfTROIHTME3krFKPALrGF2aK+oBpHx3/TZ
HN5UvWK/jmGtDL9jekKCAaeAaVIKlESUS6DIxZY+FaO3re/1fbmBNRz8Cnn1raAw
/4YZRDPvblooH4Nt5m7uooGAIIDPft3fInhmGboOoIpXc7nMGVGOWXlDN5I9oFmm
4vby4CUMy3A/0wnHgTuMNy7Tpjgz2E/1MRAOyWQ7PZYiASs4ycZfas8058O8DI+o
rSYyum/czecIz52P6jbx5LWvcKDWac8QbJoHPelthYtxcMHee2+Nh6MWW688CBzq
HSeFAdNO3d9kMiFpAgMBAAGjgbwwgbkwHQYDVR0OBBYEFDui1OC2+2z2rHADglk5
tGOndxhoMIGJBgNVHSMEgYEwf4AUO6LU4Lb7bPascAOCWTm0Y6d3GGihXKRaMFgx
CzAJBgNVBAYTAlVTMQswCQYDVQQIEwJWQTENMAsGA1UEChMEWk9EQjERMA8GA1UE
AxMIem9kYi5vcmcxGjAYBgkqhkiG9w0BCQEWC3B3QHpvZGIub3JnggkAuin0/0wF
/MswDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQUFAAOCAQEAiEYO8MZ3OG8sqy9t
AtUZbv0aTsIUzy/QTUKKDUo8qwKNOylqyqGAZV0tZ5eCoqIGFAwRJBBymIizU3zH
U1k2MnYZMVi7uYwSy+qwg52+X7GLl/kaAfx8kNvpr274CuZQnLojJS+K8HtH5Pom
YD3gTO3OxGS4IS6uf6DD+mf+C9OBnTl47P0HA0/eHBEXVSc2vsv30H/UoW5VbZ6z
6TWkoPwSMVhCNRRRif4/eqCLh24/h5b4uvAC+tsrIPQ9If7EsqVTNMCbAkv3ib6g
OmaCdbrGkqvD3UVn7i5ci96UZoF80EWNZiwhMdvQtMfOAR4jHQ1pTepJni6JwzZP
UMNDpQ==
-----END CERTIFICATE-----
src/ZEO/tests/serverpw_key.pem 0 → 100644
View file @ c499ea7d
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,769B900D03925712
z5M/XkqEC1+PxJ1T3QrUhGG9fPTBsPxKy8WlwIytbMg0RXS0oJBiFgNYp1ktqzGo
yT+AdTCRR1hNVX8M5HbV3ksUjKxXKCL3+yaaB6JtGbNRr2qTNwosvxD92nKT/hvN
R6rHF6LcO05s8ubs9b9ON/ja7HCx69N5CjBuCbCFHUTlAXkwD9w0ScrxrtfP50EY
FOw6LAqhhzq6/KO7c1SJ7k9LYzakhL+nbw5KM9QgBk4WHlmKLbCZIZ5RWvu0F4s5
n4qk/BcuXIkbYuEv2kH0nDk5eDfA/dj7xZcMMgL5VFymQzaZLYyj4WuQYXu/7JW/
nM/ZWBkZOMaI3vnPTG1hJ9pgjLjQnjfNA/bGWwbLxjCsPmR8yvZS4v2iqdB6X3Vl
yJ9aV9r8KoU0PJk3x4v2Zp+RQxgrKSaQw59sXptaXAY3NCRR6ohvq4P5X6UB8r5S
vYdoMeVXhX1hzXeMguln7zQInwJhPZqk4wMIV3lTsCqh1eJ7NC2TGCwble+B/ClR
KtzuJBzoYPLw44ltnXPEMmVH1Fkh17+QZFgZRJrKGD9PGOAXmnzudsZ1xX9kNnOM
JLIT/mzKcqkd8S1n1Xi1x7zGA0Ng5xmKGFe8oDokPJucJO1Ou+hbLDmC0DZUGzr1
qqPJ3F/DzZZDTmD/rZF6doPJgFAZvgpVeiWS8/v1qbz/nz13uwXDLjRPgLfcKpmQ
4R3V4QlgviDilW61VTZnzV9qAOx4fG6+IwWIGBlrJnfsH/fSCDNlAStc6k12zdun
PIIRJBfbEprGig3vRWUoBASReqow1JCN9DaVCX1P27pDKY5oDe+7/HOrQpwhPoya
2HEwbKeyY0nCcCXbkWGL1bwEUs/PrJv+61rik4KxOWhKpHWkZLzbozELb44jXrJx
e8K8XKz4La2DEjsUYHc31u6T69GBQO9JDEvih15phUWq8ITvDnkHpAg+wYb1JAHD
QcqDtAulMvT/ZGN0h7qdwbHMggEsLgCCVPG4iZ5K4cXsMbePFvQqq+o4FTMF+cM5
2Dq0wir92U9cH+ooy80LIt5Kp5zqgQZzr73o9MEgwqJocCrx9ZrofKRUmTV+ZU0r
w5mfUM47Ctnqia0UNGx6SUs3CHFDPWPbzrAaqGzSvFhzR1MMoL1/rJzP1VSm3Fk3
ESWkPrg0J8dcQP/ch9MhH8eoQYyA+2q1vClUbeZLAs5KoHxgi6pSkGYqFhshrA+t
2AIrUPDPPDf0PgRoXJrzdVOiNNY1rzyql+0JqDH6DjCVcAADWY+48p9U2YFTd7Je
DvnZWihwe0qYGn1AKIkvJ4SR3bQg36etrxhMrMl/8lUn2dnT7GFrhjr9HwCpJwa7
8tv150SrQXt3FXZCHb+RMUgoWZDeksDohPiGzXkPU6kaSviZVnRMslyU4ahWp6vC
8tYUhb7K6N+is1hYkICNt6zLl2vBDuCDWmiIwopHtnH1kz8bYlp4/GBVaMIgZiCM
gM/7+p4YCc++s2sJiQ9+BqPo0zKm3bbSP+fPpeWefQVte9Jx4S36YXU52HsJxBTN
WUdHABC+aS2A45I12xMNzOJR6VfxnG6f3JLpt3MkUCEg+898vJGope+TJUhD+aJC
-----END RSA PRIVATE KEY-----
src/ZEO/tests/servertesting.py
View file @ c499ea7d
......@@ -30,9 +30,8 @@ from __future__ import print_function
# Here, we'll try to provide some testing infrastructure to isolate
# servers from the network.
import ZEO.asyncio.tests
import ZEO.StorageServer
import ZEO.zrpc.connection
import ZEO.zrpc.error
import ZODB.MappingStorage
class StorageServer(ZEO.StorageServer.StorageServer):
......@@ -42,44 +41,10 @@ class StorageServer(ZEO.StorageServer.StorageServer):
storages = {'1': ZODB.MappingStorage.MappingStorage()}
ZEO.StorageServer.StorageServer.__init__(self, addr, storages, **kw)
class DispatcherClass:
__init__ = lambda *a, **kw: None
class socket:
getsockname = staticmethod(lambda : 'socket')
class Connection:
peer_protocol_version = ZEO.zrpc.connection.Connection.current_protocol
connected = True
def __init__(self, name='connection', addr=''):
name = str(name)
self.name = name
self.addr = addr or 'test-addr-'+name
def close(self):
print(self.name, 'closed')
self.connected = False
def poll(self):
if not self.connected:
raise ZEO.zrpc.error.DisconnectedError()
def callAsync(self, meth, *args):
print(self.name, 'callAsync', meth, repr(args))
callAsyncNoPoll = callAsync
def call_from_thread(self, *args):
if args:
args[0](*args[1:])
def send_reply(self, *args):
pass
def client(server, name='client', addr=''):
def client(server, name='client'):
zs = ZEO.StorageServer.ZEOStorage(server)
zs.notifyConnected(Connection(name, addr))
protocol = ZEO.asyncio.tests.server_protocol(
zs, protocol_version=b'Z5', addr='test-addr-%s' % name)
zs.notify_connected(protocol)
zs.register('1', 0)
return zs
src/ZEO/tests/testAuth.py deleted 100644 → 0
View file @ bc92bdd7
##############################################################################
#
# Copyright (c) 2003 Zope Foundation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE
#
##############################################################################
"""Test suite for AuthZEO."""
import os
import tempfile
import time
import unittest
from ZEO import zeopasswd
from ZEO.Exceptions import ClientDisconnected
from ZEO.tests.ConnectionTests import CommonSetupTearDown
class _AuthTest(CommonSetupTearDown):
__super_getServerConfig = CommonSetupTearDown.getServerConfig
__super_setUp = CommonSetupTearDown.setUp
__super_tearDown = CommonSetupTearDown.tearDown
realm = None
def setUp(self):
fd, self.pwfile = tempfile.mkstemp('pwfile')
os.close(fd)
if self.realm:
self.pwdb = self.dbclass(self.pwfile, self.realm)
else:
self.pwdb = self.dbclass(self.pwfile)
self.pwdb.add_user("foo", "bar")
self.pwdb.save()
self._checkZEOpasswd()
self.__super_setUp()
def _checkZEOpasswd(self):
args = ["-f", self.pwfile, "-p", self.protocol]
if self.protocol == "plaintext":
from ZEO.auth.base import Database
zeopasswd.main(args + ["-d", "foo"], Database)
zeopasswd.main(args + ["foo", "bar"], Database)
else:
zeopasswd.main(args + ["-d", "foo"])
zeopasswd.main(args + ["foo", "bar"])
def tearDown(self):
os.remove(self.pwfile)
self.__super_tearDown()
def getConfig(self, path, create, read_only):
return "<mappingstorage 1/>"
def getServerConfig(self, addr, ro_svr):
zconf = self.__super_getServerConfig(addr, ro_svr)
zconf.authentication_protocol = self.protocol
zconf.authentication_database = self.pwfile
zconf.authentication_realm = self.realm
return zconf
def wait(self):
for i in range(25):
time.sleep(0.1)
if self._storage.test_connection:
return
self.fail("Timed out waiting for client to authenticate")
def testOK(self):
# Sleep for 0.2 seconds to give the server some time to start up
# seems to be needed before and after creating the storage
self._storage = self.openClientStorage(wait=0, username="foo",
password="bar", realm=self.realm)
self.wait()
self.assert_(self._storage._connection)
self._storage._connection.poll()
self.assert_(self._storage.is_connected())
# Make a call to make sure the mechanism is working
self._storage.undoInfo()
def testNOK(self):
self._storage = self.openClientStorage(wait=0, username="foo",
password="noogie",
realm=self.realm)
self.wait()
# If the test established a connection, then it failed.
self.failIf(self._storage._connection)
def testUnauthenticatedMessage(self):
# Test that an unauthenticated message is rejected by the server
# if it was sent after the connection was authenticated.
self._storage = self.openClientStorage(wait=0, username="foo",
password="bar", realm=self.realm)
# Sleep for 0.2 seconds to give the server some time to start up
# seems to be needed before and after creating the storage
self.wait()
self._storage.undoInfo()
# Manually clear the state of the hmac connection
self._storage._connection._SizedMessageAsyncConnection__hmac_send = None
# Once the client stops using the hmac, it should be disconnected.
self.assertRaises(ClientDisconnected, self._storage.undoInfo)
class PlainTextAuth(_AuthTest):
import ZEO.tests.auth_plaintext
protocol = "plaintext"
database = "authdb.sha"
dbclass = ZEO.tests.auth_plaintext.Database
realm = "Plaintext Realm"
class DigestAuth(_AuthTest):
import ZEO.auth.auth_digest
protocol = "digest"
database = "authdb.digest"
dbclass = ZEO.auth.auth_digest.DigestDatabase
realm = "Digest Realm"
test_classes = [PlainTextAuth, DigestAuth]
def test_suite():
suite = unittest.TestSuite()
for klass in test_classes:
sub = unittest.makeSuite(klass)
suite.addTest(sub)
return suite
if __name__ == "__main__":
unittest.main(defaultTest='test_suite')
src/ZEO/tests/testConfig.py
View file @ c499ea7d
......@@ -12,53 +12,115 @@
#
##############################################################################
import doctest
import tempfile
import unittest
import ZODB.config
import ZODB.tests.util
from ZODB.tests.testConfig import ConfigTestBase
class ZEOConfigTest(ConfigTestBase):
def test_zeo_config(self):
# We're looking for a port that doesn't exist so a
# connection attempt will fail. Instead of elaborate
# logic to loop over a port calculation, we'll just pick a
# simple "random", likely to not-exist port number and add
# an elaborate comment explaining this instead. Go ahead,
# grep for 9.
from ZEO.ClientStorage import ClientDisconnected
import ZConfig
from ZODB.config import getDbSchema
from ZEO._compat import StringIO
cfg = """
<zodb>
<zeoclient>
server localhost:56897
wait false
</zeoclient>
</zodb>
"""
config, handle = ZConfig.loadConfigFile(getDbSchema(), StringIO(cfg))
self.assertEqual(config.database[0].config.storage.config.blob_dir,
None)
self.assertRaises(ClientDisconnected, self._test, cfg)
cfg = """
<zodb>
<zeoclient>
blob-dir blobs
server localhost:56897
wait false
</zeoclient>
</zodb>
from zope.testing import setupstack
from ZODB.config import storageFromString
from .forker import start_zeo_server
from .threaded import threaded_server_tests
class ZEOConfigTestBase(setupstack.TestCase):
setUp = setupstack.setUpDirectory
def start_server(self, settings='', **kw):
for name, value in kw.items():
settings += '\n%s %s\n' % (name.replace('_', '-'), value)
zeo_conf = """
<zeo>
address 127.0.0.1:0
%s
</zeo>
""" % settings
return start_zeo_server("<mappingstorage>\n</mappingstorage>\n",
zeo_conf, threaded=True)
def start_client(self, addr, settings='', **kw):
settings += '\nserver %s:%s\n' % addr
for name, value in kw.items():
settings += '\n%s %s\n' % (name.replace('_', '-'), value)
return storageFromString(
"""
config, handle = ZConfig.loadConfigFile(getDbSchema(), StringIO(cfg))
self.assertEqual(config.database[0].config.storage.config.blob_dir,
'blobs')
self.assertRaises(ClientDisconnected, self._test, cfg)
%import ZEO
<clientstorage>
{}
</clientstorage>
""".format(settings))
def _client_assertions(
self, client, addr,
connected=True,
cache_size=20 * (1<<20),
cache_path=None,
blob_dir=None,
shared_blob_dir=False,
blob_cache_size=None,
blob_cache_size_check=10,
read_only=False,
read_only_fallback=False,
server_sync=False,
wait_timeout=30,
client_label=None,
storage='1',
name=None,
):
self.assertEqual(client.is_connected(), connected)
self.assertEqual(client._addr, [addr])
self.assertEqual(client._cache.maxsize, cache_size)
self.assertEqual(client._cache.path, cache_path)
self.assertEqual(client.blob_dir, blob_dir)
self.assertEqual(client.shared_blob_dir, shared_blob_dir)
self.assertEqual(client._blob_cache_size, blob_cache_size)
if blob_cache_size:
self.assertEqual(client._blob_cache_size_check,
blob_cache_size * blob_cache_size_check // 100)
self.assertEqual(client._is_read_only, read_only)
self.assertEqual(client._read_only_fallback, read_only_fallback)
self.assertEqual(client._server.timeout, wait_timeout)
self.assertEqual(client._client_label, client_label)
self.assertEqual(client._storage, storage)
self.assertEqual(client.__name__,
name if name is not None else str(client._addr))
class ZEOConfigTest(ZEOConfigTestBase):
def test_default_zeo_config(self, **client_settings):
addr, stop = self.start_server()
client = self.start_client(addr, **client_settings)
self._client_assertions(client, addr, **client_settings)
client.close()
stop()
def test_client_variations(self):
for name, value in dict(
cache_size=4200,
cache_path='test',
blob_dir='blobs',
blob_cache_size=424242,
read_only=True,
read_only_fallback=True,
server_sync=True,
wait_timeout=33,
client_label='test_client',
name='Test'
).items():
params = {name: value}
self.test_default_zeo_config(**params)
def test_blob_cache_size_check(self):
self.test_default_zeo_config(blob_cache_size=424242,
blob_cache_size_check=50)
def test_suite():
return unittest.makeSuite(ZEOConfigTest)
suite = unittest.makeSuite(ZEOConfigTest)
suite.layer = threaded_server_tests
return suite
src/ZEO/tests/testConnection.py
View file @ c499ea7d
......@@ -27,11 +27,12 @@ if os.environ.get('USE_ZOPE_TESTING_DOCTEST'):
else:
import doctest
import unittest
import ZEO.tests.forker
import ZEO.tests.testMonitor
import ZEO.zrpc.connection
import ZODB.tests.util
import ZEO
from . import forker
class FileStorageConfig:
def getConfig(self, path, create, read_only):
return """\
......@@ -90,40 +91,11 @@ class MappingStorageTimeoutTests(
):
pass
class MonitorTests(ZEO.tests.testMonitor.MonitorTests):
def check_connection_management(self):
# Open and close a few connections, making sure that
# the resulting number of clients is 0.
s1 = self.openClientStorage()
s2 = self.openClientStorage()
s3 = self.openClientStorage()
stats = self.parse(self.get_monitor_output())[1]
self.assertEqual(stats.clients, 3)
s1.close()
s3.close()
s2.close()
ZEO.tests.forker.wait_until(
"Number of clients shown in monitor drops to 0",
lambda :
self.parse(self.get_monitor_output())[1].clients == 0
)
def check_connection_management_with_old_client(self):
# Check that connection management works even when using an
# older protcool that requires a connection adapter.
test_protocol = b"Z303"
current_protocol = ZEO.zrpc.connection.Connection.current_protocol
ZEO.zrpc.connection.Connection.current_protocol = test_protocol
ZEO.zrpc.connection.Connection.servers_we_can_talk_to.append(
test_protocol)
try:
self.check_connection_management()
finally:
ZEO.zrpc.connection.Connection.current_protocol = current_protocol
ZEO.zrpc.connection.Connection.servers_we_can_talk_to.pop()
class SSLConnectionTests(
MappingStorageConfig,
ConnectionTests.SSLConnectionTests,
):
pass
test_classes = [FileStorageConnectionTests,
......@@ -132,8 +104,9 @@ test_classes = [FileStorageConnectionTests,
FileStorageTimeoutTests,
MappingStorageConnectionTests,
MappingStorageTimeoutTests,
MonitorTests,
]
if not forker.ZEO4_SERVER:
test_classes.append(SSLConnectionTests)
def invalidations_while_connecting():
r"""
......@@ -154,7 +127,7 @@ This tests tries to provoke this bug by:
- opening a client to the server that writes some objects, filling
it's cache at the same time,
>>> import ZODB.tests.MinPO, transaction
>>> import ZEO, ZODB.tests.MinPO, transaction
>>> db = ZEO.DB(addr, client='x')
>>> conn = db.open()
>>> nobs = 1000
......@@ -213,7 +186,7 @@ This tests tries to provoke this bug by:
... def _():
... if (db.storage.is_connected()
... and db.storage.lastTransaction()
... == db.storage._server.lastTransaction()
... == db.storage._call('lastTransaction')
... ):
... #logging.getLogger('ZEO').debug(
... # 'Connected %r' % db.storage.lastTransaction())
......@@ -230,9 +203,9 @@ This tests tries to provoke this bug by:
... record = handler.records.pop(0)
... print(record.name, record.levelname, end=' ')
... print(handler.format(record))
... if bad:
... with open('server-%s.log' % addr[1]) as f:
... print(f.read())
... #if bad:
... # with open('server.log') as f:
... # print(f.read())
... #else:
... # logging.getLogger('ZEO').debug('GOOD %s' % c)
... db.close()
......@@ -261,7 +234,7 @@ def test_suite():
sub = unittest.makeSuite(klass, 'check')
suite.addTest(sub)
suite.addTest(doctest.DocTestSuite(
setUp=ZEO.tests.forker.setUp, tearDown=setupstack.tearDown,
setUp=forker.setUp, tearDown=setupstack.tearDown,
))
suite.layer = ZODB.tests.util.MininalTestLayer('ZEO Connection Tests')
return suite
src/ZEO/tests/testConversionSupport.py
View file @ c499ea7d
......@@ -14,6 +14,8 @@
import doctest
import unittest
import ZEO.asyncio.testing
class FakeStorageBase:
def __getattr__(self, name):
......@@ -46,10 +48,20 @@ class FakeServer:
'1': FakeStorage(),
'2': FakeStorageBase(),
}
lock_managers = storages
def register_connection(*args):
return None, None
client_conflict_resolution = False
class FakeConnection:
protocol_version = b'Z4'
addr = 'test'
call_soon_threadsafe = lambda f, *a: f(*a)
async = async_threadsafe = None
def test_server_record_iternext():
"""
......@@ -59,6 +71,7 @@ underlying storage.
>>> import ZEO.StorageServer
>>> zeo = ZEO.StorageServer.ZEOStorage(FakeServer(), False)
>>> zeo.notify_connected(FakeConnection())
>>> zeo.register('1', False)
>>> next = None
......@@ -78,6 +91,7 @@ The storage info also reflects the fact that record_iternext is supported.
True
>>> zeo = ZEO.StorageServer.ZEOStorage(FakeServer(), False)
>>> zeo.notify_connected(FakeConnection())
>>> zeo.register('2', False)
>>> zeo.get_info()['supports_record_iternext']
......@@ -85,9 +99,8 @@ The storage info also reflects the fact that record_iternext is supported.
"""
def test_client_record_iternext():
"""\
"""Test client storage delegation to the network client
The client simply delegates record_iternext calls to it's server stub.
......@@ -96,53 +109,28 @@ stuff. I'd rather do a lame test than a really lame test, so here goes.
First, fake out the connection manager so we can make a connection:
>>> import ZEO.ClientStorage
>>> from ZEO.ClientStorage import ClientStorage
>>> oldConnectionManagerClass = ClientStorage.ConnectionManagerClass
>>> class FauxConnectionManagerClass:
... def __init__(*a, **k):
... pass
... def attempt_connect(self):
... return True
>>> ClientStorage.ConnectionManagerClass = FauxConnectionManagerClass
>>> client = ClientStorage('', wait=False)
>>> ClientStorage.ConnectionManagerClass = oldConnectionManagerClass
Now we'll have our way with it's private _server attr:
>>> client._server = FakeStorage()
>>> next = None
>>> while 1:
... oid, serial, data, next = client.record_iternext(next)
... print(oid)
... if next is None:
... break
1
2
3
4
"""
>>> import ZEO
def test_server_stub_record_iternext():
"""\
>>> class Client(ZEO.asyncio.testing.ClientRunner):
...
... def record_iternext(self, next=None):
... if next == None:
... next = '0'
... next = str(int(next) + 1)
... oid = next
... if next == '4':
... next = None
...
... return oid, oid*8, 'data ' + oid, next
The server stub simply delegates record_iternext calls to it's rpc.
>>> client = ZEO.client(
... '', wait=False, _client_factory=Client)
There's really no decent way to test ZEO without running to much crazy
stuff. I'd rather do a lame test than a really lame test, so here goes.
>>> class FauxRPC:
... storage = FakeStorage()
... def call(self, meth, *args):
... return getattr(self.storage, meth)(*args)
... peer_protocol_version = 1
Now we'll have our way with it's private _server attr:
>>> import ZEO.ServerStub
>>> stub = ZEO.ServerStub.StorageServer(FauxRPC())
>>> next = None
>>> while 1:
... oid, serial, data, next = stub.record_iternext(next)
... oid, serial, data, next = client.record_iternext(next)
... print(oid)
... if next is None:
... break
......@@ -153,44 +141,8 @@ stuff. I'd rather do a lame test than a really lame test, so here goes.
"""
def history_to_version_compatible_storage():
"""
Some storages work under ZODB <= 3.8 and ZODB >= 3.9.
This means they have a history method that accepts a version parameter:
>>> class VersionCompatibleStorage(FakeStorageBase):
... def history(self,oid,version='',size=1):
... return oid,version,size
A ZEOStorage such as the following should support this type of storage:
>>> class OurFakeServer(FakeServer):
... storages = {'1':VersionCompatibleStorage()}
>>> import ZEO.StorageServer
>>> zeo = ZEO.StorageServer.ZEOStorage(OurFakeServer(), False)
>>> zeo.register('1', False)
The ZEOStorage should sort out the following call such that the storage gets
the correct parameters and so should return the parameters it was called with:
>>> zeo.history('oid',99)
('oid', '', 99)
The same problem occurs when a Z308 client connects to a Z309 server,
but different code is executed:
>>> from ZEO.StorageServer import ZEOStorage308Adapter
>>> zeo = ZEOStorage308Adapter(VersionCompatibleStorage())
The history method should still return the parameters it was called with:
>>> zeo.history('oid','',99)
('oid', '', 99)
"""
def test_suite():
return doctest.DocTestSuite()
if __name__ == '__main__':
unittest.main(defaultTest='test_suite')
src/ZEO/tests/testMonitor.py deleted 100644 → 0
View file @ bc92bdd7
##############################################################################
#
# Copyright (c) 2003 Zope Foundation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE
#
##############################################################################
"""Test that the monitor produce sensible results.
$Id$
"""
import socket
import unittest
from ZEO.tests.ConnectionTests import CommonSetupTearDown
from ZEO.monitor import StorageStats
class MonitorTests(CommonSetupTearDown):
monitor = 1
def get_monitor_output(self):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('localhost', 42000))
L = []
while 1:
buf = s.recv(8192)
if buf:
L.append(buf)
else:
break
s.close()
return b"".join(L).decode('ascii')
def parse(self, s):
# Return a list of StorageStats, one for each storage.
lines = s.split("\n")
self.assert_(lines[0].startswith("ZEO monitor server"))
# lines[1] is a date
# Break up rest of lines into sections starting with Storage:
# and ending with a blank line.
sections = []
cur = None
for line in lines[2:]:
if line.startswith("Storage:"):
cur = [line]
elif line:
cur.append(line)
else:
if cur is not None:
sections.append(cur)
cur = None
assert cur is None # bug in the test code if this fails
d = {}
for sect in sections:
hdr = sect[0]
key, value = hdr.split(":")
storage = int(value)
s = d[storage] = StorageStats()
s.parse("\n".join(sect[1:]))
return d
def getConfig(self, path, create, read_only):
return """<mappingstorage 1/>"""
def testMonitor(self):
# Just open a client to know that the server is up and running
# TODO: should put this in setUp.
self.storage = self.openClientStorage()
s = self.get_monitor_output()
self.storage.close()
self.assert_(s.find("monitor") != -1)
d = self.parse(s)
stats = d[1]
self.assertEqual(stats.clients, 1)
self.assertEqual(stats.commits, 0)
def test_suite():
return unittest.makeSuite(MonitorTests)
src/ZEO/tests/testTransactionBuffer.py
View file @ c499ea7d
......@@ -25,46 +25,32 @@ def new_store_data():
"""Return arbitrary data to use as argument to store() method."""
return random_string(8), random_string(random.randrange(1000))
def new_invalidate_data():
"""Return arbitrary data to use as argument to invalidate() method."""
return random_string(8)
def store(tbuf, resolved=False):
data = new_store_data()
tbuf.store(*data)
if resolved:
tbuf.server_resolve(data[0])
return data
class TransBufTests(unittest.TestCase):
def checkTypicalUsage(self):
tbuf = TransactionBuffer()
tbuf.store(*new_store_data())
tbuf.invalidate(new_invalidate_data())
tbuf = TransactionBuffer(0)
store(tbuf)
store(tbuf)
for o in tbuf:
pass
def doUpdates(self, tbuf):
def checkOrderPreserved(self):
tbuf = TransactionBuffer(0)
data = []
for i in range(10):
d = new_store_data()
tbuf.store(*d)
data.append(d)
d = new_invalidate_data()
tbuf.invalidate(d)
data.append(d)
for i, x in enumerate(tbuf):
if x[1] is None:
# the tbuf add a dummy None to invalidates
x = x[0]
self.assertEqual(x, data[i])
def checkOrderPreserved(self):
tbuf = TransactionBuffer()
self.doUpdates(tbuf)
data.append((store(tbuf), False))
data.append((store(tbuf, True), True))
def checkReusable(self):
tbuf = TransactionBuffer()
self.doUpdates(tbuf)
tbuf.clear()
self.doUpdates(tbuf)
tbuf.clear()
self.doUpdates(tbuf)
for i, (oid, d, resolved) in enumerate(tbuf):
self.assertEqual((oid, d), data[i][0])
self.assertEqual(resolved, data[i][1])
def test_suite():
return unittest.makeSuite(TransBufTests, 'check')
src/ZEO/tests/testZEO.py
View file @ c499ea7d
......@@ -17,10 +17,8 @@ import multiprocessing
import re
from ZEO.ClientStorage import ClientStorage
from ZEO.tests.forker import get_port
from ZEO.tests import forker, Cache, CommitLockTests, ThreadTests
from ZEO.tests import IterationTests
from ZEO.zrpc.error import DisconnectedError
from ZEO._compat import PY3
from ZODB.tests import StorageTestBase, BasicStorage, \
TransactionalUndoStorage, \
......@@ -28,7 +26,7 @@ from ZODB.tests import StorageTestBase, BasicStorage, \
MTStorage, ReadOnlyStorage, IteratorStorage, RecoveryStorage
from ZODB.tests.MinPO import MinPO
from ZODB.tests.StorageTestBase import zodb_unpickle
from ZODB.utils import p64, u64
from ZODB.utils import maxtid, p64, u64, z64
from zope.testing import renormalizing
import doctest
......@@ -40,16 +38,15 @@ import re
import shutil
import signal
import stat
import ssl
import sys
import tempfile
import threading
import time
import transaction
import unittest
import ZEO.ServerStub
import ZEO.StorageServer
import ZEO.tests.ConnectionTests
import ZEO.zrpc.connection
import ZODB
import ZODB.blob
import ZODB.tests.hexstorage
......@@ -58,6 +55,8 @@ import ZODB.tests.util
import ZODB.utils
import zope.testing.setupstack
from . import testssl
logger = logging.getLogger('ZEO.tests.testZEO')
class DummyDB:
......@@ -102,7 +101,7 @@ class MiscZEOTests:
def checkZEOInvalidation(self):
addr = self._storage._addr
storage2 = self._wrap_client(
ClientStorage(addr, wait=1, min_disconnect_poll=0.1))
ClientStorage(addr, wait=1, **self._client_options()))
try:
oid = self._storage.new_oid()
ob = MinPO('first')
......@@ -131,114 +130,87 @@ class MiscZEOTests:
# Earlier, a ClientStorage would not have the last transaction id
# available right after successful connection, this is required now.
addr = self._storage._addr
storage2 = ClientStorage(addr)
storage2 = ClientStorage(addr, **self._client_options())
self.assert_(storage2.is_connected())
self.assertEquals(ZODB.utils.z64, storage2.lastTransaction())
storage2.close()
self._dostore()
storage3 = ClientStorage(addr)
storage3 = ClientStorage(addr, **self._client_options())
self.assert_(storage3.is_connected())
self.assertEquals(8, len(storage3.lastTransaction()))
self.assertNotEquals(ZODB.utils.z64, storage3.lastTransaction())
storage3.close()
class ConfigurationTests(unittest.TestCase):
def checkDropCacheRatherVerifyConfiguration(self):
from ZODB.config import storageFromString
# the default is to do verification and not drop the cache
cs = storageFromString('''
<zeoclient>
server localhost:9090
wait false
</zeoclient>
''')
self.assertEqual(cs._drop_cache_rather_verify, False)
cs.close()
# now for dropping
cs = storageFromString('''
<zeoclient>
server localhost:9090
wait false
drop-cache-rather-verify true
</zeoclient>
''')
self.assertEqual(cs._drop_cache_rather_verify, True)
cs.close()
class GenericTests(
class GenericTestBase(
# Base class for all ZODB tests
StorageTestBase.StorageTestBase,
# ZODB test mixin classes (in the same order as imported)
BasicStorage.BasicStorage,
PackableStorage.PackableStorage,
Synchronization.SynchronizedStorage,
MTStorage.MTStorage,
ReadOnlyStorage.ReadOnlyStorage,
# ZEO test mixin classes (in the same order as imported)
CommitLockTests.CommitLockVoteTests,
ThreadTests.ThreadTests,
# Locally defined (see above)
MiscZEOTests,
):
"""Combine tests from various origins in one class."""
StorageTestBase.StorageTestBase):
shared_blob_dir = False
blob_cache_dir = None
server_debug = False
def setUp(self):
StorageTestBase.StorageTestBase.setUp(self)
logger.info("setUp() %s", self.id())
port = get_port(self)
zconf = forker.ZEOConfig(('', port))
zport, adminaddr, pid, path = forker.start_zeo_server(self.getConfig(),
zconf, port)
self._pids = [pid]
self._servers = [adminaddr]
self._conf_path = path
zport, stop = forker.start_zeo_server(
self.getConfig(), self.getZEOConfig(), debug=self.server_debug)
self._servers = [stop]
if not self.blob_cache_dir:
# This is the blob cache for ClientStorage
self.blob_cache_dir = tempfile.mkdtemp(
'blob_cache',
dir=os.path.abspath(os.getcwd()))
self._storage = self._wrap_client(ClientStorage(
self._storage = self._wrap_client(
ClientStorage(
zport, '1', cache_size=20000000,
min_disconnect_poll=0.5, wait=1,
wait_timeout=60, blob_dir=self.blob_cache_dir,
shared_blob_dir=self.shared_blob_dir))
shared_blob_dir=self.shared_blob_dir,
**self._client_options()),
)
self._storage.registerDB(DummyDB())
def getZEOConfig(self):
return forker.ZEOConfig(('127.0.0.1', 0))
def _wrap_client(self, client):
return client
def _client_options(self):
return {}
def tearDown(self):
self._storage.close()
for server in self._servers:
forker.shutdown_zeo_server(server)
if hasattr(os, 'waitpid'):
# Not in Windows Python until 2.3
for pid in self._pids:
os.waitpid(pid, 0)
for stop in self._servers:
stop()
StorageTestBase.StorageTestBase.tearDown(self)
def runTest(self):
try:
super(GenericTests, self).runTest()
except:
self._failed = True
raise
else:
self._failed = False
class GenericTests(
GenericTestBase,
# ZODB test mixin classes (in the same order as imported)
BasicStorage.BasicStorage,
PackableStorage.PackableStorage,
Synchronization.SynchronizedStorage,
MTStorage.MTStorage,
ReadOnlyStorage.ReadOnlyStorage,
# ZEO test mixin classes (in the same order as imported)
CommitLockTests.CommitLockVoteTests,
ThreadTests.ThreadTests,
# Locally defined (see above)
MiscZEOTests,
):
"""Combine tests from various origins in one class.
"""
def open(self, read_only=0):
# Needed to support ReadOnlyStorage tests. Ought to be a
# cleaner way.
addr = self._storage._addr
self._storage.close()
self._storage = ClientStorage(addr, read_only=read_only, wait=1)
self._storage = ClientStorage(
addr, read_only=read_only, wait=1, **self._client_options())
def checkWriteMethods(self):
# ReadOnlyStorage defines checkWriteMethods. The decision
......@@ -257,7 +229,8 @@ class GenericTests(
def _do_store_in_separate_thread(self, oid, revid, voted):
def do_store():
store = ZEO.ClientStorage.ClientStorage(self._storage._addr)
store = ZEO.ClientStorage.ClientStorage(
self._storage._addr, **self._client_options())
try:
t = transaction.get()
store.tpc_begin(t)
......@@ -302,12 +275,10 @@ class FileStorageRecoveryTests(StorageTestBase.StorageTestBase,
""" % tempfile.mktemp(dir='.')
def _new_storage(self):
port = get_port(self)
zconf = forker.ZEOConfig(('', port))
zport, adminaddr, pid, path = forker.start_zeo_server(self.getConfig(),
zconf, port)
self._pids.append(pid)
self._servers.append(adminaddr)
zconf = forker.ZEOConfig(('127.0.0.1', 0))
zport, stop = forker.start_zeo_server(self.getConfig(),
zconf)
self._servers.append(stop)
blob_cache_dir = tempfile.mkdtemp(dir='.')
......@@ -320,7 +291,6 @@ class FileStorageRecoveryTests(StorageTestBase.StorageTestBase,
def setUp(self):
StorageTestBase.StorageTestBase.setUp(self)
self._pids = []
self._servers = []
self._storage = self._new_storage()
......@@ -330,12 +300,8 @@ class FileStorageRecoveryTests(StorageTestBase.StorageTestBase,
self._storage.close()
self._dst.close()
for server in self._servers:
forker.shutdown_zeo_server(server)
if hasattr(os, 'waitpid'):
# Not in Windows Python until 2.3
for pid in self._pids:
os.waitpid(pid, 0)
for stop in self._servers:
stop()
StorageTestBase.StorageTestBase.tearDown(self)
def new_dest(self):
......@@ -375,6 +341,15 @@ class FileStorageTests(FullGenericTests):
self._storage._info['interfaces']
)
class FileStorageSSLTests(FileStorageTests):
def getZEOConfig(self):
return testssl.server_config
def _client_options(self):
return {'ssl': testssl.client_ssl()}
class FileStorageHexTests(FileStorageTests):
_expected_interfaces = (
('ZODB.interfaces', 'IStorageRestoreable'),
......@@ -412,7 +387,16 @@ class FileStorageClientHexTests(FileStorageHexTests):
def _wrap_client(self, client):
return ZODB.tests.hexstorage.HexStorage(client)
class ClientConflictResolutionTests(
GenericTestBase,
ConflictResolution.ConflictResolvingStorage,
):
def getConfig(self):
return '<mappingstorage>\n</mappingstorage>\n'
def getZEOConfig(self):
return forker.ZEOConfig(('', 0), client_conflict_resolution=True)
class MappingStorageTests(GenericTests):
"""ZEO backed by a Mapping storage."""
......@@ -452,56 +436,6 @@ class DemoStorageTests(
pass # DemoStorage pack doesn't do gc
checkPackAllRevisions = checkPackWithMultiDatabaseReferences
class HeartbeatTests(ZEO.tests.ConnectionTests.CommonSetupTearDown):
"""Make sure a heartbeat is being sent and that it does no harm
This is really hard to test properly because we can't see the data
flow between the client and server and we can't really tell what's
going on in the server very well. :(
"""
def setUp(self):
# Crank down the select frequency
self.__old_client_timeout = ZEO.zrpc.client.client_timeout
ZEO.zrpc.client.client_timeout = self.__client_timeout
ZEO.tests.ConnectionTests.CommonSetupTearDown.setUp(self)
__client_timeouts = 0
def __client_timeout(self):
self.__client_timeouts += 1
return .1
def tearDown(self):
ZEO.zrpc.client.client_timeout = self.__old_client_timeout
ZEO.tests.ConnectionTests.CommonSetupTearDown.tearDown(self)
def getConfig(self, path, create, read_only):
return """<mappingstorage 1/>"""
def checkHeartbeatWithServerClose(self):
# This is a minimal test that mainly tests that the heartbeat
# function does no harm.
self._storage = self.openClientStorage()
client_timeouts = self.__client_timeouts
forker.wait_until('got a timeout',
lambda : self.__client_timeouts > client_timeouts
)
self._dostore()
if hasattr(os, 'kill') and hasattr(signal, 'SIGKILL'):
# Kill server violently, in hopes of provoking problem
os.kill(self._pids[0], signal.SIGKILL)
self._servers[0] = None
else:
self.shutdownServer()
forker.wait_until('disconnected',
lambda : not self._storage.is_connected()
)
self._storage.close()
class ZRPCConnectionTests(ZEO.tests.ConnectionTests.CommonSetupTearDown):
def getConfig(self, path, create, read_only):
......@@ -511,20 +445,15 @@ class ZRPCConnectionTests(ZEO.tests.ConnectionTests.CommonSetupTearDown):
# Test what happens when the client loop falls over
self._storage = self.openClientStorage()
class Evil:
def writable(self):
raise SystemError("I'm evil")
import zope.testing.loggingsupport
handler = zope.testing.loggingsupport.InstalledHandler(
'ZEO.zrpc.client')
'ZEO.asyncio.client')
self._storage._rpc_mgr.map[None] = Evil()
try:
self._storage._rpc_mgr.trigger.pull_trigger()
except DisconnectedError:
pass
# We no longer implement the event loop, we we no longer know
# how to break it. We'll just stop it instead for now.
self._storage._server.loop.call_soon_threadsafe(
self._storage._server.loop.stop)
forker.wait_until(
'disconnected',
......@@ -533,62 +462,31 @@ class ZRPCConnectionTests(ZEO.tests.ConnectionTests.CommonSetupTearDown):
log = str(handler)
handler.uninstall()
self.assert_("ZEO client loop failed" in log)
self.assert_("Couldn't close a dispatcher." in log)
self.assert_("Client loop stopped unexpectedly" in log)
def checkExceptionLogsAtError(self):
# Test the exceptions are logged at error
self._storage = self.openClientStorage()
conn = self._storage._connection
# capture logging
log = []
conn.logger.log = (
lambda l, m, *a, **kw: log.append((l,m % a, kw))
)
# This is a deliberately bogus call to get an exception
# logged
self._storage._connection.handle_request(
'foo', 0, 'history', (1, 2, 3, 4))
# test logging
py2_msg = (
'history() raised exception: history() takes at most '
'3 arguments (5 given)'
)
py32_msg = (
'history() raised exception: history() takes at most '
'3 positional arguments (5 given)'
)
py3_msg = (
'history() raised exception: history() takes '
'from 2 to 3 positional arguments but 5 were given'
)
for level, message, kw in log:
if (message.endswith(py2_msg) or
message.endswith(py32_msg) or
message.endswith(py3_msg)):
self.assertEqual(level,logging.ERROR)
self.assertEqual(kw,{'exc_info':True})
break
else:
self.fail("error not in log %s" % log)
# cleanup
del conn.logger.log
self._dostore(z64, data=MinPO("X" * (10 * 128 * 1024)))
from zope.testing.loggingsupport import InstalledHandler
handler = InstalledHandler('ZEO.asyncio.client')
import ZODB.POSException
self.assertRaises(TypeError, self._storage.history, z64, None)
self.assertTrue(re.search(" from server: .*TypeError", str(handler)))
# POSKeyErrors and ConflictErrors aren't logged:
handler.clear()
self.assertRaises(ZODB.POSException.POSKeyError,
self._storage.history, None, None)
handler.uninstall()
self.assertEquals(str(handler), '')
def checkConnectionInvalidationOnReconnect(self):
storage = ClientStorage(self.addr, wait=1, min_disconnect_poll=0.1)
storage = ClientStorage(self.addr, min_disconnect_poll=0.1)
self._storage = storage
# and we'll wait for the storage to be reconnected:
for i in range(100):
if storage.is_connected():
break
time.sleep(0.1)
else:
raise AssertionError("Couldn't connect to server")
assert storage.is_connected()
class DummyDB:
_invalidatedCache = 0
......@@ -596,6 +494,8 @@ class ZRPCConnectionTests(ZEO.tests.ConnectionTests.CommonSetupTearDown):
self._invalidatedCache += 1
def invalidate(*a, **k):
pass
transform_record_data = untransform_record_data = \
lambda self, data: data
db = DummyDB()
storage.registerDB(db)
......@@ -603,11 +503,14 @@ class ZRPCConnectionTests(ZEO.tests.ConnectionTests.CommonSetupTearDown):
base = db._invalidatedCache
# Now we'll force a disconnection and reconnection
storage._connection.close()
storage._server.loop.call_soon_threadsafe(
storage._server.client.protocol.connection_lost,
ValueError('test'))
# and we'll wait for the storage to be reconnected:
for i in range(100):
if storage.is_connected():
if db._invalidatedCache > base:
break
time.sleep(0.1)
else:
......@@ -635,7 +538,6 @@ class CommonBlobTests:
def checkStoreBlob(self):
import transaction
from ZODB.blob import Blob
from ZODB.tests.StorageTestBase import handle_serials
from ZODB.tests.StorageTestBase import ZERO
from ZODB.tests.StorageTestBase import zodb_pickle
......@@ -652,10 +554,9 @@ class CommonBlobTests:
t = transaction.Transaction()
try:
self._storage.tpc_begin(t)
r1 = self._storage.storeBlob(oid, ZERO, data, tfname, '', t)
r2 = self._storage.tpc_vote(t)
revid = handle_serials(oid, r1, r2)
self._storage.tpc_finish(t)
self._storage.storeBlob(oid, ZERO, data, tfname, '', t)
self._storage.tpc_vote(t)
revid = self._storage.tpc_finish(t)
except:
self._storage.tpc_abort(t)
raise
......@@ -677,8 +578,7 @@ class CommonBlobTests:
def checkLoadBlob(self):
from ZODB.blob import Blob
from ZODB.tests.StorageTestBase import zodb_pickle, ZERO, \
handle_serials
from ZODB.tests.StorageTestBase import zodb_pickle, ZERO
import transaction
somedata = b'a' * 10
......@@ -693,10 +593,9 @@ class CommonBlobTests:
t = transaction.Transaction()
try:
self._storage.tpc_begin(t)
r1 = self._storage.storeBlob(oid, ZERO, data, tfname, '', t)
r2 = self._storage.tpc_vote(t)
serial = handle_serials(oid, r1, r2)
self._storage.tpc_finish(t)
self._storage.storeBlob(oid, ZERO, data, tfname, '', t)
self._storage.tpc_vote(t)
serial = self._storage.tpc_finish(t)
except:
self._storage.tpc_abort(t)
raise
......@@ -728,7 +627,6 @@ class BlobAdaptedFileStorageTests(FullGenericTests, CommonBlobTests):
def checkStoreAndLoadBlob(self):
import transaction
from ZODB.blob import Blob
from ZODB.tests.StorageTestBase import handle_serials
from ZODB.tests.StorageTestBase import ZERO
from ZODB.tests.StorageTestBase import zodb_pickle
......@@ -760,10 +658,9 @@ class BlobAdaptedFileStorageTests(FullGenericTests, CommonBlobTests):
t = transaction.Transaction()
try:
self._storage.tpc_begin(t)
r1 = self._storage.storeBlob(oid, ZERO, data, tfname, '', t)
r2 = self._storage.tpc_vote(t)
revid = handle_serials(oid, r1, r2)
self._storage.tpc_finish(t)
self._storage.storeBlob(oid, ZERO, data, tfname, '', t)
self._storage.tpc_vote(t)
revid = self._storage.tpc_finish(t)
except:
self._storage.tpc_abort(t)
raise
......@@ -785,15 +682,7 @@ class BlobAdaptedFileStorageTests(FullGenericTests, CommonBlobTests):
check_data(server_filename)
# If we remove it from the cache and call loadBlob, it should
# come back. We can do this in many threads. We'll instrument
# the method that is used to request data from teh server to
# verify that it is only called once.
sendBlob_org = ZEO.ServerStub.StorageServer.sendBlob
calls = []
def sendBlob(self, oid, serial):
calls.append((oid, serial))
sendBlob_org(self, oid, serial)
# come back. We can do this in many threads.
ZODB.blob.remove_committed(filename)
returns = []
......@@ -817,27 +706,23 @@ class BlobWritableCacheTests(FullGenericTests, CommonBlobTests):
class FauxConn:
addr = 'x'
peer_protocol_version = ZEO.zrpc.connection.Connection.current_protocol
protocol_version = ZEO.asyncio.server.best_protocol_version
peer_protocol_version = protocol_version
class StorageServerClientWrapper:
serials = []
def async(self, method, *args):
if method == 'serialnos':
self.serials.extend(args[0])
def __init__(self):
self.serials = []
def serialnos(self, serials):
self.serials.extend(serials)
def info(self, info):
pass
call_soon_threadsafe = async_threadsafe = async
class StorageServerWrapper:
def __init__(self, server, storage_id):
self.storage_id = storage_id
self.server = ZEO.StorageServer.ZEOStorage(server, server.read_only)
self.server.notifyConnected(FauxConn())
self.server.notify_connected(FauxConn())
self.server.register(storage_id, False)
self.server.client = StorageServerClientWrapper()
def sortKey(self):
return self.storage_id
......@@ -858,24 +743,23 @@ class StorageServerWrapper:
self.server.tpc_begin(id(transaction), '', '', {}, None, ' ')
def tpc_vote(self, transaction):
vote_result = self.server.vote(id(transaction))
assert vote_result is None
result = self.server.client.serials[:]
del self.server.client.serials[:]
result = self.server.vote(id(transaction))
assert result == self.server.connection.serials[:]
del self.server.connection.serials[:]
return result
def store(self, oid, serial, data, version_ignored, transaction):
self.server.storea(oid, serial, data, id(transaction))
def send_reply(self, *args): # Masquerade as conn
pass
def send_reply(self, _, result): # Masquerade as conn
self._result = result
def tpc_abort(self, transaction):
self.server.tpc_abort(id(transaction))
def tpc_finish(self, transaction, func = lambda: None):
self.server.tpc_finish(id(transaction)).set_sender(0, self)
return self._result
def multiple_storages_invalidation_queue_is_not_insane():
"""
......@@ -886,7 +770,7 @@ def multiple_storages_invalidation_queue_is_not_insane():
>>> from transaction import commit
>>> fs1 = FileStorage('t1.fs')
>>> fs2 = FileStorage('t2.fs')
>>> server = StorageServer(('', get_port()), dict(fs1=fs1, fs2=fs2))
>>> server = StorageServer(None, storages=dict(fs1=fs1, fs2=fs2))
>>> s1 = StorageServerWrapper(server, 'fs1')
>>> s2 = StorageServerWrapper(server, 'fs2')
......@@ -915,7 +799,7 @@ def multiple_storages_invalidation_queue_is_not_insane():
>>> sorted([int(u64(oid)) for oid in oids])
[10, 11, 12, 13, 14]
>>> server.close()
>>> fs1.close(); fs2.close()
"""
def getInvalidationsAfterServerRestart():
......@@ -945,10 +829,11 @@ Let's create a file storage and stuff some data into it:
Now we'll open a storage server on the data, simulating a restart:
>>> fs = FileStorage('t.fs')
>>> sv = StorageServer(('', get_port()), dict(fs=fs))
>>> sv = StorageServer(None, dict(fs=fs))
>>> s = ZEOStorage(sv, sv.read_only)
>>> s.notifyConnected(FauxConn())
>>> s.register('fs', False)
>>> s.notify_connected(FauxConn())
>>> s.register('fs', False) == fs.lastTransaction()
True
If we ask for the last transaction, we should get the last transaction
we saved:
......@@ -957,7 +842,7 @@ we saved:
True
If a storage implements the method lastInvalidations, as FileStorage
does, then the stroage server will populate its invalidation data
does, then the storage server will populate its invalidation data
structure using lastTransactions.
......@@ -979,7 +864,6 @@ need to be invalidated. This means we'll invalidate objects that
dont' need to be invalidated, however, that's better than verifying
caches.)
>>> sv.close()
>>> fs.close()
If a storage doesn't implement lastInvalidations, a client can still
......@@ -991,7 +875,7 @@ without this method:
... lastInvalidations = property()
>>> fs = FS('t.fs')
>>> sv = StorageServer(('', get_port()), dict(fs=fs))
>>> sv = StorageServer(None, dict(fs=fs))
>>> st = StorageServerWrapper(sv, 'fs')
>>> s = st.server
......@@ -1024,108 +908,50 @@ transaction, we'll get a result:
def tpc_finish_error():
r"""Server errors in tpc_finish weren't handled properly.
>>> import ZEO.ClientStorage, ZEO.zrpc.connection
If there are errors applying changes to the client cache, don't
leave the cache in an inconsistent state.
>>> class Connection:
... peer_protocol_version = (
... ZEO.zrpc.connection.Connection.current_protocol)
... def __init__(self, client):
... self.client = client
... def get_addr(self):
... return 'server'
... def is_async(self):
... return True
... def register_object(self, ob):
... pass
... def close(self):
... print('connection closed')
... trigger = property(lambda self: self)
... pull_trigger = lambda self, func, *args: func(*args)
>>> class ConnectionManager:
... def __init__(self, addr, client, tmin, tmax):
... self.client = client
... def connect(self, sync=1):
... self.client.notifyConnected(Connection(self.client))
... def close(self):
... pass
>>> class StorageServer:
... should_fail = True
... def __init__(self, conn):
... self.conn = conn
... self.t = None
... def get_info(self):
... return {}
... def endZeoVerify(self):
... self.conn.client.endVerify()
... def lastTransaction(self):
... return b'\0'*8
... def tpc_begin(self, t, *args):
... if self.t is not None:
... raise TypeError('already trans')
... self.t = t
... print('begin', args)
... def vote(self, t):
... if self.t != t:
... raise TypeError('bad trans')
... print('vote')
... def tpc_finish(self, *args):
... if self.should_fail:
... raise TypeError()
... print('finish')
... def tpc_abort(self, t):
... if self.t != t:
... raise TypeError('bad trans')
... self.t = None
... print('abort')
... def iterator_gc(*args):
... pass
>>> class ClientStorage(ZEO.ClientStorage.ClientStorage):
... ConnectionManagerClass = ConnectionManager
... StorageServerStubClass = StorageServer
>>> class Transaction:
... user = 'test'
... description = ''
... _extension = {}
>>> cs = ClientStorage(('', ''))
>>> t1 = Transaction()
>>> cs.tpc_begin(t1)
begin ('test', '', {}, None, ' ')
>>> cs.tpc_vote(t1)
vote
>>> cs.tpc_finish(t1)
>>> addr, admin = start_server()
>>> client = ZEO.client(addr)
>>> db = ZODB.DB(client)
>>> conn = db.open()
>>> conn.root.x = 1
>>> t = conn.transaction_manager.get()
>>> conn._storage.tpc_begin(t)
>>> conn.commit(t)
>>> _ = client.tpc_vote(t)
Cause some breakage by messing with the clients transaction
buffer, sadly, using implementation details:
>>> tbuf = t.data(client)
>>> tbuf.client_resolved = None
tpc_finish will fail:
>>> client.tpc_finish(t) # doctest: +ELLIPSIS
Traceback (most recent call last):
...
TypeError
AttributeError: ...
>>> cs.tpc_abort(t1)
abort
>>> client.tpc_abort(t)
>>> t.abort()
>>> t2 = Transaction()
>>> cs.tpc_begin(t2)
begin ('test', '', {}, None, ' ')
>>> cs.tpc_vote(t2)
vote
But we can still load the saved data:
If client storage has an internal error after the storage finish
succeeeds, it will close the connection, which will force a
restart and reverification.
>>> conn2 = db.open()
>>> conn2.root.x
1
>>> StorageServer.should_fail = False
>>> cs._update_cache = lambda : None
>>> try: cs.tpc_finish(t2)
... except: pass
... else: print("Should have failed")
finish
connection closed
And we can save new data:
>>> cs.close()
>>> conn2.root.x += 1
>>> conn2.transaction_manager.commit()
>>> db.close()
>>> stop_server(admin)
"""
def client_has_newer_data_than_server():
......@@ -1157,11 +983,9 @@ def client_has_newer_data_than_server():
>>> wait_until('got enough errors', lambda:
... len([x for x in handler.records
... if x.filename.lower() == 'clientstorage.py' and
... x.funcName == 'verify_cache' and
... x.levelname == 'CRITICAL' and
... x.msg == 'client Client has seen '
... 'newer transactions than server!']) >= 2)
... if x.levelname == 'CRITICAL' and
... 'Client has seen newer transactions than server!' in x.msg
... ]) >= 2)
Note that the errors repeat because the client keeps on trying to connect.
......@@ -1187,7 +1011,7 @@ def history_over_zeo():
def dont_log_poskeyerrors_on_server():
"""
>>> addr, admin = start_server()
>>> addr, admin = start_server(log='server.log')
>>> cs = ClientStorage(addr)
>>> cs.load(ZODB.utils.p64(1))
Traceback (most recent call last):
......@@ -1196,7 +1020,7 @@ def dont_log_poskeyerrors_on_server():
>>> cs.close()
>>> stop_server(admin)
>>> with open('server-%s.log' % addr[1]) as f:
>>> with open('server.log') as f:
... 'POSKeyError' in f.read()
False
"""
......@@ -1245,7 +1069,7 @@ def runzeo_without_configfile():
>>> import subprocess, re
>>> print(re.sub(b'\d\d+|[:]', b'', subprocess.Popen(
... [sys.executable, 'runzeo', '-a:%s' % get_port(), '-ft', '--test'],
... [sys.executable, 'runzeo', '-a:0', '-ft', '--test'],
... stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
... ).stdout.read()).decode('ascii'))
... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
......@@ -1254,7 +1078,7 @@ def runzeo_without_configfile():
------
--T INFO ZEO.StorageServer StorageServer created RW with storages 1RWt
------
--T INFO ZEO.zrpc () listening on ...
--T INFO ZEO.asyncio... listening on ...
------
--T INFO ZEO.StorageServer closing storage '1'
testing exit immediately
......@@ -1300,6 +1124,7 @@ def convenient_to_pass_port_to_client_and_ZEO_dot_client():
>>> client.close()
"""
@forker.skip_if_testing_client_against_zeo4
def test_server_status():
"""
You can get server status using the server_status method.
......@@ -1319,12 +1144,12 @@ def test_server_status():
'start': 'Tue May 4 10:55:20 2010',
'stores': 1,
'timeout-thread-is-alive': True,
'verifying_clients': 0,
'waiting': 0}
>>> db.close()
"""
@forker.skip_if_testing_client_against_zeo4
def test_ruok():
"""
You can also get server status using the ruok protocol.
......@@ -1338,7 +1163,8 @@ def test_ruok():
>>> _ = writer.write(struct.pack(">I", 4)+b"ruok")
>>> writer.close()
>>> proto = s.recv(struct.unpack(">I", s.recv(4))[0])
>>> data = json.loads(s.recv(struct.unpack(">I", s.recv(4))[0]).decode("ascii"))
>>> data = json.loads(
... s.recv(struct.unpack(">I", s.recv(4))[0]).decode("ascii"))
>>> pprint.pprint(data['1'])
{u'aborts': 0,
u'active_txns': 0,
......@@ -1352,7 +1178,6 @@ def test_ruok():
u'start': u'Sun Jan 4 09:37:03 2015',
u'stores': 1,
u'timeout-thread-is-alive': True,
u'verifying_clients': 0,
u'waiting': 0}
>>> db.close(); s.close()
"""
......@@ -1366,12 +1191,12 @@ log entries with actual clients. It's possible, sort of, but tedious.
You can make this easier by passing a label to the ClientStorage
constructor.
>>> addr, _ = start_server()
>>> addr, _ = start_server(log='server.log')
>>> db = ZEO.DB(addr, client_label='test-label-1')
>>> db.close()
>>> @wait_until
... def check_for_test_label_1():
... with open('server-%s.log' % addr[1]) as f:
... with open('server.log') as f:
... for line in f:
... if 'test-label-1' in line:
... print(line.split()[1:4])
......@@ -1392,8 +1217,7 @@ You can specify the client label via a configuration file as well:
>>> db.close()
>>> @wait_until
... def check_for_test_label_2():
... with open('server-%s.log' % addr[1]) as f:
... for line in open('server-%s.log' % addr[1]):
... for line in open('server.log'):
... if 'test-label-2' in line:
... print(line.split()[1:4])
... return True
......@@ -1495,6 +1319,7 @@ def read(filename):
def runzeo_logrotate_on_sigusr2():
"""
>>> from ZEO.tests.forker import get_port
>>> port = get_port()
>>> with open('c', 'w') as r:
... _ = r.write('''
......@@ -1523,11 +1348,6 @@ def runzeo_logrotate_on_sigusr2():
>>> os.rename('l', 'o')
>>> os.kill(p.pid, signal.SIGUSR2)
>>> wait_until('new file', lambda : os.path.exists('l'))
XXX: if any of the previous commands failed, we'll hang here trying to
connect to ZEO, because doctest runs all the assertions...
>>> s = ClientStorage(port)
>>> s.close()
>>> wait_until('See logging', lambda : ('Log files ' in read('l')))
......@@ -1579,7 +1399,7 @@ Now we'll try to use the connection, mainly to wait for everything to
get processed. Before we fixed this by making tpc_finish a synchronous
call to the server. we'd get some sort of error here.
>>> _ = client._server.loadEx(b'\0'*8)
>>> _ = client._call('loadBefore', b'\0'*8, maxtid)
>>> c.close()
......@@ -1588,6 +1408,14 @@ call to the server. we'd get some sort of error here.
"""
def ClientDisconnected_errors_are_TransientErrors():
"""
>>> from ZEO.Exceptions import ClientDisconnected
>>> from transaction.interfaces import TransientError
>>> issubclass(ClientDisconnected, TransientError)
True
"""
if sys.platform.startswith('win'):
......@@ -1631,12 +1459,14 @@ class MultiprocessingTests(unittest.TestCase):
conn.close()
zope.testing.setupstack.tearDown(self)
@forker.skip_if_testing_client_against_zeo4
def quick_close_doesnt_kill_server():
r"""
Start a server:
>>> addr, _ = start_server()
>>> from .testssl import server_config, client_ssl
>>> addr, _ = start_server(zeo_conf=server_config)
Now connect and immediately disconnect. This caused the server to
die in the past:
......@@ -1649,65 +1479,18 @@ def quick_close_doesnt_kill_server():
... s.connect(addr)
... s.close()
>>> print("\n\nXXX WARNING: running quick_close_doesnt_kill_server with ssl as hack pending http://bugs.python.org/issue27386\n", file=sys.stderr) # Intentional long line to be annoying till this is fixed
Now we should be able to connect as normal:
>>> db = ZEO.DB(addr)
>>> db = ZEO.DB(addr, ssl=client_ssl())
>>> db.storage.is_connected()
True
>>> db.close()
"""
def sync_connect_doesnt_hang():
r"""
>>> import threading
>>> import ZEO.zrpc.client
>>> ConnectThread = ZEO.zrpc.client.ConnectThread
>>> ZEO.zrpc.client.ConnectThread = lambda *a, **kw: threading.Thread()
>>> class CM(ZEO.zrpc.client.ConnectionManager):
... sync_wait = 1
... _start_asyncore_loop = lambda self: None
>>> cm = CM(('', 0), object())
Calling connect results in an exception being raised, instead of hanging
indefinitely when the thread dies without setting up the connection.
>>> cm.connect(sync=1)
Traceback (most recent call last):
...
AssertionError
>>> cm.thread.isAlive()
False
>>> ZEO.zrpc.client.ConnectThread = ConnectThread
"""
def lp143344_extension_methods_not_lost_on_server_restart():
r"""
Make sure we don't lose exension methods on server restart.
>>> addr, adminaddr = start_server(keep=True)
>>> conn = ZEO.connection(addr)
>>> conn.root.x = 1
>>> transaction.commit()
>>> conn.db().storage.answer_to_the_ultimate_question()
42
>>> stop_server(adminaddr)
>>> wait_until('not connected',
... lambda : not conn.db().storage.is_connected())
>>> _ = start_server(addr=addr)
>>> wait_until('connected', conn.db().storage.is_connected)
>>> conn.root.x
1
>>> conn.db().storage.answer_to_the_ultimate_question()
42
>>> conn.close()
"""
def can_use_empty_string_for_local_host_on_client():
"""We should be able to spell localhost with ''.
......@@ -1724,30 +1507,23 @@ def can_use_empty_string_for_local_host_on_client():
slow_test_classes = [
BlobAdaptedFileStorageTests, BlobWritableCacheTests,
MappingStorageTests, DemoStorageTests,
FileStorageTests, FileStorageHexTests, FileStorageClientHexTests,
FileStorageTests,
FileStorageHexTests, FileStorageClientHexTests,
]
if not forker.ZEO4_SERVER:
slow_test_classes.append(FileStorageSSLTests)
quick_test_classes = [
FileStorageRecoveryTests, ConfigurationTests, HeartbeatTests,
ZRPCConnectionTests,
]
quick_test_classes = [FileStorageRecoveryTests, ZRPCConnectionTests]
class ServerManagingClientStorage(ClientStorage):
class StorageServerStubClass(ZEO.ServerStub.StorageServer):
# Wait for abort for the benefit of blob_transaction.txt
def tpc_abort(self, id):
self.rpc.call('tpc_abort', id)
def __init__(self, name, blob_dir, shared=False, extrafsoptions=''):
if shared:
server_blob_dir = blob_dir
else:
server_blob_dir = 'server-'+blob_dir
self.globs = {}
port = forker.get_port2(self)
addr, admin, pid, config = forker.start_zeo_server(
addr, stop = forker.start_zeo_server(
"""
<blobstorage>
blob-dir %s
......@@ -1757,12 +1533,8 @@ class ServerManagingClientStorage(ClientStorage):
</filestorage>
</blobstorage>
""" % (server_blob_dir, name+'.fs', extrafsoptions),
port=port,
)
os.remove(config)
zope.testing.setupstack.register(self, os.waitpid, pid, 0)
zope.testing.setupstack.register(
self, forker.shutdown_zeo_server, admin)
zope.testing.setupstack.register(self, stop)
if shared:
ClientStorage.__init__(self, addr, blob_dir=blob_dir,
shared_blob_dir=True)
......@@ -1789,8 +1561,6 @@ class ServerManagingClientStorageForIExternalGCTest(
def test_suite():
suite = unittest.TestSuite()
# Collect misc tests into their own layer to reduce size of
# unit test layer
zeo = unittest.TestSuite()
zeo.addTest(unittest.makeSuite(ZODB.tests.util.AAAA_Test_Runner_Hack))
patterns = [
......@@ -1821,14 +1591,19 @@ def test_suite():
"ClientDisconnected"),
)),
))
zeo.addTest(doctest.DocFileSuite(
'registerDB.test', globs={'print_function': print_function}))
if not forker.ZEO4_SERVER:
# ZEO 4 doesn't support client-side conflict resolution
zeo.addTest(unittest.makeSuite(ClientConflictResolutionTests, 'check'))
zeo.layer = ZODB.tests.util.MininalTestLayer('testZeo-misc')
suite.addTest(zeo)
zeo = unittest.TestSuite()
zeo.addTest(
doctest.DocFileSuite(
'zeo-fan-out.test', 'zdoptions.test',
'zdoptions.test',
'drop_cache_rather_than_verify.txt', 'client-config.test',
'protocols.test', 'zeo_blob_cache.test', 'invalidation-age.txt',
'dynamic_server_ports.test', 'new_addr.test', '../nagios.rst',
'../nagios.rst',
setUp=forker.setUp, tearDown=zope.testing.setupstack.tearDown,
checker=renormalizing.RENormalizing(patterns),
globs={'print_function': print_function},
......@@ -1841,7 +1616,21 @@ def test_suite():
))
for klass in quick_test_classes:
zeo.addTest(unittest.makeSuite(klass, "check"))
zeo.layer = ZODB.tests.util.MininalTestLayer('testZeo-misc')
zeo.layer = ZODB.tests.util.MininalTestLayer('testZeo-misc2')
suite.addTest(zeo)
# tests that often fail, maybe if they have their own layers
for name in 'zeo-fan-out.test', 'new_addr.test':
zeo = unittest.TestSuite()
zeo.addTest(
doctest.DocFileSuite(
name,
setUp=forker.setUp, tearDown=zope.testing.setupstack.tearDown,
checker=renormalizing.RENormalizing(patterns),
globs={'print_function': print_function},
),
)
zeo.layer = ZODB.tests.util.MininalTestLayer('testZeo-' + name)
suite.addTest(zeo)
suite.addTest(unittest.makeSuite(MultiprocessingTests))
......@@ -1857,6 +1646,17 @@ def test_suite():
suite.addTest(ZODB.tests.testblob.storage_reusable_suite(
'ClientStorageSharedBlobs', create_storage_shared))
if not forker.ZEO4_SERVER:
from .threaded import threaded_server_tests
dynamic_server_ports_suite = doctest.DocFileSuite(
'dynamic_server_ports.test',
setUp=forker.setUp, tearDown=zope.testing.setupstack.tearDown,
checker=renormalizing.RENormalizing(patterns),
globs={'print_function': print_function},
)
dynamic_server_ports_suite.layer = threaded_server_tests
suite.addTest(dynamic_server_ports_suite)
return suite
......
src/ZEO/tests/testZEO2.py
View file @ c499ea7d
......@@ -33,7 +33,7 @@ def proper_handling_of_blob_conflicts():
Conflict errors weren't properly handled when storing blobs, the
result being that the storage was left in a transaction.
We originally saw this when restarting a block transaction, although
We originally saw this when restarting a blob transaction, although
it doesn't really matter.
Set up the storage with some initial blob data.
......@@ -44,7 +44,7 @@ Set up the storage with some initial blob data.
>>> conn.root.b = ZODB.blob.Blob(b'x')
>>> transaction.commit()
Get the iod and first serial. We'll use the serial later to provide
Get the oid and first serial. We'll use the serial later to provide
out-of-date data.
>>> oid = conn.root.b._p_oid
......@@ -60,22 +60,15 @@ Create the server:
And an initial client.
>>> zs1 = ZEO.StorageServer.ZEOStorage(server)
>>> conn1 = ZEO.tests.servertesting.Connection(1)
>>> zs1.notifyConnected(conn1)
>>> zs1.register('1', 0)
>>> zs1 = ZEO.tests.servertesting.client(server, 1)
>>> zs1.tpc_begin('0', '', '', {})
>>> zs1.storea(ZODB.utils.p64(99), ZODB.utils.z64, b'x', '0')
>>> _ = zs1.vote('0') # doctest: +ELLIPSIS
1 callAsync serialnos ...
In a second client, we'll try to commit using the old serial. This
will conflict. It will be blocked at the vote call.
>>> zs2 = ZEO.StorageServer.ZEOStorage(server)
>>> conn2 = ZEO.tests.servertesting.Connection(2)
>>> zs2.notifyConnected(conn2)
>>> zs2.register('1', 0)
>>> zs2 = ZEO.tests.servertesting.client(server, 2)
>>> zs2.tpc_begin('1', '', '', {})
>>> zs2.storeBlobStart()
>>> zs2.storeBlobChunk(b'z')
......@@ -85,6 +78,8 @@ will conflict. It will be blocked at the vote call.
>>> class Sender:
... def send_reply(self, id, reply):
... print('reply', id, reply)
... def send_error(self, id, err):
... print('error', id, err)
>>> delay.set_sender(1, Sender())
>>> logger = logging.getLogger('ZEO')
......@@ -94,15 +89,20 @@ will conflict. It will be blocked at the vote call.
Now, when we abort the transaction for the first client. The second
client will be restarted. It will get a conflict error, that is
handled correctly:
raised to the client:
>>> zs1.tpc_abort('0') # doctest: +ELLIPSIS
2 callAsync serialnos ...
reply 1 None
Error raised in delayed method
Traceback (most recent call last):
...ConflictError: ...
error 1 database conflict error ...
The transaction is aborted by the server:
>>> fs.tpc_transaction() is not None
>>> fs.tpc_transaction() is None
True
>>> conn2.connected
>>> zs2.connected
True
>>> logger.setLevel(logging.NOTSET)
......@@ -122,12 +122,9 @@ storage isn't left in tpc.
And an initial client.
>>> zs1 = ZEO.StorageServer.ZEOStorage(server)
>>> conn1 = ZEO.tests.servertesting.Connection(1)
>>> zs1.notifyConnected(conn1)
>>> zs1.register('1', 0)
>>> zs1 = ZEO.tests.servertesting.client(server, 1)
>>> zs1.tpc_begin('0', '', '', {})
>>> zs1.storea(ZODB.utils.p64(99), ZODB.utils.z64, 'x', '0')
>>> zs1.storea(ZODB.utils.p64(99), ZODB.utils.z64, b'x', '0')
Intentionally break zs1:
......@@ -144,16 +141,12 @@ We're not in a transaction:
We can start another client and get the storage lock.
>>> zs1 = ZEO.StorageServer.ZEOStorage(server)
>>> conn1 = ZEO.tests.servertesting.Connection(1)
>>> zs1.notifyConnected(conn1)
>>> zs1.register('1', 0)
>>> zs1 = ZEO.tests.servertesting.client(server, 1)
>>> zs1.tpc_begin('1', '', '', {})
>>> zs1.storea(ZODB.utils.p64(99), ZODB.utils.z64, 'x', '1')
>>> zs1.storea(ZODB.utils.p64(99), ZODB.utils.z64, b'x', '1')
>>> _ = zs1.vote('1') # doctest: +ELLIPSIS
1 callAsync serialnos ...
>>> zs1.tpc_finish('1').set_sender(0, conn1)
>>> zs1.tpc_finish('1').set_sender(0, zs1.connection)
>>> fs.close()
"""
......@@ -173,12 +166,10 @@ So, we arrange to get an error in vote:
>>> server = ZEO.tests.servertesting.StorageServer(
... 'x', {'1': MappingStorage()})
>>> zs = ZEO.StorageServer.ZEOStorage(server)
>>> conn = ZEO.tests.servertesting.Connection(1)
>>> zs.notifyConnected(conn)
>>> zs.register('1', 0)
>>> zs = ZEO.tests.servertesting.client(server, 1)
>>> zs.tpc_begin('0', '', '', {})
>>> zs.storea(ZODB.utils.p64(99), ZODB.utils.z64, 'x', '0')
>>> zs.vote('0')
Traceback (most recent call last):
...
......@@ -186,7 +177,7 @@ So, we arrange to get an error in vote:
When we do, the storage server's transaction lock shouldn't be held:
>>> '1' in server._commit_locks
>>> zs.lock_manager.locked is not None
False
Of course, if vote suceeds, the lock will be held:
......@@ -195,9 +186,8 @@ Of course, if vote suceeds, the lock will be held:
>>> zs.tpc_begin('1', '', '', {})
>>> zs.storea(ZODB.utils.p64(99), ZODB.utils.z64, 'x', '1')
>>> _ = zs.vote('1') # doctest: +ELLIPSIS
1 callAsync serialnos ...
>>> '1' in server._commit_locks
>>> zs.lock_manager.locked is not None
True
>>> zs.tpc_abort('1')
......@@ -234,18 +224,25 @@ quit working in Python 3.4:
We start a transaction and vote, this leads to getting the lock.
>>> zs1 = ZEO.tests.servertesting.client(server, '1')
ZEO.asyncio.base INFO
Connected server protocol
ZEO.asyncio.server INFO
received handshake 'Z5'
>>> tid1 = start_trans(zs1)
>>> zs1.vote(tid1) # doctest: +ELLIPSIS
>>> resolved1 = zs1.vote(tid1) # doctest: +ELLIPSIS
ZEO.StorageServer DEBUG
(test-addr-1) ('1') lock: transactions waiting: 0
ZEO.StorageServer BLATHER
(test-addr-1) Preparing to commit transaction: 1 objects, ... bytes
1 callAsync serialnos ...
If another client tried to vote, it's lock request will be queued and
a delay will be returned:
>>> zs2 = ZEO.tests.servertesting.client(server, '2')
ZEO.asyncio.base INFO
Connected server protocol
ZEO.asyncio.server INFO
received handshake 'Z5'
>>> tid2 = start_trans(zs2)
>>> delay = zs2.vote(tid2)
ZEO.StorageServer DEBUG
......@@ -262,7 +259,6 @@ When we end the first transaction, the queued vote gets the lock.
(test-addr-2) ('1') lock: transactions waiting: 0
ZEO.StorageServer BLATHER
(test-addr-2) Preparing to commit transaction: 1 objects, ... bytes
2 callAsync serialnos ...
Let's try again with the first client. The vote will be queued:
......@@ -306,38 +302,73 @@ increased, so does the logging level:
... tid = start_trans(client)
... delay = client.vote(tid)
... clients.append(client)
ZEO.asyncio.base INFO
Connected server protocol
ZEO.asyncio.server INFO
received handshake 'Z5'
ZEO.StorageServer DEBUG
(test-addr-10) ('1') queue lock: transactions waiting: 2
ZEO.asyncio.base INFO
Connected server protocol
ZEO.asyncio.server INFO
received handshake 'Z5'
ZEO.StorageServer DEBUG
(test-addr-11) ('1') queue lock: transactions waiting: 3
ZEO.asyncio.base INFO
Connected server protocol
ZEO.asyncio.server INFO
received handshake 'Z5'
ZEO.StorageServer WARNING
(test-addr-12) ('1') queue lock: transactions waiting: 4
ZEO.asyncio.base INFO
Connected server protocol
ZEO.asyncio.server INFO
received handshake 'Z5'
ZEO.StorageServer WARNING
(test-addr-13) ('1') queue lock: transactions waiting: 5
ZEO.asyncio.base INFO
Connected server protocol
ZEO.asyncio.server INFO
received handshake 'Z5'
ZEO.StorageServer WARNING
(test-addr-14) ('1') queue lock: transactions waiting: 6
ZEO.asyncio.base INFO
Connected server protocol
ZEO.asyncio.server INFO
received handshake 'Z5'
ZEO.StorageServer WARNING
(test-addr-15) ('1') queue lock: transactions waiting: 7
ZEO.asyncio.base INFO
Connected server protocol
ZEO.asyncio.server INFO
received handshake 'Z5'
ZEO.StorageServer WARNING
(test-addr-16) ('1') queue lock: transactions waiting: 8
ZEO.asyncio.base INFO
Connected server protocol
ZEO.asyncio.server INFO
received handshake 'Z5'
ZEO.StorageServer WARNING
(test-addr-17) ('1') queue lock: transactions waiting: 9
ZEO.asyncio.base INFO
Connected server protocol
ZEO.asyncio.server INFO
received handshake 'Z5'
ZEO.StorageServer CRITICAL
(test-addr-18) ('1') queue lock: transactions waiting: 10
If a client with the transaction lock disconnects, it will abort and
release the lock and one of the waiting clients will get the lock.
>>> zs2.notifyDisconnected() # doctest: +ELLIPSIS
>>> zs2.notify_disconnected() # doctest: +ELLIPSIS
ZEO.StorageServer INFO
(test-addr-2) disconnected during locked transaction
(test-addr-...) disconnected during locked transaction
ZEO.StorageServer CRITICAL
(test-addr-2) ('1') unlock: transactions waiting: 10
(test-addr-...) ('1') unlock: transactions waiting: 10
ZEO.StorageServer WARNING
(test-addr-1) ('1') lock: transactions waiting: 9
(test-addr-...) ('1') lock: transactions waiting: 9
ZEO.StorageServer BLATHER
(test-addr-1) Preparing to commit transaction: 1 objects, ... bytes
1 callAsync serialnos ...
(test-addr-...) Preparing to commit transaction: 1 objects, ... bytes
(In practice, waiting clients won't necessarily get the lock in order.)
......@@ -350,61 +381,30 @@ statistics using the server_status method:
'commits': 0,
'conflicts': 0,
'conflicts_resolved': 0,
'connections': 11,
'connections': 10,
'last-transaction': '0000000000000000',
'loads': 0,
'lock_time': 1272653598.693882,
'start': 'Fri Apr 30 14:53:18 2010',
'stores': 13,
'timeout-thread-is-alive': 'stub',
'verifying_clients': 0,
'waiting': 9}
(Note that the connections count above is off by 1 due to the way the
test infrastructure works.)
If clients disconnect while waiting, they will be dequeued:
>>> for client in clients:
... client.notifyDisconnected()
ZEO.StorageServer INFO
(test-addr-10) disconnected during unlocked transaction
ZEO.StorageServer WARNING
(test-addr-10) ('1') dequeue lock: transactions waiting: 8
... client.notify_disconnected() # doctest: +ELLIPSIS
ZEO.StorageServer INFO
(test-addr-11) disconnected during unlocked transaction
(test-addr-10) disconnected during...locked transaction
ZEO.StorageServer WARNING
(test-addr-11) ('1') dequeue lock: transactions waiting: 7
ZEO.StorageServer INFO
(test-addr-12) disconnected during unlocked transaction
ZEO.StorageServer WARNING
(test-addr-12) ('1') dequeue lock: transactions waiting: 6
ZEO.StorageServer INFO
(test-addr-13) disconnected during unlocked transaction
ZEO.StorageServer WARNING
(test-addr-13) ('1') dequeue lock: transactions waiting: 5
ZEO.StorageServer INFO
(test-addr-14) disconnected during unlocked transaction
ZEO.StorageServer WARNING
(test-addr-14) ('1') dequeue lock: transactions waiting: 4
ZEO.StorageServer INFO
(test-addr-15) disconnected during unlocked transaction
ZEO.StorageServer DEBUG
(test-addr-15) ('1') dequeue lock: transactions waiting: 3
ZEO.StorageServer INFO
(test-addr-16) disconnected during unlocked transaction
ZEO.StorageServer DEBUG
(test-addr-16) ('1') dequeue lock: transactions waiting: 2
ZEO.StorageServer INFO
(test-addr-17) disconnected during unlocked transaction
ZEO.StorageServer DEBUG
(test-addr-17) ('1') dequeue lock: transactions waiting: 1
ZEO.StorageServer INFO
(test-addr-18) disconnected during unlocked transaction
ZEO.StorageServer DEBUG
(test-addr-18) ('1') dequeue lock: transactions waiting: 0
(test-addr-10) ('1') ... lock: transactions waiting: ...
>>> zs1.server_status()['waiting']
0
>>> zs1.tpc_abort(tid1)
ZEO.StorageServer DEBUG
(test-addr-1) ('1') unlock: transactions waiting: 0
>>> logging.getLogger('ZEO').setLevel(logging.NOTSET)
>>> logging.getLogger('ZEO').removeHandler(handler)
......@@ -454,34 +454,87 @@ Now, we'll start a transaction, get the lock and then mark the
ZEOStorage as closed and see if trying to get a lock cleans it up:
>>> zs1 = ZEO.tests.servertesting.client(server, '1')
ZEO.asyncio.base INFO
Connected server protocol
ZEO.asyncio.server INFO
received handshake 'Z5'
>>> tid1 = start_trans(zs1)
>>> zs1.vote(tid1) # doctest: +ELLIPSIS
>>> resolved1 = zs1.vote(tid1) # doctest: +ELLIPSIS
ZEO.StorageServer DEBUG
(test-addr-1) ('1') lock: transactions waiting: 0
ZEO.StorageServer BLATHER
(test-addr-1) Preparing to commit transaction: 1 objects, ... bytes
1 callAsync serialnos ...
>>> zs1.connection = None
>>> zs1.connection.connection_lost(None)
ZEO.StorageServer INFO
(test-addr-1) disconnected during locked transaction
ZEO.StorageServer DEBUG
(test-addr-1) ('1') unlock: transactions waiting: 0
>>> zs2 = ZEO.tests.servertesting.client(server, '2')
ZEO.asyncio.base INFO
Connected server protocol
ZEO.asyncio.server INFO
received handshake 'Z5'
>>> tid2 = start_trans(zs2)
>>> zs2.vote(tid2) # doctest: +ELLIPSIS
ZEO.StorageServer CRITICAL
(test-addr-1) Still locked after disconnected. Unlocking.
>>> resolved2 = zs2.vote(tid2) # doctest: +ELLIPSIS
ZEO.StorageServer DEBUG
(test-addr-2) ('1') lock: transactions waiting: 0
ZEO.StorageServer BLATHER
(test-addr-2) Preparing to commit transaction: 1 objects, ... bytes
2 callAsync serialnos ...
>>> zs1.txnlog.close()
>>> zs2.tpc_abort(tid2)
ZEO.StorageServer DEBUG
(test-addr-2) ('1') unlock: transactions waiting: 0
>>> logging.getLogger('ZEO').setLevel(logging.NOTSET)
>>> logging.getLogger('ZEO').removeHandler(handler)
"""
def test_prefetch(self):
"""The client storage prefetch method pre-fetches from the server
>>> count = 999
>>> import ZEO
>>> addr, stop = ZEO.server(threaded=False)
>>> conn = ZEO.connection(addr)
>>> root = conn.root()
>>> cls = root.__class__
>>> for i in range(count):
... root[i] = cls()
>>> conn.transaction_manager.commit()
>>> oids = [root[i]._p_oid for i in range(count)]
>>> conn.close()
>>> conn = ZEO.connection(addr)
>>> storage = conn.db().storage
>>> len(storage._cache)
1
>>> storage.prefetch(oids, conn._storage._start)
The prefetch returns before the cache is filled:
>>> len(storage._cache) < count
True
But it is filled eventually:
>>> from zope.testing.wait import wait
>>> wait(lambda : len(storage._cache) > count)
>>> loads = storage.server_status()['loads']
Now if we reload the data, it will be satisfied from the cache:
>>> for oid in oids:
... _ = conn._storage.load(oid)
>>> storage.server_status()['loads'] == loads
True
>>> conn.close()
>>> stop()
"""
def test_suite():
return unittest.TestSuite((
......
src/ZEO/tests/test_cache.py
View file @ c499ea7d
......@@ -429,58 +429,6 @@ LockError: Couldn't lock 'cache.lock'
>>> cache.close()
"""
def thread_safe():
r"""
>>> import ZEO.cache, ZODB.utils
>>> cache = ZEO.cache.ClientCache('cache', 1000000)
>>> for i in range(100):
... cache.store(ZODB.utils.p64(i), ZODB.utils.p64(1), None, b'0')
>>> import random2 as random, sys, threading
>>> random = random.Random(0)
>>> stop = False
>>> read_failure = None
>>> def read_thread():
... def pick_oid():
... return ZODB.utils.p64(random.randint(0,99))
...
... try:
... while not stop:
... cache.load(pick_oid())
... cache.loadBefore(pick_oid(), ZODB.utils.p64(2))
... except:
... global read_failure
... read_failure = sys.exc_info()
>>> thread = threading.Thread(target=read_thread)
>>> thread.start()
>>> for tid in range(2,10):
... for oid in range(100):
... oid = ZODB.utils.p64(oid)
... cache.invalidate(oid, ZODB.utils.p64(tid))
... cache.store(oid, ZODB.utils.p64(tid), None, str(tid).encode())
>>> stop = True
>>> thread.join()
>>> if read_failure:
... print('Read failure:')
... import traceback
... traceback.print_exception(*read_failure)
>>> expected = b'9', ZODB.utils.p64(9)
>>> for oid in range(100):
... loaded = cache.load(ZODB.utils.p64(oid))
... if loaded != expected:
... print(oid, loaded)
>>> cache.close()
"""
def broken_non_current():
r"""
......
src/ZEO/tests/test_client_credentials.py 0 → 100644
View file @ c499ea7d
"""Clients can pass credentials to a server.
This is an experimental feature to enable server authentication and
authorization.
"""
from zope.testing import setupstack
import unittest
import ZEO.StorageServer
from . import forker
from .threaded import threaded_server_tests
@unittest.skipIf(forker.ZEO4_SERVER, "ZEO4 servers don't support SSL")
class ClientAuthTests(setupstack.TestCase):
def setUp(self):
self.setUpDirectory()
self.__register = ZEO.StorageServer.ZEOStorage.register
def tearDown(self):
ZEO.StorageServer.ZEOStorage.register = self.__register
def test_passing_credentials(self):
# First, we'll temporarily swap the storage server register
# method with one that let's is see credentials that were passed:
creds_log = []
def register(zs, storage_id, read_only, credentials=self):
creds_log.append(credentials)
return self.__register(zs, storage_id, read_only)
ZEO.StorageServer.ZEOStorage.register = register
# Now start an in process server
addr, stop = ZEO.server()
# If we connect, without providing credentials, then no
# credentials will be passed to register:
client = ZEO.client(addr)
self.assertEqual(creds_log, [self])
client.close()
creds_log.pop()
# But if we pass credentials, they'll be passed to register:
creds = dict(user='me', password='123')
client = ZEO.client(addr, credentials=creds)
self.assertEqual(creds_log, [creds])
client.close()
stop()
def test_suite():
suite = unittest.makeSuite(ClientAuthTests)
suite.layer = threaded_server_tests
return suite
src/ZEO/tests/test_client_side_conflict_resolution.py 0 → 100644
View file @ c499ea7d
import unittest
import zope.testing.setupstack
from BTrees.Length import Length
from ZODB import serialize
from ZODB.DemoStorage import DemoStorage
from ZODB.utils import p64, z64, maxtid
from ZODB.broken import find_global
import ZEO
from . import forker
from .utils import StorageServer
class Var(object):
def __eq__(self, other):
self.value = other
return True
@unittest.skipIf(forker.ZEO4_SERVER, "ZEO4 servers don't support SSL")
class ClientSideConflictResolutionTests(zope.testing.setupstack.TestCase):
def test_server_side(self):
# First, verify default conflict resolution.
server = StorageServer(self, DemoStorage())
zs = server.zs
reader = serialize.ObjectReader(
factory=lambda conn, *args: find_global(*args))
writer = serialize.ObjectWriter()
ob = Length(0)
ob._p_oid = z64
# 2 non-conflicting transactions:
zs.tpc_begin(1, '', '', {})
zs.storea(ob._p_oid, z64, writer.serialize(ob), 1)
self.assertEqual(zs.vote(1), [])
tid1 = server.unpack_result(zs.tpc_finish(1))
server.assert_calls(self, ('info', {'length': 1, 'size': Var()}))
ob.change(1)
zs.tpc_begin(2, '', '', {})
zs.storea(ob._p_oid, tid1, writer.serialize(ob), 2)
self.assertEqual(zs.vote(2), [])
tid2 = server.unpack_result(zs.tpc_finish(2))
server.assert_calls(self, ('info', {'size': Var(), 'length': 1}))
# Now, a cnflicting one:
zs.tpc_begin(3, '', '', {})
zs.storea(ob._p_oid, tid1, writer.serialize(ob), 3)
# Vote returns the object id, indicating that a conflict was resolved.
self.assertEqual(zs.vote(3), [ob._p_oid])
tid3 = server.unpack_result(zs.tpc_finish(3))
p, serial, next_serial = zs.loadBefore(ob._p_oid, maxtid)
self.assertEqual((serial, next_serial), (tid3, None))
self.assertEqual(reader.getClassName(p), 'BTrees.Length.Length')
self.assertEqual(reader.getState(p), 2)
# Now, we'll create a server that expects the client to
# resolve conflicts:
server = StorageServer(
self, DemoStorage(), client_conflict_resolution=True)
zs = server.zs
# 2 non-conflicting transactions:
zs.tpc_begin(1, '', '', {})
zs.storea(ob._p_oid, z64, writer.serialize(ob), 1)
self.assertEqual(zs.vote(1), [])
tid1 = server.unpack_result(zs.tpc_finish(1))
server.assert_calls(self, ('info', {'size': Var(), 'length': 1}))
ob.change(1)
zs.tpc_begin(2, '', '', {})
zs.storea(ob._p_oid, tid1, writer.serialize(ob), 2)
self.assertEqual(zs.vote(2), [])
tid2 = server.unpack_result(zs.tpc_finish(2))
server.assert_calls(self, ('info', {'length': 1, 'size': Var()}))
# Now, a conflicting one:
zs.tpc_begin(3, '', '', {})
zs.storea(ob._p_oid, tid1, writer.serialize(ob), 3)
# Vote returns an object, indicating that a conflict was not resolved.
self.assertEqual(
zs.vote(3),
[dict(oid=ob._p_oid,
serials=(tid2, tid1),
data=writer.serialize(ob),
)],
)
# Now, it's up to the client to resolve the conflict. It can
# do this by making another store call. In this call, we use
# tid2 as the starting tid:
ob.change(1)
zs.storea(ob._p_oid, tid2, writer.serialize(ob), 3)
self.assertEqual(zs.vote(3), [])
tid3 = server.unpack_result(zs.tpc_finish(3))
server.assert_calls(self, ('info', {'size': Var(), 'length': 1}))
p, serial, next_serial = zs.loadBefore(ob._p_oid, maxtid)
self.assertEqual((serial, next_serial), (tid3, None))
self.assertEqual(reader.getClassName(p), 'BTrees.Length.Length')
self.assertEqual(reader.getState(p), 3)
def test_client_side(self):
# First, traditional:
addr, stop = ZEO.server('data.fs', threaded=False)
db = ZEO.DB(addr)
with db.transaction() as conn:
conn.root.l = Length(0)
conn2 = db.open()
conn2.root.l.change(1)
with db.transaction() as conn:
conn.root.l.change(1)
conn2.transaction_manager.commit()
self.assertEqual(conn2.root.l.value, 2)
db.close(); stop()
# Now, do conflict resolution on the client.
addr2, stop = ZEO.server(
storage_conf='<mappingstorage>\n</mappingstorage>\n',
zeo_conf=dict(client_conflict_resolution=True),
threaded=False,
)
db = ZEO.DB(addr2)
with db.transaction() as conn:
conn.root.l = Length(0)
conn2 = db.open()
conn2.root.l.change(1)
with db.transaction() as conn:
conn.root.l.change(1)
self.assertEqual(conn2.root.l.value, 1)
conn2.transaction_manager.commit()
self.assertEqual(conn2.root.l.value, 2)
db.close(); stop()
def test_suite():
return unittest.makeSuite(ClientSideConflictResolutionTests)
src/ZEO/tests/test_sync.py 0 → 100644
View file @ c499ea7d
import unittest
from zope.testing import setupstack
from .. import server, client
from . import forker
if forker.ZEO4_SERVER:
server_ping_method = 'lastTransaction'
server_zss = 'connections'
else:
server_ping_method = 'ping'
server_zss = 'zeo_storages_by_storage_id'
class SyncTests(setupstack.TestCase):
def instrument(self):
self.__ping_calls = 0
server = getattr(forker, self.__name + '_server')
[zs] = getattr(server.server, server_zss)['1']
orig_ping = getattr(zs, server_ping_method)
def ping():
self.__ping_calls += 1
return orig_ping()
setattr(zs, server_ping_method, ping)
def test_server_sync(self):
self.__name = 's%s' % id(self)
addr, stop = server(name=self.__name)
# By default the client sync method is a noop:
c = client(addr)
self.instrument()
c.sync()
self.assertEqual(self.__ping_calls, 0)
c.close()
# But if we pass server_sync:
c = client(addr, server_sync=True)
self.instrument()
c.sync()
self.assertEqual(self.__ping_calls, 1)
c.close()
stop()
src/ZEO/tests/testssl.py 0 → 100644
View file @ c499ea7d
from .._compat import PY3
import mock
import os
import ssl
import unittest
from ZODB.config import storageFromString
from ..Exceptions import ClientDisconnected
from .. import runzeo
from .testConfig import ZEOConfigTestBase
from . import forker
from .threaded import threaded_server_tests
here = os.path.dirname(__file__)
server_cert = os.path.join(here, 'server.pem')
server_key = os.path.join(here, 'server_key.pem')
serverpw_cert = os.path.join(here, 'serverpw.pem')
serverpw_key = os.path.join(here, 'serverpw_key.pem')
client_cert = os.path.join(here, 'client.pem')
client_key = os.path.join(here, 'client_key.pem')
@unittest.skipIf(forker.ZEO4_SERVER, "ZEO4 servers don't support SSL")
class SSLConfigTest(ZEOConfigTestBase):
def test_ssl_basic(self):
# This shows that configuring ssl has an actual effect on connections.
# Other SSL configuration tests will be Mockiavellian.
# Also test that an SSL connection mismatch doesn't kill
# the server loop.
# An SSL client can't talk to a non-SSL server:
addr, stop = self.start_server()
with self.assertRaises(ClientDisconnected):
self.start_client(
addr,
"""<ssl>
certificate {}
key {}
</ssl>""".format(client_cert, client_key), wait_timeout=1)
# But a non-ssl one can:
client = self.start_client(addr)
self._client_assertions(client, addr)
client.close()
stop()
# A non-SSL client can't talk to an SSL server:
addr, stop = self.start_server(
"""<ssl>
certificate {}
key {}
authenticate {}
</ssl>""".format(server_cert, server_key, client_cert)
)
with self.assertRaises(ClientDisconnected):
self.start_client(addr, wait_timeout=1)
# But an SSL one can:
client = self.start_client(
addr,
"""<ssl>
certificate {}
key {}
authenticate {}
server-hostname zodb.org
</ssl>""".format(client_cert, client_key, server_cert))
self._client_assertions(client, addr)
client.close()
stop()
def test_ssl_hostname_check(self):
addr, stop = self.start_server(
"""<ssl>
certificate {}
key {}
authenticate {}
</ssl>""".format(server_cert, server_key, client_cert)
)
# Connext with bad hostname fails:
with self.assertRaises(ClientDisconnected):
client = self.start_client(
addr,
"""<ssl>
certificate {}
key {}
authenticate {}
server-hostname example.org
</ssl>""".format(client_cert, client_key, server_cert),
wait_timeout=1)
# Connext with good hostname succeeds:
client = self.start_client(
addr,
"""<ssl>
certificate {}
key {}
authenticate {}
server-hostname zodb.org
</ssl>""".format(client_cert, client_key, server_cert))
self._client_assertions(client, addr)
client.close()
stop()
def test_ssl_pw(self):
addr, stop = self.start_server(
"""<ssl>
certificate {}
key {}
authenticate {}
password-function ZEO.tests.testssl.pwfunc
</ssl>""".format(serverpw_cert, serverpw_key, client_cert)
)
stop()
@unittest.skipIf(forker.ZEO4_SERVER, "ZEO4 servers don't support SSL")
@mock.patch(('asyncio' if PY3 else 'trollius') + '.async')
@mock.patch(('asyncio' if PY3 else 'trollius') + '.set_event_loop')
@mock.patch(('asyncio' if PY3 else 'trollius') + '.new_event_loop')
@mock.patch('ZEO.asyncio.client.new_event_loop')
@mock.patch('ZEO.asyncio.server.new_event_loop')
class SSLConfigTestMockiavellian(ZEOConfigTestBase):
@mock.patch('ssl.create_default_context')
def test_ssl_mockiavellian_server_no_ssl(self, factory, *_):
server = create_server()
self.assertFalse(factory.called)
self.assertEqual(server.acceptor.ssl_context, None)
server.close()
def assert_context(
self, factory, context,
cert=(server_cert, server_key, None),
verify_mode=ssl.CERT_REQUIRED,
check_hostname=False,
cafile=None, capath=None,
):
factory.assert_called_with(
ssl.Purpose.CLIENT_AUTH, cafile=cafile, capath=capath)
context.load_cert_chain.assert_called_with(*cert)
self.assertEqual(context, factory.return_value)
self.assertEqual(context.verify_mode, verify_mode)
self.assertEqual(context.check_hostname, check_hostname)
@mock.patch('ssl.create_default_context')
def test_ssl_mockiavellian_server_ssl_no_auth(self, factory, *_):
with self.assertRaises(SystemExit):
# auth is required
create_server(certificate=server_cert, key=server_key)
@mock.patch('ssl.create_default_context')
def test_ssl_mockiavellian_server_ssl_auth_file(self, factory, *_):
server = create_server(
certificate=server_cert, key=server_key, authenticate=__file__)
context = server.acceptor.ssl_context
self.assert_context(factory, context, cafile=__file__)
server.close()
@mock.patch('ssl.create_default_context')
def test_ssl_mockiavellian_server_ssl_auth_dir(self, factory, *_):
server = create_server(
certificate=server_cert, key=server_key, authenticate=here)
context = server.acceptor.ssl_context
self.assert_context(factory, context, capath=here)
server.close()
@mock.patch('ssl.create_default_context')
def test_ssl_mockiavellian_server_ssl_pw(self, factory, *_):
server = create_server(
certificate=server_cert,
key=server_key,
password_function='ZEO.tests.testssl.pwfunc',
authenticate=here,
)
context = server.acceptor.ssl_context
self.assert_context(
factory, context, (server_cert, server_key, pwfunc), capath=here)
server.close()
@mock.patch('ssl.create_default_context')
@mock.patch('ZEO.ClientStorage.ClientStorage')
def test_ssl_mockiavellian_client_no_ssl(self, ClientStorage, factory, *_):
client = ssl_client()
self.assertFalse('ssl' in ClientStorage.call_args[1])
self.assertFalse('ssl_server_hostname' in ClientStorage.call_args[1])
@mock.patch('ssl.create_default_context')
@mock.patch('ZEO.ClientStorage.ClientStorage')
def test_ssl_mockiavellian_client_server_signed(
self, ClientStorage, factory, *_
):
client = ssl_client(certificate=client_cert, key=client_key)
context = ClientStorage.call_args[1]['ssl']
self.assertEqual(ClientStorage.call_args[1]['ssl_server_hostname'],
None)
self.assert_context(
factory, context, (client_cert, client_key, None),
check_hostname=True)
context.load_default_certs.assert_called_with()
@mock.patch('ssl.create_default_context')
@mock.patch('ZEO.ClientStorage.ClientStorage')
def test_ssl_mockiavellian_client_auth_dir(
self, ClientStorage, factory, *_
):
client = ssl_client(
certificate=client_cert, key=client_key, authenticate=here)
context = ClientStorage.call_args[1]['ssl']
self.assertEqual(ClientStorage.call_args[1]['ssl_server_hostname'],
None)
self.assert_context(
factory, context, (client_cert, client_key, None),
capath=here,
check_hostname=True,
)
context.load_default_certs.assert_not_called()
@mock.patch('ssl.create_default_context')
@mock.patch('ZEO.ClientStorage.ClientStorage')
def test_ssl_mockiavellian_client_auth_file(
self, ClientStorage, factory, *_
):
client = ssl_client(
certificate=client_cert, key=client_key, authenticate=server_cert)
context = ClientStorage.call_args[1]['ssl']
self.assertEqual(ClientStorage.call_args[1]['ssl_server_hostname'],
None)
self.assert_context(
factory, context, (client_cert, client_key, None),
cafile=server_cert,
check_hostname=True,
)
context.load_default_certs.assert_not_called()
@mock.patch('ssl.create_default_context')
@mock.patch('ZEO.ClientStorage.ClientStorage')
def test_ssl_mockiavellian_client_pw(
self, ClientStorage, factory, *_
):
client = ssl_client(
certificate=client_cert, key=client_key,
password_function='ZEO.tests.testssl.pwfunc',
authenticate=server_cert)
context = ClientStorage.call_args[1]['ssl']
self.assertEqual(ClientStorage.call_args[1]['ssl_server_hostname'],
None)
self.assert_context(
factory, context, (client_cert, client_key, pwfunc),
cafile=server_cert,
check_hostname=True,
)
@mock.patch('ssl.create_default_context')
@mock.patch('ZEO.ClientStorage.ClientStorage')
def test_ssl_mockiavellian_client_server_hostname(
self, ClientStorage, factory, *_
):
client = ssl_client(
certificate=client_cert, key=client_key, authenticate=server_cert,
server_hostname='example.com')
context = ClientStorage.call_args[1]['ssl']
self.assertEqual(ClientStorage.call_args[1]['ssl_server_hostname'],
'example.com')
self.assert_context(
factory, context, (client_cert, client_key, None),
cafile=server_cert,
check_hostname=True,
)
@mock.patch('ssl.create_default_context')
@mock.patch('ZEO.ClientStorage.ClientStorage')
def test_ssl_mockiavellian_client_check_hostname(
self, ClientStorage, factory, *_
):
client = ssl_client(
certificate=client_cert, key=client_key, authenticate=server_cert,
check_hostname=False)
context = ClientStorage.call_args[1]['ssl']
self.assertEqual(ClientStorage.call_args[1]['ssl_server_hostname'],
None)
self.assert_context(
factory, context, (client_cert, client_key, None),
cafile=server_cert,
check_hostname=False,
)
def args(*a, **kw):
return a, kw
def ssl_conf(**ssl_settings):
if ssl_settings:
ssl_conf = '<ssl>\n' + '\n'.join(
'{} {}'.format(name.replace('_', '-'), value)
for name, value in ssl_settings.items()
) + '\n</ssl>\n'
else:
ssl_conf = ''
return ssl_conf
def ssl_client(**ssl_settings):
return storageFromString(
"""%import ZEO
<clientstorage>
server 127.0.0.1:0
{}
</clientstorage>
""".format(ssl_conf(**ssl_settings))
)
def create_server(**ssl_settings):
with open('conf', 'w') as f:
f.write(
"""
<zeo>
address 127.0.0.1:0
{}
</zeo>
<mappingstorage>
</mappingstorage>
""".format(ssl_conf(**ssl_settings)))
options = runzeo.ZEOOptions()
options.realize(['-C', 'conf'])
s = runzeo.ZEOServer(options)
s.open_storages()
s.create_server()
return s.server
pwfunc = lambda : '1234'
def test_suite():
suite = unittest.TestSuite((
unittest.makeSuite(SSLConfigTest),
unittest.makeSuite(SSLConfigTestMockiavellian),
))
suite.layer = threaded_server_tests
return suite
# Helpers for other tests:
server_config = """
<zeo>
address 127.0.0.1:0
<ssl>
certificate {}
key {}
authenticate {}
</ssl>
</zeo>
""".format(server_cert, server_key, client_cert)
def client_ssl(cafile=server_key,
client_cert=client_cert,
client_key=client_key,
):
context = ssl.create_default_context(
ssl.Purpose.CLIENT_AUTH, cafile=server_cert)
context.load_cert_chain(client_cert, client_key)
context.verify_mode = ssl.CERT_REQUIRED
context.check_hostname = False
return context
# Here's a command to create a cert/key pair:
# openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem \
# -days 999999 -nodes -batch
src/ZEO/tests/threaded.py 0 → 100644
View file @ c499ea7d
"""Test layer for threaded-server tests
uvloop currently has a bug,
https://github.com/MagicStack/uvloop/issues/39, that causes failure if
multiprocessing and threaded servers are mixed in the same
application, so we isolate the few threaded tests in their own layer.
"""
import ZODB.tests.util
threaded_server_tests = ZODB.tests.util.MininalTestLayer(
'threaded_server_tests')
src/ZEO/tests/utils.py 0 → 100644
View file @ c499ea7d
"""Testing helpers
"""
import ZEO.StorageServer
from ..asyncio.server import best_protocol_version
class ServerProtocol:
method = ('register', )
def __init__(self, zs,
protocol_version=best_protocol_version,
addr='test-address'):
self.calls = []
self.addr = addr
self.zs = zs
self.protocol_version = protocol_version
zs.notify_connected(self)
closed = False
def close(self):
if not self.closed:
self.closed = True
self.zs.notify_disconnected()
def call_soon_threadsafe(self, func, *args):
func(*args)
def async(self, *args):
self.calls.append(args)
async_threadsafe = async
class StorageServer:
"""Create a client interface to a StorageServer.
This is for testing StorageServer. It interacts with the storgr
server through its network interface, but without creating a
network connection.
"""
def __init__(self, test, storage,
protocol_version=best_protocol_version,
**kw):
self.test = test
self.storage_server = ZEO.StorageServer.StorageServer(
None, {'1': storage}, **kw)
self.zs = self.storage_server.create_client_handler()
self.protocol = ServerProtocol(self.zs,
protocol_version=protocol_version)
self.zs.register('1', kw.get('read_only', False))
def assert_calls(self, test, *argss):
if argss:
for args in argss:
test.assertEqual(self.protocol.calls.pop(0), args)
else:
test.assertEqual(self.protocol.calls, ())
def unpack_result(self, result):
"""For methods that return Result objects, unwrap the results
"""
result, callback = result.args
callback()
return result
src/ZEO/tests/zeo-fan-out.test
View file @ c499ea7d
......@@ -25,6 +25,22 @@ Now, let's create some client storages that connect to these:
>>> import os, ZEO, ZODB.blob, ZODB.POSException, transaction
>>> db0 = ZEO.DB(port0, blob_dir='cb0')
XXX Work around a ZEO4 server bug (that was fixed in the ZEO5 server)
which prevents sending invalidations, and thus tids for transactions
that only add objects. This doesn't matter for ZODB4/ZEO4 but does
for ZODB5/ZEO5, with it's greater reliance on MVCC. This mainly
affects creation of the database root object, which is typically the
only transaction that only adds objects. Exacerbating this further is
that previously, databases didn't really use MVCC when checking for
the root object, but now they do because they go through a connection.
>>> with db0.transaction() as conn:
... conn.root.work_around_zeo4_server_bug = 1
>>> with db0.transaction() as conn:
... del conn.root.work_around_zeo4_server_bug
>>> db1 = ZEO.DB(addr1, blob_dir='cb1')
>>> tm1 = transaction.TransactionManager()
>>> c1 = db1.open(transaction_manager=tm1)
......@@ -94,6 +110,8 @@ Now, let's see if we can break it. :)
>>> s2 = db2.storage
>>> start_time = time.time()
>>> import os
>>> from ZEO.ClientStorage import _lock_blob
>>> while time.time() - start_time < 999:
... t = tm2.begin()
... if r2[1].v + r2[2].v:
......@@ -103,7 +121,10 @@ Now, let's see if we can break it. :)
... path = s2.fshelper.getBlobFilename(*blob_id)
... if os.path.exists(path):
... ZODB.blob.remove_committed(path)
... s2._server.sendBlob(*blob_id)
... _ = s2.fshelper.createPathForOID(blob_id[0])
... blob_lock = _lock_blob(path)
... s2._call('sendBlob', *blob_id, timeout=9999)
... blob_lock.close()
... else: print('Dang')
>>> threadf.join()
......
src/ZEO/tests/zeoserver.py deleted 100644 → 0
View file @ bc92bdd7
##############################################################################
#
# Copyright (c) 2001, 2002 Zope Foundation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE
#
##############################################################################
"""Helper file used to launch a ZEO server cross platform"""
import asyncore
import errno
import getopt
import logging
import os
import signal
import socket
import sys
import threading
import time
import ZEO.runzeo
import ZEO.zrpc.connection
def cleanup(storage):
# FileStorage and the Berkeley storages have this method, which deletes
# all files and directories used by the storage. This prevents @-files
# from clogging up /tmp
try:
storage.cleanup()
except AttributeError:
pass
logger = logging.getLogger('ZEO.tests.zeoserver')
def log(label, msg, *args):
message = "(%s) %s" % (label, msg)
logger.debug(message, *args)
class ZEOTestServer(asyncore.dispatcher):
"""A server for killing the whole process at the end of a test.
The first time we connect to this server, we write an ack character down
the socket. The other end should block on a recv() of the socket so it
can guarantee the server has started up before continuing on.
The second connect to the port immediately exits the process, via
os._exit(), without writing data on the socket. It does close and clean
up the storage first. The other end will get the empty string from its
recv() which will be enough to tell it that the server has exited.
I think this should prevent us from ever getting a legitimate addr-in-use
error.
"""
__super_init = asyncore.dispatcher.__init__
def __init__(self, addr, server, keep):
self.__super_init()
self._server = server
self._sockets = [self]
self._keep = keep
# Count down to zero, the number of connects
self._count = 1
self._label ='%d @ %s' % (os.getpid(), addr)
if isinstance(addr, str):
self.create_socket(socket.AF_UNIX, socket.SOCK_STREAM)
else:
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
# Some ZEO tests attempt a quick start of the server using the same
# port so we have to set the reuse flag.
self.set_reuse_addr()
try:
self.bind(addr)
except:
# We really want to see these exceptions
import traceback
traceback.print_exc()
raise
self.listen(5)
self.log('bound and listening')
def log(self, msg, *args):
log(self._label, msg, *args)
def handle_accept(self):
sock, addr = self.accept()
self.log('in handle_accept()')
# When we're done with everything, close the storage. Do not write
# the ack character until the storage is finished closing.
if self._count <= 0:
self.log('closing the storage')
self._server.close()
if not self._keep:
for storage in self._server.storages.values():
cleanup(storage)
self.log('exiting')
# Close all the other sockets so that we don't have to wait
# for os._exit() to get to it before starting the next
# server process.
for s in self._sockets:
s.close()
# Now explicitly close the socket returned from accept(),
# since it didn't go through the wrapper.
sock.close()
os._exit(0)
self.log('continuing')
sock.send(b'X')
self._count -= 1
def register_socket(self, sock):
# Register a socket to be closed when server shutsdown.
self._sockets.append(sock)
class Suicide(threading.Thread):
def __init__(self, addr):
threading.Thread.__init__(self)
self._adminaddr = addr
def run(self):
# If this process doesn't exit in 330 seconds, commit suicide.
# The client threads in the ConcurrentUpdate tests will run for
# as long as 300 seconds. Set this timeout to 330 to minimize
# chance that the server gives up before the clients.
time.sleep(999)
log(str(os.getpid()), "suicide thread invoking shutdown")
# If the server hasn't shut down yet, the client may not be
# able to connect to it. If so, try to kill the process to
# force it to shutdown.
if hasattr(os, "kill"):
os.kill(pid, signal.SIGTERM)
time.sleep(5)
os.kill(pid, signal.SIGKILL)
else:
from ZEO.tests.forker import shutdown_zeo_server
# Nott: If the -k option was given to zeoserver, then the
# process will go away but the temp files won't get
# cleaned up.
shutdown_zeo_server(self._adminaddr)
def main():
global pid
pid = os.getpid()
label = str(pid)
log(label, "starting")
# We don't do much sanity checking of the arguments, since if we get it
# wrong, it's a bug in the test suite.
keep = 0
configfile = None
suicide = True
# Parse the arguments and let getopt.error percolate
opts, args = getopt.getopt(sys.argv[1:], 'dkSC:v:')
for opt, arg in opts:
if opt == '-k':
keep = 1
if opt == '-d':
ZEO.zrpc.connection.debug_zrpc = True
elif opt == '-C':
configfile = arg
elif opt == '-S':
suicide = False
elif opt == '-v':
ZEO.zrpc.connection.Connection.current_protocol = arg.encode(
'ascii')
zo = ZEO.runzeo.ZEOOptions()
zo.realize(["-C", configfile])
addr = zo.address
if zo.auth_protocol == "plaintext":
__import__('ZEO.tests.auth_plaintext')
if isinstance(addr, tuple):
test_addr = addr[0], addr[1]+1
else:
test_addr = addr + '-test'
log(label, 'creating the storage server')
mon_addr = None
if zo.monitor_address:
mon_addr = zo.monitor_address
storages = dict((s.name or '1', s.open()) for s in zo.storages)
server = ZEO.runzeo.create_server(storages, zo)
try:
log(label, 'creating the test server, keep: %s', keep)
t = ZEOTestServer(test_addr, server, keep)
except socket.error as e:
if e[0] != errno.EADDRINUSE:
raise
log(label, 'addr in use, closing and exiting')
for storage in storages.values():
storage.close()
cleanup(storage)
sys.exit(2)
t.register_socket(server.dispatcher)
if suicide:
# Create daemon suicide thread
d = Suicide(test_addr)
d.setDaemon(1)
d.start()
# Loop for socket events
log(label, 'entering asyncore loop')
server.start_thread()
asyncore.loop()
if __name__ == '__main__':
import warnings
warnings.simplefilter('ignore')
main()
src/ZEO/zconfig.py 0 → 100644
View file @ c499ea7d
"""SSL configuration support
"""
import os
import sys
def ssl_config(section, server):
import ssl
cafile = capath = None
auth = section.authenticate
if auth:
if os.path.isdir(auth):
capath=auth
elif auth != 'DYNAMIC':
cafile=auth
context = ssl.create_default_context(
ssl.Purpose.CLIENT_AUTH, cafile=cafile, capath=capath)
if not auth:
assert not server
context.load_default_certs()
if section.certificate:
password = section.password_function
if password:
module, name = password.rsplit('.', 1)
module = __import__(module, globals(), locals(), ['*'], 0)
password = getattr(module, name)
context.load_cert_chain(section.certificate, section.key, password)
context.verify_mode = ssl.CERT_REQUIRED
if sys.version_info >= (3, 4):
context.verify_flags |= ssl.VERIFY_X509_STRICT | (
context.cert_store_stats()['crl'] and ssl.VERIFY_CRL_CHECK_LEAF)
if server:
context.check_hostname = False
return context
context.check_hostname = section.check_hostname
return context, section.server_hostname
def server_ssl(section):
return ssl_config(section, True)
def client_ssl(section):
return ssl_config(section, False)
class ClientStorageConfig:
def __init__(self, config):
self.config = config
self.name = config.getSectionName()
def open(self):
from ZEO.ClientStorage import ClientStorage
# config.server is a multikey of socket-connection-address values
# where the value is a socket family, address tuple.
config = self.config
addresses = [server.address for server in config.server]
options = {}
if config.blob_cache_size is not None:
options['blob_cache_size'] = config.blob_cache_size
if config.blob_cache_size_check is not None:
options['blob_cache_size_check'] = config.blob_cache_size_check
if config.client_label is not None:
options['client_label'] = config.client_label
ssl = config.ssl
if ssl:
options['ssl'] = ssl[0]
options['ssl_server_hostname'] = ssl[1]
return ClientStorage(
addresses,
blob_dir=config.blob_dir,
shared_blob_dir=config.shared_blob_dir,
storage=config.storage,
cache_size=config.cache_size,
cache=config.cache_path,
name=config.name,
read_only=config.read_only,
read_only_fallback=config.read_only_fallback,
server_sync = config.server_sync,
wait_timeout=config.wait_timeout,
**options)
tox.ini
View file @ c499ea7d
[tox]
envlist =
py27,py33,py34,pypy,simple
py27,py34,py35,simple
[testenv]
commands =
......@@ -20,11 +20,12 @@ deps =
zope.interface
zope.testing
zope.testrunner
mock
[testenv:simple]
# Test that 'setup.py test' works
basepython =
python2.7
python3.4
commands =
python setup.py -q test -q
deps = {[testenv]deps}
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7