Commit eebae5a2 authored by Julien Muchembled's avatar Julien Muchembled

Merge pull request #54 from zopefoundation/deps-cleanup

Cleanup dependencies, remove ZEO leftovers
parents 6b707535 bf276846
......@@ -16,7 +16,7 @@ use ZoneAlarm.
Compatibility
=============
ZODB 4.0 requires Python 2.6, 2.7, 3.2, 3.3 or 3.4.
ZODB 4.3 requires Python 2.7 or Python >= 3.3.
Travis: |buildstatus|_
winbot: |winbotstatus|_
......@@ -27,11 +27,10 @@ Prerequisites
You must have Python installed. If you're using a system Python
install, make sure development support is installed too.
You also need the transaction, BTrees, persistent, zc.lockfile,
ZConfig, zdaemon, zope.event, zope.interface, zope.proxy and
zope.testing packages. If you don't have them and you can connect to
the Python Package Index, then these will be installed for you if you
don't have them.
You also need the transaction, BTrees, persistent, six, zc.lockfile, ZConfig,
zodbpickle, zope.interface packages, and optionally manuel and zope.testing.
If you don't have them and you can connect to the Python Package Index,
then these will be installed for you if you don't have them.
Installation
============
......@@ -41,9 +40,8 @@ and install it are to use `easy_install
<http://peak.telecommunity.com/DevCenter/EasyInstall>`_, or
`zc.buildout <http://www.python.org/pypi/zc.buildout>`_.
To install by hand, first install the dependencies, ZConfig, zdaemon,
zope.interface, zope.proxy and zope.testing. These can be found
in the `Python Package Index <http://www.python.org/pypi>`_.
To install by hand, first install the dependencies listed in `Prerequisites`_.
These can be found in the `Python Package Index <http://www.python.org/pypi>`_.
To run the tests, use the test setup command::
......
===========================================
How to use NFS to make Blobs more efficient
===========================================
:Author: Christian Theune <ct@gocept.com>
Overview
========
When handling blobs, the biggest goal is to avoid writing operations that
require the blob data to be transferred using up IO resources.
When bringing a blob into the system, at least one O(N) operation has to
happen, e.g. when the blob is uploaded via a network server. The blob should
be extracted as a file on the final storage volume as early as possible,
avoiding further copies.
In a ZEO setup, all data is stored on a networked server and passed to it
using zrpc. This is a major problem for handling blobs, because it will lock
all transactions from committing when storing a single large blob. As a
default, this mechanism works but is not recommended for high-volume
installations.
Shared filesystem
=================
The solution for the transfer problem is to setup various storage parameters
so that blobs are always handled on a single volume that is shared via network
between ZEO servers and clients.
Step 1: Setup a writable shared filesystem for ZEO server and client
--------------------------------------------------------------------
On the ZEO server, create two directories on the volume that will be used by
this setup (assume the volume is accessible via $SERVER/):
- $SERVER/blobs
- $SERVER/tmp
Then export the $SERVER directory using a shared network filesystem like NFS.
Make sure it's writable by the ZEO clients.
Assume the exported directory is available on the client as $CLIENT.
Step 2: Application temporary directories
-----------------------------------------
Applications (i.e. Zope) will put uploaded data in a temporary directory
first. Adjust your TMPDIR, TMP or TEMP environment variable to point to the
shared filesystem:
$ export TMPDIR=$CLIENT/tmp
Step 3: ZEO client caches
-------------------------
Edit the file `zope.conf` on the ZEO client and adjust the configuration of
the `zeoclient` storage with two new variables::
blob-dir = $CLIENT/blobs
blob-cache-writable = yes
Step 4: ZEO server
------------------
Edit the file `zeo.conf` on the ZEO server to configure the blob directory.
Assuming the published storage of the ZEO server is a file storage, then the
configuration should look like this::
<blobstorage 1>
<filestorage>
path $INSTANCE/var/Data.fs
<filestorage>
blob-dir $SERVER/blobs
</blobstorage>
(Remember to manually replace $SERVER and $CLIENT with the exported directory
as accessible by either the ZEO server or the ZEO client.)
Conclusion
----------
At this point, after restarting your ZEO server and clients, the blob
directory will be shared and a minimum amount of IO will occur when working
with blobs.
ZEO Client Cache Tracing
========================
An important question for ZEO users is: how large should the ZEO
client cache be? ZEO 2 (as of ZEO 2.0b2) has a new feature that lets
you collect a trace of cache activity and tools to analyze this trace,
enabling you to make an informed decision about the cache size.
Don't confuse the ZEO client cache with the Zope object cache. The
ZEO client cache is only used when an object is not in the Zope object
cache; the ZEO client cache avoids roundtrips to the ZEO server.
Enabling Cache Tracing
----------------------
To enable cache tracing, you must use a persistent cache (specify a ``client``
name), and set the environment variable ZEO_CACHE_TRACE to a non-empty
value. The path to the trace file is derived from the path to the persistent
cache file by appending ".trace". If the file doesn't exist, ZEO will try to
create it. If the file does exist, it's opened for appending (previous trace
information is not overwritten). If there are problems with the file, a
warning message is logged. To start or stop tracing, the ZEO client process
(typically a Zope application server) must be restarted.
The trace file can grow pretty quickly; on a moderately loaded server, we
observed it growing by 7 MB per hour. The file consists of binary records,
each 34 bytes long if 8-byte oids are in use; a detailed description of the
record lay-out is given in stats.py. No sensitive data is logged: data
record sizes (but not data records), and binary object and transaction ids
are logged, but no object pickles, object types or names, user names,
transaction comments, access paths, or machine information (such as machine
name or IP address) are logged.
Analyzing a Cache Trace
-----------------------
The stats.py command-line tool is the first-line tool to analyze a cache
trace. Its default output consists of two parts: a one-line summary of
essential statistics for each segment of 15 minutes, interspersed with lines
indicating client restarts, followed by a more detailed summary of overall
statistics.
The most important statistic is the "hit rate", a percentage indicating how
many requests to load an object could be satisfied from the cache. Hit rates
around 70% are good. 90% is excellent. If you see a hit rate under 60% you
can probably improve the cache performance (and hence your Zope application
server's performance) by increasing the ZEO cache size. This is normally
configured using key ``cache_size`` in the ``zeoclient`` section of your
configuration file. The default cache size is 20 MB, which is small.
The stats.py tool shows its command line syntax when invoked without
arguments. The tracefile argument can be a gzipped file if it has a .gz
extension. It will be read from stdin (assuming uncompressed data) if the
tracefile argument is '-'.
Simulating Different Cache Sizes
--------------------------------
Based on a cache trace file, you can make a prediction of how well the cache
might do with a different cache size. The simul.py tool runs a simulation of
the ZEO client cache implementation based upon the events read from a trace
file. A new simulation is started each time the trace file records a client
restart event; if a trace file contains more than one restart event, a
separate line is printed for each simulation, and a line with overall
statistics is added at the end.
Example, assuming the trace file is in /tmp/cachetrace.log::
$ python simul.py -s 4 /tmp/cachetrace.log
CircularCacheSimulation, cache size 4,194,304 bytes
START TIME DURATION LOADS HITS INVALS WRITES HITRATE EVICTS INUSE
Jul 22 22:22 39:09 3218856 1429329 24046 41517 44.4% 40776 99.8
This shows that with a 4 MB cache size, the cache hit rate is 44.4%, the
percentage 1429329 (number of cache hits) is of 3218856 (number of load
requests). The cache simulated 40776 evictions, to make room for new object
states. At the end, 99.8% of the bytes reserved for the cache file were in
use to hold object state (the remaining 0.2% consists of "holes", bytes freed
by object eviction and not yet reused to hold another object's state).
Let's try this again with an 8 MB cache::
$ python simul.py -s 8 /tmp/cachetrace.log
CircularCacheSimulation, cache size 8,388,608 bytes
START TIME DURATION LOADS HITS INVALS WRITES HITRATE EVICTS INUSE
Jul 22 22:22 39:09 3218856 2182722 31315 41517 67.8% 40016 100.0
That's a huge improvement in hit rate, which isn't surprising since these are
very small cache sizes. The default cache size is 20 MB, which is still on
the small side::
$ python simul.py /tmp/cachetrace.log
CircularCacheSimulation, cache size 20,971,520 bytes
START TIME DURATION LOADS HITS INVALS WRITES HITRATE EVICTS INUSE
Jul 22 22:22 39:09 3218856 2982589 37922 41517 92.7% 37761 99.9
Again a very nice improvement in hit rate, and there's not a lot of room left
for improvement. Let's try 100 MB::
$ python simul.py -s 100 /tmp/cachetrace.log
CircularCacheSimulation, cache size 104,857,600 bytes
START TIME DURATION LOADS HITS INVALS WRITES HITRATE EVICTS INUSE
Jul 22 22:22 39:09 3218856 3218741 39572 41517 100.0% 22778 100.0
It's very unusual to see a hit rate so high. The application here frequently
modified a very large BTree, so given enough cache space to hold the entire
BTree it rarely needed to ask the ZEO server for data: this application
reused the same objects over and over.
More typical is that a substantial number of objects will be referenced only
once. Whenever an object turns out to be loaded only once, it's a pure loss
for the cache: the first (and only) load is a cache miss; storing the object
evicts other objects, possibly causing more cache misses; and the object is
never loaded again. If, for example, a third of the objects are loaded only
once, it's quite possible for the theoretical maximum hit rate to be 67%, no
matter how large the cache.
The simul.py script also contains code to simulate different cache
strategies. Since none of these are implemented, and only the default cache
strategy's code has been updated to be aware of MVCC, these are not further
documented here.
Simulation Limitations
----------------------
The cache simulation is an approximation, and actual hit rate may be higher
or lower than the simulated result. These are some factors that inhibit
exact simulation:
- The simulator doesn't try to emulate versions. If the trace file contains
loads and stores of objects in versions, the simulator treats them as if
they were loads and stores of non-version data.
- Each time a load of an object O in the trace file was a cache hit, but the
simulated cache has evicted O, the simulated cache has no way to repair its
knowledge about O. This is more frequent when simulating caches smaller
than the cache used to produce the trace file. When a real cache suffers a
cache miss, it asks the ZEO server for the needed information about O, and
saves O in the client cache. The simulated cache doesn't have a ZEO server
to ask, and O continues to be absent in the simulated cache. Further
requests for O will continue to be simulated cache misses, although in a
real cache they'll likely be cache hits. On the other hand, the
simulated cache doesn't need to evict any objects to make room for O, so it
may enjoy further cache hits on objects a real cache would have evicted.
ZEO Client Cache
The client cache provides a disk based cache for each ZEO client. The
client cache allows reads to be done from local disk rather than by remote
access to the storage server.
The cache may be persistent or transient. If the cache is persistent, then
the cache file is retained for use after process restarts. A non-
persistent cache uses a temporary file.
The client cache is managed in a single file, of the specified size.
The life of the cache is as follows:
- The cache file is opened (if it already exists), or created and set to
the specified size.
- Cache records are written to the cache file, as transactions commit
locally, and as data are loaded from the server.
- Writes are to "the current file position". This is a pointer that
travels around the file, circularly. After a record is written, the
pointer advances to just beyond it. Objects starting at the current
file position are evicted, as needed, to make room for the next record
written.
A distinct index file is not created, although indexing structures are
maintained in memory while a ClientStorage is running. When a persistent
client cache file is reopened, these indexing structures are recreated
by analyzing the file contents.
Persistent cache files are created in the directory named in the ``var``
argument to the ClientStorage, or if ``var`` is None, in the current
working directory. Persistent cache files have names of the form::
client-storage.zec
where:
client -- the client name, as given by the ClientStorage's ``client``
argument
storage -- the storage name, as given by the ClientStorage's ``storage``
argument; this is typically a string denoting a small integer,
"1" by default
For example, the cache file for client '8881' and storage 'spam' is named
"8881-spam.zec".
This diff is collapsed.
......@@ -160,7 +160,6 @@ setup(name="ZODB",
'transaction >= 1.4.4',
'six',
'zc.lockfile',
'zdaemon >= 4.0.0a1',
'zope.interface',
'zodbpickle >= 0.6.0',
],
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment