Merge pull request #24 from zopefoundation/setup-and-reference

Setup and reference

buildout.cfg

 [buildout]
-develop = .
+develop = . ../zodb
 parts =
     stxpy
     test
@@ -13,6 +13,10 @@ recipe = zc.recipe.egg
 eggs =
     Sphinx
     docutils
+    ZODB
+    j1m.sphinxautointerface
+    j1m.sphinxautozconfig
+
 interpreter = stxpy
 scripts =
     sphinx-build

conf.py

@@ -24,7 +24,9 @@
 
 # Add any Sphinx extension module names here, as strings. They can be extensions
 # coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
-extensions = []
+extensions = ['sphinx.ext.autodoc',
+              'j1m.sphinxautointerface',
+              'j1m.sphinxautozconfig']
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['.templates']

documentation/guide/index.rst

@@ -12,7 +12,11 @@ If you haven't yet, you should read the :ref:`Tutorial <tutorial-label>`.
 .. toctree::
    :maxdepth: 2
 
+   install-and-run
    writing-persistent-objects.rst
+
+.. todo:
+
    transaction.rst
    storages.rst
    configuration.rst

documentation/guide/install-and-run.rst

+==========================
+Installing and running ZODB
+===========================
+
+This topic discusses some boring nitty-gritty details needed to
+actually run ZODB.
+
+Installation
+============
+
+Installation of ZODB is pretty straightforward using Python's
+packaging system. For example, using pip::
+
+  pip install ZODB
+
+You may need additional optional packages, such as `ZEO
+<https://pypi.python.org/pypi/ZEO>`_ or `RelStorage
+<https://pypi.python.org/pypi/RelStorage>`_, depending your deployment
+choices.
+
+Configuration
+=============
+
+You can set up ZODB in your application using either Python, or
+ZODB's configuration language.  For simple database setup, and
+especially for exploration, the Python APIs are sufficient.
+
+For more complex configurations, you'll probably find ZODB's
+configuration language easier to use.
+
+To understand database setup, it's important to understand ZODB's
+architecture.  ZODB separates database functionality
+from storage concerns. When you create a database object,
+you specify a storage object for it to use, as in::
+
+    import ZODB, ZODB.FileStorage
+
+    storage = ZODB.FileStorage.FileStorage('mydata.fs')
+    db = ZODB.DB(storage)
+
+So when you define a database, you'll also define a storage. In the
+example above, we define a :class:`file storage
+<ZODB.FileStorage.FileStorage.FileStorage>` and then use it to define
+a database.
+
+Sometimes, storages are created through composition.  For example, if
+we want to save space, we could layer a ``ZlibStorage``
+[#zlibstoragefn]_ over the file storage::
+
+    import ZODB, ZODB.FileStorage, zc.zlibstorage
+
+    storage = ZODB.FileStorage.FileStorage('mydata.fs')
+    compressed_storage = zc.zlibstorage.ZlibStorage(storage)
+    db = ZODB.DB(compressed_storage)
+
+`ZlibStorage <https://pypi.python.org/pypi/zc.zlibstorage>`_
+compresses database records [#zlib]_.
+
+Python configuration
+--------------------
+
+To set up a database with Python, you'll construct a storage using the
+:ref:`storage APIs <included-storages-label>`, and then pass the
+storage to the :class:`~ZODB.DB` class to create a database, as shown
+in the examples in the previous section.
+
+The :class:`~ZODB.DB` class also accepts a string path name as its
+storage argument to automatically create a file storage.  You can also
+pass ``None`` as the storage to automatically use a
+:class:`~ZODB.MappingStorage.MappingStorage`, which is convenient when
+exploring ZODB::
+
+  db = ZODB.DB(None) # Create an in-memory database.
+
+Text configuration
+------------------
+
+ZODB supports a text-based configuration language.  It uses a syntax
+similar to Apache configuration files.  The syntax was chosen to be
+familiar to site administrators.
+
+ZODB's text configuration uses `ZConfig
+<https://pypi.python.org/pypi/ZConfig>`_. You can use ZConfig to
+create your application's configuration, but it's more common to
+include ZODB configuration strings in their own files or embedded in
+simpler configuration files, such as `configarser
+<https://docs.python.org/3/library/configparser.html#module-configparser>`_
+files.
+
+A database configuration string has a ``zodb`` section wrapping a
+storage section, as in::
+
+  <zodb>
+    cache-size-bytes 100MB
+    <mappingstorage>
+    </mappingstorage>
+  </zodb>
+
+.. -> snippet
+
+In the example above, the :ref:`mappingstorage
+<mappingstorage-text-configuration>` section defines the storage used
+by the database.
+
+To create a database from a string, use
+:func:`ZODB.config.databaseFromString`::
+
+    >>> import ZODB.config
+    >>> db = ZODB.config.databaseFromString(snippet)
+
+To load databases from file names or URLs, use
+:func:`ZODB.config.databaseFromURL`.
+
+URI-based configuration
+-----------------------
+
+Another database configuration option is provided by the `zodburi
+<https://pypi.python.org/pypi/zodburi>`_ package. See:
+http://docs.pylonsproject.org/projects/zodburi.  It's less powerful
+than the Python or text configuration options, but allows
+configuration to be reduced to a single URI and handles most cases.
+
+Using databases: connections
+============================
+
+Once you have a database, you need to get a database connection to to
+much of anything.  Connections take care of loading and saving objects
+and manage object caches. Each connection has it's own cache
+[#caches-are-expensive]_.
+
+Getting connections
+-------------------
+
+Amongst [#amongst]_ the common ways of getting a connection:
+
+db.open()
+   The database :meth:`~ZODB.DB.open` method opens a
+   connection, returning a connection object::
+
+      >>> conn = db.open()
+
+   It's up to the application to call
+   :meth:`~ZODB.Connection.Connection.close` when the application is
+   done using the connection.
+
+   If changes are made, the application :ref:`commits transactions
+   <commit-transactions>` to make them permanent.
+
+db.transaction()
+   The database :meth:`~ZODB.DB.transaction` method
+   returns a context manager that can be used with the `python with
+   statement
+   <https://docs.python.org/3/reference/compound_stmts.html#grammar-token-with_stmt>`_
+   to execute a block of code in a transaction::
+
+     with db.transaction() as connection:
+         connection.root.foo = 1
+
+   .. -> src
+
+      >>> exec(src)
+      >>> with db.transaction() as connection:
+      ...     print connection.root.foo
+      1
+
+      >>> _ = conn.transaction_manager.begin() # get updates on conn
+
+   In the example above, we used ``as connection`` to get the database
+   connection used in the variable ``connection``.
+
+some_object._p_jar
+   For code that's already running in the context of an open
+   connection, you can get the current connection as the ``_p_jar``
+   attribute of some persistent object that was accessed via the
+   connection.
+
+Getting objects
+---------------
+
+Once you have a connection, you access objects by traversing the
+object graph from the root object.
+
+The database root object is a mapping object that holds the top level
+objects in the database.  There should only be a small number of
+top-level objects (often only one).  You can get the root object by calling a
+connection's ``root`` attribute::
+
+    >>> root = conn.root()
+    >>> root
+    {'foo': 1}
+    >>> root['foo']
+    1
+
+For convenience [#root-convenience]_, you can also get top-level
+objects by accessing attributes of the connection root object:
+
+    >>> conn.root.foo
+    1
+
+Once you have a top-level object, you use its methods, attributes, or
+operations to access other objects and so on to get the objects you
+need.  Often indexing data structures like BTrees_ are used to
+make it possible to search objects in large collections.
+
+.. [#zlibstoragefn] `zc.zlibstorage
+   <https://pypi.python.org/pypi/zc.zlibstorage>`_ is an optional
+   package that you need to install separately.
+
+.. [#zlib] ZlibStorage uses the :mod:`zlib` standard module, which
+   uses the `zlib library <http://www.zlib.net/>`_.
+
+.. [#caches-are-expensive] ZODB can be very efficient at caching data
+   in memory, especially if your `working set
+   <https://en.wikipedia.org/wiki/Working_set>`_ is small enough to
+   fit in memory, because the cache is simply an object tree and
+   accessing a cached object typically requires no database
+   interaction.  Because each connection has its own cache,
+   connections can be expensive, depending on their cache sizes.  For
+   this reason, you'll generally want to limit the number of open
+   connections you have at any one time.  Connections are pooled, so
+   opening a connection is inexpensive.
+
+.. [#amongst] https://www.youtube.com/watch?v=7WJXHY2OXGE
+
+.. [#root-convenience] The ability to access top-level objects of the
+   database as root attributes is a recent convenience. Originally,
+   the ``root()`` method was used to access the root object which was
+   then accessed as a mapping.  It's still potentially useful to
+   access top-level objects using the mapping interface if their names
+   aren't valid attribute names.
+
+.. _BTrees: https://pythonhosted.org/BTrees/

documentation/guide/writing-persistent-objects.rst

@@ -157,7 +157,7 @@ record from the book record and is managed by ZODB independent of the
 management of the book.
 
 In addition to ``PersistentList`` and ``PersistentMapping``, general
-persistent data structures are provided by the ``BTrees`` package,
+persistent data structures are provided by the BTrees_ package,
 most notably ``BTree`` and ``TreeSet`` objects.  Unlike
 ``PersistentList`` and ``PersistentMapping``, ``BTree`` and
 ``TreeSet`` objects are scalable and can easily hold millions of
@@ -566,3 +566,5 @@ framework for managing schema-migration scripts.
    <https://pypi.python.org/pypi/zope.cachedescriptors>`_ package
    provides some descriptors that help implement attributes that cache
    data.
+
+.. _BTrees: https://pythonhosted.org/BTrees/

documentation/reference/index.rst

+=======================
+Reference Documentation
+=======================
+
+
+.. toctree::
+   :maxdepth: 2
+
+   zodb.rst
+   storages.rst
+ 

documentation/reference/storages.rst

+=============
+Storage APIs
+=============
+
+.. contents::
+
+Storage interfaces
+==================
+
+There are various storage implementations that implement standard
+storage interfaces. Thet differ primarily in their constructors.
+
+Application code rarely calls storage methods, and those it calls are
+generally called indirectly through databases.  There are
+interface-defined methods that are called internally by ZODB. These
+aren't shown below.
+
+
+IStorage
+--------
+
+.. autointerface:: ZODB.interfaces.IStorage
+   :members: close, getName, getSize, history, isReadOnly, lastTransaction,
+             __len__, pack, sortKey
+
+IStorageIteration
+-----------------
+
+.. autointerface:: ZODB.interfaces.IStorageIteration
+
+IStorageUndoable
+----------------
+
+.. autointerface:: ZODB.interfaces.IStorageUndoable
+   :members: undoLog, undoInfo
+
+IStorageCurrentRecordIteration
+------------------------------
+
+.. autointerface:: ZODB.interfaces.IStorageCurrentRecordIteration
+
+IBlobStorage
+------------
+
+.. autointerface:: ZODB.interfaces.IBlobStorage
+   :members: temporaryDirectory
+
+IStorageRecordInformation
+-------------------------
+
+.. autointerface:: ZODB.interfaces.IStorageRecordInformation
+
+IStorageTransactionInformation
+------------------------------
+
+.. autointerface:: ZODB.interfaces.IStorageTransactionInformation
+
+.. _included-storages-label:
+
+Included storages
+=================
+
+FileStorage
+-----------
+
+
+.. autoclass:: ZODB.FileStorage.FileStorage.FileStorage
+   :members: __init__
+
+
+.. autointerface:: ZODB.FileStorage.interfaces.IFileStoragePacker
+
+FileStorage text configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+File storages are configured using the ``filestorage`` section::
+
+  <filestorage>
+    path Data.fs
+  </filestorage>
+
+which accepts the following options:
+
+.. zconfigsectionkeys:: ZODB component.xml filestorage
+
+MappingStorage
+--------------
+
+.. autoclass:: ZODB.MappingStorage.MappingStorage
+   :members: __init__
+
+.. _mappingstorage-text-configuration:
+
+MappingStorage text configuration
+---------------------------------
+
+File storages are configured using the ``mappingstorage`` section::
+
+  <mappingstorage>
+  </mappingstorage>
+
+Options:
+
+.. zconfigsectionkeys:: ZODB component.xml mappingstorage
+
+DemoStorage
+-----------
+
+.. autoclass:: ZODB.DemoStorage.DemoStorage
+   :members: __init__, push, pop
+
+DemoStorage text configuration
+------------------------------
+
+Demo storages are configured using the ``demostorage`` section::
+
+  <demostorage>
+    <filestorage base>
+      path base.fs
+    </filestorage>
+    <mappingstorage changes>
+      name Changes
+    </mappingstorage>
+  </demostorage>
+
+.. -> src
+
+   >>> import ZODB.config
+   >>> storage = ZODB.config.storageFromString(src)
+   >>> storage.base.getName()
+   'base.fs'
+   >>> storage.changes.getName()
+   'Changes'
+
+``demostorage`` sections can contain up to 2 storage subsections,
+named ``base`` and ``changes``, specifying the demo storage's base and
+changes storages.  See :meth:`ZODB.DemoStorage.DemoStorage.__init__`
+for more on the base anc changes storages.
+
+Options:
+
+.. zconfigsectionkeys:: ZODB component.xml demostorage
+
+Noteworthy non-included storages
+================================
+
+A number of important ZODB storages are distriubuted separately, including:
+
+RelStorage
+  `RelStorage <http://relstorage.readthedocs.io/en/latest/>`_
+  stores data in relational databases.  This is especially
+  useful when you have requirements or existing infrastructure for
+  storing data in relational databases.  Unlike the included storages,
+  multiple processes can share the same database.
+
+  For more imformation, see http://relstorage.readthedocs.io/en/latest/.
+
+ZEO
+  `ZEO <https://github.com/zopefoundation/ZEO>`_ is a client-server
+  database implementation for ZODB.  To use ZEO, you run a ZEO server,
+  and use ZEO clients in your application.  Unlike the included
+  storages, multiple processes can share the same database.
+
+  For more imformation, see https://github.com/zopefoundation/ZEO.
+
+ZRS
+  `ZRS <https://github.com/zc/zrs>`_
+  provides replication from one database to another.  It's most
+  commonly used with ZEO.  With ZRS, you create a ZRS primary database
+  around a :class:`~ZODB.FileStorage.FileStorage.FileStorage` and in a
+  separate process, you creatre a ZRS secondary storage around any
+  :interface:`storage <ZODB.interfaces.IStorage>`. As transactions are
+  committed on the primary, they're copied asynchronously to
+  secondaries.
+
+  For more imformation, see https://github.com/zc/zrs.
+
+zlibstorage
+  `zlibstorage <https://pypi.python.org/pypi/zc.zlibstorage>`_
+  compresses database records using the compression
+  algorithm used by `gzip <http://www.gzip.org/>`_.
+
+  For more imformation, see https://pypi.python.org/pypi/zc.zlibstorage.
+
+beforestorage
+  `beforestorage <https://pypi.python.org/pypi/zc.beforestorage>`_
+  provides a point-in-time view of a database that might
+  be changing.  This can be useful to provide a non-changing view of a
+  production database for use with a :class:`~ZODB.DemoStorage.DemoStorage`.
+
+  For more imformation, see https://pypi.python.org/pypi/zc.beforestorage.
+
+cipher.encryptingstorage
+  `cipher.encryptingstorage
+  <https://pypi.python.org/pypi/cipher.encryptingstorage/>`_ provided
+  compression and encryption of database records.
+
+  For more informayion see,
+  https://pypi.python.org/pypi/cipher.encryptingstorage/.

documentation/reference/zodb.rst

+=========
+ZODB APIs
+=========
+
+.. contents::
+
+ZODB module functions
+=====================
+
+.. method:: DB(storage, *args, **kw)
+
+      Create a databse. See :py:class:`ZODB.DB`.
+
+.. autofunction:: ZODB.connection
+
+Databases
+=========
+
+.. autoclass:: ZODB.DB
+   :members: __init__, open, close, pack,
+             cacheDetail, cacheExtremeDetail, cacheMinimize,
+             cacheSize, cacheDetailSize, getCacheSize, getCacheSizeBytes,
+             lastTransaction, getName, getPoolSize, getSize,
+             getHistoricalCacheSize, getHistoricalCacheSizeBytes,
+             getHistoricalPoolSize, getHistoricalTimeout,
+             objectCount, connectionDebugInfo,
+             setCacheSize, setCacheSizeBytes,
+             setHistoricalCacheSize, setHistoricalCacheSizeBytes,
+             setPoolSize, setHistoricalPoolSize, setHistoricalTimeout,
+             history,
+             supportsUndo, undoLog, undoInfo, undoMultiple, undo,
+             transaction, storage
+
+.. _database-text-configuration:
+
+Database text configuration
+---------------------------
+
+Databases are configured with ``zodb`` sections::
+
+  <zodb>
+    cache-size-bytes 100MB
+    <mappingstorage>
+    </mappingstorage>
+  </zodb>
+
+A ``zodb`` section must have a storage sub-section specifying a
+storage and any of the following options:
+
+.. zconfigsectionkeys:: ZODB component.xml zodb
+
+.. _multidatabase-text-configuration:
+
+For a multi-database configuration, use multiple ``zodb`` sections and
+give the sections names::
+
+  <zodb first>
+    cache-size-bytes 100MB
+    <mappingstorage>
+    </mappingstorage>
+  </zodb>
+
+  <zodb second>
+    <mappingstorage>
+    </mappingstorage>
+  </zodb>
+
+.. -> src
+
+   >>> import ZODB.config
+   >>> db = ZODB.config.databaseFromString(src)
+   >>> sorted(db.databases)
+   ['first', 'second']
+   >>> db._cache_size_bytes
+   104857600
+
+When the configuration is loaded, a single database will be returned,
+but all of the databases will be available through the returned
+database's ``databases`` attribute.
+
+Connections
+===========
+
+.. autoclass:: ZODB.Connection.Connection
+   :members: add, cacheGC, cacheMinimize, close, db, get,
+             getDebugInfo, get_connection, isReadOnly, oldstate,
+             onCloseCallback, root, setDebugInfo, sync
+
+TimeStamp (transaction ids)
+===========================
+
+.. class:: ZODB.TimeStamp.TimeStamp(year, month, day, hour, minute, seconds)
+
+   Create a time-stamp object. Time stamps facilitate the computation
+   of transaction ids, which are based on times. The arguments are
+   integers, except for seconds, which may be a floating-point
+   number. Time stamps have microsecond precision. Time stamps are
+   implicitly UTC based.
+
+   Time stamps are orderable and hashable.
+
+   .. method:: day()
+
+      Return the time stamp's day.
+
+   .. method:: hour()
+
+      Return the time stamp's hour.
+
+   .. method:: laterThan(other)
+
+      Return a timestamp instance which is later than 'other'.
+
+      If self already qualifies, return self.
+
+      Otherwise, return a new instance one moment later than 'other'.
+
+   .. method:: minute()
+
+      Return the time stamp's minute.
+
+   .. method:: month()
+
+      Return the time stamp's month.
+
+   .. method:: raw()
+
+      Get an 8-byte representatin of the time stamp for use in APIs
+      that require a time stamp.
+
+   .. method:: second()
+
+      Return the time stamp's second.
+
+   .. method:: timeTime()
+
+      Return the time stamp as seconds since the epoc, as used by the
+      ``time`` module.
+
+   .. method:: year()
+
+      Return the time stamp's year.
+
+Loading configuration
+=====================
+
+.. automodule:: ZODB.config
+   :members: databaseFromString, databaseFromFile, databaseFromURL,
+             storageFromString, storageFromFile, storageFromURL

index.rst

@@ -198,6 +198,7 @@ Learning more
 
    documentation/tutorial
    documentation/guide/index
+   documentation/reference/index
    documentation/articles/index
 
 * `The ZODB Book (in progress) <http://zodb.readthedocs.org/en/latest/>`_ 

requirements.txt

+Sphinx
+docutils
+ZODB
+j1m.sphinxautointerface
+j1m.sphinxautozconfig

zodbdocumentationtests/tests.py

@@ -34,11 +34,15 @@ def tearDown(test):
 def test_suite():
     here = os.path.dirname(__file__)
     guide = join(here, '..', 'documentation', 'guide')
+    reference = join(here, '..', 'documentation', 'reference')
 
     return unittest.TestSuite((
         manuel.testing.TestSuite(
             manuel.doctest.Manuel() + manuel.capture.Manuel(),
             join(guide, 'writing-persistent-objects.rst'),
+            join(guide, 'install-and-run.rst'),
+            join(reference, 'zodb.rst'),
+            join(reference, 'storages.rst'),
             setUp=setUp, tearDown=tearDown,
             ),
         ))