Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Z
ZODB
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Analytics
Analytics
CI / CD
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Kirill Smelkov
ZODB
Commits
52a9c479
Commit
52a9c479
authored
Jul 24, 2011
by
Hanno Schlichting
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Tweaks
parent
ccade6e5
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
20 additions
and
21 deletions
+20
-21
documentation/articles/multi-zodb-gc.rst
documentation/articles/multi-zodb-gc.rst
+20
-21
No files found.
documentation/articles/multi-zodb-gc.rst
View file @
52a9c479
Using zc.zodbdgc (fix PosKeyError's)
Using zc.zodbdgc (fix PosKeyError's)
====================================
====================================
The `zc.zodbdgc <http://pypi.python.org/pypi/zc.zodbdgc>`_ library contains two
*This article was written by Hanno Schlichting*
useful features.
On the one hand it supports advanced ZODB packing and garbage collection
The `zc.zodbdgc <http://pypi.python.org/pypi/zc.zodbdgc>`_ library contains two
approaches and on the other hand it includes the ability to create a database
useful features. On the one hand it supports advanced ZODB packing and garbage
of all persistent references.
collection approaches and on the other hand it includes the ability to create a
database of all persistent references.
The second feature allows us to debug and repair PosKeyErrors by finding the
The second feature allows us to debug and repair PosKeyErrors by finding the
persistent object(s) that point to the lost object.
persistent object(s) that point to the lost object.
...
@@ -24,8 +24,8 @@ same information as we are creating in the reference database.
...
@@ -24,8 +24,8 @@ same information as we are creating in the reference database.
Setup
Setup
-----
-----
We'll assume you are familiar with a buildout setup. A
simple example config
We'll assume you are familiar with a buildout setup. A
typical config might
might
look like this::
look like this::
[buildout]
[buildout]
parts =
parts =
...
@@ -84,7 +84,7 @@ Garbage collection
...
@@ -84,7 +84,7 @@ Garbage collection
------------------
------------------
We configured the ZEO server to skip garbage collection as part of the normal
We configured the ZEO server to skip garbage collection as part of the normal
pack in the
exampl
e config (`pack-gc = false`). Instead we use explicit garbage
pack in the
aboc
e config (`pack-gc = false`). Instead we use explicit garbage
collection via a different job::
collection via a different job::
bin/multi-zodb-gc etc/zeo.conf
bin/multi-zodb-gc etc/zeo.conf
...
@@ -112,8 +112,8 @@ Reference analysis and POSKeyErrors
...
@@ -112,8 +112,8 @@ Reference analysis and POSKeyErrors
If our database has any POSKeyErrors, we can find and repair those.
If our database has any POSKeyErrors, we can find and repair those.
If we haven't gotten the oid's of lost objects already, we can check our
Either we already have the oid's of lost objects or we can check the entire
database for any errors
, by running the following script
::
database for any errors
. To check everything we run the following command
::
$ bin/multi-zodb-check-refs etc/zeo.conf
$ bin/multi-zodb-check-refs etc/zeo.conf
...
@@ -121,8 +121,8 @@ This can take about 15 to 30 minutes on moderately sized databases of up to
...
@@ -121,8 +121,8 @@ This can take about 15 to 30 minutes on moderately sized databases of up to
10gb but depends on disk speed. We'll write down the reported errors, as we'll
10gb but depends on disk speed. We'll write down the reported errors, as we'll
need them later on to analyze them.
need them later on to analyze them.
If there are
some errors, we can create a reference database to make it easier
If there are
any lost objects, we can create a reference database to make it
to debug and find those error
s::
easier to debug and find those lost object
s::
$ bin/multi-zodb-check-refs -r var/refdb.fs etc/zeo.conf
$ bin/multi-zodb-check-refs -r var/refdb.fs etc/zeo.conf
...
@@ -141,23 +141,18 @@ If we've gotten this error report::
...
@@ -141,23 +141,18 @@ If we've gotten this error report::
!!! main 13184375 ?
!!! main 13184375 ?
POSKeyError: 0xc92d77
POSKeyError: 0xc92d77
We can look up the poid it was referenced from via::
We can look up the p
ersistent
oid it was referenced from via::
>>> parent = list(refs['main'][13184375])
>>> parent = list(refs['main'][13184375])
>>> parent
>>> parent
[13178389]
[13178389]
If you prefer the hex representation, that's easy as well
::
We can also get the hex representation
::
>>> from ZODB.utils import p64
>>> from ZODB.utils import p64
>>> p64(parent[0])
>>> p64(parent[0])
'\x00\x00\x00\x00\x00\xc9\x16\x15'
'\x00\x00\x00\x00\x00\xc9\x16\x15'
Once we are finished we should be nice and close the database::
>>> conn.close()
>>> db.close()
With this information, we should get back to our actual database and look
With this information, we should get back to our actual database and look
up this object. We'll leave the ref db open, as we might need to recursively
up this object. We'll leave the ref db open, as we might need to recursively
look up some more objects, until we get one we can identify and work on.
look up some more objects, until we get one we can identify and work on.
...
@@ -188,7 +183,7 @@ one::
...
@@ -188,7 +183,7 @@ one::
>>> import transaction
>>> import transaction
>>> transaction.begin()
>>> transaction.begin()
The poid that was reported missing was ``13184375``::
The p
ersistent
oid that was reported missing was ``13184375``::
>>> from ZODB.utils import p64
>>> from ZODB.utils import p64
>>> p64(13184375)
>>> p64(13184375)
...
@@ -199,7 +194,7 @@ The poid that was reported missing was ``13184375``::
...
@@ -199,7 +194,7 @@ The poid that was reported missing was ``13184375``::
>>> a._p_oid = '\x00\x00\x00\x00\x00\xc9-w'
>>> a._p_oid = '\x00\x00\x00\x00\x00\xc9-w'
We cannot use the ``add`` method of the connection, as this would assign the
We cannot use the ``add`` method of the connection, as this would assign the
object a new poid. So we replicate its internals here::
object a new p
ersistent
oid. So we replicate its internals here::
>>> a._p_jar = app._p_jar
>>> a._p_jar = app._p_jar
>>> app._p_jar._register(a)
>>> app._p_jar._register(a)
...
@@ -215,6 +210,10 @@ Both getting the object as well as its parent will work now::
...
@@ -215,6 +210,10 @@ Both getting the object as well as its parent will work now::
>>> app._p_jar.get('\x00\x00\x00\x00\x00\xc9\x16\x15')
>>> app._p_jar.get('\x00\x00\x00\x00\x00\xc9\x16\x15')
BTrees.IOBTree.IOBucket([(39078692, <five.intid.keyreference...
BTrees.IOBTree.IOBucket([(39078692, <five.intid.keyreference...
Once we are finished we should be nice and close all databases::
>>> conn.close()
>>> db.close()
Depending on the class of object that went missing, we might need to use a
Depending on the class of object that went missing, we might need to use a
different persistent class, like a persistent mapping or a BTree bucket.
different persistent class, like a persistent mapping or a BTree bucket.
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment