Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
W
wendelin.core
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Labels
Merge Requests
0
Merge Requests
0
Analytics
Analytics
Repository
Value Stream
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Commits
Open sidebar
Kirill Smelkov
wendelin.core
Commits
45db6007
Commit
45db6007
authored
Sep 24, 2015
by
Kirill Smelkov
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
bigfile/zodb: General overview how virtmem -> ZODB propagation works
parent
65f6a5db
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
22 additions
and
8 deletions
+22
-8
bigfile/file_zodb.py
bigfile/file_zodb.py
+22
-8
No files found.
bigfile/file_zodb.py
View file @
45db6007
...
@@ -50,14 +50,30 @@ will be our future approach after we teach NEO about object deduplication.
...
@@ -50,14 +50,30 @@ will be our future approach after we teach NEO about object deduplication.
~~~~
~~~~
TODO big picture module description
As file pages are changed in RAM with changes being managed by virtmem
subsystem, we need to propagate the changes to ZODB objects back at some time.
T
hings are done this way (vs more ZODB-like way) because
:
T
wo approaches exist
:
- compatibility with FS approach (to be able to switch to fuse)
1) on every RAM page dirty, in a callback invoked by virtmem, mark
- only one manager (vs two managers - bigfile & Connection and synchronizing them)
corresponding ZODB object as dirty, and at commit time, in
- on abort no need to synchronize changes (= "only 1 manager" as above)
obj.__getstate__ retrieve memory content.
- TODO ...
2) hook into commit process, and before committing, synchronize RAM page
state to ZODB objects state, propagating all dirtied pages to ZODB objects
and then do the commit process as usual.
"1" is more natural to how ZODB works, but requires tight integration between
virtmem subsystem and ZODB (to be able to receive callback on a page dirtying).
"2" is less natural to how ZODB works, but requires less-tight integration
between virtmem subsystem and ZODB, and virtmem->ZODB propagation happens only
at commit time.
Since, for performance reasons, virtmem subsystem is going away and BigFiles
will be represented by real FUSE-based filesystem with virtual memory being
done by kernel, where we cannot get callback on a page-dirtying, it is more
natural to also use "2" here.
"""
"""
from
wendelin.bigfile
import
BigFile
,
WRITEOUT_STORE
,
WRITEOUT_MARKSTORED
from
wendelin.bigfile
import
BigFile
,
WRITEOUT_STORE
,
WRITEOUT_MARKSTORED
...
@@ -76,8 +92,6 @@ import os
...
@@ -76,8 +92,6 @@ import os
# FIXME peak 2·Ndirty memory consumption on write (should be 1·NDirty)
# FIXME peak 2·Ndirty memory consumption on write (should be 1·NDirty)
# XXX write about why we can't memory-dirty -> ZBlk (memory could be modified
# later one more time)
# Base class for data of 1 file block as stored in ZODB
# Base class for data of 1 file block as stored in ZODB
class
ZBlkBase
(
Persistent
):
class
ZBlkBase
(
Persistent
):
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment