• Kirill Smelkov's avatar
    bigfile/zodb: Teach ZBlk1 not to waste memory on memory-page -> DB path · 1bf0cf31
    Kirill Smelkov authored
    ZBlk* objects are intermediate ZODB object in between data stored in
    ZODB and memory pages managed by virtmem. As such, after they do their
    job to either load data from DB to memory, or store from memory to DB,
    it is not needed to keep them alive with duplicate content thus only
    wasting memory.
    
    ZBlk0 cares about this detail via "deactivating" ._v_blkdata in
    loadblkdata() and __getstate__() prologues.
    
    ZBlk1 did the same for load path in loadblkdata() prologue, but for
    .__getstate__() it was not directly possible, because for ZBlk1 the
    state is IOBTree, not one non-persistent object, and thus it first needs
    to be processed by ZODB together with its subobjects on its way to
    storage and only then all they deactivated.
    
    So 13c0c17c (bigfile/zodb: Format #1 which is optimized for small
    changes) only put TODO for memory-page -> DB path about not wasting
    memory this way.
    
    But the problem is relatively easy to solve:
    
        - we can deactivate ZData objects (leaf objects in ZBlk1.chunktab
          btree) by hooking into ZData.__getstate__() prologue;
    
        - we also need to care to deactivate chunks right away, which
          setblkdata() loaded to compare .data and found them to be not
          changed
    
    This way we do not waste memory keeping intermediate ZData objects alive
    with the same content as memory page after commit.
    
    /cc @Tyagov
    1bf0cf31
file_zodb.py 27 KB