Commit ae2567a6 authored by Kirill Smelkov's avatar Kirill Smelkov

fixup! fixup! fixup! fixup! ZBigFile: Add ZBlk format option 'h' (heuristic) (4)

We can return to migrating previous block by amending the condition when
we can do so: if that previous block is also modified by current
transaction, then migrating it is ok without breaking the invariant.

Also rework append criteria to be really adding data to tail of the file
instead of growing data of a block somewhere in the middle of the file.
This should prevent false-positive ZBlk0 activations.
parent 898b2e87
...@@ -596,19 +596,39 @@ class ZBigFile(LivePersistent): ...@@ -596,19 +596,39 @@ class ZBigFile(LivePersistent):
old_data = bytes(zblk.loadblkdata()).rstrip(b'\0') old_data = bytes(zblk.loadblkdata()).rstrip(b'\0')
ndelta = memdelta(old_data, new_data) ndelta = memdelta(old_data, new_data)
append = (new_data[:len(old_data)] == old_data) try:
last_blk = self.blktab.maxKey()
except ValueError: # empty tree
last_blk = -1
append_oldblk = ((blk == last_blk) and (new_data[:len(old_data)] == old_data))
append_newblk = ((blk == last_blk+1) and (len(old_data) == 0))
append = append_oldblk or append_newblk
small = (ndelta < 0.5*self.blksize) small = (ndelta < 0.5*self.blksize)
filled = (len(new_data) == self.blksize) # filled full with non-zeros at the end
# append - if block is already full, use ZBlk0 for fast reads # append - migrate previously filled-up block to ZBlk0 for fast reads
# - otherwise use ZBlk1 if the append is small and ZBlk0 otherwise # - for current block use ZBlk1 if the append is small and not fully filled, and ZBlk0 otherwise
#
# do the migration of previous block only if it is also changed in
# current transaction. This preserves the invairant that "transaction
# changes ZBlk objects only for modified blocks of the file".
# NOTE: this misses a case when append stops exactly on blocks boundary
# after appending some zeros. For now we ignore such case as improbable.
#
# For the implementation we rely on that zfileh.dirty_writeout()
# invokes storeblk in ascending order of blk, so that when we are here,
# we can be sure that if previous block is also modified, then .blktab
# already has corresponding entry for it.
if append: if append:
# NOTE: There can always be (stripped) 0 at the end of a block, so that if append_newblk:
# a full block isn't necesarrily as big as blksize. We therefore use an zblk_prev = self.blktab.get(blk-1)
# approximate definition of fullness here: if a block is mostly filled if zblk_prev is not None and \
# with != 0 data, we assume it's full. zblk_prev._p_changed and \
if len(new_data) >= (self.blksize * 0.99): type(zblk_prev) is not ZBlk0:
return ZBlk0 self._setzblk(blk-1, zblk_prev, zblk_prev.loadblkdata(), ZBlk0)
return ZBlk1 if small else ZBlk0 return ZBlk1 if (small and not filled) else ZBlk0
# all other changes - use ZBlk1 if the change is small and ZBlk0 otherwise # all other changes - use ZBlk1 if the change is small and ZBlk0 otherwise
else: else:
......
...@@ -388,7 +388,8 @@ int fileh_dirty_writeout(BigFileH *fileh, enum WriteoutFlags flags) ...@@ -388,7 +388,8 @@ int fileh_dirty_writeout(BigFileH *fileh, enum WriteoutFlags flags)
BUG_ON(fileh->writeout_inprogress); BUG_ON(fileh->writeout_inprogress);
fileh->writeout_inprogress = 1; fileh->writeout_inprogress = 1;
/* pages are stored (if stored) in sorted order */ /* pages are stored (if stored) in sorted order
* NOTE ZBlk format 'auto' relies on this */
if (flags & WRITEOUT_STORE) if (flags & WRITEOUT_STORE)
list_sort(&fileh->dirty_pages, hpage_indirty_cmp_bypgoffset, NULL); list_sort(&fileh->dirty_pages, hpage_indirty_cmp_bypgoffset, NULL);
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment