Commit c306df9b authored by Kirill Smelkov's avatar Kirill Smelkov

bigfile/zodb: Allow to pass both buffer and any object with buffer interface to ZBlk.setblkdata

Since wendelin.core beginning PyBigFile.storeblk(blk, buf) is passed with
buf being buffer object. See 35eb95c2 (bigfile: Python wrapper around
virtual memory subsystem) for details about that. In its turn
ZBigFile.storeblk was initially making bytes copy snapshot of that
buffer, but later pushed that job to ZBlk.setblkdata in
e0b00bda (bigfile/zodb: Make "set blkdata to be later saved to DB"
explicit method of ZBlk). So on that codepath ZBlk.setblkdata(buf) is
invoked with buf being python buffer as argument name suggests.

However later, when working on WCFS, for preparing test data I also
explicitly used ZBlk.setblkdata in

    wcfs/internal/zdata/testdata/zblk_test_gen.py  2c152d41 (wcfs: Add zdata package to load ZBlk/ZBigFile data)
    wcfs/internal/xbtree/xbtreetest/treegen.py     a8595565 (wcfs: tests: Treegen functionality)

and there test data is prepared by regularly using ZODB without virtmem
layer involvement at all. ZBlk.setblkdata calls are made with bytes
and strings which happen to work by chance on py2 more or less but
turned out to break a bit on py3.

-> Prepare to make all that to work robustly by allowing ZBlk.setblkdata
to be called with buf being either buffer object, or any other object
that provides buffer interface.

Everything continues to work for py2 and we will fix issues around
treegen.py on py3 in the next patch.
parent 49908e3c
# -*- coding: utf-8 -*-
# Wendelin.bigfile | BigFile ZODB backend
# Copyright (C) 2014-2024 Nexedi SA and Contributors.
# Copyright (C) 2014-2025 Nexedi SA and Contributors.
# Kirill Smelkov <kirr@nexedi.com>
#
# This program is free software: you can Use, Study, Modify and Redistribute
......@@ -201,6 +201,8 @@ class ZBlkBase(Persistent):
# client requests us to set blkdata to be later saved to DB
# (DB <- ) .blkdata <- memory-page
#
# buf: buffer object, or, for convenience anything with buffer interface
#
# return: blkchanged=(True|False) - whether blk should be considered changed
# after setting its data.
#
......@@ -291,7 +293,8 @@ class ZBlk0(ZBlkBase):
# (DB <- ) ._v_blkdata <- memory-page
def setblkdata(self, buf):
blkdata = bytes(buf) # FIXME does memcpy
buf = memoryview(buf)
blkdata = buf.tobytes() # FIXME does memcpy
# trim trailing \0
self._v_blkdata = blkdata.rstrip(b'\0') # FIXME copy
......@@ -407,6 +410,7 @@ class ZBlk1(ZBlkBase):
# (DB <- ) .chunktab <- memory-page
def setblkdata(self, buf):
buf = memoryview(buf)
chunktab = self.chunktab
CHUNKSIZE = self.CHUNKSIZE
blkchanged= False
......@@ -420,11 +424,10 @@ class ZBlk1(ZBlkBase):
# scan over buf and update/delete changed chunks
for start in range(0, len(buf), CHUNKSIZE):
data = buf[start:start+CHUNKSIZE] # FIXME copy on py2
data = buf[start:start+CHUNKSIZE]
# make sure data is bytes
# (else we cannot .rstrip() it below)
if not isinstance(data, bytes):
data = bytes(data) # FIXME copy on py3
data = data.tobytes() # FIXME copy
# trim trailing \0
data = data.rstrip(b'\0') # FIXME copy
chunk = chunktab.get(start)
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment