go: Proper support for py3-generated data and py3

- Add robust support for data generated by python3 based on ogórek [StrictUnicode mode](https://github.com/kisielk/og-rek/commit/b28613c2). - Test handling of all ZODB kinds of data we care about: py2/pickle{1,2,3} and py3/pickle3, fix issues discovered along the way. - Fix interoperability with py3 programs: for example previously index, that FileStorage/go saves, was not readable on py3 side. - Switch encoding protocol for pickles, that ZODB/go and other go places emit, from 2 to 3. - Switch unpickling to work in [PyDict=y](https://github.com/kisielk/og-rek/pull/75) mode. - Add tests for zodb.Map and zodb.List . - minor fixes. Robust support for py3-generated data is needed for Wendelin.core . This work depends on recent zodbtools update from nexedi/zodbtools!29. /cc @jerome, @levin.zimmermann, @vnmabus /proposed-for-review-on !8

go: Proper support for py3-generated data and py3
- Add robust support for data generated by python3 based on ogórek [StrictUnicode mode](https://github.com/kisielk/og-rek/commit/b28613c2). - Test handling of all ZODB kinds of data we care about: py2/pickle{1,2,3} and py3/pickle3, fix issues discovered along the way. - Fix interoperability with py3 programs: for example previously index, that FileStorage/go saves, was not readable on py3 side. - Switch encoding protocol for pickles, that ZODB/go and other go places emit, from 2 to 3. - Switch unpickling to work in [PyDict=y](https://github.com/kisielk/og-rek/pull/75) mode. - Add tests for zodb.Map and zodb.List . - minor fixes. Robust support for py3-generated data is needed for Wendelin.core . This work depends on recent zodbtools update from nexedi/zodbtools!29. /cc @jerome, @levin.zimmermann, @vnmabus /proposed-for-review-on !8
48f8faef · Kirill Smelkov · ee23551d · 0a58eb7a · 48f8faef · 48f8faef
Commit 48f8faef authored Feb 20, 2025 by Kirill Smelkov
145 changed files
--- a/go/go.mod
+++ b/go/go.mod
+module lab.nexedi.com/kirr/neo/go
+
+go 1.18
+
+require (
+	github.com/DataDog/czlib v0.0.0-20210322182103-8087f4e14ae7
+	github.com/cznic/strutil v0.0.0-20181122101858-275e90344537
+	github.com/fsnotify/fsnotify v1.7.0
+	github.com/golang/glog v1.2.2
+	github.com/kisielk/og-rek v1.2.1-0.20240923165241-e691997e3596
+	github.com/kylelemons/godebug v1.1.0
+	github.com/philhofer/fwd v1.1.3-0.20240612014219-fbbf4953d986
+	github.com/pkg/errors v0.9.1
+	github.com/shamaton/msgpack v1.2.1
+	github.com/someonegg/gocontainer v1.0.0
+	github.com/stretchr/testify v1.9.0
+	github.com/tinylib/msgp v1.2.0
+	lab.nexedi.com/kirr/go123 v0.0.0-20240626173136-48920809d24c
+)
+
+require (
+	crawshaw.io/sqlite v0.3.2 // indirect
+	github.com/aristanetworks/gomap v0.0.0-20230726210543-f4e41046dced // indirect
+	github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 // indirect
+	github.com/davecgh/go-spew v1.1.1 // indirect
+	github.com/pmezard/go-difflib v1.0.0 // indirect
+	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
+	golang.org/x/exp v0.0.0-20230725093048-515e97ebf090 // indirect
+	golang.org/x/sys v0.4.0 // indirect
+	gopkg.in/yaml.v3 v3.0.1 // indirect
+)
--- a/go/go.sum
+++ b/go/go.sum
+crawshaw.io/iox v0.0.0-20181124134642-c51c3df30797 h1:yDf7ARQc637HoxDho7xjqdvO5ZA2Yb+xzv/fOnnvZzw=
+crawshaw.io/iox v0.0.0-20181124134642-c51c3df30797/go.mod h1:sXBiorCo8c46JlQV3oXPKINnZ8mcqnye1EkVkqsectk=
+crawshaw.io/sqlite v0.3.2 h1:N6IzTjkiw9FItHAa0jp+ZKC6tuLzXqAYIv+ccIWos1I=
+crawshaw.io/sqlite v0.3.2/go.mod h1:igAO5JulrQ1DbdZdtVq48mnZUBAPOeFzer7VhDWNtW4=
+github.com/DataDog/czlib v0.0.0-20210322182103-8087f4e14ae7 h1:6ZJZdzkbvKb6HRXmZ12ICZ0IbqfR+0Cd2C0IutWHHIA=
+github.com/DataDog/czlib v0.0.0-20210322182103-8087f4e14ae7/go.mod h1:ROY4muaTWpoeQAx/oUkvxe9zKCmgU5xDGXsfEbA+omc=
+github.com/aristanetworks/gomap v0.0.0-20230726210543-f4e41046dced h1:HxlRMDx/VeRqzj3nvqX9k4tjeBcEIkoNHDJPsS389hs=
+github.com/aristanetworks/gomap v0.0.0-20230726210543-f4e41046dced/go.mod h1:p7lmI+ecoe1RTyD11SPXWsSQ3H+pJ4cp5y7vtKW4QdM=
+github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 h1:iwZdTE0PVqJCos1vaoKsclOGD3ADKpshg3SRtYBbwso=
+github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548/go.mod h1:e6NPNENfs9mPDVNRekM7lKScauxd5kXTr1Mfyig6TDM=
+github.com/cznic/strutil v0.0.0-20181122101858-275e90344537 h1:MZRmHqDBd0vxNwenEbKSQqRVT24d3C05ft8kduSwlqM=
+github.com/cznic/strutil v0.0.0-20181122101858-275e90344537/go.mod h1:AHHPPPXTw0h6pVabbcbyGRK1DckRn7r/STdZEeIDzZc=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nosvA=
+github.com/fsnotify/fsnotify v1.7.0/go.mod h1:40Bi/Hjc2AVfZrqy+aj+yEI+/bRxZnMJyTJwOpGvigM=
+github.com/golang/glog v1.2.2 h1:1+mZ9upx1Dh6FmUTFR1naJ77miKiXgALjWOZ3NVFPmY=
+github.com/golang/glog v1.2.2/go.mod h1:6AhwSGph0fcJtXVM/PEHPqZlFeoLxhs7/t5UDAwmO+w=
+github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
+github.com/kisielk/og-rek v1.2.1-0.20240716192710-7d520718eea5 h1:txaR41tUwcahEx3KrKg6JcI+wl2sP3SK2k4nhhb51Zs=
+github.com/kisielk/og-rek v1.2.1-0.20240716192710-7d520718eea5/go.mod h1:6ihsOSzSAxR/65S3Bn9zNihoEqRquhDQZ2c6I2+MG3c=
+github.com/kisielk/og-rek v1.2.1-0.20240717143235-465792bb0690 h1:bTP2GHBnWOuZx6OH9DlJNCZDauVqPX1Jkkxia+6kNzU=
+github.com/kisielk/og-rek v1.2.1-0.20240717143235-465792bb0690/go.mod h1:6ihsOSzSAxR/65S3Bn9zNihoEqRquhDQZ2c6I2+MG3c=
+github.com/kisielk/og-rek v1.2.1-0.20240819044129-b1bd3f2e3d5e h1:Pbl49BVQ1h9V3fPv6m82nu/+X0L5jHw4mS7gCCDHJFk=
+github.com/kisielk/og-rek v1.2.1-0.20240819044129-b1bd3f2e3d5e/go.mod h1:6ihsOSzSAxR/65S3Bn9zNihoEqRquhDQZ2c6I2+MG3c=
+github.com/kisielk/og-rek v1.2.1-0.20240923165241-e691997e3596 h1:m4FSNNEFnhXcUfaMmPphtrIGPhuGdxmVS744izSg9uI=
+github.com/kisielk/og-rek v1.2.1-0.20240923165241-e691997e3596/go.mod h1:4at7oxyfBTDilURhNCf7irHWtosJlJl9uyqUqAkrP4w=
+github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
+github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
+github.com/philhofer/fwd v1.1.3-0.20240612014219-fbbf4953d986 h1:jYi87L8j62qkXzaYHAQAhEapgukhenIMZRBKTNRLHJ4=
+github.com/philhofer/fwd v1.1.3-0.20240612014219-fbbf4953d986/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM=
+github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
+github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
+github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
+github.com/shamaton/msgpack v1.2.1 h1:40cwW7YAEdOIxcxIsUkAxSMUyYWZUyNiazI5AyiBntI=
+github.com/shamaton/msgpack v1.2.1/go.mod h1:ibiaNQRTCUISAYkkyOpaSCEBiCAxXe6u6Mu1sQ6945U=
+github.com/someonegg/gocontainer v1.0.0 h1:9MMUFbQf7g+g9sMG4ggBHPDS1+Iz+wd9Ee/O4BNRdw0=
+github.com/someonegg/gocontainer v1.0.0/go.mod h1:zGJcXRK0ikzEYPFKTaFXi6UU/ulNuJypfADX4UQGtMw=
+github.com/someonegg/gox v1.0.0/go.mod h1:pngAcWxBFnyYM4oY+h9Rgv0WaikLkSfY5dBYuabUnBE=
+github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
+github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
+github.com/tinylib/msgp v1.2.0 h1:0uKB/662twsVBpYUPbokj4sTSKhWFKB7LopO2kWK8lY=
+github.com/tinylib/msgp v1.2.0/go.mod h1:2vIGs3lcUo8izAATNobrCHevYZC/LMsJtw4JPiYPHro=
+github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
+golang.org/x/exp v0.0.0-20230725093048-515e97ebf090 h1:Di6/M8l0O2lCLc6VVRWhgCiApHV8MnQurBnFSHsQtNY=
+golang.org/x/exp v0.0.0-20230725093048-515e97ebf090/go.mod h1:FXUEEKJgO7OQYeo8N01OfiKP8RXMtf6e8aTskBGqWdc=
+golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
+golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
+golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
+golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20210220032951-036812b2e83c h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
+golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210301091718-77cc2087c03b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.4.0 h1:Zr2JFtRQNX3BCZ8YtxRE9hNJYC8J6I1MVbMg6owUp18=
+golang.org/x/sys v0.4.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0=
+golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+lab.nexedi.com/kirr/go123 v0.0.0-20240626173136-48920809d24c h1:GboU09uDjU09xORiCgCco5hPI4pmMOw6Qye2hJNal70=
+lab.nexedi.com/kirr/go123 v0.0.0-20240626173136-48920809d24c/go.mod h1:pwDpdCuvtz0QxisDzV/z9eUb9zc/rMQec520h4i8VWQ=
--- a/go/internal/xmaps/xmaps.go
+++ b/go/internal/xmaps/xmaps.go
+// Copyright (C) 2024  Nexedi SA and Contributors.
+//                     Kirill Smelkov <kirr@nexedi.com>
+//
+// This program is free software: you can Use, Study, Modify and Redistribute
+// it under the terms of the GNU General Public License version 3, or (at your
+// option) any later version, as published by the Free Software Foundation.
+//
+// You can also Link and Combine this program with other software covered by
+// the terms of any of the Free Software licenses or any of the Open Source
+// Initiative approved licenses and Convey the resulting work. Corresponding
+// source of such a combination shall include the source code for all other
+// software used.
+//
+// This program is distributed WITHOUT ANY WARRANTY; without even the implied
+// warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+//
+// See COPYING file for full licensing terms.
+// See https://www.nexedi.com/licensing for rationale and options.
+
+// Package xmaps complements standard package maps.
+package xmaps
+
+import (
+	"cmp"
+	"slices"
+)
+
+// SortedKeys returns all keys of map m in their ascending order.
+func SortedKeys[M ~map[K]V, K cmp.Ordered, V any](m M) []K {
+	keyv := make([]K, 0, len(m))
+	for k := range m {
+		keyv = append(keyv, k)
+	}
+	slices.Sort(keyv)
+	return keyv
+}
--- a/go/zodb/internal/pickletools/pickletools.go
+++ b/go/zodb/internal/pickletools/pickletools.go
-// Copyright (C) 2017-2019  Nexedi SA and Contributors.
-//                          Kirill Smelkov <kirr@nexedi.com>
+// Copyright (C) 2024  Nexedi SA and Contributors.
+//                     Kirill Smelkov <kirr@nexedi.com>
 //
 // This program is free software: you can Use, Study, Modify and Redistribute
 // it under the terms of the GNU General Public License version 3, or (at your
@@ -17,62 +17,30 @@
 // See COPYING file for full licensing terms.
 // See https://www.nexedi.com/licensing for rationale and options.

-// Package pickletools provides utilities related to python pickles.
-//
-// It complements package ogórek (github.com/kisielk/og-rek).
-package pickletools
+package xmaps

 import (
-	"encoding/binary"
-	"fmt"
-	"math/big"
-
-	pickle "github.com/kisielk/og-rek"
+	"slices"
+	"testing"
 )

-// Xint64 tries to convert unpickled value to int64.
-//
-// (ogórek decodes python long as big.Int)
-func Xint64(xv interface{}) (v int64, ok bool) {
-	switch v := xv.(type) {
-	case int64:
-		return v, true
-	case *big.Int:
-		if v.IsInt64() {
-			return v.Int64(), true
-		}
+func TestSortedKeys(t *testing.T) {
+	m := map[string]int{
+		"c":   1,
+		"b":   2,
+		"a":   3,
+		"zzz": 4,
+		"rr":  5,
+		"p":   6,
+		"j":   7,
 	}

-	return 0, false
-}
-
-// Xstrbytes verifies and extacts str|bytes from unpickled value.
-func Xstrbytes(x interface{}) (string, error) {
-	var s string
-	switch x := x.(type) {
-	default:
-		return "", fmt.Errorf("expect str|bytes; got %T", x)
+	kok := []string{"a", "b", "c", "j", "p", "rr", "zzz"}

-	case string:
-		s = x
-
-	case pickle.Bytes:
-		s = string(x)
-	}
-
-	return s, nil
-}
-
-// Xstrbytes8 verifies and extracts [8](str|bytes) from unpickled value as big-endian u64.
-func Xstrbytes8(x interface{}) (uint64, error) {
-	s, err := Xstrbytes(x)
-	if err != nil {
-		return 0, err
-	}
-
-	if len(s) != 8 {
-		return 0, fmt.Errorf("expect [8]bytes; got [%d]bytes", len(s))
+	for i := 0; i < 100; i++ {
+		keyv := SortedKeys(m)
+		if !slices.Equal(keyv, kok) {
+			t.Fatalf("SortedKeys:\nhave: %s\nwant: %s", keyv, kok)
+		}
 	}
-
-	return binary.BigEndian.Uint64([]byte(s)), nil
 }
--- a/go/internal/xtesting/xtesting.go
+++ b/go/internal/xtesting/xtesting.go
-// Copyright (C) 2017-2021  Nexedi SA and Contributors.
+// Copyright (C) 2017-2024  Nexedi SA and Contributors.
 //                          Kirill Smelkov <kirr@nexedi.com>
 //
 // This program is free software: you can Use, Study, Modify and Redistribute
@@ -39,9 +39,35 @@ import (
 	"lab.nexedi.com/kirr/neo/go/zodb"
 )

+// WithEachPy runs f for all Python versions we care about.
+//
+// For each version f is run separately under corresponding environment where
+// `python` on $PATH runs that python version.
+//
+// We currently support the following versions:
+//
+// - python2
+// - python3
+func WithEachPy(t *testing.T, f func(t *testing.T)) {
+	tmpd := t.TempDir()
+	for major := 2; major <= 3; major++ {
+		t.Run(fmt.Sprintf("py%d", major), func(t *testing.T) {
+			err := os.WriteFile(tmpd + "/python", []byte("#!/bin/sh\n" +
+					fmt.Sprintf("exec python%d \"$@\"\n", major)), 0777)
+			if err != nil {
+				t.Fatal(err)
+			}
+
+			t.Setenv("PATH", fmt.Sprintf("%s%c%s", tmpd, os.PathListSeparator, os.Getenv("PATH")))
+			f(t)
+		})
+	}
+}
+
+
 var (
-	pyMu   sync.Mutex
-	pyHave = map[string]bool{} // {} pymod -> y/n  ; ".python" indicates python presence
+	pyMu        sync.Mutex
+	pyHaveByVer = map[string]map[string]bool{} // {} pyver -> {} pymod -> y/n
 )

 // NeedPy skips current test if python and specified modules are not available.
@@ -59,24 +85,30 @@ func NeedPy(t testing.TB, modules ...string) {
 	pyMu.Lock()
 	defer pyMu.Unlock()

-	// verify if python is present
-	havePy, know := pyHave[".python"]
-	if !know {
-		cmd := exec.Command("python2", "-c", "0")
-		err := cmd.Run()
-		havePy = (err == nil)
-		pyHave[".python"] = havePy
+	// verify if python is present, retrieve its version
+	//
+	// Under different WithEachPy executions python might point to
+	// different python versions, so we cannot cache whether we have python
+	// or not. Retrieving `python --version` should be relatively fast - ~5ms -
+	// contrary to 100-200-300ms when running python code.
+	cmd := exec.Command("python", "--version")
+	out, err := cmd.CombinedOutput() // py2 emits version to stderr, py3 to stdout
+	if (err != nil) {
+		t.Skipf("skipping: python is not available: %s\n%s", err, out)
 	}
+	pyver := string(out) // e.g. "Python 2.7.18"

-	if !havePy {
-		t.Skipf("skipping: python is not availble")
+	pyHave, ok := pyHaveByVer[pyver]
+	if !ok {
+		pyHave = map[string]bool{}
+		pyHaveByVer[pyver] = pyHave
 	}

 	var donthave []string
 	for _, pymod := range modules {
 		have, know := pyHave[pymod]
 		if !know {
-			cmd := exec.Command("python2", "-c", "import "+pymod)
+			cmd := exec.Command("python", "-c", "import "+pymod)
 			err := cmd.Run()
 			have = (err == nil)
 			pyHave[pymod] = have
@@ -125,7 +157,7 @@ func ZPyCommitRaw(zurl string, at zodb.Tid, objv ...ZRawObject) (_ zodb.Tid, err
 	zin.WriteString("\n")

 	// run py `zodb commit`
-	cmd:= exec.Command("python2", "-m", "zodbtools.zodb", "commit", zurl, at.String())
+	cmd:= exec.Command("python", "-m", "zodbtools.zodb", "commit", zurl, at.String())
 	cmd.Stdin  = zin
 	cmd.Stderr = os.Stderr
 	out, err := cmd.Output()
@@ -149,7 +181,7 @@ func ZPyRestore(zurl string, zin string) (tidv []zodb.Tid, err error) {
 	defer xerr.Contextf(&err, "%s: zpyrestore", zurl)

 	// run py `zodb restore`
-	cmd:= exec.Command("python2", "-m", "zodbtools.zodb", "restore", zurl)
+	cmd:= exec.Command("python", "-m", "zodbtools.zodb", "restore", zurl)
 	cmd.Stdin  = strings.NewReader(zin)
 	cmd.Stderr = os.Stderr
 	out, err := cmd.Output()

--- a/go/internal/xtesting/xtesting_test.go
+++ b/go/internal/xtesting/xtesting_test.go
-// Copyright (C) 2020  Nexedi SA and Contributors.
-//                     Kirill Smelkov <kirr@nexedi.com>
+// Copyright (C) 2020-2024  Nexedi SA and Contributors.
+//                          Kirill Smelkov <kirr@nexedi.com>
 //
 // This program is free software: you can Use, Study, Modify and Redistribute
 // it under the terms of the GNU General Public License version 3, or (at your
@@ -29,7 +29,7 @@ import (

 func TestLoadDBHistory(t *testing.T) {
 	X := FatalIf(t)
-	data := "../../zodb/storage/fs1/testdata/1.fs"
+	data := "../../zodb/storage/fs1/testdata/py2_pickle3/1.fs" // one zkind is enough for hereby test

 	txnv, err := LoadDBHistory(data); X(err)


--- a/go/internal/xtesting/ztestdata.go
+++ b/go/internal/xtesting/ztestdata.go
+// Copyright (C) 2024  Nexedi SA and Contributors.
+//                     Kirill Smelkov <kirr@nexedi.com>
+//
+// This program is free software: you can Use, Study, Modify and Redistribute
+// it under the terms of the GNU General Public License version 3, or (at your
+// option) any later version, as published by the Free Software Foundation.
+//
+// You can also Link and Combine this program with other software covered by
+// the terms of any of the Free Software licenses or any of the Open Source
+// Initiative approved licenses and Convey the resulting work. Corresponding
+// source of such a combination shall include the source code for all other
+// software used.
+//
+// This program is distributed WITHOUT ANY WARRANTY; without even the implied
+// warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+//
+// See COPYING file for full licensing terms.
+// See https://www.nexedi.com/licensing for rationale and options.
+
+package xtesting
+// ZTestData + friends
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"testing"
+
+	"lab.nexedi.com/kirr/neo/go/internal/xmaps"
+)
+
+
+// ZTestData[T] represents one ZODB testdata case.
+//
+// Test files are located at .Prefix on the filesystem and were generated with
+// specified zkind (python major and pickle protocol versions). T represents
+// additional information about test data, for example which values to expect
+// when loading data from ZODB files.
+type ZTestData[T any] struct {
+	Kind   string // e.g. py2_pickle3
+	Prefix string // e.g. testdata/py2_pickle3/
+	Misc   *T     // better embed, but https://github.com/golang/go/issues/49030
+}
+
+
+// ZTestDataRegistry maintains registry of available ZTestData.
+//
+// Each testdata case needs to be registered via .Register method.
+type ZTestDataRegistry[T any] struct {
+	dir map[string/*zkind*/]*ZTestData[T]
+}
+
+// Register adds ZODB testdata case to the registry.
+//
+// ZODB test files are of specified zkind are located under prefix on the filesystem.
+// Additional information about testdata goes in misc.
+//
+// It is invalid to register the same zkind twice.
+func (r *ZTestDataRegistry[T]) Register(zkind, prefix string, misc *T) {
+	if r.dir == nil {
+		r.dir = make(map[string]*ZTestData[T])
+	}
+	_, already := r.dir[zkind]
+	if already {
+		panic(fmt.Sprintf("zkind %q already registered, zkind", zkind))
+	}
+	r.dir[zkind] = &ZTestData[T]{Kind: zkind, Prefix: prefix, Misc: misc}
+}
+
+// RunWithEach tests f with each registered ztestdata separately.
+func (r *ZTestDataRegistry[T]) RunWithEach(t *testing.T, f func(t *testing.T, ztestdata *ZTestData[T])) {
+	for _, zkind := range xmaps.SortedKeys(r.dir) {
+		ztestdata := r.dir[zkind]
+		t.Run(zkind, func(t *testing.T) {
+			f(t, ztestdata)
+		})
+	}
+}
+
+// BenchWithEach benchmarks f with each registered ztestdata separately.
+func (r *ZTestDataRegistry[T]) BenchWithEach(b *testing.B, f func(b *testing.B, ztestdata *ZTestData[T])) {
+	for _, zkind := range xmaps.SortedKeys(r.dir) {
+		ztestdata := r.dir[zkind]
+		b.Run(zkind, func(b *testing.B) {
+			f(b, ztestdata)
+		})
+	}
+}
+
+// Path returns filesystem path for a file located as path under ztestdata prefix.
+func (z *ZTestData[T]) Path(path string) string {
+	return fmt.Sprintf("%s/%s", z.Prefix, path)
+}
+
+
+// LoadZTestData loads ZODB testdata cases from the filesystem.
+//
+// Every subdirectory under dir counts as one testdata case.
+func LoadZTestData(dir string) ZTestDataRegistry[struct{}] {
+	ztestdataReg := ZTestDataRegistry[struct{}]{}
+	zprefixv, err := filepath.Glob(dir + "/[^.]*")
+	if err != nil {
+		panic(err) // the only possible error is ErrBadPattern
+	}
+	for _, zprefix := range zprefixv {
+		i, err := os.Stat(zprefix)
+		if err != nil {
+			continue // same behaviour as glob
+		}
+		if !i.IsDir() {
+			continue
+		}
+		zkind := filepath.Base(zprefix)
+		ztestdataReg.Register(zkind, zprefix, nil)
+	}
+	return ztestdataReg
+}
--- a/go/internal/zodbpickle/zodbpickle.go
+++ b/go/internal/zodbpickle/zodbpickle.go
+// Copyright (C) 2016-2024  Nexedi SA and Contributors.
+//                          Kirill Smelkov <kirr@nexedi.com>
+//
+// This program is free software: you can Use, Study, Modify and Redistribute
+// it under the terms of the GNU General Public License version 3, or (at your
+// option) any later version, as published by the Free Software Foundation.
+//
+// You can also Link and Combine this program with other software covered by
+// the terms of any of the Free Software licenses or any of the Open Source
+// Initiative approved licenses and Convey the resulting work. Corresponding
+// source of such a combination shall include the source code for all other
+// software used.
+//
+// This program is distributed WITHOUT ANY WARRANTY; without even the implied
+// warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+//
+// See COPYING file for full licensing terms.
+// See https://www.nexedi.com/licensing for rationale and options.
+
+// Package zodbpickle provides pickling facility tailored to ZODB/go needs.
+//
+// Contrary to zodbpickle/py it merely wraps [github.com/kisielk/og-rek] without
+// forking code from anywhere.
+//
+// Use [NewPickler]/[NewUnpickler] to create pickle encoder and decoder
+// correspondingly. The decoder uses StrictUnicode=y mode to prevent loading
+// *STRING and *UNICODE opcodes as the same type "string" on Go side and thus
+// loosing information. The encoder uses StrictUnicode=y mode as well for
+// symmetry so that decode/encode cycle is identity. As the result
+//
+//   - py bytestring (str from py2) is represented as Go type ogórek.ByteString;
+//   - py unicode (unicode from py2 and str from py3) is represented as Go type string;
+//   - py bytes (bytes from py3 and zodbpickle.binary from py2) is represented as Go type ogórek.Bytes.
+//
+// At application level utilities like ogórek.AsBytes and ogórek.AsString are
+// handy to work with unpickled data for pickles generated by either py2 or py3.
+//
+// The decoder also uses PyDict=y mode to generally adhere to Python semantic
+// and to allow programs to access unpickled dictionaries via string/bytes keys
+// following py3 model, while still being able to handle dictionaries
+// generated from under py2.
+//
+// The encoder emits pickles with protocol=3 to natively support all py bytes
+// and strings types, and to stay interoperable with both py2 and py3: both ZODB4
+// and ZODB5 use zodbpickle which supports protocol=3 on py2 and py3, and ZODB5
+// actually saves pickles with protocol=3 on both py2 and py3 starting from
+// ZODB 5.4 . Pickles saved with protocol=3 are thus universally readable.
+//
+// See package [github.com/kisielk/og-rek] for details of pickling/unpickling on Go side.
+package zodbpickle
+
+import (
+	"encoding/binary"
+	"fmt"
+	"io"
+
+	pickle "github.com/kisielk/og-rek"
+)
+
+// NewPickler creates new pickle encoder.
+//
+// The encoder adheres to semantic explained in top-level documentation of
+// zodbpickle package.
+//
+// getref, if !nil, will be used by the encoder to see if a referenced object
+// should be represented by persistent reference in generated pickle stream.
+// See documentation of ogórek.EncoderConfig.PersistentRef for details.
+func NewPickler(w io.Writer, getref func(obj any) *pickle.Ref) *pickle.Encoder {
+	return pickle.NewEncoderWithConfig(w, &pickle.EncoderConfig{
+		Protocol:      3,    // see top-level doc
+		StrictUnicode: true, // see top-level doc
+		PersistentRef: getref,
+	})
+}
+
+// NewUnpickler creates new pickle decoder.
+//
+// The decoder adheres to semantic explained in top-level documentation of
+// zodbpickle package.
+//
+// loadref, if !nil, will be used by the decoder to handle persistent
+// references. See documentation of ogórek.DecoderConfig.PersistentLoad for details.
+func NewUnpickler(r io.Reader, loadref func(ref pickle.Ref) (any, error)) *pickle.Decoder {
+	return pickle.NewDecoderWithConfig(r, &pickle.DecoderConfig{
+		StrictUnicode:  true, // see top-level doc
+		PyDict:         true, // see top-level doc
+		PersistentLoad: loadref,
+	})
+}
+
+
+// ---- pack/unpack uint64 into [8]bytes ----
+
+// Pack64 packs uint64 into big-endian bytes.
+func Pack64(x uint64) pickle.Bytes {
+	var b [8]byte
+	binary.BigEndian.PutUint64(b[:], x)
+	return pickle.Bytes(b[:])
+}
+
+// Unpack64 tries to extract [8](bytestr|bytes) from unpickled value as big-endian uint64.
+func Unpack64(x interface{}) (uint64, error) {
+	s, err := pickle.AsBytes(x)
+	if err != nil {
+		return 0, err
+	}
+
+	if len(s) != 8 {
+		return 0, fmt.Errorf("expect [8]bytes; got [%d]bytes", len(s))
+	}
+
+	return binary.BigEndian.Uint64([]byte(s)), nil
+}
--- a/go/zodb/btree/btree.go
+++ b/go/zodb/btree/btree.go
-// Copyright (C) 2018-2020  Nexedi SA and Contributors.
+// Copyright (C) 2018-2024  Nexedi SA and Contributors.
 //                          Kirill Smelkov <kirr@nexedi.com>

 // Package btree provides B⁺ Trees for ZODB.
@@ -45,7 +45,7 @@ import (
 	"reflect"

 	"lab.nexedi.com/kirr/neo/go/zodb"
-	"lab.nexedi.com/kirr/neo/go/zodb/internal/pickletools"
+	pickle "github.com/kisielk/og-rek"
 )

 // Length is equivalent of BTrees.Length.Length in BTree/py.
@@ -69,9 +69,9 @@ func (l *lengthState) PyGetState() interface{} {

 // PySetState implements zodb.PyStateful.
 func (l *lengthState) PySetState(pystate interface{}) (err error) {
-	v, ok := pickletools.Xint64(pystate)
-	if !ok {
-		return fmt.Errorf("state must be int; got %T", pystate)
+	v, err := pickle.AsInt64(pystate)
+	if err != nil {
+		return fmt.Errorf("state: %s", err)
 	}

 	l.value = int(v)

--- a/go/zodb/btree/btree.go.in
+++ b/go/zodb/btree/btree.go.in
 // Copyright (c) 2001, 2002 Zope Foundation and Contributors.
 // All Rights Reserved.
 //
-// Copyright (C) 2018-2021  Nexedi SA and Contributors.
+// Copyright (C) 2018-2024  Nexedi SA and Contributors.
 //                          Kirill Smelkov <kirr@nexedi.com>
 //
 // This software is subject to the provisions of the Zope Public License,
@@ -26,7 +26,6 @@ import (
 	pickle "github.com/kisielk/og-rek"
 	"lab.nexedi.com/kirr/go123/xerr"
 	"lab.nexedi.com/kirr/neo/go/zodb"
-	"lab.nexedi.com/kirr/neo/go/zodb/internal/pickletools"
 )

 // Node represents a tree node - either BTree or Bucket.
@@ -493,9 +492,9 @@ func (b *bucketState) PySetState(pystate interface{}) (err error) {
 		xk := t[2*i]
 		v := t[2*i+1]

-		k, ok := pickletools.Xint64(xk)
-		if !ok {
-			return fmt.Errorf("data: [%d]: key must be integer; got %T", i, xk)
+		k, err := pickle.AsInt64(xk)
+		if err != nil {
+			return fmt.Errorf("data: [%d]: key: %s", i, err)
 		}

 		kk := KEY(k)

--- a/go/zodb/btree/btree_test.go
+++ b/go/zodb/btree/btree_test.go
-// Copyright (C) 2018-2021  Nexedi SA and Contributors.
+// Copyright (C) 2018-2024  Nexedi SA and Contributors.
 //                          Kirill Smelkov <kirr@nexedi.com>
 //
 // This program is free software: you can Use, Study, Modify and Redistribute
@@ -19,7 +19,8 @@

 package btree

-//go:generate ./testdata/gen-testdata
+//go:generate python2 testdata/gen-testdata
+//go:generate python3 testdata/gen-testdata

 import (
 	"context"
@@ -28,11 +29,31 @@ import (

 	"lab.nexedi.com/kirr/go123/exc"
 	"lab.nexedi.com/kirr/go123/xerr"
+	"lab.nexedi.com/kirr/neo/go/internal/xtesting"
 	"lab.nexedi.com/kirr/neo/go/transaction"
 	"lab.nexedi.com/kirr/neo/go/zodb"
 	_ "lab.nexedi.com/kirr/neo/go/zodb/wks"
+
+	pickle "github.com/kisielk/og-rek"
 )

+// ztestdataReg maintains registry of all entries under testdata/ .
+var  ztestdataReg = xtesting.ZTestDataRegistry[_TestDataOK]{}
+type ZTestData    = xtesting.ZTestData[_TestDataOK]
+
+// _TestDataOK describes expected data for one entry under testdata/ .
+type _TestDataOK struct {
+	smallTestv []testEntry // which data is stored in small b0-b2 and B0-B2 nodes
+
+	B3_oid    zodb.Oid // information about large B3 tree
+	B3_maxkey int64
+
+	Bv_oid  zodb.Oid // information about Bv tree, used to verify visit callbacks
+	Bv_kmin int64
+	Bv_kmax int64
+	Bvdict  map[int64][]tVisit
+}
+
 // kv is one (key, value) pair.
 type kv struct {
 	key   int64
@@ -106,9 +127,14 @@ type tVisit struct {
 }

 func TestBTree(t *testing.T) {
+	ztestdataReg.RunWithEach(t, _TestBTree)
+}
+func _TestBTree(t *testing.T, z *ZTestData) {
+	zz := z.Misc
+
 	X := exc.Raiseif
 	ctx := context.Background()
-	stor, err := zodb.Open(ctx, "testdata/1.fs", &zodb.OpenOptions{ReadOnly: true})
+	stor, err := zodb.Open(ctx, z.Path("1.fs"), &zodb.OpenOptions{ReadOnly: true})
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -130,7 +156,7 @@ func TestBTree(t *testing.T) {
 	}

 	// go through small test Buckets/BTrees and verify that Get(key) is as expected.
-	for _, tt := range smallTestv {
+	for _, tt := range zz.smallTestv {
 		xobj, err := conn.Get(ctx, tt.oid)
 		if err != nil {
 			t.Fatal(err)
@@ -168,12 +194,12 @@ func TestBTree(t *testing.T) {
 			}

 			if !ok {
-				t.Errorf("%s: get %v -> ø;  want %v", tt.oid, kv.key, kv.value)
+				t.Errorf("%s: get %v -> ø;  want %#v", tt.oid, kv.key, kv.value)
 				continue
 			}

 			if value != kv.value {
-				t.Errorf("%s: get %v -> %v;  want %v", tt.oid, kv.key, value, kv.value)
+				t.Errorf("%s: get %v -> %#v;  want %#v", tt.oid, kv.key, value, kv.value)
 			}

 			// XXX .next == nil
@@ -205,16 +231,16 @@ func TestBTree(t *testing.T) {

 	// B3 is a large BTree with {i: i} data.
 	// verify Get(key), {Min,Max}Key and that different bucket links lead to the same in-RAM object.
-	xB3, err := conn.Get(ctx, B3_oid)
+	xB3, err := conn.Get(ctx, zz.B3_oid)
 	if err != nil {
 		t.Fatal(err)
 	}
 	B3, ok := xB3.(*LOBTree)
 	if !ok {
-		t.Fatalf("B3: %v; got %T;  want LOBTree", B3_oid, xB3)
+		t.Fatalf("B3: %v; got %T;  want LOBTree", zz.B3_oid, xB3)
 	}

-	for i := int64(0); i <= B3_maxkey; i++ {
+	for i := int64(0); i <= zz.B3_maxkey; i++ {
 		v, ok, err := B3.Get(ctx, i)
 		if err != nil {
 			t.Fatal(err)
@@ -232,9 +258,9 @@ func TestBTree(t *testing.T) {
 	if err := xerr.Merge(emin, emax); err != nil {
 		t.Fatalf("B3: min/max key: %s", err)
 	}
-	if !(kmin == 0 && kmax == B3_maxkey && okmin && okmax) {
+	if !(kmin == 0 && kmax == zz.B3_maxkey && okmin && okmax) {
 		t.Fatalf("B3: min/max key wrong: got [%v, %v] (%v, %v);  want [%v, %v] (%v, %v)",
-			kmin, kmax, okmin, okmax, 0, B3_maxkey, true, true)
+			kmin, kmax, okmin, okmax, 0, zz.B3_maxkey, true, true)
 	}

 	// verifyFirstBucket verifies that b.firstbucket is correct and returns it.
@@ -272,13 +298,13 @@ func TestBTree(t *testing.T) {
 	verifyFirstBucket(B3)

 	// verify nodes/keycov visited through VGet/V{Min,Max}Key
-	xBv, err := conn.Get(ctx, Bv_oid);  X(err)
+	xBv, err := conn.Get(ctx, zz.Bv_oid);  X(err)
 	Bv, ok := xBv.(*LOBTree)
 	if !ok {
-		t.Fatalf("Bv: %v; got %T;  want LOBTree", Bv_oid, xBv)
+		t.Fatalf("Bv: %v; got %T;  want LOBTree", zz.Bv_oid, xBv)
 	}

-	for k, visitOK := range Bvdict {
+	for k, visitOK := range zz.Bvdict {
 		visit := []tVisit{}
 		_, _, err := Bv.VGet(ctx, k, func(node LONode, keycov LKeyRange) {
 			visit = append(visit, tVisit{node.POid(), keycov})
@@ -288,8 +314,8 @@ func TestBTree(t *testing.T) {
 		}
 	}

-	visitMinOK := Bvdict[Bv_kmin]
-	visitMaxOK := Bvdict[Bv_kmax]
+	visitMinOK := zz.Bvdict[zz.Bv_kmin]
+	visitMaxOK := zz.Bvdict[zz.Bv_kmax]
 	visitMin := []tVisit{}
 	visitMax := []tVisit{}
 	_, _, err = Bv.VMinKey(ctx, func(node LONode, keycov LKeyRange) {
@@ -306,3 +332,9 @@ func TestBTree(t *testing.T) {
 		t.Errorf("VMaxKey(): visit:\nhave: %v\nwant: %v", visitMax, visitMaxOK)
 	}
 }
+
+
+// ---- misc ----
+
+// ztestdata_py2_* use bstr
+type bstr = pickle.ByteString
--- a/go/zodb/btree/testdata/.gitignore
+++ b/go/zodb/btree/testdata/.gitignore
-/*.lock
-/*.tmp
-/*.tr[0-9]
+*.lock
+*.tmp
+*.tr[0-9]
--- a/go/zodb/btree/testdata/gen-testdata
+++ b/go/zodb/btree/testdata/gen-testdata
-#!/usr/bin/env python2
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
-# Copyright (C) 2018-2021  Nexedi SA and Contributors.
+# Copyright (C) 2018-2024  Nexedi SA and Contributors.
 #                          Kirill Smelkov <kirr@nexedi.com>
 #
 # This program is free software: you can Use, Study, Modify and Redistribute
@@ -20,25 +20,27 @@
 # See https://www.nexedi.com/licensing for rationale and options.
 """generate test data for btree serialization tests"""

+from __future__ import print_function
+
 from ZODB.DB import DB
 from BTrees.LOBTree import LOBucket, LOBTree
 from BTrees.check import check as bcheck
 from ZODB.utils import u64
-from zodbtools.test.gen_testdata import run_with_zodb4py2_compat
-import os, os.path, transaction
+from zodbtools.test.gen_testdata import run_with_all_zodb_pickle_kinds, current_zkind
+import os, os.path, shutil, transaction

 from golang.gcompat import qq

-def rm_f(path):
-    if os.path.exists(path):
-        os.remove(path)
-
 def main2():
    import zodbtools.test.gen_testdata  # to make time predictable (XXX)

-    outfs = "testdata/1.fs"
-    rm_f(outfs)
-    rm_f(outfs + ".index")
+    zkind = current_zkind()
+    prefix = "testdata/%s" % zkind
+    if os.path.exists(prefix):
+        shutil.rmtree(prefix)
+    os.makedirs(prefix)
+
+    outfs = "%s/1.fs" % prefix
    db = DB(outfs)
    conn = db.open()
    root = conn.root()
@@ -79,18 +81,23 @@ def main2():
    assert Bv[9] == "e"


-    with open("ztestdata_expect_test.go", "w") as f:
+    with open("ztestdata_expect_%s_test.go" % zkind, "w") as f:
        def emit(v):
-            print >>f, v
-        emit("// Code generated by %s; DO NOT EDIT." % __file__)
+            print(v, file=f)
+        emit("// Code generated by %s; DO NOT EDIT." % os.path.relpath(__file__))
        emit("package btree\n")

+        emit("func init() {")
+        emit("\ttestdataok := _TestDataOK{")
+
        def emititems(b):
            s = "testEntry{oid: %s, kind: %s, itemv: []kv{" \
                    % (u64(b._p_oid), "kind%s" % type(b).__name__[2:])
            for k, v in b.items():
                if isinstance(v, str):
                    v = qq(v)
+                    if str is bytes:
+                        v = "bstr(%s)" % v
                elif isinstance(v, int):
                    v = "int64(%d)" % v
                else:
@@ -98,20 +105,20 @@ def main2():

                s += "{%s, %s}, " % (k, v)
            s += "}},"
-            emit("\t"+s)
+            emit("\t\t\t"+s)

-        emit("\nvar smallTestv = [...]testEntry{")
+        emit("\t\tsmallTestv: []testEntry{")
        for b in (b0, b1, b2, B0, B1, B2):
            emititems(b)
-        emit("}")
+        emit("\t\t},")

-        emit("\nconst B3_oid = %s" % u64(B3._p_oid))
-        emit("const B3_maxkey = %d" % B3.maxKey())
+        emit("\n\t\tB3_oid: %s,"  % u64(B3._p_oid))
+        emit("\t\tB3_maxkey: %d," % B3.maxKey())

-        emit("\nconst Bv_oid = %s" % u64(Bv._p_oid))
-        emit("const Bv_kmin = %d"  % Bv.minKey())
-        emit("const Bv_kmax = %d"  % Bv.maxKey())
-        emit("var Bvdict = map[int64][]tVisit{")
+        emit("\n\t\tBv_oid: %s," % u64(Bv._p_oid))
+        emit("\t\tBv_kmin: %d,"  % Bv.minKey())
+        emit("\t\tBv_kmax: %d,"  % Bv.maxKey())
+        emit("\t\tBvdict: map[int64][]tVisit{")
        noo = "_LKeyMin"
        oo  = "_LKeyMax"
        def emitVisit(key, *visitv): # visitv = [](node, lo,hi)
@@ -122,20 +129,24 @@ def main2():
                else:
                    hi_ = hi-1
                vstr.append("{%d, LKeyRange{%s, %s}}" % (u64(node._p_oid), lo, hi_))
-            emit("\t%d: []tVisit{%s}," % (key, ", ".join(vstr)))
+            emit("\t\t\t%d: []tVisit{%s}," % (key, ", ".join(vstr)))

        emitVisit(1, (T4, noo,oo), (T2, noo,4), (v1, noo,2))
        emitVisit(2, (T4, noo,oo), (T2, noo,4), (v2, 2,4))
        emitVisit(5, (T4, noo,oo), (T,  4,oo),  (T79, 4,oo), (v5, 4,7))
        emitVisit(8, (T4, noo,oo), (T,  4,oo),  (T79, 4,oo), (v8, 7,9))
        emitVisit(9, (T4, noo,oo), (T,  4,oo),  (T79, 4,oo), (v9, 9,oo))
+        emit("\t\t},")
+        emit("\t}")
+
+        emit("\n\tztestdataReg.Register(%s, %s, &testdataok)" % (qq(zkind), qq(prefix)))
        emit("}")

    conn.close()
    db.close()

 def main():
-    run_with_zodb4py2_compat(main2)
+    run_with_all_zodb_pickle_kinds(main2)


 if __name__ == '__main__':

--- a/go/zodb/btree/testdata/py2_pickle1/1.fs
+++ b/go/zodb/btree/testdata/py2_pickle1/1.fs
--- a/go/zodb/btree/testdata/py2_pickle1/1.fs.index
+++ b/go/zodb/btree/testdata/py2_pickle1/1.fs.index
--- a/go/zodb/btree/testdata/1.fs
+++ b/go/zodb/btree/testdata/1.fs
--- a/go/zodb/btree/testdata/1.fs.index
+++ b/go/zodb/btree/testdata/1.fs.index
--- a/go/zodb/btree/testdata/py2_pickle3/1.fs
+++ b/go/zodb/btree/testdata/py2_pickle3/1.fs
--- a/go/zodb/btree/testdata/py2_pickle3/1.fs.index
+++ b/go/zodb/btree/testdata/py2_pickle3/1.fs.index
--- a/go/zodb/btree/testdata/py3_pickle3/1.fs
+++ b/go/zodb/btree/testdata/py3_pickle3/1.fs
--- a/go/zodb/btree/testdata/py3_pickle3/1.fs.index
+++ b/go/zodb/btree/testdata/py3_pickle3/1.fs.index
--- a/go/zodb/btree/ziobtree.go
+++ b/go/zodb/btree/ziobtree.go
@@ -3,7 +3,7 @@
 // Copyright (c) 2001, 2002 Zope Foundation and Contributors.
 // All Rights Reserved.
 //
-// Copyright (C) 2018-2021  Nexedi SA and Contributors.
+// Copyright (C) 2018-2024  Nexedi SA and Contributors.
 //                          Kirill Smelkov <kirr@nexedi.com>
 //
 // This software is subject to the provisions of the Zope Public License,
@@ -28,7 +28,6 @@ import (
 	pickle "github.com/kisielk/og-rek"
 	"lab.nexedi.com/kirr/go123/xerr"
 	"lab.nexedi.com/kirr/neo/go/zodb"
-	"lab.nexedi.com/kirr/neo/go/zodb/internal/pickletools"
 )

 // IONode represents a tree node - either IOBTree or IOBucket.
@@ -495,9 +494,9 @@ func (b *iobucketState) PySetState(pystate interface{}) (err error) {
 		xk := t[2*i]
 		v := t[2*i+1]

-		k, ok := pickletools.Xint64(xk)
-		if !ok {
-			return fmt.Errorf("data: [%d]: key must be integer; got %T", i, xk)
+		k, err := pickle.AsInt64(xk)
+		if err != nil {
+			return fmt.Errorf("data: [%d]: key: %s", i, err)
 		}

 		kk := int32(k)

--- a/go/zodb/btree/zlobtree.go
+++ b/go/zodb/btree/zlobtree.go
@@ -3,7 +3,7 @@
 // Copyright (c) 2001, 2002 Zope Foundation and Contributors.
 // All Rights Reserved.
 //
-// Copyright (C) 2018-2021  Nexedi SA and Contributors.
+// Copyright (C) 2018-2024  Nexedi SA and Contributors.
 //                          Kirill Smelkov <kirr@nexedi.com>
 //
 // This software is subject to the provisions of the Zope Public License,
@@ -28,7 +28,6 @@ import (
 	pickle "github.com/kisielk/og-rek"
 	"lab.nexedi.com/kirr/go123/xerr"
 	"lab.nexedi.com/kirr/neo/go/zodb"
-	"lab.nexedi.com/kirr/neo/go/zodb/internal/pickletools"
 )

 // LONode represents a tree node - either LOBTree or LOBucket.
@@ -495,9 +494,9 @@ func (b *lobucketState) PySetState(pystate interface{}) (err error) {
 		xk := t[2*i]
 		v := t[2*i+1]

-		k, ok := pickletools.Xint64(xk)
-		if !ok {
-			return fmt.Errorf("data: [%d]: key must be integer; got %T", i, xk)
+		k, err := pickle.AsInt64(xk)
+		if err != nil {
+			return fmt.Errorf("data: [%d]: key: %s", i, err)
 		}

 		kk := int64(k)

--- a/go/zodb/btree/ztestdata_expect_py2_pickle1_test.go
+++ b/go/zodb/btree/ztestdata_expect_py2_pickle1_test.go
+// Code generated by testdata/gen-testdata; DO NOT EDIT.
+package btree
+
+func init() {
+	testdataok := _TestDataOK{
+		smallTestv: []testEntry{
+			testEntry{oid: 7, kind: kindBucket, itemv: []kv{}},
+			testEntry{oid: 4, kind: kindBucket, itemv: []kv{{10, int64(17)}, }},
+			testEntry{oid: 1, kind: kindBucket, itemv: []kv{{15, int64(1)}, {23, bstr("hello")}, }},
+			testEntry{oid: 3, kind: kindBTree, itemv: []kv{}},
+			testEntry{oid: 8, kind: kindBTree, itemv: []kv{{5, int64(4)}, }},
+			testEntry{oid: 5, kind: kindBTree, itemv: []kv{{7, int64(3)}, {9, bstr("world")}, }},
+		},
+
+		B3_oid: 6,
+		B3_maxkey: 9999,
+
+		Bv_oid: 2,
+		Bv_kmin: 1,
+		Bv_kmax: 9,
+		Bvdict: map[int64][]tVisit{
+			1: []tVisit{{2, LKeyRange{_LKeyMin, _LKeyMax}}, {342, LKeyRange{_LKeyMin, 3}}, {344, LKeyRange{_LKeyMin, 1}}},
+			2: []tVisit{{2, LKeyRange{_LKeyMin, _LKeyMax}}, {342, LKeyRange{_LKeyMin, 3}}, {349, LKeyRange{2, 3}}},
+			5: []tVisit{{2, LKeyRange{_LKeyMin, _LKeyMax}}, {343, LKeyRange{4, _LKeyMax}}, {345, LKeyRange{4, _LKeyMax}}, {346, LKeyRange{4, 6}}},
+			8: []tVisit{{2, LKeyRange{_LKeyMin, _LKeyMax}}, {343, LKeyRange{4, _LKeyMax}}, {345, LKeyRange{4, _LKeyMax}}, {347, LKeyRange{7, 8}}},
+			9: []tVisit{{2, LKeyRange{_LKeyMin, _LKeyMax}}, {343, LKeyRange{4, _LKeyMax}}, {345, LKeyRange{4, _LKeyMax}}, {348, LKeyRange{9, _LKeyMax}}},
+		},
+	}
+
+	ztestdataReg.Register("py2_pickle1", "testdata/py2_pickle1", &testdataok)
+}
--- a/go/zodb/btree/ztestdata_expect_py2_pickle2_test.go
+++ b/go/zodb/btree/ztestdata_expect_py2_pickle2_test.go
+// Code generated by testdata/gen-testdata; DO NOT EDIT.
+package btree
+
+func init() {
+	testdataok := _TestDataOK{
+		smallTestv: []testEntry{
+			testEntry{oid: 7, kind: kindBucket, itemv: []kv{}},
+			testEntry{oid: 4, kind: kindBucket, itemv: []kv{{10, int64(17)}, }},
+			testEntry{oid: 1, kind: kindBucket, itemv: []kv{{15, int64(1)}, {23, bstr("hello")}, }},
+			testEntry{oid: 3, kind: kindBTree, itemv: []kv{}},
+			testEntry{oid: 8, kind: kindBTree, itemv: []kv{{5, int64(4)}, }},
+			testEntry{oid: 5, kind: kindBTree, itemv: []kv{{7, int64(3)}, {9, bstr("world")}, }},
+		},
+
+		B3_oid: 6,
+		B3_maxkey: 9999,
+
+		Bv_oid: 2,
+		Bv_kmin: 1,
+		Bv_kmax: 9,
+		Bvdict: map[int64][]tVisit{
+			1: []tVisit{{2, LKeyRange{_LKeyMin, _LKeyMax}}, {342, LKeyRange{_LKeyMin, 3}}, {344, LKeyRange{_LKeyMin, 1}}},
+			2: []tVisit{{2, LKeyRange{_LKeyMin, _LKeyMax}}, {342, LKeyRange{_LKeyMin, 3}}, {349, LKeyRange{2, 3}}},
+			5: []tVisit{{2, LKeyRange{_LKeyMin, _LKeyMax}}, {343, LKeyRange{4, _LKeyMax}}, {345, LKeyRange{4, _LKeyMax}}, {346, LKeyRange{4, 6}}},
+			8: []tVisit{{2, LKeyRange{_LKeyMin, _LKeyMax}}, {343, LKeyRange{4, _LKeyMax}}, {345, LKeyRange{4, _LKeyMax}}, {347, LKeyRange{7, 8}}},
+			9: []tVisit{{2, LKeyRange{_LKeyMin, _LKeyMax}}, {343, LKeyRange{4, _LKeyMax}}, {345, LKeyRange{4, _LKeyMax}}, {348, LKeyRange{9, _LKeyMax}}},
+		},
+	}
+
+	ztestdataReg.Register("py2_pickle2", "testdata/py2_pickle2", &testdataok)
+}
--- a/go/zodb/btree/ztestdata_expect_py2_pickle3_test.go
+++ b/go/zodb/btree/ztestdata_expect_py2_pickle3_test.go
+// Code generated by testdata/gen-testdata; DO NOT EDIT.
+package btree
+
+func init() {
+	testdataok := _TestDataOK{
+		smallTestv: []testEntry{
+			testEntry{oid: 7, kind: kindBucket, itemv: []kv{}},
+			testEntry{oid: 4, kind: kindBucket, itemv: []kv{{10, int64(17)}, }},
+			testEntry{oid: 1, kind: kindBucket, itemv: []kv{{15, int64(1)}, {23, bstr("hello")}, }},
+			testEntry{oid: 3, kind: kindBTree, itemv: []kv{}},
+			testEntry{oid: 8, kind: kindBTree, itemv: []kv{{5, int64(4)}, }},
+			testEntry{oid: 5, kind: kindBTree, itemv: []kv{{7, int64(3)}, {9, bstr("world")}, }},
+		},
+
+		B3_oid: 6,
+		B3_maxkey: 9999,
+
+		Bv_oid: 2,
+		Bv_kmin: 1,
+		Bv_kmax: 9,
+		Bvdict: map[int64][]tVisit{
+			1: []tVisit{{2, LKeyRange{_LKeyMin, _LKeyMax}}, {342, LKeyRange{_LKeyMin, 3}}, {344, LKeyRange{_LKeyMin, 1}}},
+			2: []tVisit{{2, LKeyRange{_LKeyMin, _LKeyMax}}, {342, LKeyRange{_LKeyMin, 3}}, {349, LKeyRange{2, 3}}},
+			5: []tVisit{{2, LKeyRange{_LKeyMin, _LKeyMax}}, {343, LKeyRange{4, _LKeyMax}}, {345, LKeyRange{4, _LKeyMax}}, {346, LKeyRange{4, 6}}},
+			8: []tVisit{{2, LKeyRange{_LKeyMin, _LKeyMax}}, {343, LKeyRange{4, _LKeyMax}}, {345, LKeyRange{4, _LKeyMax}}, {347, LKeyRange{7, 8}}},
+			9: []tVisit{{2, LKeyRange{_LKeyMin, _LKeyMax}}, {343, LKeyRange{4, _LKeyMax}}, {345, LKeyRange{4, _LKeyMax}}, {348, LKeyRange{9, _LKeyMax}}},
+		},
+	}
+
+	ztestdataReg.Register("py2_pickle3", "testdata/py2_pickle3", &testdataok)
+}
--- a/go/zodb/btree/ztestdata_expect_py3_pickle3_test.go
+++ b/go/zodb/btree/ztestdata_expect_py3_pickle3_test.go
+// Code generated by testdata/gen-testdata; DO NOT EDIT.
+package btree
+
+func init() {
+	testdataok := _TestDataOK{
+		smallTestv: []testEntry{
+			testEntry{oid: 1, kind: kindBucket, itemv: []kv{}},
+			testEntry{oid: 2, kind: kindBucket, itemv: []kv{{10, int64(17)}, }},
+			testEntry{oid: 3, kind: kindBucket, itemv: []kv{{15, int64(1)}, {23, "hello"}, }},
+			testEntry{oid: 4, kind: kindBTree, itemv: []kv{}},
+			testEntry{oid: 5, kind: kindBTree, itemv: []kv{{5, int64(4)}, }},
+			testEntry{oid: 6, kind: kindBTree, itemv: []kv{{7, int64(3)}, {9, "world"}, }},
+		},
+
+		B3_oid: 7,
+		B3_maxkey: 9999,
+
+		Bv_oid: 8,
+		Bv_kmin: 1,
+		Bv_kmax: 9,
+		Bvdict: map[int64][]tVisit{
+			1: []tVisit{{8, LKeyRange{_LKeyMin, _LKeyMax}}, {9, LKeyRange{_LKeyMin, 3}}, {11, LKeyRange{_LKeyMin, 1}}},
+			2: []tVisit{{8, LKeyRange{_LKeyMin, _LKeyMax}}, {9, LKeyRange{_LKeyMin, 3}}, {16, LKeyRange{2, 3}}},
+			5: []tVisit{{8, LKeyRange{_LKeyMin, _LKeyMax}}, {10, LKeyRange{4, _LKeyMax}}, {12, LKeyRange{4, _LKeyMax}}, {13, LKeyRange{4, 6}}},
+			8: []tVisit{{8, LKeyRange{_LKeyMin, _LKeyMax}}, {10, LKeyRange{4, _LKeyMax}}, {12, LKeyRange{4, _LKeyMax}}, {14, LKeyRange{7, 8}}},
+			9: []tVisit{{8, LKeyRange{_LKeyMin, _LKeyMax}}, {10, LKeyRange{4, _LKeyMax}}, {12, LKeyRange{4, _LKeyMax}}, {15, LKeyRange{9, _LKeyMax}}},
+		},
+	}
+
+	ztestdataReg.Register("py3_pickle3", "testdata/py3_pickle3", &testdataok)
+}
--- a/go/zodb/btree/ztestdata_expect_test.go
+++ b/go/zodb/btree/ztestdata_expect_test.go
-// Code generated by ./testdata/gen-testdata; DO NOT EDIT.
-package btree
-
-
-var smallTestv = [...]testEntry{
-	testEntry{oid: 7, kind: kindBucket, itemv: []kv{}},
-	testEntry{oid: 4, kind: kindBucket, itemv: []kv{{10, int64(17)}, }},
-	testEntry{oid: 1, kind: kindBucket, itemv: []kv{{15, int64(1)}, {23, "hello"}, }},
-	testEntry{oid: 3, kind: kindBTree, itemv: []kv{}},
-	testEntry{oid: 8, kind: kindBTree, itemv: []kv{{5, int64(4)}, }},
-	testEntry{oid: 5, kind: kindBTree, itemv: []kv{{7, int64(3)}, {9, "world"}, }},
-}
-
-const B3_oid = 6
-const B3_maxkey = 9999
-
-const Bv_oid = 2
-const Bv_kmin = 1
-const Bv_kmax = 9
-var Bvdict = map[int64][]tVisit{
-	1: []tVisit{{2, LKeyRange{_LKeyMin, _LKeyMax}}, {342, LKeyRange{_LKeyMin, 3}}, {344, LKeyRange{_LKeyMin, 1}}},
-	2: []tVisit{{2, LKeyRange{_LKeyMin, _LKeyMax}}, {342, LKeyRange{_LKeyMin, 3}}, {349, LKeyRange{2, 3}}},
-	5: []tVisit{{2, LKeyRange{_LKeyMin, _LKeyMax}}, {343, LKeyRange{4, _LKeyMax}}, {345, LKeyRange{4, _LKeyMax}}, {346, LKeyRange{4, 6}}},
-	8: []tVisit{{2, LKeyRange{_LKeyMin, _LKeyMax}}, {343, LKeyRange{4, _LKeyMax}}, {345, LKeyRange{4, _LKeyMax}}, {347, LKeyRange{7, 8}}},
-	9: []tVisit{{2, LKeyRange{_LKeyMin, _LKeyMax}}, {343, LKeyRange{4, _LKeyMax}}, {345, LKeyRange{4, _LKeyMax}}, {348, LKeyRange{9, _LKeyMax}}},
-}
--- a/go/zodb/impexp_test.go
+++ b/go/zodb/impexp_test.go
-// Copyright (C) 2019-2021  Nexedi SA and Contributors.
+// Copyright (C) 2019-2024  Nexedi SA and Contributors.
 //                          Kirill Smelkov <kirr@nexedi.com>
 //
 // This program is free software: you can Use, Study, Modify and Redistribute
@@ -20,6 +20,8 @@
 package zodb

 import (
+	"testing"
+
 	"lab.nexedi.com/kirr/go123/mem"
 )

@@ -38,3 +40,13 @@ type ZRawObject struct { // keep in sync with xtesting.ZRawObject
 func PSerialize(obj IPersistent) *mem.Buf {
 	return obj.persistent().pSerialize()
 }
+
+// OpenTestDB opens zurl via tDB.
+func OpenTestDB(t0 *testing.T, zurl string) *tDB {
+	// reuse tDB repointing its .zurl to requested zurl
+	// TODO instruct tDB not to create/populate its own temporary database first.
+	tdb := testdb(t0, /*rawcache=*/false)
+	tdb.zurl = zurl
+	tdb.Reopen()
+	return tdb
+}
--- a/go/zodb/persistent.go
+++ b/go/zodb/persistent.go
-// Copyright (C) 2018-2021  Nexedi SA and Contributors.
+// Copyright (C) 2018-2024  Nexedi SA and Contributors.
 //                          Kirill Smelkov <kirr@nexedi.com>
 //
 // This program is free software: you can Use, Study, Modify and Redistribute
@@ -26,6 +26,8 @@ import (
 	"reflect"
 	"sync"

+	pickle "github.com/kisielk/og-rek"
+
 	"lab.nexedi.com/kirr/go123/mem"
 	"lab.nexedi.com/kirr/go123/xerr"
 )
@@ -578,22 +580,19 @@ var brokenZClass = &zclass{
 type Map struct {
 	Persistent

-	// XXX it is not possible to embed map. And even if we embed a map via
-	// another type = map, then it is not possible to use indexing and
-	// range over Map. -> just provide access to the map as .Data .
-	Data map[interface{}]interface{}
+	pickle.Dict
 }

 type mapState Map // hide state methods from public API

 func (m *mapState) DropState() {
-	m.Data = nil
+	m.Dict = pickle.Dict{}
 }

 func (m *mapState) PyGetState() interface{} {
-	return map[interface{}]interface{}{
-		"data": m.Data,
-	}
+	return pickle.NewDictWithData(
+		"data", m.Dict,
+	)
 }

 func (m *mapState) PySetState(pystate interface{}) error {
@@ -604,12 +603,12 @@ func (m *mapState) PySetState(pystate interface{}) error {
 		return err
 	}

-	data, ok := xdata.(map[interface{}]interface{})
+	data, ok := xdata.(pickle.Dict)
 	if !ok {
 		return fmt.Errorf("state data must be dict, not %T", xdata)
 	}

-	m.Data = data
+	m.Dict = data
 	return nil
 }

@@ -617,7 +616,9 @@ func (m *mapState) PySetState(pystate interface{}) error {
 type List struct {
 	Persistent

-	// XXX it is not possible to embed slice - see Map for similar issue and more details.
+	// XXX it is not possible to embed slice. And even if we embed a slice via
+	// another type = slice, then it is not possible to use indexing and
+	// range over List. -> just provide access to the slice as .Data .
 	Data []interface{}
 }

@@ -628,9 +629,9 @@ func (l *listState) DropState() {
 }

 func (l *listState) PyGetState() interface{} {
-	return map[interface{}]interface{}{
-		"data": l.Data,
-	}
+	return pickle.NewDictWithData(
+		"data", l.Data,
+	)
 }

 func (l *listState) PySetState(pystate interface{}) error {
@@ -651,17 +652,17 @@ func (l *listState) PySetState(pystate interface{}) error {
 // pystateDict1 decodes pystate that is expected to be {} with single key and
 // returns data for that key.
 func pystateDict1(pystate interface{}, acceptKeys ...string) (data interface{}, _ error) {
-	d, ok := pystate.(map[interface{}]interface{})
+	d, ok := pystate.(pickle.Dict)
 	if !ok {
 		return nil, fmt.Errorf("state must be dict, not %T", pystate)
 	}

-	if l := len(d); l != 1 {
+	if l := d.Len(); l != 1 {
 		return nil, fmt.Errorf("state dict has %d keys, must be only 1", l)
 	}

 	for _, key := range acceptKeys {
-		data, ok := d[key]
+		data, ok := d.Get_(key)
 		if ok {
 			return data, nil
 		}

--- a/go/zodb/persistent_test.go
+++ b/go/zodb/persistent_test.go
@@ -362,14 +362,11 @@ func (t *tDB) Open(opt *ConnOptions) *tConnection {
 	}
 }

-// Get gets oid from t.conn and asserts its type.
+// Get gets oid from t.conn and asserts it to be MyObject.
 func (t *tConnection) Get(oid Oid) *MyObject {
 	t.Helper()
-	xobj, err := t.conn.Get(t.ctx, oid)
-	if err != nil {
-		t.Fatal(err)
-	}

+	xobj   := t.GetAny(oid)
 	zclass := ClassOf(xobj)
 	zmy    := "t.zodb.MyObject"
 	if zclass != zmy {
@@ -379,6 +376,17 @@ func (t *tConnection) Get(oid Oid) *MyObject {
 	return xobj.(*MyObject)
 }

+// GetAny gets oid from t.conn.
+func (t *tConnection) GetAny(oid Oid) IPersistent {
+	t.Helper()
+	xobj, err := t.conn.Get(t.ctx, oid)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return xobj
+}
+
+
 // PActivate activates obj in t environment.
 func (t *tConnection) PActivate(obj IPersistent) {
 	t.Helper()
@@ -924,8 +932,6 @@ func TestLiveCache(t0 *testing.T) {
 	// TODO reclassify tests
 }

-// TODO Map & List tests.
-

 // TODO PyGetState vs PySetState tests (general - for any type):
 //

--- a/go/zodb/persistent_x_test.go
+++ b/go/zodb/persistent_x_test.go
+// Copyright (C) 2024  Nexedi SA and Contributors.
+//                     Kirill Smelkov <kirr@nexedi.com>
+//
+// This program is free software: you can Use, Study, Modify and Redistribute
+// it under the terms of the GNU General Public License version 3, or (at your
+// option) any later version, as published by the Free Software Foundation.
+//
+// You can also Link and Combine this program with other software covered by
+// the terms of any of the Free Software licenses or any of the Open Source
+// Initiative approved licenses and Convey the resulting work. Corresponding
+// source of such a combination shall include the source code for all other
+// software used.
+//
+// This program is distributed WITHOUT ANY WARRANTY; without even the implied
+// warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+//
+// See COPYING file for full licensing terms.
+// See https://www.nexedi.com/licensing for rationale and options.
+
+package zodb_test
+
+import (
+	"testing"
+
+	"lab.nexedi.com/kirr/neo/go/zodb"
+	pickle "github.com/kisielk/og-rek"
+
+	"lab.nexedi.com/kirr/neo/go/internal/xtesting"
+	assert "github.com/stretchr/testify/require"
+)
+
+
+// ztestdataReg maintains registry of all entries under testdata/ .
+var  ztestdataReg = xtesting.ZTestDataRegistry[struct{}]{}
+type ZTestData    = xtesting.ZTestData[struct{}]
+
+
+// Verify that Map and List can load from data saved by py.
+// TODO go saves Map/List -> py loads and checks.
+func TestPersistentMapListLoad(t *testing.T) {
+	ztestdataReg.RunWithEach(t, _TestPersistentMapListLoad)
+}
+func _TestPersistentMapListLoad(t0 *testing.T, z *ZTestData) {
+	assert := assert.New(t0)
+
+	tdb := zodb.OpenTestDB(t0, z.Path("data.fs"))
+	defer tdb.Close()
+
+	t := tdb.Open(&zodb.ConnOptions{})
+	X := xtesting.FatalIf(t0)
+
+	xroot := t.GetAny(0)
+	root, ok := xroot.(*zodb.Map)
+	if !ok {
+		t.Fatalf("root: got %T  ; expect Map", xroot)
+	}
+
+	t.PActivate(root)
+	defer root.PDeactivate()
+
+	// see py/pydata-gen-testdata
+	assert.Equal(root.Len(), 3)
+	assert.Equal(root.Get("int1"), int64(1))
+
+	xabc := root.Get("strABC")
+	abc, err := pickle.AsString(xabc); X(err)
+	assert.Equal(abc, "abc")
+
+	xplist := root.Get("plist")
+	plist, ok := xplist.(*zodb.List)
+	if !ok {
+		t.Fatalf("plist: got %T  ; expect List", xplist)
+	}
+
+	t.PActivate(plist)
+	defer plist.PDeactivate()
+
+	assert.Equal(len(plist.Data), 3)
+	xa := plist.Data[0]
+	a, err := pickle.AsString(xa); X(err)
+	assert.Equal(a, "a")
+
+	assert.Equal(plist.Data[1], int64(1))
+	assert.Equal(plist.Data[2], pickle.None{})
+}
--- a/go/zodb/py/pydata-gen-testdata
+++ b/go/zodb/py/pydata-gen-testdata
-#!/usr/bin/env python2
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
-# Copyright (C) 2017-2021  Nexedi SA and Contributors.
+# Copyright (C) 2017-2024  Nexedi SA and Contributors.
 #                          Kirill Smelkov <kirr@nexedi.com>
 #
 # This program is free software: you can Use, Study, Modify and Redistribute
@@ -18,21 +18,73 @@
 #
 # See COPYING file for full licensing terms.
 # See https://www.nexedi.com/licensing for rationale and options.
-"""generate reference pickle objects encoding for tests"""
+"""generate reference database and pickle objects encoding for tests"""

-from ZODB import serialize
-from zodbtools.test.gen_testdata import run_with_zodb4py2_compat
+from __future__ import print_function
+
+from ZODB import DB, serialize
+from ZODB.utils import z64
+from persistent.mapping import PersistentMapping
+from persistent.list import PersistentList
+import transaction
+from zodbtools.test.gen_testdata import run_with_all_zodb_pickle_kinds, current_zkind
 from golang.gcompat import qq
+from os import makedirs
+from os.path import exists, relpath
+from shutil import rmtree
+
+from six import PY3
+if PY3:
+    from importlib import reload
+else:
+    reload  # available as builtin
+
+
+# gen_ztestdata generates test databases that ZODB/go will try to load.
+def gen_ztestdata():
+    zkind = current_zkind()
+    prefix = "testdata/%s" % zkind
+    if exists(prefix):
+        rmtree(prefix)
+    makedirs(prefix)
+
+    outfs = "%s/data.fs" % prefix
+    db = DB(outfs)
+    conn = db.open()
+    root = conn.root()
+    assert type(root) is PersistentMapping
+    assert root._p_oid == z64
+
+    # NOTE keep in sync with TestPersistentMapListLoad
+    root['int1']   = 1
+    root['strABC'] = 'abc'
+    root['plist']  = PersistentList(["a", 1, None])
+
+    transaction.commit()
+
+    with open("ztestdata_%s_x_test.go" % zkind, "w") as f:
+        def emit(v):
+            print(v, file=f)
+        emit("// Code generated by %s; DO NOT EDIT." % relpath(__file__))
+        emit("package zodb_test")
+        emit("func init() {")
+        emit("\tztestdataReg.Register(%s, %s, nil)" % (qq(zkind), qq(prefix)))
+        emit("}")
+

-def main2():
+# gen_test_pydata generates testdata for PyData serialization tests.
+def gen_test_pydata():
    # import ZODB.tests at runtime after ZODB.X._protocol is patched
+    # reload the module each time because SerializerTestCase uses make_pickle to initialize class-level attributes
    from ZODB.tests import testSerialize
+    reload(testSerialize)

    # dump to go what to expect
-    with open("ztestdata_pydata_test.go", "w") as f:
+    zkind = current_zkind()
+    with open("ztestdata_pydata_%s_test.go" % zkind, "w") as f:
        def emit(v):
-            print >>f, v
-        emit("// Code generated by %s; DO NOT EDIT." % __file__)
+            print(v, file=f)
+        emit("// Code generated by %s; DO NOT EDIT." % relpath(__file__))
        emit("package zodb")

        # [] of pickle
@@ -46,17 +98,20 @@ def main2():

        r = serialize.ObjectReader(factory=testSerialize._factory)

-        emit("\nvar _PyData_ClassName_Testv = [...]_PyDataClassName_TestEntry{")
+        emit("func init() {")
+        emit("\t_PyDataClassName_TestDataRegistry[%s] = []_PyDataClassName_TestEntry{" % qq(zkind))
        for test in testv:
-            emit("\t{")
-            emit("\t\t%s," % qq(test))
-            emit("\t\t%s," % qq(r.getClassName(test)))
-            emit("\t},")
-        emit('\t{"aaa", "?.?"},') # invalid
+            emit("\t\t{")
+            emit("\t\t\t%s," % qq(test))
+            emit("\t\t\t%s," % qq(r.getClassName(test)))
+            emit("\t\t},")
+        emit('\t\t{"aaa", "?.?"},') # invalid
+        emit("\t}")
        emit("}")

 def main():
-    run_with_zodb4py2_compat(main2)
+    run_with_all_zodb_pickle_kinds(gen_ztestdata)
+    run_with_all_zodb_pickle_kinds(gen_test_pydata)

 if __name__ == '__main__':
    main()
--- a/go/zodb/pydata.go
+++ b/go/zodb/pydata.go
-// Copyright (C) 2016-2019  Nexedi SA and Contributors.
+// Copyright (C) 2016-2024  Nexedi SA and Contributors.
 //                          Kirill Smelkov <kirr@nexedi.com>
 //
 // This program is free software: you can Use, Study, Modify and Redistribute
@@ -26,7 +26,7 @@ import (
 	"strings"

 	pickle "github.com/kisielk/og-rek"
-	"lab.nexedi.com/kirr/neo/go/zodb/internal/pickletools"
+	"lab.nexedi.com/kirr/neo/go/internal/zodbpickle"

 	"lab.nexedi.com/kirr/go123/xerr"
 )
@@ -47,7 +47,7 @@ type PyData []byte
 // If pickle decoding fails - "?.?" is returned.
 func (d PyData) ClassName() string {
 	// see ObjectReader.getClassName & get_pickle_metadata in zodb/py
-	p := pickle.NewDecoder(bytes.NewReader([]byte(d)))
+	p := zodbpickle.NewUnpickler(bytes.NewReader([]byte(d)), nil)
 	xklass, err := p.Decode()
 	if err != nil {
 		return "?.?"
@@ -66,12 +66,7 @@ func encodePyData(pyclass pickle.Class, pystate interface{}) PyData {
 	// XXX better return mem.Buf instead of PyData?
 	buf := &bytes.Buffer{}

-	p := pickle.NewEncoderWithConfig(buf, &pickle.EncoderConfig{
-		// allow pristine python2 to decode the pickle.
-		// TODO 2 -> 3 since ZODB5 switched to it and uses zodbpickle.
-		Protocol:      2,
-		PersistentRef: persistentRef,
-	})
+	p := zodbpickle.NewPickler(buf, persistentRef)

 	// emit: object type
 	err := p.Encode(pyclass)
@@ -99,10 +94,7 @@ func encodePyData(pyclass pickle.Class, pystate interface{}) PyData {
 func (d PyData) decode(jar *Connection) (pyclass pickle.Class, pystate interface{}, err error) {
 	defer xerr.Context(&err, "pydata: decode")

-	p := pickle.NewDecoderWithConfig(
-		bytes.NewReader([]byte(d)),
-		&pickle.DecoderConfig{PersistentLoad: jar.loadref},
-	)
+	p := zodbpickle.NewUnpickler(bytes.NewReader([]byte(d)), jar.loadref)

 	xklass, err := p.Decode()
 	if err != nil {
@@ -214,10 +206,13 @@ func xpyclass(xklass interface{}) (_ pickle.Class, err error) {
 			if len(t) != 2 {
 				return pickle.Class{}, fmt.Errorf("xklass: expect [2](); got [%d]()", len(t))
 			}
-			modname, ok1 := t[0].(string)
-			classname, ok2 := t[1].(string)
-			if !(ok1 && ok2) {
-				return pickle.Class{}, fmt.Errorf("xklass: expect (str, str); got (%T, %T)", t[0], t[1])
+			modname, err := pickle.AsString(t[0])
+			if err != nil {
+				return pickle.Class{}, fmt.Errorf("xklass: module: %s", err)
+			}
+			classname, err := pickle.AsString(t[1])
+			if err != nil {
+				return pickle.Class{}, fmt.Errorf("xklass: class: %s", err)
 			}

 			return pickle.Class{Module: modname, Name: classname}, nil
@@ -238,7 +233,7 @@ func xoid(x interface{}) (_ Oid, err error) {

 	// ZODB >= 5.4 encodes oid as bytes; before - as str:
 	// https://github.com/zopefoundation/ZODB/commit/12ee41c473
-	v, err := pickletools.Xstrbytes8(x)
+	v, err := zodbpickle.Unpack64(x)
 	if err != nil {
 		return InvalidOid, err
 	}

--- a/go/zodb/pydata_test.go
+++ b/go/zodb/pydata_test.go
-// Copyright (C) 2016-2017  Nexedi SA and Contributors.
+// Copyright (C) 2016-2024  Nexedi SA and Contributors.
 //                          Kirill Smelkov <kirr@nexedi.com>
 //
 // This program is free software: you can Use, Study, Modify and Redistribute
@@ -19,10 +19,13 @@

 package zodb

-//go:generate ./py/pydata-gen-testdata
+//go:generate python2 py/pydata-gen-testdata
+//go:generate python3 py/pydata-gen-testdata

 import (
 	"testing"
+
+	"lab.nexedi.com/kirr/neo/go/internal/xmaps"
 )

 type _PyDataClassName_TestEntry struct {
@@ -30,7 +33,17 @@ type _PyDataClassName_TestEntry struct {
 	className string
 }

+var _PyDataClassName_TestDataRegistry = map[/*zkind*/string][]_PyDataClassName_TestEntry{}
+
 func TestPyClassName(t *testing.T) {
+	for _, zkind := range xmaps.SortedKeys(_PyDataClassName_TestDataRegistry) {
+		t.Run(zkind, func(t *testing.T) {
+			_TestPyClassName(t, _PyDataClassName_TestDataRegistry[zkind])
+		})
+	}
+}
+
+func _TestPyClassName(t *testing.T, _PyData_ClassName_Testv []_PyDataClassName_TestEntry) {
 	for _, tt := range _PyData_ClassName_Testv {
 		className := PyData(tt.pydata).ClassName()
 		if className != tt.className {

--- a/go/zodb/storage/demo/demo_test.go
+++ b/go/zodb/storage/demo/demo_test.go
-// Copyright (C) 2021  Nexedi SA and Contributors.
-//                     Kirill Smelkov <kirr@nexedi.com>
+// Copyright (C) 2021-2024  Nexedi SA and Contributors.
+//                          Kirill Smelkov <kirr@nexedi.com>
 //
 // This program is free software: you can Use, Study, Modify and Redistribute
 // it under the terms of the GNU General Public License version 3, or (at your
@@ -160,10 +160,21 @@ func TestEmptyDB(t *testing.T) {
 	})
 }

+// ztestdataReg keeps registry of ZODB test data we use in tests with non-empty Preload.
+// we take the data from fs1 testdata.
+var  ztestdataReg = xtesting.ZTestDataRegistry[struct{}]{}
+type ZTestData    = xtesting.ZTestData[struct{}]
+func init() {
+        ztestdataReg = xtesting.LoadZTestData("../fs1/testdata")
+}
+
 func TestLoad(t *testing.T) {
+	ztestdataReg.RunWithEach(t, _TestLoad)
+}
+func _TestLoad(t *testing.T, z *ZTestData) {
 	X := xtesting.FatalIf(t)

-	data := "../fs1/testdata/1.fs"
+	data := z.Path("1.fs")
 	txnvOk, err := xtesting.LoadDBHistory(data); X(err)

 	withDemo(t, func(t *testing.T, _ *DemoData, ddrv *Storage) {

--- a/go/zodb/storage/fs1/filestorage_test.go
+++ b/go/zodb/storage/fs1/filestorage_test.go
-// Copyright (C) 2017-2021  Nexedi SA and Contributors.
+// Copyright (C) 2017-2024  Nexedi SA and Contributors.
 //                          Kirill Smelkov <kirr@nexedi.com>
 //
 // This program is free software: you can Use, Study, Modify and Redistribute
@@ -33,6 +33,17 @@ import (
 	"lab.nexedi.com/kirr/go123/exc"
 )

+// ztestdataReg maintains registry of all entries under testdata/ .
+var ztestdataReg = xtesting.ZTestDataRegistry[_TestDataOK]{}
+type ZTestData   = xtesting.ZTestData[_TestDataOK]
+
+// _TestDataOK describes expected data for one entry under testdata/ .
+type _TestDataOK struct {
+	_1fs_indexTopPos int64	      // topPos of 1fs.index
+	_1fs_indexEntryv []indexEntry // index content
+	_1fs_dbEntryv    []dbEntry    // database content
+}
+
 // one database transaction record
 type dbEntry struct {
 	Header TxnHeader
@@ -73,21 +84,27 @@ func xfsopenopt(t testing.TB, path string, opt *zodb.DriverOptions) (*FileStorag
 }

 func TestEmptyDB(t *testing.T) {
-	fs, _ := xfsopen(t, "testdata/empty.fs")
+	ztestdataReg.RunWithEach(t, _TestEmptyDB)
+}
+func _TestEmptyDB(t *testing.T, z *ZTestData) {
+	fs, _ := xfsopen(t, z.Path("empty.fs"))
 	defer exc.XRun(fs.Close)

 	xtesting.DrvTestEmptyDB(t, fs)
 }

 func TestLoad(t *testing.T) {
-	fs, _ := xfsopen(t, "testdata/1.fs")
+	ztestdataReg.RunWithEach(t, _TestLoad)
+}
+func _TestLoad(t *testing.T, z *ZTestData) {
+	fs, _ := xfsopen(t, z.Path("1.fs"))
 	defer exc.XRun(fs.Close)

 	// NOTE don't use xtesting.LoadDBHistory here - it is itself tested
 	// with the assumption that fs1.Load and fs1.Iterate work correctly.
 	// Use what testdata generator gave use with what to expect.
 	txnv := []xtesting.Txn{}
-	for _, dbe := range _1fs_dbEntryv {
+	for _, dbe := range z.Misc._1fs_dbEntryv {
 		txn := xtesting.Txn{Header: &zodb.TxnInfo{
 			Tid: dbe.Header.Tid,

@@ -221,12 +238,17 @@ func testIterate(t *testing.T, fs *FileStorage, tidMin, tidMax zodb.Tid, expectv

 // TODO -> xtesting
 func TestIterate(t *testing.T) {
-	fs, _ := xfsopen(t, "testdata/1.fs")
+	ztestdataReg.RunWithEach(t, _TestIterate)
+}
+func _TestIterate(t *testing.T, z *ZTestData) {
+	zz := z.Misc
+
+	fs, _ := xfsopen(t, z.Path("1.fs"))
 	defer exc.XRun(fs.Close)

 	// all []tids in test database
 	tidv := []zodb.Tid{}
-	for _, dbe := range _1fs_dbEntryv {
+	for _, dbe := range zz._1fs_dbEntryv {
 		tidv = append(tidv, dbe.Header.Tid)
 	}

@@ -248,17 +270,20 @@ func TestIterate(t *testing.T) {
 			}

 			//fmt.Printf("%d%+d .. %d%+d\t -> %d steps\n", i, ii-1, j, jj-1, nsteps)
-			testIterate(t, fs, tmin, tmax, _1fs_dbEntryv[i + ii/2:][:nsteps])
+			testIterate(t, fs, tmin, tmax, zz._1fs_dbEntryv[i + ii/2:][:nsteps])
 		}}
 	}}

 	// also check 0..tidMax
-	testIterate(t, fs, 0, zodb.TidMax, _1fs_dbEntryv[:])
+	testIterate(t, fs, 0, zodb.TidMax, zz._1fs_dbEntryv[:])
 }

 // TODO -> xtesting
 func BenchmarkIterate(b *testing.B) {
-	fs, _ := xfsopen(b, "testdata/1.fs")
+	ztestdataReg.BenchWithEach(b, _BenchmarkIterate)
+}
+func _BenchmarkIterate(b *testing.B, z *ZTestData) {
+	fs, _ := xfsopen(b, z.Path("1.fs"))
 	defer exc.XRun(fs.Close)

 	ctx := context.Background()
@@ -307,12 +332,16 @@ func TestWatch(t *testing.T) {
 // TestOpenRecovery verifies how Open handles data file with not-finished voted
 // transaction in the end.
 func TestOpenRecovery(t *testing.T) {
+	ztestdataReg.RunWithEach(t, _TestOpenRecovery)
+}
+func _TestOpenRecovery(t *testing.T, z *ZTestData) {
 	X := exc.Raiseif
-	main, err := ioutil.ReadFile("testdata/1.fs"); X(err)
-	index, err := ioutil.ReadFile("testdata/1.fs.index"); X(err)
-	headOk := _1fs_dbEntryv[len(_1fs_dbEntryv)-1].Header.Tid
-	topPos := int64(_1fs_indexTopPos)
-	voteTail, err := ioutil.ReadFile("testdata/1voted.tail"); X(err)
+	zz := z.Misc
+	main, err := ioutil.ReadFile(z.Path("1.fs")); X(err)
+	index, err := ioutil.ReadFile(z.Path("1.fs.index")); X(err)
+	headOk := zz._1fs_dbEntryv[len(zz._1fs_dbEntryv)-1].Header.Tid
+	topPos := int64(zz._1fs_indexTopPos)
+	voteTail, err := ioutil.ReadFile(z.Path("1voted.tail")); X(err)

 	workdir := xworkdir(t)
 	ctx := context.Background()
@@ -381,7 +410,10 @@ func TestOpenRecovery(t *testing.T) {
 // FileStorage/py.deleteObject allows to create whiteouts instead of raising
 // POSKeyError.
 func TestLoadWhiteout(t *testing.T) {
-	fs, _ := xfsopen(t, "testdata/whiteout.fs")
+	ztestdataReg.RunWithEach(t, _TestLoadWhiteout)
+}
+func _TestLoadWhiteout(t *testing.T, z *ZTestData) {
+	fs, _ := xfsopen(t, z.Path("whiteout.fs"))
 	defer exc.XRun(fs.Close)

 	xid := zodb.Xid{At: zodb.Tid(0x17), Oid: zodb.Oid(1)}

--- a/go/zodb/storage/fs1/format.go
+++ b/go/zodb/storage/fs1/format.go
-// Copyright (C) 2017-2021  Nexedi SA and Contributors.
+// Copyright (C) 2017-2024  Nexedi SA and Contributors.
 //                          Kirill Smelkov <kirr@nexedi.com>
 //
 // This program is free software: you can Use, Study, Modify and Redistribute
@@ -68,7 +68,8 @@ type DataHeader struct {
 }

 const (
-	Magic = "FS21" // every FileStorage file starts with this
+	Magic21 = "FS21" // FileStorage file produced by Python2 starts with this
+	Magic30	= "FS30" // ----//---- by Python3

 	// on-disk sizes
 	FileHeaderSize    = 4
@@ -153,8 +154,12 @@ func (fh *FileHeader) Load(r io.ReaderAt) error {
 	if err != nil {
 		return fmt.Errorf("%sread magic: %s", ioprefix(r), err)
 	}
-	if string(fh.Magic[:]) != Magic {
+	switch string(fh.Magic[:]) {
+	default:
 		return fmt.Errorf("%sinvalid fs1 magic %q", ioprefix(r), fh.Magic)
+
+	case Magic21, Magic30:
+		// ok
 	}

 	return nil

--- a/go/zodb/storage/fs1/fs1tools/dump.go
+++ b/go/zodb/storage/fs1/fs1tools/dump.go
-// Copyright (C) 2017-2021  Nexedi SA and Contributors.
+// Copyright (C) 2017-2024  Nexedi SA and Contributors.
 //                          Kirill Smelkov <kirr@nexedi.com>
 //
 // This program is free software: you can Use, Study, Modify and Redistribute
@@ -157,6 +157,10 @@ func (d *DumperFsDump) DumpTxn(buf *xfmt.Buffer, it *fs1.Iter) error {
 	buf .S("Trans #")
 	buf .S(fmt.Sprintf("%05d", d.ntxn))	// XXX -> .D_f("05", d.ntxn)
 	buf .S(" tid=") .V(txnh.Tid)
+
+	// XXX here fsdump/py prints size of transaction record without "redundant lenght" field
+	buf .S(" size=") .D64(txnh.Len - 8)
+
 	buf .S(" time=") .V(txnh.Tid.Time())

 	// XXX here fsdump/py prints position of first data record, NOT transaction start!

--- a/go/zodb/storage/fs1/fs1tools/dump_test.go
+++ b/go/zodb/storage/fs1/fs1tools/dump_test.go
-// Copyright (C) 2017-2021  Nexedi SA and Contributors.
+// Copyright (C) 2017-2024  Nexedi SA and Contributors.
 //                          Kirill Smelkov <kirr@nexedi.com>
 //
 // This program is free software: you can Use, Study, Modify and Redistribute
@@ -19,31 +19,26 @@

 package fs1tools

-//go:generate sh -c "python2 -m ZODB.scripts.fstail -n 1000000 ../testdata/1.fs >testdata/1.fstail.ok"
-//go:generate sh -c "python2 -c 'from ZODB.FileStorage import fsdump; fsdump.main()' ../testdata/1.fs >testdata/1.fsdump.ok"
-//go:generate sh -c "python2 -c 'from ZODB.FileStorage.fsdump import Dumper; import sys; d = Dumper(sys.argv[1]); d.dump()' ../testdata/1.fs >testdata/1.fsdumpv.ok"
-
-// fstail.py crashes on empty.fs . The output should be empty file so generate it with just echo.
-////go:generate sh -c "python2 -m ZODB.scripts.fstail -n 1000000 ../testdata/empty.fs >testdata/empty.fstail.ok"
-//go:generate sh -c "echo -n >testdata/empty.fstail.ok"
-//go:generate sh -c "python2 -c 'from ZODB.FileStorage import fsdump; fsdump.main()' ../testdata/empty.fs >testdata/empty.fsdump.ok"
-//go:generate sh -c "python2 -c 'from ZODB.FileStorage.fsdump import Dumper; import sys; d = Dumper(sys.argv[1]); d.dump()' ../testdata/empty.fs >testdata/empty.fsdumpv.ok"
-
-//go:generate sh -c "python2 -m ZODB.scripts.fstail -n 1000000 ../testdata/whiteout.fs >testdata/whiteout.fstail.ok"
-//go:generate sh -c "python2 -c 'from ZODB.FileStorage import fsdump; fsdump.main()' ../testdata/whiteout.fs >testdata/whiteout.fsdump.ok"
-//go:generate sh -c "python2 -c 'from ZODB.FileStorage.fsdump import Dumper; import sys; d = Dumper(sys.argv[1]); d.dump()' ../testdata/whiteout.fs >testdata/whiteout.fsdumpv.ok"
+//go:generate ./gen-testdata

 import (
 	"bytes"
 	"fmt"
 	"io/ioutil"
+	"strings"
 	"testing"

+	"lab.nexedi.com/kirr/neo/go/internal/xtesting"
 	"lab.nexedi.com/kirr/neo/go/zodb/storage/fs1"

 	"github.com/kylelemons/godebug/diff"
 )

+// ztestdataReg maintains registry of all entries under testdata/ .
+var  ztestdataReg = xtesting.ZTestDataRegistry[struct{}]{}
+type ZTestData    = xtesting.ZTestData[struct{}]
+
+
 func loadFile(t *testing.T, path string) string {
 	data, err := ioutil.ReadFile(path)
 	if err != nil {
@@ -53,18 +48,36 @@ func loadFile(t *testing.T, path string) string {
 }

 func testDump(t *testing.T, dir fs1.IterDir, newd func() Dumper) {
+	ztestdataReg.RunWithEach(t, func(t *testing.T, z *ZTestData) {
+		_testDump(t, dir, newd, z)
+	})
+}
+
+func _testDump(t *testing.T, dir fs1.IterDir, newd func() Dumper, z *ZTestData) {
 	testv := []string{"1", "empty", "whiteout"}
 	for _, tt := range testv {
 		t.Run("db=" + tt, func(t *testing.T) {
 			d   := newd()
 			buf := bytes.Buffer{}

-			err := Dump(&buf, fmt.Sprintf("../testdata/%s.fs", tt), dir, d)
+			err := Dump(&buf, "../"+z.Path(tt+".fs"), dir, d)
 			if err != nil {
 				t.Fatalf("%s: %v", d.DumperName(), err)
 			}

-			dumpOk := loadFile(t, fmt.Sprintf("testdata/%s.%s.ok", tt, d.DumperName()))
+			dumpOk := loadFile(t, z.Path(fmt.Sprintf("/%s.%s.ok", tt, d.DumperName())))
+
+			// normalize dumpOk from py3 to py2 and go outputs:
+			// bytestrings come as '...' on py2 and b'...' on py3
+			if strings.HasPrefix(z.Kind, "py3") {
+				sub := func(old, new string) {
+					dumpOk = strings.ReplaceAll(dumpOk, old, new)
+				}
+				sub(`: b'`, `: '`)
+				sub(`: b"`, `: "`)
+				sub(`=b'`,  `='`)
+				sub(`=b"`,  `="`)
+			}

 			if dumpOk != buf.String() {
 				t.Errorf("%s: dump different:\n%v", d.DumperName(), diff.Diff(dumpOk, buf.String()))

--- a/go/zodb/storage/fs1/fs1tools/gen-testdata
+++ b/go/zodb/storage/fs1/fs1tools/gen-testdata
+#!/bin/bash -e
+# generate files in testdata/
+#
+# Copyright (C) 2017-2024  Nexedi SA and Contributors.
+#                          Kirill Smelkov <kirr@nexedi.com>
+#
+# This program is free software: you can Use, Study, Modify and Redistribute
+# it under the terms of the GNU General Public License version 3, or (at your
+# option) any later version, as published by the Free Software Foundation.
+#
+# You can also Link and Combine this program with other software covered by
+# the terms of any of the Free Software licenses or any of the Open Source
+# Initiative approved licenses and Convey the resulting work. Corresponding
+# source of such a combination shall include the source code for all other
+# software used.
+#
+# This program is distributed WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# See COPYING file for full licensing terms.
+# See https://www.nexedi.com/licensing for rationale and options.
+
+rm -rf testdata ztestdata_test.go
+
+emit() {
+	echo -e "$@" >>ztestdata_test.go
+}
+
+emit "// Code generated by gen-testdata; DO NOT EDIT."
+emit "package fs1tools"
+emit "func init() {"
+
+for zin in ../testdata/*/ ; do
+	zkind=`basename $zin`	# py2_pickle3
+	py=(${zkind//_/ })	# (py2 pickle3)
+	py=${py[0]}		# py2
+	python=${py/py/python}	# python2
+
+	out=testdata/$zkind
+	mkdir -p "$out"
+
+	emit "\tztestdataReg.Register(\"$zkind\", \"$out\", nil)"
+
+	$python -m ZODB.scripts.fstail -n 1000000 $zin/1.fs >$out/1.fstail.ok
+	$python -c 'from ZODB.FileStorage import fsdump; fsdump.main()' $zin/1.fs >$out/1.fsdump.ok
+	$python -c 'from ZODB.FileStorage.fsdump import Dumper; import sys; d = Dumper(sys.argv[1]); d.dump()' $zin/1.fs >$out/1.fsdumpv.ok
+
+	# fstail.py crashes on empty.fs . The output should be empty file so generate it with just echo.
+	# $python -m ZODB.scripts.fstail -n 1000000 $zin/empty.fs >$out/empty.fstail.ok
+	echo -n >$out/empty.fstail.ok
+	$python -c 'from ZODB.FileStorage import fsdump; fsdump.main()' $zin/empty.fs >$out/empty.fsdump.ok
+	$python -c 'from ZODB.FileStorage.fsdump import Dumper; import sys; d = Dumper(sys.argv[1]); d.dump()' $zin/empty.fs >$out/empty.fsdumpv.ok
+
+	$python -m ZODB.scripts.fstail -n 1000000 $zin/whiteout.fs >$out/whiteout.fstail.ok
+	$python -c 'from ZODB.FileStorage import fsdump; fsdump.main()' $zin/whiteout.fs >$out/whiteout.fsdump.ok
+	$python -c 'from ZODB.FileStorage.fsdump import Dumper; import sys; d = Dumper(sys.argv[1]); d.dump()' $zin/whiteout.fs >$out/whiteout.fsdumpv.ok
+done
+
+emit "}"
--- a/go/zodb/storage/fs1/fs1tools/testdata/1.fstail.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/1.fstail.ok
-1979-01-03 21:01:31.300002: hash=94252f1f1d30b1a3f1f7503c266a713f02b4ba52
-user="root1\nYour\nRoyal\nMagesty' \x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" description='delete 1\nalpha beta gamma\'delta"lambda\n\nqqq ...' length=240 offset=13455 (+190)
-
-1979-01-03 21:01:30.200002: hash=25f7492a49cc22ef77900de9eb3b83de7266e140
-user='' description='predelete 6' length=401 offset=13046 (+34)
-
-1979-01-03 21:01:29.100002: hash=fb51ce7bfafa31fed95763a9717d4634b48e7e63
-user='root1.1\nYour\nMagesty ' description='undo 1.1\nmore detailed description\n\nzzz ...\t\t' length=210 offset=12828 (+160)
-
-1979-01-03 21:01:28.000002: hash=9404e6448e47b8091b1d92bcf0f312fed8f4ad0d
-user='root1.0\nYour\nMagesty ' description='undo 1.0\nmore detailed description\n\nzzz ...\t' length=209 offset=12611 (+159)
-
-1979-01-03 21:01:25.800002: hash=897458fb84e889f88e7061041959ab98a371181b
-user='user1.24' description='step 1.24' length=190 offset=12413 (+95)
-
-1979-01-03 21:01:24.700002: hash=a56e6de8176172eb09883e48ccbf28cf33b7a75b
-user='user1.23' description='step 1.23' length=190 offset=12215 (+95)
-
-1979-01-03 21:01:23.600002: hash=f6dd3e02bd8e02c52558864e1cfbdcf898d847ab
-user='user1.22' description='step 1.22' length=190 offset=12017 (+95)
-
-1979-01-03 21:01:22.500002: hash=0fe8ae8e0ee1f5d82e354299ebf653f8cc15205f
-user='user1.21' description='step 1.21' length=190 offset=11819 (+95)
-
-1979-01-03 21:01:21.400002: hash=bcc9fce2b221b2e1e372f281c7497ccd0caddbf7
-user='user1.20' description='step 1.20' length=190 offset=11621 (+95)
-
-1979-01-03 21:01:20.300002: hash=9727adacbf4ad8c956ec3264d2420d1f5d96fa3b
-user='user1.19' description='step 1.19' length=190 offset=11423 (+95)
-
-1979-01-03 21:01:19.200002: hash=10546257706e21d2e15ccc08c33c65e17bd82657
-user='user1.18' description='step 1.18' length=190 offset=11225 (+95)
-
-1979-01-03 21:01:18.100002: hash=ad0151c6c824c3bcd7ff5ddfd679446743250910
-user='user1.17' description='step 1.17' length=190 offset=11027 (+95)
-
-1979-01-03 21:01:17.000002: hash=f3bebd8d1edeb250862df7623f20618dfe5c58c8
-user='user1.16' description='step 1.16' length=190 offset=10829 (+95)
-
-1979-01-03 21:01:15.900002: hash=7516e440ce0d427170c939cae6fcde1994c5afc9
-user='user1.15' description='step 1.15' length=190 offset=10631 (+95)
-
-1979-01-03 21:01:14.800002: hash=2163b0daccd414189f332c3a9791ef254e45074e
-user='user1.14' description='step 1.14' length=190 offset=10433 (+95)
-
-1979-01-03 21:01:13.700002: hash=88576c57888c89ae5d39c2abd884ec759688edfd
-user='user1.13' description='step 1.13' length=190 offset=10235 (+95)
-
-1979-01-03 21:01:12.600002: hash=2bd5dd2dc35b052c5c48a255fa2be3ffd1ecd7db
-user='user1.12' description='step 1.12' length=190 offset=10037 (+95)
-
-1979-01-03 21:01:11.500002: hash=e9f461a8d3006d364a1e66850cf9578c20b7ab21
-user='user1.11' description='step 1.11' length=190 offset=9839 (+95)
-
-1979-01-03 21:01:10.400002: hash=5bb3dae37c2952dd72a3c9fc6854acc1354c6033
-user='user1.10' description='step 1.10' length=190 offset=9641 (+95)
-
-1979-01-03 21:01:09.300001: hash=b603984d5114835b5f02eceb0864fad8c05e4c18
-user='user1.9' description='step 1.9' length=187 offset=9446 (+93)
-
-1979-01-03 21:01:08.200001: hash=82f456b31e0a04ba25ebdd4ff87901e10668c2e7
-user='user1.8' description='step 1.8' length=187 offset=9251 (+93)
-
-1979-01-03 21:01:07.100001: hash=0418d908951c18fefd261442f77c3195c2bfcb22
-user='user1.7' description='step 1.7' length=187 offset=9056 (+93)
-
-1979-01-03 21:01:06.000001: hash=9890b78dd666542a03d29236c59f0e07604618c2
-user='user1.6' description='step 1.6' length=187 offset=8861 (+93)
-
-1979-01-03 21:01:04.900001: hash=4eb74b859985a34203df28884752ac779846f8e8
-user='user1.5' description='step 1.5' length=187 offset=8666 (+93)
-
-1979-01-03 21:01:03.800001: hash=2a6ecb00068bce7aec7b3fe86e01e74be2304275
-user='user1.4' description='step 1.4' length=187 offset=8471 (+93)
-
-1979-01-03 21:01:02.700001: hash=dd57eeec7984452cf83ea92f5ef1971dab21dab7
-user='user1.3' description='step 1.3' length=187 offset=8276 (+93)
-
-1979-01-03 21:01:01.600001: hash=91807ee2469781ff1ab3190d9315ffc4376ef456
-user='user1.2' description='step 1.2' length=187 offset=8081 (+93)
-
-1979-01-03 21:01:00.500001: hash=200d3846826c608b580d940ccce3993d026fd5d4
-user='user1.1' description='step 1.1' length=187 offset=7886 (+93)
-
-1979-01-03 21:00:59.400001: hash=8731a3aedc212671194b3b1283aa97822e5d2e1c
-user='user1.0' description='step 1.0' length=187 offset=7691 (+93)
-
-1979-01-03 21:00:45.100001: hash=f09c9050d5e5bff8b7d14c7b0909a574825bfcc7
-user="root0\nYour\nRoyal\nMagesty' \x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" description='delete 0\nalpha beta gamma\'delta"lambda\n\nqqq ...' length=240 offset=7443 (+190)
-
-1979-01-03 21:00:44.000001: hash=d9b220897f846e886da76735f68dd935ace92c4b
-user='' description='predelete 7' length=401 offset=7034 (+34)
-
-1979-01-03 21:00:42.900001: hash=cd47f9f155d5e5699d8a039b5d5bfe87b5269996
-user='root0.1\nYour\nMagesty ' description='undo 0.1\nmore detailed description\n\nzzz ...\t' length=209 offset=6817 (+159)
-
-1979-01-03 21:00:41.800001: hash=11b63b0eb9d3da7480882da9641279b26d1ba6e8
-user='root0.0\nYour\nMagesty ' description='undo 0.0\nmore detailed description\n\nzzz ...' length=208 offset=6601 (+158)
-
-1979-01-03 21:00:40.700001: hash=09f89c9aabd271e61e9b8659f5a433de453d8dbe
-user='user0.24' description='step 0.24' length=190 offset=6403 (+95)
-
-1979-01-03 21:00:39.600001: hash=8faa12a66b7bea90ed50c3718cefb66f3d51efa1
-user='user0.23' description='step 0.23' length=190 offset=6205 (+95)
-
-1979-01-03 21:00:38.500001: hash=80e78a0c9f6a4df6488d83c976ffef7ca2bde4a6
-user='user0.22' description='step 0.22' length=464 offset=5733 (+95)
-
-1979-01-03 21:00:37.400001: hash=ff8650ead93eebb87eb6b3eda9739c038e8f1f6d
-user='user0.21' description='step 0.21' length=190 offset=5535 (+95)
-
-1979-01-03 21:00:36.300001: hash=157bcb480d13190aba47f77745b3a228a2fe47ee
-user='user0.20' description='step 0.20' length=190 offset=5337 (+95)
-
-1979-01-03 21:00:35.200001: hash=65f947a791d6db121504812661a4f3f7b7f59df1
-user='user0.19' description='step 0.19' length=190 offset=5139 (+95)
-
-1979-01-03 21:00:34.100001: hash=734ecdb6b80fad959d57c6335f6766574f85bd5a
-user='user0.18' description='step 0.18' length=190 offset=4941 (+95)
-
-1979-01-03 21:00:33.000001: hash=af345f3f6264a22fe07f9593640ade8f6eb0f1e5
-user='user0.17' description='step 0.17' length=190 offset=4743 (+95)
-
-1979-01-03 21:00:31.900001: hash=e62086be7154c9da3a99ddcdc5d40eebfad73423
-user='user0.16' description='step 0.16' length=190 offset=4545 (+95)
-
-1979-01-03 21:00:30.800001: hash=60880e954e9c43f4f6cd66f348e1258d5a72bfcc
-user='user0.15' description='step 0.15' length=190 offset=4347 (+95)
-
-1979-01-03 21:00:29.700001: hash=4fafec0da6d66941a66f2034abc54707d2aef9a0
-user='user0.14' description='step 0.14' length=190 offset=4149 (+95)
-
-1979-01-03 21:00:28.600001: hash=58d64fdc19a3990a9bcd5b1642d1a31e2fcf0e68
-user='user0.13' description='step 0.13' length=190 offset=3951 (+95)
-
-1979-01-03 21:00:27.500001: hash=9e23abe95e8519f96b207dc227cc81c0e32b350a
-user='user0.12' description='step 0.12' length=190 offset=3753 (+95)
-
-1979-01-03 21:00:26.400001: hash=dbfb884fb3c8d41c62a25b08682997a3b994ae27
-user='user0.11' description='step 0.11' length=190 offset=3555 (+95)
-
-1979-01-03 21:00:25.300001: hash=08d94d9c33ef5b6b298fee1227ad5e793706d97a
-user='user0.10' description='step 0.10' length=190 offset=3357 (+95)
-
-1979-01-03 21:00:24.200001: hash=d9526d1e85dac9785bb689781a65ff0b2e6eecc5
-user='user0.9' description='step 0.9' length=187 offset=3162 (+93)
-
-1979-01-03 21:00:23.100000: hash=bb402ec5dd9c412adce959b62b5490f5ba97f116
-user='user0.8' description='step 0.8' length=442 offset=2712 (+93)
-
-1979-01-03 21:00:22.000000: hash=96877aeb66269349dc15e5108adc1bc3d66cb25b
-user='user0.7' description='step 0.7' length=187 offset=2517 (+93)
-
-1979-01-03 21:00:20.900000: hash=ee1c47b451653e98ba04dd24e2c348635201238b
-user='user0.6' description='step 0.6' length=187 offset=2322 (+93)
-
-1979-01-03 21:00:19.800000: hash=954218c38364ea2963568221de593a4cfbd3269e
-user='user0.5' description='step 0.5' length=423 offset=1891 (+93)
-
-1979-01-03 21:00:18.700000: hash=46814b832d84153c9372f2f80d24d756b2b36c2b
-user='user0.4' description='step 0.4' length=404 offset=1479 (+93)
-
-1979-01-03 21:00:17.600000: hash=9b7a1c17f57b2edf602dd940989b2eb622f53727
-user='user0.3' description='step 0.3' length=187 offset=1284 (+93)
-
-1979-01-03 21:00:16.500000: hash=6bb00a8cbe03fb054be5e5f9531f5bf32e793645
-user='user0.2' description='step 0.2' length=385 offset=891 (+93)
-
-1979-01-03 21:00:15.400000: hash=e323362838d636477bc3efca7fa320a84e379eee
-user='user0.1' description='step 0.1' length=366 offset=517 (+93)
-
-1979-01-03 21:00:14.300000: hash=01a03576dde5a9b1bdb0df6ac2746677dc1ebd2a
-user='user0.0' description='step 0.0' length=346 offset=163 (+93)
-
-1979-01-03 21:00:08.800000: hash=2838c499aba0cfd9c477d9a313e715c513c71dba
-user='' description='initial database creation' length=151 offset=4 (+48)
-
--- a/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle1/1.fsdump.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle1/1.fsdump.ok
--- a/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle1/1.fsdumpv.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle1/1.fsdumpv.ok
--- a/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle1/1.fstail.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle1/1.fstail.ok
--- a/go/zodb/storage/fs1/fs1tools/testdata/empty.fsdump.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/empty.fsdump.ok
--- a/go/zodb/storage/fs1/fs1tools/testdata/empty.fsdumpv.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/empty.fsdumpv.ok
--- a/go/zodb/storage/fs1/fs1tools/testdata/empty.fstail.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/empty.fstail.ok
--- a/go/zodb/storage/fs1/fs1tools/testdata/whiteout.fsdump.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/whiteout.fsdump.ok
-Trans #00000 tid=0000000000000017 time=1900-01-01 00:00:00.000000 offset=27
+Trans #00000 tid=0000000000000017 size=73 time=1900-01-01 00:00:00.000000 offset=27
    status=' ' user='' description=''
  data #00000 oid=0000000000000001 class=undo or abort of object creation
--- a/go/zodb/storage/fs1/fs1tools/testdata/whiteout.fsdumpv.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/whiteout.fsdumpv.ok
--- a/go/zodb/storage/fs1/fs1tools/testdata/whiteout.fstail.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/whiteout.fstail.ok
--- a/go/zodb/storage/fs1/fs1tools/testdata/1.fsdump.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/1.fsdump.ok
--- a/go/zodb/storage/fs1/fs1tools/testdata/1.fsdumpv.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/1.fsdumpv.ok
--- a/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle2/1.fstail.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle2/1.fstail.ok
--- a/go/zodb/zodbtools/testdata/empty.zdump.pyok
+++ b/go/zodb/zodbtools/testdata/empty.zdump.pyok
--- a/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle2/empty.fsdumpv.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle2/empty.fsdumpv.ok
+************************************************************
+file identifier: 'FS21'
--- a/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle2/empty.fstail.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle2/empty.fstail.ok
--- a/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle2/whiteout.fsdump.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle2/whiteout.fsdump.ok
+Trans #00000 tid=0000000000000017 size=73 time=1900-01-01 00:00:00.000000 offset=27
+    status=' ' user='' description=''
+  data #00000 oid=0000000000000001 class=undo or abort of object creation
--- a/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle2/whiteout.fsdumpv.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle2/whiteout.fsdumpv.ok
+************************************************************
+file identifier: 'FS21'
+============================================================
+offset: 4
+end pos: 77
+transaction id: 0000000000000017
+trec len: 73
+status: ' '
+user: ''
+description: ''
+len(extra): 0
+------------------------------------------------------------
+offset: 27
+oid: 0000000000000001
+revid: 0000000000000017
+previous record offset: 0
+transaction offset: 4
+len(data): 0
+backpointer: 0
+redundant trec len: 73
--- a/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle2/whiteout.fstail.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle2/whiteout.fstail.ok
+1900-01-01 00:00:00.000000: hash=e3e0ff6c686f2fa02266e744f6629d592d432061
+user='' description='' length=73 offset=4 (+23)
+
--- a/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle3/1.fsdump.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle3/1.fsdump.ok
--- a/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle3/1.fsdumpv.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle3/1.fsdumpv.ok
--- a/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle3/1.fstail.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle3/1.fstail.ok
--- a/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle3/empty.fsdump.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle3/empty.fsdump.ok
--- a/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle3/empty.fsdumpv.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle3/empty.fsdumpv.ok
+************************************************************
+file identifier: 'FS21'
--- a/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle3/empty.fstail.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle3/empty.fstail.ok
--- a/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle3/whiteout.fsdump.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle3/whiteout.fsdump.ok
+Trans #00000 tid=0000000000000017 size=73 time=1900-01-01 00:00:00.000000 offset=27
+    status=' ' user='' description=''
+  data #00000 oid=0000000000000001 class=undo or abort of object creation
--- a/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle3/whiteout.fsdumpv.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle3/whiteout.fsdumpv.ok
+************************************************************
+file identifier: 'FS21'
+============================================================
+offset: 4
+end pos: 77
+transaction id: 0000000000000017
+trec len: 73
+status: ' '
+user: ''
+description: ''
+len(extra): 0
+------------------------------------------------------------
+offset: 27
+oid: 0000000000000001
+revid: 0000000000000017
+previous record offset: 0
+transaction offset: 4
+len(data): 0
+backpointer: 0
+redundant trec len: 73
--- a/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle3/whiteout.fstail.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py2_pickle3/whiteout.fstail.ok
+1900-01-01 00:00:00.000000: hash=e3e0ff6c686f2fa02266e744f6629d592d432061
+user='' description='' length=73 offset=4 (+23)
+
--- a/go/zodb/storage/fs1/fs1tools/testdata/py3_pickle3/1.fsdump.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py3_pickle3/1.fsdump.ok
--- a/go/zodb/storage/fs1/fs1tools/testdata/py3_pickle3/1.fsdumpv.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py3_pickle3/1.fsdumpv.ok
--- a/go/zodb/storage/fs1/fs1tools/testdata/py3_pickle3/1.fstail.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py3_pickle3/1.fstail.ok
--- a/go/zodb/storage/fs1/fs1tools/testdata/py3_pickle3/empty.fsdump.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py3_pickle3/empty.fsdump.ok
--- a/go/zodb/storage/fs1/fs1tools/testdata/py3_pickle3/empty.fsdumpv.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py3_pickle3/empty.fsdumpv.ok
+************************************************************
+file identifier: b'FS30'
--- a/go/zodb/storage/fs1/fs1tools/testdata/py3_pickle3/empty.fstail.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py3_pickle3/empty.fstail.ok
--- a/go/zodb/storage/fs1/fs1tools/testdata/py3_pickle3/whiteout.fsdump.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py3_pickle3/whiteout.fsdump.ok
+Trans #00000 tid=0000000000000017 size=73 time=1900-01-01 00:00:00.000000 offset=27
+    status=' ' user=b'' description=b''
+  data #00000 oid=0000000000000001 class=undo or abort of object creation
--- a/go/zodb/storage/fs1/fs1tools/testdata/py3_pickle3/whiteout.fsdumpv.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py3_pickle3/whiteout.fsdumpv.ok
+************************************************************
+file identifier: b'FS30'
+============================================================
+offset: 4
+end pos: 77
+transaction id: 0000000000000017
+trec len: 73
+status: b' '
+user: ''
+description: ''
+len(extra): 0
+------------------------------------------------------------
+offset: 27
+oid: 0000000000000001
+revid: 0000000000000017
+previous record offset: 0
+transaction offset: 4
+len(data): 0
+backpointer: 0
+redundant trec len: 73
--- a/go/zodb/storage/fs1/fs1tools/testdata/py3_pickle3/whiteout.fstail.ok
+++ b/go/zodb/storage/fs1/fs1tools/testdata/py3_pickle3/whiteout.fstail.ok
+1900-01-01 00:00:00.000000: hash=e3e0ff6c686f2fa02266e744f6629d592d432061
+user='' description='' length=73 offset=4 (+23)
+
--- a/go/zodb/storage/fs1/fs1tools/ztestdata_test.go
+++ b/go/zodb/storage/fs1/fs1tools/ztestdata_test.go
+// Code generated by gen-testdata; DO NOT EDIT.
+package fs1tools
+func init() {
+	ztestdataReg.Register("py2_pickle1", "testdata/py2_pickle1", nil)
+	ztestdataReg.Register("py2_pickle2", "testdata/py2_pickle2", nil)
+	ztestdataReg.Register("py2_pickle3", "testdata/py2_pickle3", nil)
+	ztestdataReg.Register("py3_pickle3", "testdata/py3_pickle3", nil)
+}
--- a/go/zodb/storage/fs1/index.go
+++ b/go/zodb/storage/fs1/index.go
--- a/go/zodb/storage/fs1/index_test.go
+++ b/go/zodb/storage/fs1/index_test.go
--- a/go/zodb/storage/fs1/py/gen-testdata
+++ b/go/zodb/storage/fs1/py/gen-testdata
--- a/go/zodb/storage/fs1/py/indexcmp
+++ b/go/zodb/storage/fs1/py/indexcmp
-#!/usr/bin/env python2
-# Copyright (C) 2017  Nexedi SA and Contributors.
-#                     Kirill Smelkov <kirr@nexedi.com>
+#!/usr/bin/env python
+# Copyright (C) 2017-2024  Nexedi SA and Contributors.
+#                          Kirill Smelkov <kirr@nexedi.com>
 #
 # This program is free software: you can Use, Study, Modify and Redistribute
 # it under the terms of the GNU General Public License version 3, or (at your
@@ -17,23 +17,53 @@
 #
 # See COPYING file for full licensing terms.
 # See https://www.nexedi.com/licensing for rationale and options.
-"""compare two ZODB FileStorage v1 index files"""
+"""compare two ZODB FileStorage v1 index files.

-from ZODB.fsIndex import fsIndex
+If a file is prefixed with py2compat: for that file index loader is adjusted to
+handle py2->py3 compatibility by loading *STRING opcodes as bytes on py3.
+"""
+
+from __future__ import print_function
+
+from ZODB import fsIndex
+from golang import func, defer
 import sys

+
+@func
+def loadFSIndex(path):
+    # apply py2compat logic
+    if ':' in path:
+        mode, path = path.split(':', 1)
+        if mode != 'py2compat':
+            raise ValueError('invalid mode %r' % mode)
+        if sys.version_info.major > 2:
+            origUnpickler_init = fsIndex.Unpickler.__init__
+            def xUnpickler_init(self, f):
+                # ZODB._compat.Unpickler does not expose `encoding` argument
+                # -> use zodbpickle.Unpickler directly. This is ok to do
+                # because _compat.Unpickler __init__ is trivial:
+                # https://github.com/zopefoundation/ZODB/blob/5.8.1-0-g72cebe6bc/src/ZODB/_compat.py#L55-L57
+                fsIndex.Unpickler.__base__.__init__(self, f, encoding='bytes')
+            fsIndex.Unpickler.__init__ = xUnpickler_init
+            def _():
+                fsIndex.Unpickler.__init__ = origUnpickler_init
+            defer(_)
+
+    return fsIndex.fsIndex.load(path)
+
 def main():
    path1, path2 = sys.argv[1:]

-    d1 = fsIndex.load(path1)
-    d2 = fsIndex.load(path2)
+    d1 = loadFSIndex(path1)
+    d2 = loadFSIndex(path2)

    topPos1, fsi1 = d1["pos"], d1["index"]
    topPos2, fsi2 = d2["pos"], d2["index"]

-    #print topPos1, topPos2
-    #print fsi1.items()
-    #print fsi2.items()
+    #print(topPos1, topPos2)
+    #print(fsi1.items())
+    #print(fsi2.items())
    equal = (topPos1 == topPos2 and fsi1.items() == fsi2.items())
    sys.exit(int(not equal))


--- a/go/zodb/storage/fs1/testdata/.gitignore
+++ b/go/zodb/storage/fs1/testdata/.gitignore
--- a/go/zodb/storage/fs1/testdata/1.fs
+++ b/go/zodb/storage/fs1/testdata/1.fs
--- a/go/zodb/storage/fs1/testdata/1.fs.index
+++ b/go/zodb/storage/fs1/testdata/1.fs.index
--- a/go/zodb/storage/fs1/testdata/1voted.tail
+++ b/go/zodb/storage/fs1/testdata/1voted.tail
--- a/go/zodb/storage/fs1/testdata/py2_pickle1/1.fs
+++ b/go/zodb/storage/fs1/testdata/py2_pickle1/1.fs
--- a/go/zodb/storage/fs1/testdata/py2_pickle1/1.fs.index
+++ b/go/zodb/storage/fs1/testdata/py2_pickle1/1.fs.index
--- a/go/zodb/storage/fs1/testdata/py2_pickle1/1voted.tail
+++ b/go/zodb/storage/fs1/testdata/py2_pickle1/1voted.tail
--- a/go/zodb/storage/fs1/testdata/empty.fs
+++ b/go/zodb/storage/fs1/testdata/empty.fs
--- a/go/zodb/storage/fs1/testdata/whiteout.fs
+++ b/go/zodb/storage/fs1/testdata/whiteout.fs
--- a/go/zodb/storage/fs1/testdata/py2_pickle2/1.fs
+++ b/go/zodb/storage/fs1/testdata/py2_pickle2/1.fs
--- a/go/zodb/storage/fs1/testdata/py2_pickle2/1.fs.index
+++ b/go/zodb/storage/fs1/testdata/py2_pickle2/1.fs.index
--- a/go/zodb/storage/fs1/testdata/py2_pickle2/1voted.tail
+++ b/go/zodb/storage/fs1/testdata/py2_pickle2/1voted.tail
--- a/go/zodb/storage/fs1/testdata/py2_pickle2/empty.fs
+++ b/go/zodb/storage/fs1/testdata/py2_pickle2/empty.fs
--- a/go/zodb/storage/fs1/testdata/py2_pickle2/whiteout.fs
+++ b/go/zodb/storage/fs1/testdata/py2_pickle2/whiteout.fs
--- a/go/zodb/storage/fs1/testdata/py2_pickle3/1.fs
+++ b/go/zodb/storage/fs1/testdata/py2_pickle3/1.fs
--- a/go/zodb/storage/fs1/testdata/py2_pickle3/1.fs.index
+++ b/go/zodb/storage/fs1/testdata/py2_pickle3/1.fs.index
--- a/go/zodb/storage/fs1/testdata/py2_pickle3/1voted.tail
+++ b/go/zodb/storage/fs1/testdata/py2_pickle3/1voted.tail
--- a/go/zodb/storage/fs1/testdata/py2_pickle3/empty.fs
+++ b/go/zodb/storage/fs1/testdata/py2_pickle3/empty.fs
--- a/go/zodb/storage/fs1/testdata/py2_pickle3/whiteout.fs
+++ b/go/zodb/storage/fs1/testdata/py2_pickle3/whiteout.fs
--- a/go/zodb/storage/fs1/testdata/py3_pickle3/1.fs
+++ b/go/zodb/storage/fs1/testdata/py3_pickle3/1.fs
--- a/go/zodb/storage/fs1/testdata/py3_pickle3/1.fs.index
+++ b/go/zodb/storage/fs1/testdata/py3_pickle3/1.fs.index
--- a/go/zodb/storage/fs1/testdata/py3_pickle3/1voted.tail
+++ b/go/zodb/storage/fs1/testdata/py3_pickle3/1voted.tail
--- a/go/zodb/storage/fs1/testdata/py3_pickle3/empty.fs
+++ b/go/zodb/storage/fs1/testdata/py3_pickle3/empty.fs
--- a/go/zodb/storage/fs1/testdata/py3_pickle3/whiteout.fs
+++ b/go/zodb/storage/fs1/testdata/py3_pickle3/whiteout.fs
--- a/go/zodb/storage/fs1/ztestdata_expect_py2_pickle1_test.go
+++ b/go/zodb/storage/fs1/ztestdata_expect_py2_pickle1_test.go
--- a/go/zodb/storage/fs1/ztestdata_expect_py2_pickle2_test.go
+++ b/go/zodb/storage/fs1/ztestdata_expect_py2_pickle2_test.go
--- a/go/zodb/storage/fs1/ztestdata_expect_py2_pickle3_test.go
+++ b/go/zodb/storage/fs1/ztestdata_expect_py2_pickle3_test.go
--- a/go/zodb/storage/fs1/ztestdata_expect_py3_pickle3_test.go
+++ b/go/zodb/storage/fs1/ztestdata_expect_py3_pickle3_test.go
--- a/go/zodb/storage/fs1/ztestdata_expect_test.go
+++ b/go/zodb/storage/fs1/ztestdata_expect_test.go
--- a/go/zodb/storage/zeo/proto.go
+++ b/go/zodb/storage/zeo/proto.go
--- a/go/zodb/storage/zeo/zeo.go
+++ b/go/zodb/storage/zeo/zeo.go
--- a/go/zodb/storage/zeo/zeo_test.go
+++ b/go/zodb/storage/zeo/zeo_test.go
--- a/go/zodb/testdata/.gitignore
+++ b/go/zodb/testdata/.gitignore
--- a/go/zodb/testdata/py2_pickle1/data.fs
+++ b/go/zodb/testdata/py2_pickle1/data.fs
--- a/go/zodb/testdata/py2_pickle1/data.fs.index
+++ b/go/zodb/testdata/py2_pickle1/data.fs.index
--- a/go/zodb/testdata/py2_pickle2/data.fs
+++ b/go/zodb/testdata/py2_pickle2/data.fs
--- a/go/zodb/testdata/py2_pickle2/data.fs.index
+++ b/go/zodb/testdata/py2_pickle2/data.fs.index
--- a/go/zodb/testdata/py2_pickle3/data.fs
+++ b/go/zodb/testdata/py2_pickle3/data.fs
--- a/go/zodb/testdata/py2_pickle3/data.fs.index
+++ b/go/zodb/testdata/py2_pickle3/data.fs.index
--- a/go/zodb/testdata/py3_pickle3/data.fs
+++ b/go/zodb/testdata/py3_pickle3/data.fs
--- a/go/zodb/testdata/py3_pickle3/data.fs.index
+++ b/go/zodb/testdata/py3_pickle3/data.fs.index
--- a/go/zodb/zodb.go
+++ b/go/zodb/zodb.go
--- a/go/zodb/zodbtools/dump_test.go
+++ b/go/zodb/zodbtools/dump_test.go
--- a/go/zodb/zodbtools/gen-testdata
+++ b/go/zodb/zodbtools/gen-testdata
--- a/go/zodb/zodbtools/testdata/py2_pickle1/1.zdump.pyok
+++ b/go/zodb/zodbtools/testdata/py2_pickle1/1.zdump.pyok
--- a/go/zodb/zodbtools/testdata/py2_pickle1/empty.zdump.pyok
+++ b/go/zodb/zodbtools/testdata/py2_pickle1/empty.zdump.pyok
--- a/go/zodb/zodbtools/testdata/1.zdump.pyok
+++ b/go/zodb/zodbtools/testdata/1.zdump.pyok
--- a/go/zodb/zodbtools/testdata/py2_pickle2/empty.zdump.pyok
+++ b/go/zodb/zodbtools/testdata/py2_pickle2/empty.zdump.pyok
--- a/go/zodb/zodbtools/testdata/py2_pickle3/1.zdump.pyok
+++ b/go/zodb/zodbtools/testdata/py2_pickle3/1.zdump.pyok
--- a/go/zodb/zodbtools/testdata/py2_pickle3/empty.zdump.pyok
+++ b/go/zodb/zodbtools/testdata/py2_pickle3/empty.zdump.pyok
--- a/go/zodb/zodbtools/testdata/py3_pickle3/1.zdump.pyok
+++ b/go/zodb/zodbtools/testdata/py3_pickle3/1.zdump.pyok
--- a/go/zodb/zodbtools/testdata/py3_pickle3/empty.zdump.pyok
+++ b/go/zodb/zodbtools/testdata/py3_pickle3/empty.zdump.pyok
--- a/go/zodb/zodbtools/ztestdata_test.go
+++ b/go/zodb/zodbtools/ztestdata_test.go
--- a/go/zodb/ztestdata_py2_pickle1_x_test.go
+++ b/go/zodb/ztestdata_py2_pickle1_x_test.go
--- a/go/zodb/ztestdata_py2_pickle2_x_test.go
+++ b/go/zodb/ztestdata_py2_pickle2_x_test.go
--- a/go/zodb/ztestdata_py2_pickle3_x_test.go
+++ b/go/zodb/ztestdata_py2_pickle3_x_test.go
--- a/go/zodb/ztestdata_py3_pickle3_x_test.go
+++ b/go/zodb/ztestdata_py3_pickle3_x_test.go
--- a/go/zodb/ztestdata_pydata_py2_pickle1_test.go
+++ b/go/zodb/ztestdata_pydata_py2_pickle1_test.go
--- a/go/zodb/ztestdata_pydata_py2_pickle2_test.go
+++ b/go/zodb/ztestdata_pydata_py2_pickle2_test.go
--- a/go/zodb/ztestdata_pydata_py2_pickle3_test.go
+++ b/go/zodb/ztestdata_pydata_py2_pickle3_test.go
--- a/go/zodb/ztestdata_pydata_py3_pickle3_test.go
+++ b/go/zodb/ztestdata_pydata_py3_pickle3_test.go
--- a/go/zodb/ztestdata_pydata_test.go
+++ b/go/zodb/ztestdata_pydata_test.go