go/neo/t/neotest: ZODB benchmarks

Add to neotest zbench-local and zbench-cluster commands that perform
ZODB benchmarks on FileStorage, ZEO and NEO with Python and Go clients
either locally, or with a server and client running on 2 different nodes.

There are 2 client programs: tzodb.py and tzodb.go which for now compute
hash of whole latest objects stream in a ZODB database. On server side
neotest is taught to launch ZEO and various NEO clusters and to execute
client load on them.

Two test datasets are used: wczblk1-8 - the dataset with wendelin.core ZBlk1
objects covering 8M array, and prod1-1024 - synthethic dataset that tries to
represent regular ERP5 instance. Both datasets are very small and so we can
assume they reside completely in server disk cache while running benchmarks.
Benchmark timings will thus give pure storage software processing latency, as
pagecache hit time is on par, or less, to 1µs.

Example output:

	x/src/lab.nexedi.com/kirr/neo/go/neo/t$ ./neotest zbench-local
	dataset:	wczblk1-8
	node:
	cluster:	deco

	*** generating fs1 data...
	I: RAM:  7.47GB
	I: WORK: 0.01GB
	gen signal t=0...1.05e+06  float64  (= 0.01GB)
	gen signal blk [0:1048576]  (100.0%)
	VIRT: 297 MB	RSS: 48MB

	*** generating sqlite data...
	I: RAM:  7.47GB
	I: WORK: 0.01GB
	gen signal t=0...1.05e+06  float64  (= 0.01GB)
	gen signal blk [0:1048576]  (100.0%)
	VIRT: 386 MB	RSS: 58MB
	2018-07-10 19:57:35.7065 ERROR     NEO        [           app: 91] primary master is down
	Cluster state changed

	*** generating sql data...
	2018-07-10 19:57:35 140115116649600 [Note] /usr/sbin/mysqld (mysqld 10.1.29-MariaDB-6+b1) starting as process 27574 ...
	2018-07-10 19:57:39 140205509999744 [Note] /usr/sbin/mysqld (mysqld 10.1.29-MariaDB-6+b1) starting as process 27603 ...
	2018-07-10 19:57:42 139692109810816 [Note] /usr/sbin/mysqld (mysqld 10.1.29-MariaDB-6+b1) starting as process 27633 ...
	2018-07-10 19:57:45 139759221546112 [Note] mysqld (mysqld 10.1.29-MariaDB-6+b1) starting as process 27662 ...
	I: RAM:  7.47GB
	I: WORK: 0.01GB
	gen signal t=0...1.05e+06  float64  (= 0.01GB)
	gen signal blk [0:1048576]  (100.0%)
	VIRT: 387 MB	RSS: 59MB
	2018-07-10 19:57:48.2565 ERROR     NEO        [           app: 91] primary master is down
	Cluster state changed

	*** FileStorage
	Benchmarkfs1-zhash.py 2127 16.3 µs/object	# crc32:14640593  nread=8540363  t=0.035s	# POLL·2 C1·73 C1E·38 C3·12 C6·36 C7s·0 C8·112 C9·0 C10·62
	Benchmarkfs1-zhash.py 2127 16.6 µs/object	# crc32:14640593  nread=8540363  t=0.035s	# POLL·0 C1·113 C1E·21 C3·16 C6·56 C7s·0 C8·136 C9·0 C10·41
	Benchmarkfs1-zhash.py 2127 15.9 µs/object	# crc32:14640593  nread=8540363  t=0.034s	# POLL·0 C1·71 C1E·36 C3·22 C6·50 C7s·0 C8·167 C9·0 C10·47
	Benchmarkfs1-zhash.py 2127 15.9 µs/object	# crc32:14640593  nread=8540363  t=0.034s	# POLL·0 C1·77 C1E·32 C3·11 C6·55 C7s·0 C8·184 C9·0 C10·31
	Benchmarkfs1-zhash.py 2127 16.0 µs/object	# crc32:14640593  nread=8540363  t=0.034s	# POLL·0 C1·78 C1E·15 C3·12 C6·51 C7s·0 C8·140 C9·0 C10·44

	# 16 clients in parallel
	Benchmarkfs1-zhash.py·P16 2127 129.0 µs/object	# crc32:14640593  nread=8540363  t=0.274s
	Benchmarkfs1-zhash.py·P16 2127 132.6 µs/object	# crc32:14640593  nread=8540363  t=0.282s
	Benchmarkfs1-zhash.py·P16 2127 135.0 µs/object	# crc32:14640593  nread=8540363  t=0.287s
	Benchmarkfs1-zhash.py·P16 2127 135.3 µs/object	# crc32:14640593  nread=8540363  t=0.288s
	Benchmarkfs1-zhash.py·P16 2127 136.6 µs/object	# crc32:14640593  nread=8540363  t=0.291s
	Benchmarkfs1-zhash.py·P16 2127 122.8 µs/object	# crc32:14640593  nread=8540363  t=0.261s
	Benchmarkfs1-zhash.py·P16 2127 130.9 µs/object	# crc32:14640593  nread=8540363  t=0.279s
	Benchmarkfs1-zhash.py·P16 2127 126.4 µs/object	# crc32:14640593  nread=8540363  t=0.269s
	Benchmarkfs1-zhash.py·P16 2127 125.8 µs/object	# crc32:14640593  nread=8540363  t=0.268s
	Benchmarkfs1-zhash.py·P16 2127 108.3 µs/object	# crc32:14640593  nread=8540363  t=0.230s
	Benchmarkfs1-zhash.py·P16 2127 131.0 µs/object	# crc32:14640593  nread=8540363  t=0.279s
	Benchmarkfs1-zhash.py·P16 2127 124.1 µs/object	# crc32:14640593  nread=8540363  t=0.264s
	Benchmarkfs1-zhash.py·P16 2127 129.3 µs/object	# crc32:14640593  nread=8540363  t=0.275s
	Benchmarkfs1-zhash.py·P16 2127 125.0 µs/object	# crc32:14640593  nread=8540363  t=0.266s
	Benchmarkfs1-zhash.py·P16 2127 131.5 µs/object	# crc32:14640593  nread=8540363  t=0.280s
	Benchmarkfs1-zhash.py·P16 2127 131.4 µs/object	# crc32:14640593  nread=8540363  t=0.280s
	# POLL·0 C1·4 C1E·13 C3·11 C6·79 C7s·0 C8·14 C9·0 C10·0

	...

And its summary via benchstat:

	x/src/lab.nexedi.com/kirr/neo/go/neo/t$ benchstat -split node,cluster,dataset x.log
	name                                 time/object
	cluster:deco dataset:wczblk1-8
	fs1-zhash.py                         16.1µs ± 3%
	fs1-zhash.py·P16                      130µs ± 5%
	fs1-zhash.go                         3.00µs ±10%
	fs1-zhash.go+prefetch128             3.40µs ±18%
	fs1-zhash.go·P16                     10.2µs ±71%
	zeo/py/fs1-zhash.py                   336µs ± 3%
	zeo/py/fs1-zhash.py·P16              3.22ms ± 6%
	zeo/py/fs1-zhash.go                   112µs ± 2%
	zeo/py/fs1-zhash.go+prefetch128      60.9µs ± 1%
	zeo/py/fs1-zhash.go·P16              1.07ms ± 5%
	neo/py(!log)/sqlite·P1-zhash.py       291µs ± 2%
	neo/py(!log)/sqlite·P1-zhash.py·P16  2.86ms ± 1%
	neo/py(!log)/sql·P1-zhash.py          318µs ± 4%
	neo/py(!log)/sql·P1-zhash.py·P16     3.99ms ± 0%
	cluster:deco dataset:prod1-1024
	fs1-zhash.py                         12.3µs ± 1%
	fs1-zhash.py·P16                      106µs ±10%
	fs1-zhash.go                         2.56µs ±10%
	fs1-zhash.go+prefetch128             2.68µs ± 8%
	fs1-zhash.go·P16                     9.48µs ±43%
	zeo/py/fs1-zhash.py                   319µs ± 3%
	zeo/py/fs1-zhash.py·P16              3.13ms ± 3%
	zeo/py/fs1-zhash.go                   101µs ± 5%
	zeo/py/fs1-zhash.go+prefetch128      56.9µs ± 1%
	zeo/py/fs1-zhash.go·P16              1.19ms ± 4%
	neo/py(!log)/sqlite·P1-zhash.py       281µs ± 3%
	neo/py(!log)/sqlite·P1-zhash.py·P16  2.80ms ± 1%
	neo/py(!log)/sql·P1-zhash.py          316µs ± 1%
	neo/py(!log)/sql·P1-zhash.py·P16     3.91ms ± 1%

Since there is no NEO/go support yet, corresponding neotest parts are merged,
but commented-out with appropriate remark.

Parallel access is simulated with spawning many OS processes for now.
This will change in the nearby followup patch to zwrk.

Results of ZODB benchmarking were discussed in

	http://navytux.spb.ru/~kirr/neo.html#performance-tests		, and
	http://navytux.spb.ru/~kirr/neo.html#results-and-discussion

Some draft history related to this patch:

	lab.nexedi.com/kirr/neo/commit/e0d875bc	X neotest: Teach it to benchmark NEO with storage partitioned to several nodes
	lab.nexedi.com/kirr/neo/commit/590f0a46	X neo/py uses n(replica) as n(real-replica) - 1
	lab.nexedi.com/kirr/neo/commit/b655da26	X save time not benchmarking things we do not show
	lab.nexedi.com/kirr/neo/commit/f834f40d	X zhash: Show N(obj) read, not 1, in place of N(iter)
	lab.nexedi.com/kirr/neo/commit/a16e8d52	X teach golang to access ZEO
	lab.nexedi.com/kirr/neo/commit/b9827725	X switch to using no compression, because this way it is more fair for comparing storage latencies
	lab.nexedi.com/kirr/neo/commit/c0067335	X neotest: Don't depend on killall
	lab.nexedi.com/kirr/neo/commit/2bcd6ebb	X neotest: add zbench-local & zbench-cluster subcomands
	lab.nexedi.com/kirr/neo/commit/fb165ad9	X neotest: Also benchmark NEO/py with logging disabled
	lab.nexedi.com/kirr/neo/commit/2118ba38	X neotest: Help mysqlk_install_db find its basedir under SlapOS
	lab.nexedi.com/kirr/neo/commit/80eaa05e	X zgenprod1 tool
	lab.nexedi.com/kirr/neo/commit/eb0e516f	X check hash result and error if mismatch (zhash.* part); neotest part pending
	lab.nexedi.com/kirr/neo/commit/046370db	X benchify rest of bench-cluster
	lab.nexedi.com/kirr/neo/commit/2d13818e	X bench-local + zhash: Add output in std bench format
	lab.nexedi.com/kirr/neo/commit/1d692a3b	X add NEO/go with SHA1 disabled (both Sgo and Cgo to regular benchmarks)

go/neo/t/.gitignore

@@ -2,4 +2,6 @@
 /var
 /tcpu
 /tcpu_go
+/tzodb
+/tzodb_go
 /ioping.tmp

go/neo/t/neotest

@@ -22,6 +22,8 @@
 
 set -o pipefail
 
+# FIXME warn if/when thermal throttling activates
+
 # ---- deploy NEO for tests/benchmarks at a node ----
 
 die() {
@@ -85,6 +87,8 @@ EOF
 
 	. env.sh
 
+	pip install git+https://lab.nexedi.com/nexedi/wendelin.core.git@master	# XXX does not show git in ver
+	pip install zodbtools
 	pip install pygolang	# for tcpu.py
 
 	mkdir -p src/lab.nexedi.com/kirr
@@ -177,6 +181,19 @@ init_net() {
 
 	# prefer ipv4 for now
 	myaddr="${myaddr4v[0]}"
+
+	# port allocations ([] works for IPv4 too)
+	Abind=[$myaddr]:5551	# NEO admin
+	Mbind=[$myaddr]:5552	# NEO master
+	Zbind=[$myaddr]:5553	# ZEO
+
+}
+
+# Sbind n	- return bind address for storage #n	(n=1...)
+Sbind() {
+	# for NEO storage. bind not strictly needed but this way we also make sure
+	# no 2 storages are started at the same time.
+	echo "[$myaddr]:$((5554 + $1 - 1))"
 }
 
 # init_fs	- do initial disk allocations
@@ -185,10 +202,82 @@ init_fs() {
 	var=`pwd`/var;		mkdir -p $var
 }
 
+dataset=	# name of current dataset
+dataset_size=	# size parameter for current dataset
+ds=		# top-level dir of current dataset
+neocluster=	# NEO cluster name (different for different datasets to avoid confusion)
+
+# switch_dataset <name> <size>	- switch benchmarking dataset to named with size
+switch_dataset() {
+	dataset=$1
+	dataset_size=$2
+	ds=`echo $dataset $dataset_size |tr ' ' '-'`	# wczblk1 8 -> wczblk1-8
+	neocluster=pygotest-$ds
+	echo -e "\ndataset:\t$ds"
+	ds=$var/$ds
+
+	fs1=$ds/fs1;		mkdir -p $fs1		# FileStorage (and so ZEO and NEO/go) data
+}
+
+Pneo=1		# N(partition) for NEO (= N(S) nodes to run)
+# XXX switch_neo_P (like switch_dataset)
+
+
+# neolite n	- NEO: sqlite for partition n
+neolite() {
+	echo $ds/neo.sqlite·P$Pneo-$1
+}
+
+# neosql n	- NEO: mariadb directory for partition n
+neosql() {
+	local d=$ds/neo.sql·P$Pneo-$1;	mkdir -p $d
+	echo $d
+}
+
+# mycnf n	- NEO: mariadb config for partition n
+mycnf() {
+	echo `neosql $1`/mariadb.cnf
+}
+
+# mysock n	- NEO: mariadb socket for partition n
+mysock() {
+	echo $(realpath `neosql $1`)/my.sock
+}
+
+# mylog n	- NEO: mariadb log for partition n
+mylog() {
+	echo $log/mdb.log·P$Pneo-$1
+}
+
+
+# foreach_dataset <command>	- run command once for each dataset serially
+foreach_dataset() {
+	for d in "${datasetv[@]}" ; do
+		switch_dataset $d
+		"$@"
+	done
+}
+
+# xneoctl ...	- control started NEO cluster
+xneoctl() {
+	neoctl -a $Abind "$@"
+}
+
+# xmysql ...	- control all started MariaDB servers
+xmysql() {
+	for i in `seq $Pneo`; do
+		mysql --defaults-file=`mycnf $i` "$@"
+	done
+}
+
 # if we are abnormally terminating
 install_trap() {
 	trap 'set +e
 echo "E: abnormal termination - stopping..."
+xneoctl set cluster stopping
+sleep 1
+xmysql -e "SHUTDOWN"
+sleep 1
 j="$(jobs -p)"
 test -z "$j" && exit
 echo "E: killing left jobs..."
@@ -196,6 +285,247 @@ jobs -l
 kill $j' EXIT
 }
 
+# ---- start NEO/ZEO nodes ----
+
+# neopy_log logname	- `--logfile ...` argument for logging from a NEO/py node
+# empty if $X_NEOPY_LOG_SKIP set
+neopy_log() {
+	test -z "$X_NEOPY_LOG_SKIP" || return
+	echo --logfile=$log/$1.log
+}
+
+# M{py,go} n ...	- spawn master with N(partition) = n
+Mpy() {
+	local n=$1
+	shift
+
+	# XXX neo/py meaning for --replicas is `n(real-replica) - 1`
+	exec -a Mpy \
+		neomaster --cluster=$neocluster --bind=$Mbind --masters=$Mbind	\
+		--replicas 0 --partitions $n `neopy_log Mpy` "$@" &
+}
+
+Mgo() {
+	exec -a Mgo \
+		neo --log_dir=$log master -cluster=$neocluster -bind=$Mbind "$@" &
+}
+
+# Spy <n> ...	- spawn NEO/py storage handling partition #n
+Spy() {
+	local n=$1
+	shift
+
+	# --adapter=...
+	# --database=...
+	# --engine=...
+	exec -a Spy \
+		neostorage --cluster=$neocluster --bind=`Sbind $n` --masters=$Mbind `neopy_log Spy` "$@" &
+}
+
+# Sgo <n>	- spawn NEO/go storage
+Sgo() {
+	local n=$1
+	shift
+	test $n = 1 || die "Sgo: TODO ·P<n>"
+
+	# -alsologtostderr
+	exec -a Sgo \
+		neo -log_dir=$log storage -cluster=$neocluster -bind=`Sbind $n` -masters=$Mbind "$@" &
+}
+
+# Apy ...	- spawn NEO/py admin
+Apy() {
+	exec -a Apy \
+		neoadmin --cluster=$neocluster --bind=$Abind --masters=$Mbind `neopy_log Apy` "$@" &
+}
+
+# Zpy <data.fs> ...	- spawn ZEO
+Zpy() {
+	exec -a Zpy \
+		runzeo --address $Zbind --filename "$@" 2>>$log/Zpy.log &
+}
+
+
+# ---- start NEO clusters ----
+
+# spawn NEO/go cluster (Sgo+Mpy+Apy) working on data.fs
+NEOgofs1() {
+	Mpy 1 --autostart=1
+	Sgo 1 fs1://$fs1/data.fs
+	Apy
+}
+
+# spawn NEO/go cluster (Sgo+Mpy+Apy) working on sqlite db
+NEOgolite() {
+	test $Pneo == 1 || die "NEOgolite: Sgo not ready to handle ·P<n>"
+	Mpy 1 --autostart=1
+	Sgo 1 sqlite://`neolite 1`
+	Apy
+}
+
+# spawn NEO/py cluster working on sqlite db
+NEOpylite() {
+	Mpy $Pneo --autostart=$Pneo
+	for i in `seq $Pneo`; do
+		Spy $i --adapter=SQLite --database=`neolite $i`
+	done
+	Apy
+}
+
+# spawn NEO/py cluster working on mariadb
+NEOpysql() {
+	for i in `seq $Pneo`; do
+		MDB $i
+	done
+	sleep 1	# XXX fragile
+	xmysql -e "CREATE DATABASE IF NOT EXISTS neo"
+
+	Mpy $Pneo --autostart=$Pneo
+	for i in `seq $Pneo`; do
+		Spy $i --adapter=MySQL --engine=InnoDB --database=root@neo`mysock $i`
+	done
+	Apy
+}
+
+
+# MDB <n>	- setup/spawn mariadb for partition n
+MDB() {
+	local n=$1
+	cat >`mycnf $n` <<EOF
+[mysqld]
+skip_networking
+socket		= `mysock $n`
+datadir		= `neosql $n`/data
+log_error	= `mylog $n`
+
+# the following comes from
+# https://lab.nexedi.com/nexedi/slapos/blob/bd197876/software/neoppod/my.cnf.in#L18
+# ---- 8< ----
+
+# kirr: disabled
+#plugin-load = ha_tokudb;ha_rocksdb
+
+log_warnings = 1
+disable-log-bin
+
+## The following settings come from ERP5 configuration.
+
+max_allowed_packet = 128M
+query_cache_size = 32M
+innodb_locks_unsafe_for_binlog = 1
+
+# Some dangerous settings you may want to uncomment temporarily
+# if you only want performance or less disk access.
+#innodb_flush_log_at_trx_commit = 0
+#innodb_flush_method = nosync
+#innodb_doublewrite = 0
+#sync_frm = 0
+
+# Extra parameters.
+log_slow_verbosity = explain,query_plan
+# kirr: rocksb disabled
+# rocksdb_block_cache_size = 10G
+# rocksdb_max_background_compactions = 3
+long_query_time = 1
+innodb_file_per_table = 1
+
+# Force utf8 usage
+collation_server = utf8_unicode_ci
+character_set_server = utf8
+skip_character_set_client_handshake
+
+[client]
+socket = `mysock $n`
+user = root
+EOF
+
+	# setup system tables on first run
+	if ! test -e `neosql $n`/data ; then
+		# XXX --cross-bootstrap only to avoid final large print notice
+		# XXX but cross-bootstrap filters out current host name from installed tables - is it ok?
+		mysql_bin=$(dirname `which mysql`)	# XXX under slapos mysql_install_db cannot find its base automatically
+		mysql_install_db --basedir=${mysql_bin%/bin} --defaults-file=`mycnf $n` --cross-bootstrap
+	fi
+
+	mysqld --defaults-file=`mycnf $n` &
+}
+
+# ---- generate test data ----
+
+# dataset 1: wendelin.core array with many small (4K) objects
+wczblk1_gen_data() {
+	url=$1
+	size=$2
+	WENDELIN_CORE_ZBLK_FMT=ZBlk1 demo-zbigarray --worksize=$size gen $url
+}
+
+# dataset 2: synthetic test database according to NEO PROD1 statistics
+prod1_gen_data() {
+	url=$1
+	size=$2
+	zgenprod1.py $url $size
+}
+
+
+# run benchmarks on this datasets:
+datasetv=(
+	"wczblk1 8"	# size: array size (MB)		XXX raise 32,64,512...
+	"prod1   1024"	# size: # of transactions	XXX raise size
+	# TODO + wczblk0
+)
+
+# generate data in data.fs
+GENfs() {
+	test -e $ds/generated.fs && return
+	echo -e '\n*** generating fs1 data...'
+	${dataset}_gen_data $fs1/data.fs $dataset_size
+	sync
+	touch $ds/generated.fs
+
+	# remember correct hash to later check in benchmarks
+	# crc32:1552c530   ; oid=0..2127  nread=8534126  t=0.033s (15.7μs / object)  x=zhash.py
+	tzodb.py zhash --$zhashfunc $fs1/data.fs |awk '{print $1}' >$ds/zhash.ok
+}
+
+# generate data in sqlite
+GENsqlite() {
+	test -e $ds/generated.sqlite·P$Pneo && return
+	echo -e '\n*** generating sqlite data...'
+	NEOpylite
+	# NOTE compression is disabled because when benchmarking server latency
+	# we do not want the time client(s) take to decompress data to interfere.
+	${dataset}_gen_data neo://$neocluster@$Mbind?compress=false $dataset_size
+	xneoctl set cluster stopping
+	wait	# XXX fragile - won't work if there are children spawned outside
+	sync
+	touch $ds/generated.sqlite·P$Pneo
+}
+
+# generate data in mariadb
+GENsql() {
+	test -e $ds/generated.sql·P$Pneo && return
+	echo -e '\n*** generating sql data...'
+	NEOpysql
+	# NOTE compression is disabled - see ^^^ (sqlite) for rationale.
+	${dataset}_gen_data neo://$neocluster@$Mbind?compress=false $dataset_size
+	xneoctl set cluster stopping
+	sleep 1	# XXX fragile
+	xmysql -e "SHUTDOWN"
+	wait	# XXX fragile
+	sync
+	touch $ds/generated.sql·P$Pneo
+}
+
+# generate all test databases
+gen_data() {
+	GENfs
+	GENsqlite
+	GENsql
+	wait
+	sync
+}
+
+
 # ---- information about system ----
 
 # pyver <egg> (<showas>) - print version of egg
@@ -540,6 +870,7 @@ system_info() {
 	pyver zodb
 	pyver zeo
 	pyver mysqlclient
+	pyver wendelin.core
 }
 
 
@@ -600,6 +931,7 @@ cpustat() {
 }
 
 Nrun=5			# repeat benchmarks N time
+Npar=16		# run so many parallel clients in parallel phase
 
 #profile=
 profile=cpustat
@@ -611,6 +943,20 @@ nrun() {
 	done
 }
 
+# nrunpar ...	- run $Npar ... instances in parallel and wait for completion
+nrunpar() {
+	$profile _nrunpar "$@"
+}
+
+_nrunpar() {
+	local jobv
+	for i in `seq $Npar`; do
+		"$@" &
+		jobv="$jobv $!"
+	done
+	wait $jobv
+}
+
 # bench_cpu	- microbenchmark CPU
 bench_cpu() {
 	echo -ne "node:\t"; xhostname
@@ -760,6 +1106,253 @@ Benchmark$1-avg 1 \\2 ms/op\
 }
 
 
+# hash function to compute via zhash in tests/benchmarks
+zhashfunc=crc32	# sha1, adler32, null, ...
+
+# zbench <url> <topic> <zhashok>	- run ZODB client benchmarks against URL
+zbench() {
+	url=$1
+	topic=$2
+	zhashok=$3
+
+	#nrun time demo-zbigarray read $url
+
+	nrun tzodb.py zhash --check=$zhashok --bench=$topic-%s --$zhashfunc $url
+	echo -e "\n# ${Npar} clients in parallel"
+	nrunpar tzodb.py zhash --check=$zhashok --bench=$topic-%s·P$Npar --$zhashfunc $url
+	echo
+	zbench_go $url $topic $zhashok
+}
+
+# go-only part of zbench
+zbench_go() {
+	url=$1
+	topic=$2-%s
+	zhashok=$3
+	test "$X_NEOGO_SHA1_SKIP" = y && topic="$topic(!sha1)" || :
+
+	# XXX remove once NEO/go client support lands in
+	if [[ $url == neo://* ]]; then
+		echo "(skipping tzodb.go on NEO -- Cgo does not yet support neo:// protocol)"
+		return
+	fi
+
+	nrun tzodb_go zhash -check=$zhashok -bench=$topic -$zhashfunc $url
+	nrun tzodb_go zhash -check=$zhashok -bench=$topic -$zhashfunc -useprefetch $url
+
+	echo -e "\n# ${Npar} clients in parallel"
+	nrunpar tzodb_go zhash -check=$zhashok -bench=$topic·P$Npar -$zhashfunc $url
+}
+
+
+# command: benchmark ZODB client/server over localhost
+cmd_zbench-local() {
+	install_trap
+
+	foreach_dataset zbench_local
+
+	# all ok
+	trap - EXIT
+}
+
+# zodb part of cmd_bench-local
+zbench_local() {
+	echo "node:"
+	echo -ne "cluster:\t"; xhostname
+
+	gen_data
+	zhashok=`cat $ds/zhash.ok`
+
+	echo -e "\n*** FileStorage"
+	zbench $fs1/data.fs fs1 $zhashok
+
+	echo -e "\n*** ZEO"
+	Zpy $fs1/data.fs
+	Zpy_job=$!
+	zbench zeo://$Zbind zeo/py/fs1 $zhashok
+	kill $Zpy_job
+	wait $Zpy_job
+
+	# XXX save time - we show only neo/py(!log)/sqlite
+	#echo -e "\n*** NEO/py sqlite"
+	#NEOpylite
+	#zbench neo://$neocluster@$Mbind neo/py/sqlite·P$Pneo $zhashok
+	#xneoctl set cluster stopping
+	#wait
+
+	# XXX JM asked to also have NEO/py with logging disabled
+	echo -e "\n*** NEO/py sqlite (logging disabled)"
+	X_NEOPY_LOG_SKIP=y NEOpylite
+	zbench neo://$neocluster@$Mbind "neo/py(!log)/sqlite·P$Pneo" $zhashok
+	xneoctl set cluster stopping
+	wait
+
+	# XXX save time - we show only neo/py(!log)/sql
+	#echo -e "\n*** NEO/py sql"
+	#NEOpysql
+	#zbench neo://$neocluster@$Mbind neo/py/sql·P$Pneo $zhashok
+	#xneoctl set cluster stopping
+	#xmysql -e "SHUTDOWN"
+	#wait
+
+	# XXX JM asked to also have NEO/py with logging disabled
+	echo -e "\n*** NEO/py sql (logging disabled)"
+	X_NEOPY_LOG_SKIP=y NEOpysql
+	zbench neo://$neocluster@$Mbind "neo/py(!log)/sql·P$Pneo" $zhashok
+	xneoctl set cluster stopping
+	xmysql -e "SHUTDOWN"
+	wait
+
+	# XXX reenable once NEO/go server support lands in
+: <<EOF
+	echo -e "\n*** NEO/go fs1"
+	NEOgofs1
+	zbench neo://$neocluster@$Mbind neo/go/fs1·P1 $zhashok
+	xneoctl set cluster stopping
+	wait
+
+	echo -e "\n*** NEO/go fs1 (sha1 disabled on: storage, client)"
+	X_NEOGO_SHA1_SKIP=y NEOgofs1
+	X_NEOGO_SHA1_SKIP=y zbench_go neo://$neocluster@$Mbind "neo/go/fs1(!sha1)·P1" $zhashok
+	xneoctl set cluster stopping
+	wait
+
+	echo -e "\n*** NEO/go sqlite"
+	if [ $Pneo == 1 ]; then
+		NEOgolite
+		zbench neo://$neocluster@$Mbind neo/go/sqlite·P$Pneo $zhashok
+		xneoctl set cluster stopping
+		wait
+	else
+		echo "# skipping (NEO/go sqlite not yet ready to handle ·P<n>"
+	fi
+
+	echo -e "\n*** NEO/go sqlite (sha1 disabled on: client)"
+	if [ $Pneo == 1 ]; then
+		NEOgolite
+		X_NEOGO_SHA1_SKIP=y zbench_go neo://$neocluster@$Mbind "neo/go/sqlite·P$Pneo" $zhashok
+		xneoctl set cluster stopping
+		wait
+	else
+		echo "# skipping (NEO/go sqlite not yet ready to handle ·P<n>"
+	fi
+EOF
+}
+
+# command: benchmark ZODB client/server over network
+cmd_zbench-cluster() {
+	url=$1
+	test -z "$url" && die "Usage: neotest zbench-cluster [user@]<host>:<path>"
+
+	install_trap
+
+	foreach_dataset zbench_cluster $url
+
+	# all ok
+	trap - EXIT
+}
+
+# zbench_cluster <url>	- zodb part of cmd_bench-cluster
+zbench_cluster() {
+	url=$1
+	peer=`hostof $url`
+
+	echo "node:"
+	echo -e "cluster:\t`xhostname`-$peer"
+
+	gen_data
+	zhashok=`cat $ds/zhash.ok`
+
+	echo -e "\n*** ZEO"
+	Zpy $fs1/data.fs
+	Zpy_job=$!
+	on $url ./neotest zbench-client zeo://$Zbind zeo/py/fs1 $zhashok
+	kill $Zpy_job
+	wait $Zpy_job
+
+	# XXX save time - we show only neo/py(!log)/sqlite
+	#echo -e "\n*** NEO/py sqlite"
+	#NEOpylite
+	#on $url ./neotest zbench-client neo://$neocluster@$Mbind neo/py/sqlite·P$Pneo $zhashok
+	#xneoctl set cluster stopping
+	#wait
+
+	# XXX JM asked to also have NEO/py with logging disabled
+	echo -e "\n*** NEO/py sqlite (logging disabled)"
+	X_NEOPY_LOG_SKIP=y NEOpylite
+	on $url ./neotest zbench-client neo://$neocluster@$Mbind "\\\"neo/py(!log)/sqlite\\\"·P$Pneo" $zhashok
+	xneoctl set cluster stopping
+	wait
+
+	# XXX save time - we show only neo/py(!log)/sql
+	#echo -e "\n*** NEO/py sql"
+	#NEOpysql
+	#on $url ./neotest zbench-client neo://$neocluster@$Mbind neo/py/sql·P$Pneo $zhashok
+	#xneoctl set cluster stopping
+	#xmysql -e "SHUTDOWN"
+	#wait
+
+	# XXX JM asked to also have NEO/py with logging disabled
+	echo -e "\n*** NEO/py sql (logging disabled)"
+	X_NEOPY_LOG_SKIP=y NEOpysql
+	on $url ./neotest zbench-client neo://$neocluster@$Mbind "\\\"neo/py(!log)/sql\\\"·P$Pneo" $zhashok
+	xneoctl set cluster stopping
+	xmysql -e "SHUTDOWN"
+	wait
+
+	# XXX reenable once NEO/go server support lands in
+: <<EOF
+	echo -e "\n*** NEO/go fs"
+	NEOgofs1
+	on $url ./neotest zbench-client neo://$neocluster@$Mbind neo/go/fs1·P1 $zhashok
+	xneoctl set cluster stopping
+	wait
+
+	echo -e "\n*** NEO/go fs1 (sha1 disabled on: storage, client)"
+	X_NEOGO_SHA1_SKIP=y NEOgofs1
+	on $url X_NEOGO_SHA1_SKIP=y ./neotest zbench-client --goonly neo://$neocluster@$Mbind "\\\"neo/go/fs1(!sha1)\\\"·P1" $zhashok
+	xneoctl set cluster stopping
+	wait
+
+	echo -e "\n*** NEO/go sqlite"
+	if [ $Pneo == 1 ]; then
+		NEOgolite
+		on $url ./neotest zbench-client neo://$neocluster@$Mbind neo/go/sqlite·P$Pneo $zhashok
+		xneoctl set cluster stopping
+		wait
+	else
+		echo "# skipping (NEO/go sqlite not yet ready to handle ·P<n>"
+	fi
+
+	echo -e "\n*** NEO/go sqlite (sha1 disabled on: client)"
+	if [ $Pneo == 1 ]; then
+		NEOgolite
+		on $url X_NEOGO_SHA1_SKIP=y ./neotest zbench-client --goonly neo://$neocluster@$Mbind "\\\"neo/go/sqlite\\\"·P$Pneo" $zhashok
+		xneoctl set cluster stopping
+		wait
+	else
+		echo "# skipping (NEO/go sqlite not yet ready to handle ·P<n>"
+	fi
+EOF
+}
+
+# command: benchmark client workload against separate server
+cmd_zbench-client() {
+	goonly=""
+	case "$1" in
+	--goonly)
+		goonly=y
+		shift
+		;;
+	esac
+
+	url=$1
+	topic=$2
+	zhashok=$3
+	test -z "$url" -o -z "$topic" -o -z "$zhashok" && die "Usage: neotest zbench-client <url> <topic> <zhashok>"
+
+	test -z "$goonly" && zbench $url $topic $zhashok || zbench_go $url $topic $zhashok
+}
 
 # command: benchmark local disk
 cmd_bench-disk() {
@@ -818,6 +1411,10 @@ The commands are:
 	bench-disk	benchmark local disk
 	bench-net	benchmark network
 
+	zbench-local	run ZODB benchmarks on localhost
+	zbench-cluster	run ZODB benchmarks via network
+	zbench-client	run ZODB client benchmarks against separate server (part of zbench-cluster)
+
 
 	deploy		deploy NEO & needed software for tests to remote host
 	deploy-local	deploy NEO & needed software for tests locally
@@ -845,6 +1442,9 @@ test-py)	f=(                     );;
 bench-cpu)	f=(build                );;
 bench-disk)	f=(          fs         );;
 bench-net)	f=(      net            );;
+zbench-local)	f=(build net fs         );;
+zbench-cluster)	f=(build net fs         );;
+zbench-client)	f=(build                );;
 
 info)		f=(                     );;
 info-local)	f=(      net            );;
@@ -865,13 +1465,14 @@ esac
 for flag in ${f[*]}; do
 	case "$flag" in
 	build)
-		# make sure tcpu* is on PATH (because we could be invoked from another dir)
+		# make sure tzodb*, tcpu* and zgenprod are on PATH (because we could be invoked from another dir)
 		X=$(cd `dirname $0` && pwd)
 		export PATH=$X:$PATH
 
 		# rebuild go bits
 		# neo/py, wendelin.core, ... - must be pip install'ed - `neotest deploy` cares about that
 		go install -v lab.nexedi.com/kirr/neo/go/...
+		go build -o $X/tzodb_go $X/tzodb.go
 		go build -o $X/tcpu_go $X/tcpu.go
 		;;
 

go/neo/t/tzodb.go

+// Copyright (C) 2017-2018  Nexedi SA and Contributors.
+//                          Kirill Smelkov <kirr@nexedi.com>
+//
+// This program is free software: you can Use, Study, Modify and Redistribute
+// it under the terms of the GNU General Public License version 3, or (at your
+// option) any later version, as published by the Free Software Foundation.
+//
+// You can also Link and Combine this program with other software covered by
+// the terms of any of the Free Software licenses or any of the Open Source
+// Initiative approved licenses and Convey the resulting work. Corresponding
+// source of such a combination shall include the source code for all other
+// software used.
+//
+// This program is distributed WITHOUT ANY WARRANTY; without even the implied
+// warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+//
+// See COPYING file for full licensing terms.
+// See https://www.nexedi.com/licensing for rationale and options.
+
+// +build ignore
+
+// tzodb - ZODB-related benchmarks
+package main
+
+import (
+	"context"
+	"crypto/sha1"
+	"crypto/sha256"
+	"crypto/sha512"
+	"flag"
+	"fmt"
+	"hash"
+	"hash/adler32"
+	"hash/crc32"
+	"io"
+	"log"
+	"os"
+	"time"
+
+	"lab.nexedi.com/kirr/go123/prog"
+	"lab.nexedi.com/kirr/go123/xerr"
+
+	"lab.nexedi.com/kirr/neo/go/zodb"
+	_ "lab.nexedi.com/kirr/neo/go/zodb/wks"
+
+	"github.com/pkg/errors"
+)
+
+// hasher is hash.Hash which also knows its name
+type hasher struct {
+	hash.Hash
+	name string
+}
+
+// hasher that discards data
+type nullHasher struct {}
+
+func (h nullHasher) Write(b []byte) (int, error) { return len(b), nil }
+func (h nullHasher) Sum(b []byte) []byte         { return []byte{0} }
+func (h nullHasher) Reset()                      {}
+func (h nullHasher) Size() int                   { return 1 }
+func (h nullHasher) BlockSize() int              { return 1 }
+
+
+// hashFlags installs common hash flags and returns function to retrieve selected hasher.
+func hashFlags(flags *flag.FlagSet) func() hasher {
+	fnull    := flags.Bool("null",	false, "don't compute hash - just read data")
+	fadler32 := flags.Bool("adler32",false, "compute Adler32 checksum")
+	fcrc32   := flags.Bool("crc32",	false, "compute CRC32 checksum")
+	fsha1    := flags.Bool("sha1",	false, "compute SHA1 cryptographic hash")
+	fsha256  := flags.Bool("sha256", false, "compute SHA256 cryptographic hash")
+	fsha512  := flags.Bool("sha512", false, "compute SHA512 cryptographic hash")
+
+	return func() hasher {
+		var h hasher
+		inith := func(name string, ctor func() hash.Hash) {
+			if h.name != "" {
+				log.Fatalf("duplicate hashes: %s and %s specified", h.name, name)
+			}
+
+			h.name = name
+			h.Hash = ctor()
+		}
+
+		switch {
+		case *fnull:	inith("null",	 func() hash.Hash { return nullHasher{} })
+		case *fadler32:	inith("adler32", func() hash.Hash { return adler32.New() })
+		case *fcrc32:	inith("crc32",   func() hash.Hash { return crc32.NewIEEE() })
+		case *fsha1:	inith("sha1",    sha1.New)
+		case *fsha256:	inith("sha256",  sha256.New)
+		case *fsha512:	inith("sha512",  sha512.New)
+		}
+
+		return h
+	}
+}
+
+
+// ----------------------------------------
+
+const zhashSummary = "compute hash of whole latest objects stream in a ZODB database."
+
+func zhashUsage(w io.Writer) {
+	fmt.Fprintf(w,
+`Usage: tzodb zhash [options] <url>
+`)
+}
+
+func zhashMain(argv []string) {
+	ctx := context.Background()
+	flags := flag.NewFlagSet("", flag.ExitOnError)
+	flags.Usage = func() { zhashUsage(os.Stderr); flags.PrintDefaults() }
+
+	fhash    := hashFlags(flags)
+	fcheck   := flags.String("check", "",   "verify resulting hash to be = expected")
+	fbench   := flags.String("bench", "",   "use benchmarking format for output")
+	useprefetch := flags.Bool("useprefetch", false, "prefetch loaded objects")
+
+	flags.Parse(argv[1:])
+
+	if flags.NArg() != 1 {
+		flags.Usage()
+		os.Exit(1)
+	}
+
+	url := flags.Arg(0)
+
+	h := fhash()
+	if h.Hash == nil {
+		log.Fatal("no hash function specified")
+	}
+
+	err := zhash(ctx, url, h, *useprefetch, *fbench, *fcheck)
+	if err != nil {
+		log.Fatal(err)
+	}
+}
+
+func zhash(ctx context.Context, url string, h hasher, useprefetch bool, bench, check string) (err error) {
+	defer xerr.Context(&err, "zhash")
+
+	stor, err := zodb.OpenStorage(ctx, url, &zodb.OpenOptions{ReadOnly: true})
+	if err != nil {
+		return err
+	}
+	defer func() {
+		err2 := stor.Close()
+		err = xerr.First(err, err2)
+	}()
+
+	const nprefetch = 128 // XXX -> 512 ?
+
+	// prefetchBlk prefetches block of nprefetch objects starting from xid
+	//var tprevLoadBlkStart time.Time
+	prefetchBlk := func(ctx context.Context, xid zodb.Xid) {
+		if !useprefetch {
+			return
+		}
+
+		//t1 := time.Now()
+		for i := 0; i < nprefetch; i++ {
+			//fmt.Printf("prefetch %v\n", xid)
+			stor.Prefetch(ctx, xid)
+			xid.Oid++
+		}
+		//t2 := time.Now()
+		//δt := t2.Sub(t1)
+		//fmt.Printf("tprefetch: %s", δt)
+		//if !tprevLoadBlkStart.IsZero() {
+		//	fmt.Printf("\ttprevload: %s", t1.Sub(tprevLoadBlkStart))
+		//}
+		//fmt.Printf("\n")
+		//
+		//tprevLoadBlkStart = t2
+	}
+
+
+
+	lastTid, err := stor.LastTid(ctx)
+	if err != nil {
+		return err
+	}
+
+	tstart := time.Now()
+
+	oid := zodb.Oid(0)
+	nread := 0
+loop:
+	for {
+		xid := zodb.Xid{Oid: oid, At: lastTid}
+		if xid.Oid % nprefetch == 0 {
+			prefetchBlk(ctx, xid)
+		}
+		buf, _, err := stor.Load(ctx, xid)
+		if err != nil {
+			switch errors.Cause(err).(type) {
+			case *zodb.NoObjectError:
+				break loop
+			default:
+				return err
+			}
+		}
+
+		h.Write(buf.Data)
+
+		//fmt.Fprintf(os.Stderr, "%d @%s\t%s: %x\n", uint(oid), serial, h.name, h.Sum(nil))
+		//fmt.Fprintf(os.Stderr, "\tdata: %x\n", buf.Data)
+
+		nread += len(buf.Data)
+		oid += 1
+
+		buf.Release()
+	}
+
+	tend := time.Now()
+	δt := tend.Sub(tstart)
+
+	x := "zhash.go"
+	if useprefetch {
+		x += fmt.Sprintf("+prefetch%d", nprefetch)
+	}
+	hresult := fmt.Sprintf("%s:%x", h.name, h.Sum(nil))
+	if bench == "" {
+		fmt.Printf("%s   ; oid=0..%d  nread=%d  t=%s (%s / object)  x=%s\n",
+			hresult, oid-1, nread, δt, δt / time.Duration(oid), x) // XXX /oid cast ?
+	} else {
+		topic := fmt.Sprintf(bench, x)	// XXX -> better text/template
+		fmt.Printf("Benchmark%s %d %.1f µs/object\t# %s  nread=%d  t=%s\n",
+			topic, oid-1, float64(δt) / float64(oid) / float64(time.Microsecond),
+			hresult, nread, δt)
+	}
+
+	if check != "" && hresult != check {
+		return fmt.Errorf("%s: hash mismatch: expected %s  ; got %s\t# x=%s", url, check, hresult, x)
+	}
+
+	return nil
+}
+
+
+// ----------------------------------------
+
+var commands = prog.CommandRegistry{
+	{"zhash", zhashSummary, zhashUsage, zhashMain},
+}
+
+func main() {
+	log.SetPrefix("tzodb: ")
+	log.SetFlags(0)
+
+	prog := prog.MainProg{
+		Name:       "tzodb",
+		Summary:    "tzodb is a tool to run ZODB-related benchmarks",
+		Commands:   commands,
+		HelpTopics: nil,
+	}
+
+	prog.Main()
+}

go/neo/t/tzodb.py

+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# Copyright (C) 2017-2018  Nexedi SA and Contributors.
+#                          Kirill Smelkov <kirr@nexedi.com>
+#
+# This program is free software: you can Use, Study, Modify and Redistribute
+# it under the terms of the GNU General Public License version 3, or (at your
+# option) any later version, as published by the Free Software Foundation.
+#
+# You can also Link and Combine this program with other software covered by
+# the terms of any of the Free Software licenses or any of the Open Source
+# Initiative approved licenses and Convey the resulting work. Corresponding
+# source of such a combination shall include the source code for all other
+# software used.
+#
+# This program is distributed WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# See COPYING file for full licensing terms.
+# See https://www.nexedi.com/licensing for rationale and options.
+"""tzodb - ZODB-related benchmarks"""
+
+from __future__ import print_function
+
+import zodbtools.util
+from ZODB.POSException import POSKeyError
+from ZODB.utils import p64, u64
+
+import hashlib
+from tcpu import Adler32Hasher, CRC32Hasher
+import sys
+import logging
+from time import time
+from getopt import getopt, GetoptError
+
+# hasher that discards data
+class NullHasher:
+    name = "null"
+
+    def update(self, data):
+        pass
+
+    def hexdigest(self):
+        return "00"
+
+# {} name -> hasher
+hashRegistry = {
+    "null":     NullHasher,
+    "adler32":  Adler32Hasher,
+    "crc32":    CRC32Hasher,
+    "sha1":     hashlib.sha1,
+    "sha256":   hashlib.sha256,
+    "sha512":   hashlib.sha512,
+}
+
+def usage(w):
+    print(\
+"""Usage: tzodb zhash [options] url
+
+options:
+
+    --null              don't compute hash - just read data
+    --adler32           compute Adler32 checksum
+    --crc32             compute CRC32 checksum
+    --sha1              compute SHA1 cryptographic hash
+    --sha256            compute SHA256 cryptographic hash
+    --sha512            compute SHA512 cryptographic hash
+
+    --check=<expected>  verify resulting hash to be = expected
+
+    --bench=<topic>     use benchmarking format for output
+""", file=w)
+
+def zhash():
+    """zhash - compute hash of whole latest objects stream in a ZODB database"""
+    if len(sys.argv) < 2 or sys.argv[1] != "zhash":
+        usage(sys.stderr)
+        exit(1)
+
+    try:
+        optv, argv = getopt(sys.argv[2:], "h", ["help", "check=", "bench="] + hashRegistry.keys())
+    except GetoptError as e:
+        print("E: %s" % e, file=sys.stderr)
+        usage(sys.stderr)
+        exit(1)
+
+    bench=None
+    check=None
+    for opt, arg in optv:
+        if opt in ("-h", "--help"):
+            print(__doc__)
+            usage(sys.stdout)
+            sys.exit()
+
+        if opt in ("--bench"):
+            bench=arg
+            continue
+
+        if opt in ("--check"):
+            check=arg
+            continue
+
+        opt = opt.lstrip("-")
+        hctor = hashRegistry[opt]
+        h = hctor()
+
+    if len(argv) != 1:
+        print(__doc__)
+        usage(sys.stderr)
+        sys.exit(1)
+
+    url = argv[0]
+
+    # log -> stderr
+    l = logging.getLogger()
+    l.addHandler(logging.StreamHandler())
+
+    stor = zodbtools.util.storageFromURL(url, read_only=True)
+    last_tid = stor.lastTransaction()
+    before = p64(u64(last_tid) + 1)
+
+    tstart = time()
+
+    oid = 0
+    nread = 0
+    while 1:
+        try:
+            data, serial, _ = stor.loadBefore(p64(oid), before)
+        except POSKeyError:
+            break
+
+        h.update(data)
+
+        #print('%s @%s\t%s: %s' % (oid, u64(serial), h.name, h.hexdigest()), file=sys.stderr)
+        #print('\tdata: %s' % (data.encode('hex'),), file=sys.stderr)
+
+        nread += len(data)
+        oid += 1
+
+    tend = time()
+    dt = tend - tstart
+
+    x = "zhash.py"
+    hresult = "%s:%s" % (h.name, h.hexdigest())
+    if bench is None:
+        print('%s   ; oid=0..%d  nread=%d  t=%.3fs (%.1fµs / object)  x=%s' % \
+                (hresult, oid-1, nread, dt, dt * 1E6 / oid, x))
+    else:
+        topic = bench % x
+        print('Benchmark%s %d %.1f µs/object\t# %s  nread=%d  t=%.3fs' % \
+                (topic, oid-1, dt * 1E6 / oid, hresult, nread, dt))
+
+    if check != None and hresult != check:
+        print("%s: hash mismatch: expected %s  ; got %s\t# x=%s" % (url, check, hresult, x), file=sys.stderr)
+        sys.exit(1)
+
+
+def main():
+    # XXX stub (no commands dispatching here)
+    zhash()
+
+if __name__ == '__main__':
+    main()

go/neo/t/zgenprod1.py

+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# Copyright (C) 2017  Nexedi SA and Contributors.
+#
+# This program is free software: you can Use, Study, Modify and Redistribute
+# it under the terms of the GNU General Public License version 3, or (at your
+# option) any later version, as published by the Free Software Foundation.
+#
+# You can also Link and Combine this program with other software covered by
+# the terms of any of the Free Software licenses or any of the Open Source
+# Initiative approved licenses and Convey the resulting work. Corresponding
+# source of such a combination shall include the source code for all other
+# software used.
+#
+# This program is distributed WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# See COPYING file for full licensing terms.
+# See https://www.nexedi.com/licensing for rationale and options.
+"""zgenprod1 - generate synthetic test database according to NEO PROD1 statistics"""
+
+from __future__ import print_function
+
+from neo.tests.stat_zodb import PROD1
+from random import Random
+import zodbtools.util
+import sys
+import logging
+
+def usage(w):
+    print(\
+"""Usage: zgenprod1 url n""", file=w)
+
+def main():
+    argv = sys.argv[1:]
+
+    if len(argv) != 2:
+        print(__doc__)
+        usage(sys.stderr)
+        sys.exit(1)
+
+    url = argv[0]
+    n   = int(argv[1])
+
+    # log -> stderr
+    l = logging.getLogger()
+    l.addHandler(logging.StreamHandler())
+
+    zin  = PROD1(Random(0)).as_storage(n)
+    zout = zodbtools.util.storageFromURL(url)
+
+    zout.copyTransactionsFrom(zin)
+    zout.close()
+
+if __name__ == '__main__':
+    main()