go/neo/t: Neotest start

For testing and benchmarking how NEO/py & NEO/go interact with each
other we need corresponding test driver. Neotest will be that driver and
present patch begins it:

- neotest can deploy NEO/{go,py} checkout either locally or on remote node;
- it can run NEO/{go,py} unit tests either locally or on remote node;
- it can also show information about a system - either local or remote.

Examples in action. Banner:

    x/src/lab.nexedi.com/kirr/neo/go/neo/t$ ./neotest
    Neotest is a tool to test and benchmark NEO.

    Usage:

            neotest command [arguments]

    The commands are:

            test            run all tests on a remote host
            test-local      run all tests locally

            test-go         run NEO/go unit tests   (part of test-local)
            test-py         run NEO/py unit tests   (part of test-local)

            deploy          deploy NEO & needed software for tests to remote host
            deploy-local    deploy NEO & needed software for tests locally

            info            print information about a node
            info-local      print information about local deployment

Deploy to another node:

    x/src/lab.nexedi.com/kirr/neo/go/neo/t$ ./neotest deploy neotest@rio.kirr.nexedi.com:3
    *** deploying to neotest@rio.kirr.nexedi.com:3 ...
    ...
    # deployed ok

Print information about that node:

    x/lab.nexedi.com/kirr/neo/go/neo/t$ ./neotest info neotest@rio.kirr.nexedi.com:3
    date:   Sun, 08 Jul 2018 21:30:43 +0300
    xnode:  neotest@rio.kirr.nexedi.com (2401:5180:0:2a::1 192.168.0.8)
    uname:  Linux rio 4.16.0-2-amd64 #1 SMP Debian 4.16.16-2 (2018-06-22) x86_64 GNU/Linux
    sw/python:          Python 2.7.15
    sw/go:              go version go1.10.3 linux/amd64
    sw/sqlite:          sqlite 3.24.0 (py mod 2.6.0)
    sw/mysqld:          mysqld  Ver 10.1.29-MariaDB-6+b1 for debian-linux-gnu on x86_64 (Debian buildd-unstable)
    sw/neo:             v1.9-42-g972ff5f9
    sw/zodb:            5.4.0
    sw/zeo:             5.2.0
    sw/mysqlclient:     1.3.13

Run NEO/{py,go} unit tests there:

    x/src/lab.nexedi.com/kirr/neo/go/neo/t$ ./neotest test neotest@rio.kirr.nexedi.com:4
    ?       lab.nexedi.com/kirr/neo/go/internal/packed      [no test files]
    ok      lab.nexedi.com/kirr/neo/go/neo/neonet   (cached)
    ok      lab.nexedi.com/kirr/neo/go/neo/proto    (cached)
    ok      lab.nexedi.com/kirr/neo/go/zodb (cached)
    ?       lab.nexedi.com/kirr/neo/go/zodb/cmd/zodb        [no test files]
    ?       lab.nexedi.com/kirr/neo/go/zodb/internal/pickletools    [no test files]
    ?       lab.nexedi.com/kirr/neo/go/zodb/storage [no test files]
    ok      lab.nexedi.com/kirr/neo/go/zodb/storage/fs1     (cached)
    ?       lab.nexedi.com/kirr/neo/go/zodb/storage/fs1/cmd/fs1     [no test files]
    ok      lab.nexedi.com/kirr/neo/go/zodb/storage/fs1/fs1tools    (cached)
    ?       lab.nexedi.com/kirr/neo/go/zodb/storage/fs1/fsb [no test files]
    ?       lab.nexedi.com/kirr/neo/go/zodb/storage/zeo     [no test files]
    ?       lab.nexedi.com/kirr/neo/go/zodb/wks     [no test files]
    ok      lab.nexedi.com/kirr/neo/go/zodb/zodbtools       (cached)
    ...........................................................................E.EE....c....^C

Neotest will be the driver that was used to prepare benchmarks in
http://navytux.spb.ru/~kirr/neo.html and http://navytux.spb.ru/~kirr/misc/neo·P4.html .

Some draft history related to this patch:

lab.nexedi.com/kirr/neo/commit/d46afb3e		X neotest: Teach it to also run go & py unit tests; hook it into nxd/runTestSuite
lab.nexedi.com/kirr/neo/commit/f694d643		X neotest: Don't fail silently if network address detection fails
lab.nexedi.com/kirr/neo/commit/fa78290a		X neotest: FQDN host name might be not configured
lab.nexedi.com/kirr/neo/commit/56faccad		X neotest/info-local: Don't crash if a prog could not be found
lab.nexedi.com/kirr/neo/commit/f2932247		X neotest/info-local: Don't crash if an egg could not be found
lab.nexedi.com/kirr/neo/commit/f06b7302		X neotest: Determine machine IP addresses via `ip ...` directly, not `getent ...`
lab.nexedi.com/kirr/neo/commit/42e5fe71		X neotest info

go/neo/t/neotest

+#!/bin/bash -e
+# neotest: run tests and benchmarks against FileStorage, ZEO and various NEO/py and NEO/go clusters
+
+# Copyright (C) 2017-2018  Nexedi SA and Contributors.
+#                          Kirill Smelkov <kirr@nexedi.com>
+#
+# This program is free software: you can Use, Study, Modify and Redistribute
+# it under the terms of the GNU General Public License version 3, or (at your
+# option) any later version, as published by the Free Software Foundation.
+#
+# You can also Link and Combine this program with other software covered by
+# the terms of any of the Free Software licenses or any of the Open Source
+# Initiative approved licenses and Convey the resulting work. Corresponding
+# source of such a combination shall include the source code for all other
+# software used.
+#
+# This program is distributed WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# See COPYING file for full licensing terms.
+# See https://www.nexedi.com/licensing for rationale and options.
+
+set -o pipefail
+
+# ---- deploy NEO for tests/benchmarks at a node ----
+
+die() {
+	echo 2>&1 "$@"
+	exit 1
+}
+
+# cmd_deploy [user@]<host>:<path>	- deploy NEO & needed software for tests there
+# ssh-key or password for access should be available
+cmd_deploy() {
+	host=`echo $1 |sed -e 's/:[^:]*$//'`	# user@host
+	path=${1:$((${#host} + 1))}		# path
+	test -z "$host" -o -z "$path" && die "Usage: neotest deploy [user@]<host>:<path>"
+	echo -e "\n*** deploying to $@ ..."
+	scp $0 $host:neotest
+	ssh $host ./neotest deploy-local "$path"
+}
+
+# cmd_deploy-local <path>		- deploy NEO & needed software for tests @path
+cmd_deploy-local() {
+	path=$1
+	test -z "$path" && die "Usage: neotest deploy-local <path>"
+	test -e $path/deployed && echo "# already deployed" && return
+	mkdir -p $path
+	cd $path
+
+	# python part
+	virtualenv venv
+
+	# env.sh for deployment
+	cat >env.sh << 'EOF'
+X=${1:-${BASH_SOURCE[0]}}       # path to original env.sh is explicitly passed
+X=$(cd `dirname $X` && pwd)     # when there is other env.sh wrapping us
+
+export GOPATH=$X:$GOPATH
+export PATH=$X/bin:$PATH
+export PS1="(`basename $X`) $PS1"
+
+# strip trailing : from $GOPATH
+GOPATH=${GOPATH%:}
+
+# python
+. $X/venv/bin/activate
+
+# XXX for mysqld
+export PATH=$PATH:/sbin:/usr/sbin
+EOF
+
+	. env.sh
+
+	mkdir -p src/lab.nexedi.com/kirr
+	pushd src/lab.nexedi.com/kirr
+	test -d neo || git clone -o kirr https://lab.nexedi.com/kirr/neo.git neo
+	cd neo
+	pip install -e .[admin,client,ctl,master,storage-sqlite,storage-mysqldb]
+	popd
+
+	go get -v -t lab.nexedi.com/kirr/neo/go/...
+	go get -v golang.org/x/perf/cmd/benchstat		# to summarize/diff benchmark results
+	#go get -v github.com/aclements/perflock/cmd/perflock	# handy to fix CPU frequency/etc
+
+	echo ok >deployed
+	echo "# deployed ok"
+}
+
+# jump to deploy command early if we have to
+case "$1" in
+deploy|deploy-local)
+	cmd="$1"
+	shift
+	cmd_$cmd "$@"
+	exit
+	;;
+esac
+
+# on <url> ...		- run ... on deployed url from inside dir of neotest
+on() {
+	#echo "on $@"
+	host=`echo $1 |sed -e 's/:[^:]*$//'`	# user@host
+	path=${1:$((${#host} + 1))}		# path
+	test -z "$host" -o -z "$path" && die "on $1: invalid URL"
+	shift
+	ssh $host "bash -c \"test -e $path/deployed || { echo 1>&2 '$url not yet deployed'; exit 1; }
+cd $path
+. env.sh
+#set -x
+cd src/lab.nexedi.com/kirr/neo/go/neo/t
+$@
+\""
+}
+
+# ---- go/py unit tests ----
+cmd_test-go() {
+	go test lab.nexedi.com/kirr/neo/go/...
+}
+
+cmd_test-py() {
+	# NOTE testing with only sqlite should be ok to check for client correctness
+	NEO_TESTS_ADAPTER=SQLite python -m neo.scripts.runner -ufz
+}
+
+cmd_test-local() {
+	cmd_test-go
+	cmd_test-py
+}
+
+cmd_test() {
+	url="$1"
+	test -z "$url" && die "Usage neotest test [user@]<host>:<path>"
+	on $url ./neotest test-local
+}
+
+
+# ---- net/fs setup + processes control/teardown ----
+
+# init_net	- initialize networking
+init_net() {
+	# determine our external addresses IPv4 or IPv6
+	# 2: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
+	#     inet 192.168.102.52/24 brd 192.168.102.255 scope global dynamic wlan0
+	#        valid_lft 82495sec preferred_lft 82495sec
+	#
+	# 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
+	#     inet6 2401:5180:0:37::1/64 scope global
+	#        valid_lft forever preferred_lft forever
+	# 2: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
+	#     inet6 2401:5180:0:1d:429a:e612:c957:29/64 scope global noprefixroute dynamic
+	#        valid_lft 86232sec preferred_lft 14232sec
+	myaddr4v=(`ip -4 addr show scope global |grep inet |awk '{print $2}' |sed -e 's|/.*$||'`) || :
+	myaddr6v=(`ip -6 addr show scope global |grep inet |awk '{print $2}' |sed -e 's|/.*$||'`) || :
+	test "${#myaddr4v[@]}" -gt 0 || die "init_net: cannot determine my IPv4 network addresses"
+	test "${#myaddr6v[@]}" -gt 0 || die "init_net: cannot determine my IPv6 network addresses"
+
+	# prefer ipv4 for now
+	myaddr="${myaddr4v[0]}"
+}
+
+# if we are abnormally terminating
+install_trap() {
+	trap 'set +e
+echo "E: abnormal termination - stopping..."
+j="$(jobs -p)"
+test -z "$j" && exit
+echo "E: killing left jobs..."
+jobs -l
+kill $j' EXIT
+}
+
+# ---- information about system ----
+
+# pyver <egg> (<showas>) - print version of egg
+pyver() {
+	local egg=$1
+	local showas=$2
+	test "$showas" == "" && showas=$egg
+	local loc
+	local pyver
+	{
+		read loc
+		read pyver
+	} < <(python -c "
+import pkg_resources as p
+try:
+	e=p.require(\"$egg\")[0]
+except p.DistributionNotFound:
+	print(\"\nø\")
+else:
+	print(\"%s\n%s\" % (e.location, e.version))
+	")
+	local gitver=$(git -C $loc describe --long --dirty 2>/dev/null)
+	local ver
+	test "$gitver" != "" && ver="$gitver" || ver="$pyver"
+	printf "sw/%-16s %s\n" "${showas}:" "$ver"
+}
+
+# proginfo <prog> ...	- run `prog ...` or print that prog is missing
+proginfo() {
+	prog=$1
+	shift
+	which $prog >/dev/null 2>&1 && $prog "$@" || printf "%-16s: ø\n" "$prog"
+}
+
+# show information about local system (os, hardware, versions, ...)
+system_info() {
+	echo -ne "date:\t"; date --rfc-2822
+	echo -ne "xnode:\t`whoami`@`hostname --fqdn 2>/dev/null || hostname` ("
+	echo -n "${myaddr6v[0]}"
+	test "${#myaddr6v[@]}" -eq 1 || echo -n " (+ $((${#myaddr6v[@]} - 1))·ipv6)"
+	echo -n " ${myaddr4v[0]}"
+	test "${#myaddr4v[@]}" -eq 1 || echo -n " (+ $((${#myaddr4v[@]} - 1))·ipv4)"
+	echo ")"
+	echo -ne "uname:\t"; uname -a
+
+	printf "%-20s" "sw/python:";	proginfo python --version 2>&1	# https://bugs.python.org/issue18338
+	printf "%-20s" "sw/go:";	proginfo go version
+	printf "%-20s" "sw/sqlite:";	proginfo python -c \
+		'import sqlite3 as s; print "sqlite %s (py mod %s)" % (s.sqlite_version, s.version)'
+	printf "%-20s" "sw/mysqld:";	proginfo mysqld --version
+
+	pyver neoppod neo
+	pyver zodb
+	pyver zeo
+	pyver mysqlclient
+}
+
+
+
+# command: print information about local node
+cmd_info-local() {
+	system_info
+}
+
+# command: print information about remote node
+cmd_info() {
+	url="$1"
+	test -z "$url" && die "Usage neotest info [user@]<host>:<path>"
+	on $url ./neotest info-local
+}
+
+# ---- main driver ----
+
+usage() {
+cat 1>&2 << EOF
+Neotest is a tool to test and benchmark NEO.
+
+Usage:
+
+	neotest command [arguments]
+
+The commands are:
+
+	test		run all tests on a remote host
+	test-local	run all tests locally
+
+	test-go		run NEO/go unit tests	(part of test-local)
+	test-py		run NEO/py unit tests	(part of test-local)
+
+
+	deploy		deploy NEO & needed software for tests to remote host
+	deploy-local	deploy NEO & needed software for tests locally
+
+	info		print information about a node
+	info-local	print information about local deployment
+EOF
+}
+
+# commands and their flags:
+#
+# build: needs to rebuild NEO stuff
+# net:   needs init_net
+case "$1" in
+test)		f=(                     );;
+test-local)	f=(build                );;
+test-go)	f=(build                );;
+test-py)	f=(                     );;
+
+info)		f=(                     );;
+info-local)	f=(      net            );;
+
+-h)
+	usage
+	exit 0
+	;;
+*)
+	usage
+	exit 1
+	;;
+esac
+
+
+for flag in ${f[*]}; do
+	case "$flag" in
+	build)
+		# rebuild go bits
+		# neo/py, wendelin.core, ... - must be pip install'ed - `neotest deploy` cares about that
+		go install -v lab.nexedi.com/kirr/neo/go/...
+		;;
+
+	net)
+		# setup network environment
+		init_net
+		;;
+
+	*)
+		die "internal-error: command $1: invalid flag: $flag"
+		;;
+	esac
+done
+
+# run the command
+cmd="$1"
+shift
+cmd_$cmd "$@"