xnet/lonet: New package to provide TCP network simulated on top of localhost TCP loopback.

Lonet is the virtnet network that, contrary to pipenet, could be used
when there are several OS-level processes involved.

It uses SQLite for its registry and native OS-level TCP over loopback
for data exchange. There is small text-based connection handshake
protocol prelude that have to be carried out when a connection is tried
to be established to host via its subnetwork, but after it data exchange
goes directly through OS TCP stack.

Lonet documentation follows:

"""
Package lonet provides TCP network simulated on top of localhost TCP loopback.

For testing distributed systems it is sometimes handy to imitate network of
several TCP hosts. It is also handy that ports allocated on Dial/Listen/Accept on
that hosts be predictable - that would help tests to verify network events
against expected sequence. When whole system could be imitated in 1 OS-level
process, package lab.nexedi.com/kirr/go123/xnet/pipenet serves the task via
providing TCP-like synchronous in-memory network of net.Pipes. However
pipenet cannot be used for cases where tested system consists of 2 or more
OS-level processes. This is where lonet comes into play:

Similarly to pipenet addresses on lonet are host:port pairs and several
hosts could be created with different names. A host is xnet.Networker and
so can be worked with similarly to regular TCP network access-point with
Dial/Listen/Accept. Host's ports allocation is predictable: ports of a host
are contiguous integer sequence starting from 1 that are all initially free,
and whenever autobind is requested the first free port of the host will be
used.

Internally lonet network maintains registry of hosts so that lonet
addresses could be resolved to OS-level addresses, for example α:1 and β:1
to 127.0.0.1:4567 and 127.0.0.1:8765, and once lonet connection is
established it becomes served by OS-level TCP connection over loopback.

Example:

net, err := lonet.Join(ctx, "mynet")
hα, err := net.NewHost(ctx, "α")
hβ, err := net.NewHost(ctx, "β")

// starts listening on address "α:10"
l, err := hα.Listen(":10")
go func() {
 csrv, err := l.Accept()   // csrv will have LocalAddr "α:1"
}()
ccli, err := hβ.Dial(ctx, "α:10") // ccli will be connection between "β:1" - "α:1"

Once again lonet is similar to pipenet, but since it works via OS TCP stack
it could be handy for testing networked application when there are several
OS-level processes involved.
"""

"""
Lonet organization

For every lonet network there is a registry with information about hosts
available on the network, and for each host its OS-level listening address.
The registry is kept as SQLite database under

/<tmp>/lonet/<network>/registry.db

Whenever host α needs to establish connection to address on host β, it
queries the registry for β and further talks to β on that address.
Correspondingly when a host joins the network, it announces itself to the
registry so that other hosts could see it.

Handshake protocol

After α establishes OS-level connection to β via main β address, it sends
request to further establish lonet connection on top of that:

> lonet "<network>" dial "<α:portα>" "<β:portβ>"\n

β checks whether portβ is listening, and if yes, accepts the connection on
corresponding on-β listener with giving feedback to α that connection was
accepted:

< lonet "<network>" connected "<β:portβ'>"\n

After that connection is considered to be lonet-established and all further
exchange on it is directly controlled by corresponding lonet-level
Read/Write on α and β.

If, on the other hand, lonet-level connection cannot be established, β replies:

< lonet "<networkβ>" E "<error>"\n

where <error> could be:

- connection refused if <β:portβ> is not listening
- network mismatch if β thinks it works on different lonet network than α
- protocol error if β thinks that α send incorrect dial request
- ...
"""

xnet/lonet/lonet_test.go

+// Copyright (C) 2018  Nexedi SA and Contributors.
+//                     Kirill Smelkov <kirr@nexedi.com>
+//
+// This program is free software: you can Use, Study, Modify and Redistribute
+// it under the terms of the GNU General Public License version 3, or (at your
+// option) any later version, as published by the Free Software Foundation.
+//
+// You can also Link and Combine this program with other software covered by
+// the terms of any of the Free Software licenses or any of the Open Source
+// Initiative approved licenses and Convey the resulting work. Corresponding
+// source of such a combination shall include the source code for all other
+// software used.
+//
+// This program is distributed WITHOUT ANY WARRANTY; without even the implied
+// warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+//
+// See COPYING file for full licensing terms.
+// See https://www.nexedi.com/licensing for rationale and options.
+
+package lonet
+
+import (
+	"context"
+	"io/ioutil"
+	"log"
+	"os"
+	"testing"
+
+	"lab.nexedi.com/kirr/go123/xnet/internal/virtnettest"
+)
+
+func TestLonet(t *testing.T) {
+	subnet, err := Join(context.Background(), "")
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	// XXX TestBasic shutdows subnet, but /tmp/lonet/<name> is left alive.
+	virtnettest.TestBasic(t, subnet)
+}
+
+
+
+var workRoot string
+
+func TestMain(m *testing.M) {
+	// setup workroot for all tests
+	workRoot, err := ioutil.TempDir("", "t-lonet")
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	exit := m.Run()
+	os.RemoveAll(workRoot)
+	os.Exit(exit)
+}
+
+// xworkdir creates temp directory inside workRoot.
+func xworkdir(t testing.TB) string {
+	work, err := ioutil.TempDir(workRoot, "")
+	if err != nil {
+		t.Fatal(err)
+	}
+	return work
+}

xnet/lonet/registry_sqlite.go

+// Copyright (C) 2018  Nexedi SA and Contributors.
+//                     Kirill Smelkov <kirr@nexedi.com>
+//
+// This program is free software: you can Use, Study, Modify and Redistribute
+// it under the terms of the GNU General Public License version 3, or (at your
+// option) any later version, as published by the Free Software Foundation.
+//
+// You can also Link and Combine this program with other software covered by
+// the terms of any of the Free Software licenses or any of the Open Source
+// Initiative approved licenses and Convey the resulting work. Corresponding
+// source of such a combination shall include the source code for all other
+// software used.
+//
+// This program is distributed WITHOUT ANY WARRANTY; without even the implied
+// warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+//
+// See COPYING file for full licensing terms.
+// See https://www.nexedi.com/licensing for rationale and options.
+
+package lonet
+// registry implemented as shared SQLite file.
+
+import (
+	"context"
+	"errors"
+	"fmt"
+
+	"crawshaw.io/sqlite"
+	"crawshaw.io/sqlite/sqliteutil"
+
+	"lab.nexedi.com/kirr/go123/xerr"
+	"lab.nexedi.com/kirr/go123/xnet/virtnet"
+)
+
+// registry schema	(keep in sync wrt .setup())
+//
+// hosts:
+//	hostname	text !null PK
+//	osladdr		text !null
+//
+// meta:
+//	name		text !null PK
+//	value		text !null
+//
+// "schemaver"  text	- version of db schema.
+// "network"    text	- name of lonet network this registry serves.
+
+const schemaVer = "lonet.1"
+
+type sqliteRegistry struct {
+	dbpool *sqlite.Pool
+
+	uri    string	// URI db was originally opened with
+}
+
+// openRegistrySQLite opens SQLite registry located at dburi.
+//
+// the registry is setup/verified to be serving specified lonet network.
+func openRegistrySQLite(ctx context.Context, dburi, network string) (_ *sqliteRegistry, err error) {
+	r := &sqliteRegistry{uri: dburi}
+	defer r.regerr(&err, "open")
+
+	dbpool, err := sqlite.Open(dburi, 0, /* poolSize= */16)	// XXX pool size ok?
+	if err != nil {
+		return nil, err
+	}
+
+	r.dbpool = dbpool
+
+	// initialize/check db
+	err = r.setup(ctx, network)
+	if err != nil {
+		r.Close()
+		return nil, err
+	}
+
+	return r, nil
+}
+
+// Close implements registry.
+func (r *sqliteRegistry) Close() (err error) {
+	defer r.regerr(&err, "close")
+	return r.dbpool.Close()
+}
+
+// withConn runs f on a dbpool connection.
+//
+// connection is first allocated from dbpool and put back after call to f.
+func (r *sqliteRegistry) withConn(ctx context.Context, f func(*sqlite.Conn) error) error {
+	conn := r.dbpool.Get(ctx.Done())
+	if conn == nil {
+		// either ctx cancel or dbpool close
+		if ctx.Err() != nil {
+			return ctx.Err()
+		}
+		return virtnet.ErrRegistryDown // db closed
+	}
+	defer r.dbpool.Put(conn)
+	return f(conn)
+}
+
+var errNoRows   = errors.New("query: empty result")
+var errManyRows = errors.New("query: multiple results")
+
+// query1 is like sqliteutil.Exec but checks that exactly 1 row is returned.
+//
+// if query results in no rows - errNoRows is returned.
+// if query results in more than 1 row - errManyRows is returned.
+func query1(conn *sqlite.Conn, query string, resultf func(stmt *sqlite.Stmt), argv ...interface{}) error {
+	nrow := 0
+	err := sqliteutil.Exec(conn, query, func(stmt *sqlite.Stmt) error {
+		if nrow == 1 {
+			return errManyRows
+		}
+		nrow++
+		resultf(stmt)
+		return nil
+	}, argv...)
+	if err != nil {
+		return err
+	}
+	if nrow == 0 {
+		return errNoRows
+	}
+	return nil
+}
+
+// setup initializes/checks registry database to be of expected schema and configuration.
+func (r *sqliteRegistry) setup(ctx context.Context, network string) (err error) {
+	defer xerr.Contextf(&err, "setup %q", network)
+
+	return r.withConn(ctx, func(conn *sqlite.Conn) (err error) {
+		// NOTE: keep in sync wrt top-level text.
+		err = sqliteutil.ExecScript(conn, `
+			CREATE TABLE IF NOT EXISTS hosts (
+				hostname	TEXT NON NULL PRIMARY KEY,
+				osladdr		TEXT NON NULL
+			);
+
+			CREATE TABLE IF NOT EXISTS meta (
+				name		TEXT NON NULL PRIMARY KEY,
+				value		TEXT NON NULL
+			);
+		`)
+		if err != nil {
+			return err
+		}
+
+		// do whole checks/init under transaction, so that there is
+		// e.g. no race wrt another process setting config.
+		defer sqliteutil.Save(conn)(&err)
+
+		// check/init schema version
+		ver, err := r.config(conn, "schemaver")
+		if err != nil {
+			return err
+		}
+		if ver == "" {
+			ver = schemaVer
+			err = r.setConfig(conn, "schemaver", ver)
+			if err != nil {
+				return err
+			}
+		}
+		if ver != schemaVer {
+			return fmt.Errorf("schema version mismatch: want %q; have %q", schemaVer, ver)
+		}
+
+		// check/init network name
+		dbnetwork, err := r.config(conn, "network")
+		if err != nil {
+			return err
+		}
+		if dbnetwork == "" {
+			dbnetwork = network
+			err = r.setConfig(conn, "network", dbnetwork)
+			if err != nil {
+				return err
+			}
+		}
+		if dbnetwork != network {
+			return fmt.Errorf("network name mismatch: want %q; have %q", network, dbnetwork)
+		}
+
+
+		return nil
+	})
+}
+
+// config gets one registry configuration value by name.
+//
+// if there is no record corresponding to name - ("", nil) is returned.
+// XXX add ok ret to indicate presence of value?
+func (r *sqliteRegistry) config(conn *sqlite.Conn, name string) (value string, err error) {
+	defer xerr.Contextf(&err, "config: get %q", name)
+
+	err = query1(conn, "SELECT value FROM meta WHERE name = ?", func(stmt *sqlite.Stmt) {
+		value = stmt.ColumnText(0)
+	}, name)
+
+	switch err {
+	case errNoRows:
+		return "", nil
+	case errManyRows:
+		value = ""
+	}
+
+	return value, err
+}
+
+// setConfig sets one registry configuration value by name.
+func (r *sqliteRegistry) setConfig(conn *sqlite.Conn, name, value string) (err error) {
+	defer xerr.Contextf(&err, "config: set %q = %q", name, value)
+
+	err = sqliteutil.Exec(conn,
+		"INSERT OR REPLACE INTO meta (name, value) VALUES (?, ?)", nil,
+		name, value)
+	return err
+}
+
+// Announce implements registry.
+func (r *sqliteRegistry) Announce(ctx context.Context, hostname, osladdr string) (err error) {
+	defer r.regerr(&err, "announce", hostname, osladdr)
+
+	return r.withConn(ctx, func(conn *sqlite.Conn) error {
+		err := sqliteutil.Exec(conn,
+			"INSERT INTO hosts (hostname, osladdr) VALUES (?, ?)", nil,
+			hostname, osladdr)
+
+		switch sqlite.ErrCode(err) {
+		case sqlite.SQLITE_CONSTRAINT_UNIQUE:
+			err = virtnet.ErrHostDup
+		}
+
+		return err
+	})
+}
+
+var errRegDup = errors.New("registry broken: duplicate host entries")
+
+// Query implements registry.
+func (r *sqliteRegistry) Query(ctx context.Context, hostname string) (osladdr string, err error) {
+	defer r.regerr(&err, "query", hostname)
+
+	err = r.withConn(ctx, func(conn *sqlite.Conn) error {
+		err := query1(conn, "SELECT osladdr FROM hosts WHERE hostname = ?",
+			func (stmt *sqlite.Stmt) {
+				osladdr = stmt.ColumnText(0)
+			}, hostname)
+
+		switch err {
+		case errNoRows:
+			return virtnet.ErrNoHost
+
+		case errManyRows:
+			// hostname is PK - we should not get several results
+			osladdr = ""
+			return errRegDup
+		}
+
+		return err
+	})
+
+	return osladdr, err
+}
+
+// regerr is syntactic sugar to wrap !nil *errp into RegistryError.
+//
+// intended too be used like
+//
+//	defer r.regerr(&err, "operation", arg1, arg2, ...)
+func (r *sqliteRegistry) regerr(errp *error, op string, args ...interface{}) {
+	if *errp == nil {
+		return
+	}
+
+	*errp = &virtnet.RegistryError{
+		Registry: r.uri,
+		Op:       op,
+		Args:     args,
+		Err:      *errp,
+	}
+}

xnet/lonet/registry_sqlite_test.go

+// Copyright (C) 2018  Nexedi SA and Contributors.
+//                     Kirill Smelkov <kirr@nexedi.com>
+//
+// This program is free software: you can Use, Study, Modify and Redistribute
+// it under the terms of the GNU General Public License version 3, or (at your
+// option) any later version, as published by the Free Software Foundation.
+//
+// You can also Link and Combine this program with other software covered by
+// the terms of any of the Free Software licenses or any of the Open Source
+// Initiative approved licenses and Convey the resulting work. Corresponding
+// source of such a combination shall include the source code for all other
+// software used.
+//
+// This program is distributed WITHOUT ANY WARRANTY; without even the implied
+// warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+//
+// See COPYING file for full licensing terms.
+// See https://www.nexedi.com/licensing for rationale and options.
+
+package lonet
+
+import (
+	"context"
+	"fmt"
+	"testing"
+
+	"lab.nexedi.com/kirr/go123/exc"
+	"lab.nexedi.com/kirr/go123/xnet/virtnet"
+)
+
+// registryTester is handy utility to test sqliteRegistry
+type registryTester struct {
+	*testing.T
+	r *sqliteRegistry
+}
+
+// Query checks that result of Query(hostname) is as expected.
+//
+// if expect is error - it checks that Query returns error with cause == expect.
+// otherwise expect must be string and it will check that Query
+// succeeds and returns osladdr == expect.
+func (t *registryTester) Query(hostname string, expect interface{}) {
+	t.Helper()
+	r := t.r
+
+	osladdr, err := r.Query(context.Background(), hostname)
+	if ewant, iserr := expect.(error); iserr {
+		// error expected
+		// XXX construct full registry error around ewant + reflect.DeepCompare?
+		e, ok := err.(*virtnet.RegistryError)
+		if !(ok && e.Err == ewant && osladdr == "") {
+			t.Fatalf("%s: query %q:\nwant: \"\", %v\nhave: %q, %v",
+				r.uri, hostname, ewant, osladdr, err)
+		}
+	} else {
+		// !error expected
+		laddr := expect.(string)
+		if !(osladdr == laddr && err == nil) {
+			t.Fatalf("%s: query %q:\nwant: %q, nil\nhave: %q, %v",
+				r.uri, hostname, laddr, osladdr, err)
+		}
+	}
+}
+
+// Announce checks that result of Announce(hostname, osladdr) is as expected.
+//
+// if len(errv) == 1 - it checks that Announce returns error with cause == errv[0].
+// otherwise it will check that Announce succeeds and returns nil error.
+func (t *registryTester) Announce(hostname, osladdr string, errv ...error) {
+	t.Helper()
+	r := t.r
+
+	err := r.Announce(context.Background(), hostname, osladdr)
+	var ewant error
+	if len(errv) > 0 {
+		ewant = errv[0]
+		if len(errv) > 1 {
+			panic("only 1 error allowed in announce check")
+		}
+	}
+	if ewant != nil {
+		// error expected
+		// XXX construct full registry error around ewant + reflect.DeepCompare?
+		e, ok := err.(*virtnet.RegistryError)
+		if (!ok && e.Err == ewant) {
+			t.Fatalf("%s: announce %q %q:\nwant %v\nhave: %v",
+				r.uri, hostname, osladdr, ewant, err)
+		}
+	} else {
+		// !error expected
+		if err != nil {
+			t.Fatalf("%s: announce %q %q: %s", r.uri, hostname, osladdr, err)
+		}
+	}
+}
+
+// handy shortcuts for registry errors, ...
+var ø     = virtnet.ErrNoHost
+var DUP   = virtnet.ErrHostDup
+var RDOWN = virtnet.ErrRegistryDown
+
+var X  = exc.Raiseif
+var bg = context.Background()
+
+
+func TestRegistrySQLite(t *testing.T) {
+	work := xworkdir(t)
+	dbpath := work + "/1.db"
+
+	r1, err := openRegistrySQLite(bg, dbpath, "aaa")
+	X(err)
+
+	t1 := &registryTester{t, r1}
+	t1.Query("α", ø)
+	t1.Announce("α", "alpha:1234")
+	t1.Announce("α", "alpha:1234", DUP)
+	t1.Announce("α", "alpha:1235", DUP)
+	t1.Query("α", "alpha:1234")
+	t1.Query("β", ø)
+
+	r2, err := openRegistrySQLite(bg, dbpath, "aaa")
+	X(err)
+
+	t2 := &registryTester{t, r2}
+	t2.Query("α", "alpha:1234")
+	t2.Query("β", ø)
+	t2.Announce("β", "beta:zzz")
+	t2.Query("β", "beta:zzz")
+
+	t1.Query("β", "beta:zzz")
+
+	X(r1.Close())
+
+	t1.Query("α", RDOWN)
+	t1.Query("β", RDOWN)
+	t1.Announce("γ", "gamma:qqq", RDOWN)
+	t1.Query("γ", RDOWN)
+
+	t2.Query("α", "alpha:1234")
+
+	X(r2.Close())
+
+	t2.Query("α", RDOWN)
+
+
+	// verify network mismatch detection works
+	r3, err := openRegistrySQLite(bg, dbpath, "bbb")
+	if !(r3 == nil && err != nil) {
+		t.Fatalf("network mismatch: not detected")
+	}
+	errWant := fmt.Sprintf(`%s: open []: setup "bbb": network name mismatch: want "bbb"; have "aaa"`, dbpath)
+	if err.Error() != errWant {
+		t.Fatalf("network mismatch: error:\nhave: %q\nwant: %q", err.Error(), errWant)
+	}
+}

xnet/pipenet/pipenet.go

@@ -36,6 +36,9 @@
 //
 // Pipenet might be handy for testing interaction of networked applications in 1
 // process without going to OS networking stack.
+//
+// See also package lab.nexedi.com/kirr/go123/xnet/lonet for similar network
+// that can work across several OS-level processes.
 package pipenet
 
 import (

xnet/virtnet/virtnet.go

@@ -45,8 +45,9 @@
 // Starting from SubNetwork one can create Hosts and from those exchange data
 // throughout whole network.
 //
-// Please see package lab.nexedi.com/kirr/go123/xnet/pipenet for particular
-// well-known virtnet-based network.
+// Please see packages lab.nexedi.com/kirr/go123/xnet/lonet and
+// lab.nexedi.com/kirr/go123/xnet/pipenet for particular well-known
+// virtnet-based networks.
 //
 //
 // Implementing virtnet networks