xnet/pipenet: Generalize it into xnet/virtnet

As we are going to implement another virtual network it would be good to
share common code between implementations. For this generalize pipenet
implementation to also cover the case when one does not own full network
and owns only some hosts of it.

An example of such situation is when one process handles one group of
virtual hosts and another process handles another group of virtual
hosts. Below a group of virtual hosts handled as part of network is
called subnetwork.

If hosts are not created in one place, we need a way to communicate
information about new hosts in between subnetworks. This leads to using
some kind of "registry" (see Registry interface).

Then for the common code to be reused by a virtual network
implementation it has to provide its working in the form of Engine
interface to that common code. In turn the common code exposes another
- Notifier - interface for particular network implementation to notify
common code of events that come from outside to the subnetwork.

Pipenet is reworked to be just a client of the common virtnet
infrastructure.

Virtnet documentation follows:

"""
Package virtnet provides infrastructure for TCP-like virtual networks.

For testing distributed systems it is sometimes handy to imitate network of
several TCP hosts. It is also handy that ports allocated on Dial/Listen/Accept
on that hosts be predictable - that would help tests to verify network
events against expected sequence.

Package virtnet provides infrastructure for using and implementing such
TCP-like virtual networks.

Using virtnet networks

Addresses on a virtnet network are host:port pairs represented by Addr.
A network conceptually consists of several SubNetworks each being home for
multiple Hosts. Host is xnet.Networker and so can be worked with similarly
to regular TCP network access-point with Dial/Listen/Accept. Host's ports
allocation is predictable: ports of a host are contiguous integer sequence
starting from 1 that are all initially free, and whenever autobind is
requested the first free port of the host will be used.
Virtnet ensures that host names are unique throughout whole network.

To work with a virtnet network, one uses corresponding package for
particular virtnet network implementation. Such packages provide a way to
join particular network and after joining give back SubNetwork to user.
Starting from SubNetwork one can create Hosts and from those exchange data
throughout whole network.

Please see package lab.nexedi.com/kirr/go123/xnet/pipenet for particular
well-known virtnet-based network.

Implementing virtnet networks

To implement a virtnet-based network one need to implement Engine and Registry.

A virtnet network implementation should provide Engine and Registry
instances to SubNetwork when creating it. The subnetwork will use provided
engine and registry for its operations. A virtnet network implementation
receives instance of Notifier together with SubNetwork when creating it. The
network implementation should use provided Notifier to notify the subnetwork
to handle incoming events.

Please see Engine, Registry and Notifier documentation for details.
"""

Another virtnet-based network that is not limited to be used only in 1
OS process will follow next.

internal/xtesting/xtesting.go

+// Copyright (C) 2018  Nexedi SA and Contributors.
+//                     Kirill Smelkov <kirr@nexedi.com>
+//
+// This program is free software: you can Use, Study, Modify and Redistribute
+// it under the terms of the GNU General Public License version 3, or (at your
+// option) any later version, as published by the Free Software Foundation.
+//
+// You can also Link and Combine this program with other software covered by
+// the terms of any of the Free Software licenses or any of the Open Source
+// Initiative approved licenses and Convey the resulting work. Corresponding
+// source of such a combination shall include the source code for all other
+// software used.
+//
+// This program is distributed WITHOUT ANY WARRANTY; without even the implied
+// warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+//
+// See COPYING file for full licensing terms.
+// See https://www.nexedi.com/licensing for rationale and options.
+
+// Package xtesting provides addons to std package testing.
+//
+// The tools provided are mostly useful when doing tests with exceptions.
+package xtesting
+
+import (
+	"fmt"
+	"reflect"
+	"testing"
+
+	"lab.nexedi.com/kirr/go123/exc"
+)
+
+// Asserter is handy objects to make asserts in tests.
+//
+// For example:
+//
+//	assert := xtesting.Assert(t)
+//	assert.Eq(a, b)
+//	..
+//
+// Contrary to t.Fatal* and e.g. github.com/stretchr/testify/require.Assert it
+// is safe to use Asserter from non-main goroutine.
+type Asserter struct {
+	t testing.TB
+}
+
+// Assert creates Asserter bound to t for reporting.
+func Assert(t testing.TB) *Asserter {
+	return &Asserter{t}
+}
+
+// Eq asserts that a == b and raises exception if not.
+func (x *Asserter) Eq(a, b interface{}) {
+	x.t.Helper()
+	if !reflect.DeepEqual(a, b) {
+		fmt.Printf("not equal:\nhave: %v\nwant: %v\n", a, b)
+		x.t.Errorf("not equal:\nhave: %v\nwant: %v", a, b)
+		exc.Raise(0)
+	}
+}

xnet/internal/virtnettest/vnettest.go

+// Copyright (C) 2017-2018  Nexedi SA and Contributors.
+//                          Kirill Smelkov <kirr@nexedi.com>
+//
+// This program is free software: you can Use, Study, Modify and Redistribute
+// it under the terms of the GNU General Public License version 3, or (at your
+// option) any later version, as published by the Free Software Foundation.
+//
+// You can also Link and Combine this program with other software covered by
+// the terms of any of the Free Software licenses or any of the Open Source
+// Initiative approved licenses and Convey the resulting work. Corresponding
+// source of such a combination shall include the source code for all other
+// software used.
+//
+// This program is distributed WITHOUT ANY WARRANTY; without even the implied
+// warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+//
+// See COPYING file for full licensing terms.
+// See https://www.nexedi.com/licensing for rationale and options.
+
+// Package virtnettest provides basic tests to be run on virtnet network implementations.
+package virtnettest
+
+import (
+	"context"
+	"io"
+	"net"
+	"testing"
+
+	"golang.org/x/sync/errgroup"
+
+	"lab.nexedi.com/kirr/go123/exc"
+	"lab.nexedi.com/kirr/go123/internal/xtesting"
+	"lab.nexedi.com/kirr/go123/xnet/virtnet"
+)
+
+
+type mklistener interface {
+	Listen(string) (net.Listener, error)
+}
+
+func xlisten(n mklistener, laddr string) net.Listener {
+	l, err := n.Listen(laddr)
+	exc.Raiseif(err)
+	return l
+}
+
+func xaccept(l net.Listener) net.Conn {
+	c, err := l.Accept()
+	exc.Raiseif(err)
+	return c
+}
+
+type dialer interface {
+	Dial(context.Context, string) (net.Conn, error)
+}
+
+func xdial(n dialer, addr string) net.Conn {
+	c, err := n.Dial(context.Background(), addr)
+	exc.Raiseif(err)
+	return c
+}
+
+func xread(r io.Reader) string {
+	buf := make([]byte, 4096)
+	n, err := r.Read(buf)
+	exc.Raiseif(err)
+	return string(buf[:n])
+}
+
+func xwrite(w io.Writer, data string) {
+	_, err := w.Write([]byte(data))
+	exc.Raiseif(err)
+}
+
+func xwait(w interface { Wait() error }) {
+	err := w.Wait()
+	exc.Raiseif(err)
+}
+
+// TestBasic runs basic tests on a virtnet network implementation.
+func TestBasic(t *testing.T, subnet *virtnet.SubNetwork) {
+	X := exc.Raiseif
+	ctx := context.Background()
+	assert := xtesting.Assert(t)
+
+	defer func() {
+		err := subnet.Close()
+		X(err)
+	}()
+
+	xaddr := func(addr string) *virtnet.Addr {
+		a, err := virtnet.ParseAddr(subnet.Network(), addr)
+		X(err)
+		return a
+	}
+
+	hα, err := subnet.NewHost(ctx, "α")
+	X(err)
+
+	hβ, err := subnet.NewHost(ctx, "β")
+	X(err)
+
+	assert.Eq(hα.Network(), subnet.Network())
+	assert.Eq(hβ.Network(), subnet.Network())
+	assert.Eq(hα.Name(), "α")
+	assert.Eq(hβ.Name(), "β")
+
+	_, err = hα.Dial(ctx, ":0")
+	assert.Eq(err, &net.OpError{Op: "dial", Net: subnet.Network(), Source: xaddr("α:1"), Addr: xaddr("α:0"), Err: virtnet.ErrConnRefused})
+
+	l1, err := hα.Listen("")
+	X(err)
+	assert.Eq(l1.Addr(), xaddr("α:1"))
+
+	// zero port always stays unused even after autobind
+	_, err = hα.Dial(ctx, ":0")
+	assert.Eq(err, &net.OpError{Op: "dial", Net: subnet.Network(), Source: xaddr("α:2"), Addr: xaddr("α:0"), Err: virtnet.ErrConnRefused})
+
+	wg := &errgroup.Group{}
+	wg.Go(exc.Funcx(func() {
+		c1s := xaccept(l1)
+		assert.Eq(c1s.LocalAddr(), xaddr("α:2"))
+		assert.Eq(c1s.RemoteAddr(), xaddr("β:1"))
+
+		assert.Eq(xread(c1s), "ping")		// XXX for !pipe could read less
+		xwrite(c1s, "pong")
+
+		c2s := xaccept(l1)
+		assert.Eq(c2s.LocalAddr(), xaddr("α:3"))
+		assert.Eq(c2s.RemoteAddr(), xaddr("β:2"))
+
+		assert.Eq(xread(c2s), "hello")
+		xwrite(c2s, "world")
+	}))
+
+	c1c := xdial(hβ, "α:1")
+	assert.Eq(c1c.LocalAddr(), xaddr("β:1"))
+	assert.Eq(c1c.RemoteAddr(), xaddr("α:2"))
+
+	xwrite(c1c, "ping")
+	assert.Eq(xread(c1c), "pong")
+
+	c2c := xdial(hβ, "α:1")
+	assert.Eq(c2c.LocalAddr(), xaddr("β:2"))
+	assert.Eq(c2c.RemoteAddr(), xaddr("α:3"))
+
+	xwrite(c2c, "hello")
+	assert.Eq(xread(c2c), "world")
+
+	xwait(wg)
+
+	l2 := xlisten(hα, ":0") // autobind again
+	assert.Eq(l2.Addr(), xaddr("α:4"))
+}

xnet/pipenet/pipenet_test.go

@@ -20,133 +20,38 @@
 package pipenet
 
 import (
-	"context"
-	"fmt"
-	"io"
-	"net"
-	"reflect"
 	"testing"
 
-	"golang.org/x/sync/errgroup"
-
-	"lab.nexedi.com/kirr/go123/exc"
+	"lab.nexedi.com/kirr/go123/internal/xtesting"
+	"lab.nexedi.com/kirr/go123/xnet/internal/virtnettest"
 )
 
-// we assume net.Pipe works ok; here we only test Listen/Accept/Dial routing
-// XXX tests are ugly, non-robust and small coverage
-
-type mklistener interface {
-	Listen(string) (net.Listener, error)
-}
-
-func xlisten(n mklistener, laddr string) net.Listener {
-	l, err := n.Listen(laddr)
-	exc.Raiseif(err)
-	return l
-}
-
-func xaccept(l net.Listener) net.Conn {
-	c, err := l.Accept()
-	exc.Raiseif(err)
-	return c
-}
-
-type dialer interface {
-	Dial(context.Context, string) (net.Conn, error)
-}
-
-func xdial(n dialer, addr string) net.Conn {
-	c, err := n.Dial(context.Background(), addr)
-	exc.Raiseif(err)
-	return c
-}
-
-func xread(r io.Reader) string {
-	buf := make([]byte, 4096)
-	n, err := r.Read(buf)
-	exc.Raiseif(err)
-	return string(buf[:n])
-}
-
-func xwrite(w io.Writer, data string) {
-	_, err := w.Write([]byte(data))
-	exc.Raiseif(err)
-}
-
-func xwait(w interface { Wait() error }) {
-	err := w.Wait()
-	exc.Raiseif(err)
-}
-
-func assertEq(t *testing.T, a, b interface{}) {
-	t.Helper()
-	if !reflect.DeepEqual(a, b) {
-		fmt.Printf("not equal:\nhave: %v\nwant: %v\n", a, b)
-		t.Errorf("not equal:\nhave: %v\nwant: %v", a, b)
-		exc.Raise(0)
-	}
+func TestPipeNet(t *testing.T) {
+	virtnettest.TestBasic(t, New("t").vnet)
 }
 
-
-func TestPipeNet(t *testing.T) {
+// pipenet has a bit different API than virtnet: Host has no error and returns
+// same instance if called twice, not dup, etc. Test it.
+func TestPipeNet2(t *testing.T) {
+	assert := xtesting.Assert(t)
 	pnet := New("t")
-	xaddr := func(addr string) *Addr {
-		a, err := pnet.ParseAddr(addr)
-		exc.Raiseif(err)
-		return a
-	}
 
 	hα := pnet.Host("α")
 	hβ := pnet.Host("β")
 
-	assertEq(t, hα.Network(), "pipet")
-	assertEq(t, hβ.Network(), "pipet")
-	assertEq(t, hα.Name(), "α")
-	assertEq(t, hβ.Name(), "β")
-
-	_, err := hα.Dial(context.Background(), ":0")
-	assertEq(t, err, &net.OpError{Op: "dial", Net: "pipet", Addr: xaddr("α:0"), Err: errConnRefused})
-
-	l1 := xlisten(hα, "")
-	assertEq(t, l1.Addr(), xaddr("α:1"))
-
-	// zero port always stays unused even after autobind
-	_, err = hα.Dial(context.Background(), ":0")
-	assertEq(t, err, &net.OpError{Op: "dial", Net: "pipet", Addr: xaddr("α:0"), Err: errConnRefused})
-
-	wg := &errgroup.Group{}
-	wg.Go(exc.Funcx(func() {
-		c1s := xaccept(l1)
-		assertEq(t, c1s.LocalAddr(), xaddr("α:2"))
-		assertEq(t, c1s.RemoteAddr(), xaddr("β:1"))
+	assert.Eq(hα.Network(), "pipet")
+	assert.Eq(hβ.Network(), "pipet")
+	assert.Eq(hα.Name(), "α")
+	assert.Eq(hβ.Name(), "β")
 
-		assertEq(t, xread(c1s), "ping")
-		xwrite(c1s, "pong")
+	hα2 := pnet.Host("α")
+	hβ2 := pnet.Host("β")
 
-		c2s := xaccept(l1)
-		assertEq(t, c2s.LocalAddr(), xaddr("α:3"))
-		assertEq(t, c2s.RemoteAddr(), xaddr("β:2"))
-
-		assertEq(t, xread(c2s), "hello")
-		xwrite(c2s, "world")
-	}))
-
-	c1c := xdial(hβ, "α:1")
-	assertEq(t, c1c.LocalAddr(), xaddr("β:1"))
-	assertEq(t, c1c.RemoteAddr(), xaddr("α:2"))
-
-	xwrite(c1c, "ping")
-	assertEq(t, xread(c1c), "pong")
-
-	c2c := xdial(hβ, "α:1")
-	assertEq(t, c2c.LocalAddr(), xaddr("β:2"))
-	assertEq(t, c2c.RemoteAddr(), xaddr("α:3"))
-
-	xwrite(c2c, "hello")
-	assertEq(t, xread(c2c), "world")
-
-	xwait(wg)
+	if !(hα == hα2 && hβ == hβ2) {
+		t.Fatalf("Host(x) is not idempotent")
+	}
 
-	l2 := xlisten(hα, ":0") // autobind again
-	assertEq(t, l2.Addr(), xaddr("α:4"))
+	if AsVirtNet(pnet) != pnet.vnet {
+		t.Fatal("AsVirtNet broken")
+	}
 }

xnet/virtnet/export_test.go

+// Copyright (C) 2018  Nexedi SA and Contributors.
+//                     Kirill Smelkov <kirr@nexedi.com>
+//
+// This program is free software: you can Use, Study, Modify and Redistribute
+// it under the terms of the GNU General Public License version 3, or (at your
+// option) any later version, as published by the Free Software Foundation.
+//
+// You can also Link and Combine this program with other software covered by
+// the terms of any of the Free Software licenses or any of the Open Source
+// Initiative approved licenses and Convey the resulting work. Corresponding
+// source of such a combination shall include the source code for all other
+// software used.
+//
+// This program is distributed WITHOUT ANY WARRANTY; without even the implied
+// warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+//
+// See COPYING file for full licensing terms.
+// See https://www.nexedi.com/licensing for rationale and options.
+
+package virtnet
+
+func SubnetShutdown(n *SubNetwork, err error) {
+	n.shutdown(err)
+}

xnet/virtnet/interfaces.go

+// Copyright (C) 2018  Nexedi SA and Contributors.
+//                     Kirill Smelkov <kirr@nexedi.com>
+//
+// This program is free software: you can Use, Study, Modify and Redistribute
+// it under the terms of the GNU General Public License version 3, or (at your
+// option) any later version, as published by the Free Software Foundation.
+//
+// You can also Link and Combine this program with other software covered by
+// the terms of any of the Free Software licenses or any of the Open Source
+// Initiative approved licenses and Convey the resulting work. Corresponding
+// source of such a combination shall include the source code for all other
+// software used.
+//
+// This program is distributed WITHOUT ANY WARRANTY; without even the implied
+// warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+//
+// See COPYING file for full licensing terms.
+// See https://www.nexedi.com/licensing for rationale and options.
+
+package virtnet
+// interfaces that virtnet uses in its working.
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"net"
+)
+
+// Engine is the interface for particular virtnet network implementation to be
+// used by SubNetwork.
+//
+// A virtnet network implementation should provide Engine instance to
+// SubNetwork when creating it. The subnetwork will use provided engine for its
+// operations.
+//
+// It should be safe to access Engine from multiple goroutines simultaneously.
+type Engine interface {
+	// VNetNewHost creates resources for host and announces it to registry.
+	//
+	// VNetNewHost should create resources for new host and announce
+	// hostname to provided registry. When announcing it should encode in
+	// hostdata a way for VNetDial - potentially run on another subnetwork
+	// - to find out where to connect to when dialing to this host.
+	//
+	// On error the returned error will be wrapped by virtnet with "new
+	// host" operation and hostname.
+	VNetNewHost(ctx context.Context, hostname string, registry Registry) error
+
+	// VNetDial creates outbound virtnet connection.
+	//
+	// VNetDial, given destination virtnet address and destination
+	// hostdata, should establish connection to destination. It should let
+	// remote side know that its peer virtnet address is src.
+	//
+	// On success net.Conn that will be handling data exchange via its
+	// Read/Write should be returned. This net.Conn will be wrapped by
+	// virtnet with overwritten LocalAddr and RemoteAddr to be src and
+	// addrAccept correspondingly.
+	//
+	// On error the returned error will be wrapped by virtnet with
+	// corresponding net.OpError{"dial", src, dst}.
+	//
+	// Virtnet always passes to VNetDial src and dst with the same network
+	// name that was used when creating corresponding SubNetwork.
+	VNetDial(ctx context.Context, src, dst *Addr, dsthostdata string) (_ net.Conn, addrAccept *Addr, _ error)
+
+	// Close shuts down subnetwork engine.
+	//
+	// Close should close engine resources and return corresponding error.
+	//
+	// There is no need to explicitly interrupt other engine operations -
+	// to those virtnet always passes ctx that is canceled before
+	// engine.Close is called.
+	Close() error
+}
+
+
+// Notifier is the interface to be used by particular virtnet network
+// implementation for notifying SubNetwork.
+//
+// A virtnet network implementation receives instance of Notifier together with
+// SubNetwork when creating it. The network implementation should use provided
+// Notifier to notify the subnetwork to handle incoming events.
+//
+// It should be safe to access Notifier from multiple goroutines simultaneously.
+type Notifier interface {
+	// VNetAccept notifies virtnet about incoming connection.
+	//
+	// VNetAccept, given destination virtnet address, should make decision
+	// to either accept or reject provided connection.
+	//
+	// On success the connection is pre-accepted and corresponding Accept
+	// is returned to virtnet network implementation.
+	//
+	// On error an error is returned without any "accept" prefix, e.g.
+	// ErrConnRefused. Such accept prefix should be provided by network
+	// implementation that is using VNetAccept.
+	VNetAccept(ctx context.Context, src, dst *Addr, netconn net.Conn) (*Accept, error)
+
+	// VNetDown notifies virtnet that underlying network is down.
+	//
+	// Provided err describes the cause of why the network is down.
+	VNetDown(err error)
+}
+
+// Accept represents successful acceptance decision from Notifier.VNetAccept .
+//
+// On successful accept decision corresponding virtnet-level Accept() is
+// waiting on .Ack to continue and will check the error from there.
+//
+// On success the connection will be finally accepted and corresponding virtnet
+// listener will be notified. Provided netconn will be adjusted by virtnet
+// internally with overwritten LocalAddr and RemoteAddr to be correspondingly
+// .Addr and src that was originally passed to VNetAccept.
+//
+// On error the acceptance will be canceled.
+type Accept struct {
+	Addr *Addr      // accepting with this local address
+	Ack  chan error
+}
+
+
+// Registry represents access to a virtnet network registry.
+//
+// A virtnet network implementation should provide Registry instance to
+// SubNetwork when creating it. The subnetwork will eventually use it when
+// creating hosts via NewHost, and in turn creating outbound connections on
+// them via Host.Dial.
+//
+// The registry holds information about hosts available on the network, and
+// for each host additional data about it. Whenever host α needs to establish
+// connection to address on host β, it queries the registry for β.
+// Correspondingly when a host joins the network, it announces itself to the
+// registry so that other hosts could see it.
+//
+// The registry could be implemented in several ways, for example:
+//
+//	- dedicated network server,
+//	- hosts broadcasting information to each other similar to ARP,
+//	- shared memory or file,
+//	- ...
+//
+// It should be safe to access registry from multiple goroutines simultaneously.
+type Registry interface {
+	// Announce announces host to registry.
+	//
+	// Returned error, if !nil, is *RegistryError with .Err describing the
+	// error cause:
+	//
+	//	- ErrRegistryDown  if registry cannot be accessed,
+	//	- ErrHostDup       if hostname was already announced,
+	//	- some other error indicating e.g. IO problem.
+	Announce(ctx context.Context, hostname, hostdata string) error
+
+	// Query queries registry for host.
+	//
+	// Returned error, if !nil, is *RegistryError with .Err describing the
+	// error cause:
+	//
+	//	- ErrRegistryDown  if registry cannot be accessed,
+	//	- ErrNoHost        if hostname was not announced to registry,
+	//	- some other error indicating e.g. IO problem.
+	Query(ctx context.Context, hostname string) (hostdata string, _ error)
+
+	// Close closes access to registry.
+	//
+	// Close should close registry resources and return corresponding error.
+	//
+	// There is no need to explicitly interrupt other registry operations -
+	// to those virtnet always passes ctx that is canceled before
+	// registry.Close is called.
+	Close() error
+}
+
+var (
+	ErrRegistryDown = errors.New("registry is down")
+	ErrNoHost       = errors.New("no such host")
+	ErrHostDup      = errors.New("host already registered")
+)
+
+// RegistryError represents an error of a registry operation.
+type RegistryError struct {
+	Registry string      // name of the registry
+	Op       string      // operation that failed
+	Args     interface{} // operation arguments, if any
+	Err      error       // actual error that occurred during the operation
+}
+
+func (e *RegistryError) Error() string {
+	s := e.Registry + ": " + e.Op
+	if e.Args != nil {
+		s += fmt.Sprintf(" %q", e.Args)
+	}
+	s += ": " + e.Err.Error()
+	return s
+}
+
+func (e *RegistryError) Cause() error {
+	return e.Err
+}

xnet/virtnet/virtnet_test.go

+// Copyright (C) 2018  Nexedi SA and Contributors.
+//                     Kirill Smelkov <kirr@nexedi.com>
+//
+// This program is free software: you can Use, Study, Modify and Redistribute
+// it under the terms of the GNU General Public License version 3, or (at your
+// option) any later version, as published by the Free Software Foundation.
+//
+// You can also Link and Combine this program with other software covered by
+// the terms of any of the Free Software licenses or any of the Open Source
+// Initiative approved licenses and Convey the resulting work. Corresponding
+// source of such a combination shall include the source code for all other
+// software used.
+//
+// This program is distributed WITHOUT ANY WARRANTY; without even the implied
+// warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+//
+// See COPYING file for full licensing terms.
+// See https://www.nexedi.com/licensing for rationale and options.
+
+package virtnet_test
+
+import (
+	"context"
+	"io"
+	"net"
+	"strings"
+	"testing"
+	"time"
+
+	"golang.org/x/sync/errgroup"
+
+	"lab.nexedi.com/kirr/go123/exc"
+	"lab.nexedi.com/kirr/go123/internal/xtesting"
+	"lab.nexedi.com/kirr/go123/xnet/pipenet"
+	. "lab.nexedi.com/kirr/go123/xnet/virtnet"
+
+	"github.com/pkg/errors"
+)
+
+// testNet is testing network environment.
+//
+// It consists of a subnetwork backed by pipenet with 2 hosts: hα and hβ. On
+// both hosts a listener is started at "" (i.e. it will have ":1" address).
+// There is a connection established in between α:2-β:2.
+type testNet struct {
+	testing.TB
+
+	net      *SubNetwork
+	hα, hβ   *Host
+	lα, lβ   net.Listener
+	cαβ, cβα net.Conn
+}
+
+// newTestNet creates new testing network environment.
+func newTestNet(t0 testing.TB) *testNet {
+	t := &testNet{TB: t0}
+	t.Helper()
+
+	var err error
+	t.net = pipenet.AsVirtNet(pipenet.New("t"))
+	t.hα, err = t.net.NewHost(context.Background(), "α")
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.hβ, err = t.net.NewHost(context.Background(), "β")
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.lα, err = t.hα.Listen("")
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.lβ, err = t.hβ.Listen("")
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	// preestablish α:2-β:2 connection
+	wg := &errgroup.Group{}
+	defer func() {
+		err := wg.Wait()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}()
+
+	wg.Go(func() error {
+		c, err := t.lβ.Accept()
+		if err != nil {
+			return err
+		}
+		t.cβα = c
+		return nil
+	})
+
+	c, err := t.hα.Dial(context.Background(), "β:1")
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.cαβ = c
+
+	return t
+}
+
+// xneterr constructs net.OpError for testNet network.
+//
+// if addr is of form "α:1" - only .Addr is set.
+// if addr is of form "α:1->β:1" - both .Source and .Addr are set.
+func xneterr(op, addr string, err error) *net.OpError {
+	addrv := strings.Split(addr, "->")
+	if len(addrv) > 2 {
+		exc.Raisef("xneterr: invalid addr %q", addr)
+	}
+
+	operr := &net.OpError{
+		Op:  op,
+		Net: "pipet", // matches newTestNet
+		Err: err,
+	}
+
+	for i, addr := range addrv {
+		a, e := ParseAddr("pipet", addr)
+		exc.Raiseif(e)
+
+		if i == len(addrv)-1 {
+			operr.Addr = a
+		} else {
+			operr.Source = a
+		}
+	}
+
+	return operr
+}
+
+// xobject lookups testNet object by name.
+func (t *testNet) xobject(name string) io.Closer {
+	switch name {
+	case "subnet":	return t.net
+	case "hα":	return t.hα
+	case "hβ":	return t.hβ
+	case "lα":	return t.lα
+	case "lβ":	return t.lβ
+	case "cαβ":	return t.cαβ
+	case "cβα":	return t.cβα
+	}
+
+	exc.Raisef("invalid object: %q", name)
+	panic(0)
+}
+
+type testFlag int
+const serialOnly testFlag = 1
+
+// testClose verifies object.Close vs test func.
+//
+// object to close is specified by name, e.g. "hβ". test func should try to do
+// an action and verify it gets expected error given object is closed.
+//
+// two scenarios are verified:
+//
+//	- serial case: first close, then test, and
+//	- concurrent case: close is run in parallel to test.
+//
+// if concurrent case is not applicable for test (e.g. it tries to run a
+// function that does not block, like e.g. NewHost in pipenet case), it can be
+// disabled via passing optional serialOnly flag.
+func testClose(t0 testing.TB, object string, test func(*testNet), flagv ...testFlag) {
+	t0.Helper()
+
+	// serial case
+	t := newTestNet(t0)
+	obj := t.xobject(object)
+
+	err := obj.Close()
+	if err != nil {
+		t.Fatal(err)
+	}
+	test(t)
+
+	if len(flagv) > 0 && flagv[0] == serialOnly {
+		return
+	}
+
+	// concurrent case
+	t = newTestNet(t0)
+	obj = t.xobject(object)
+
+	wg := &errgroup.Group{}
+	wg.Go(func() error {
+		tdelay()
+		return obj.Close()
+	})
+
+	test(t)
+	err = wg.Wait()
+	if err != nil {
+		t.Fatal(err)
+	}
+}
+
+// tdelay delays a bit.
+//
+// needed e.g. to test Close interaction with waiting read or write
+// (we cannot easily sync and make sure e.g. read is started and became asleep)
+func tdelay() {
+	time.Sleep(1 * time.Millisecond)
+}
+
+// TestClose verifies that for all virtnet objects Close properly interrupt /
+// errors all corresponding operations.
+func TestClose(t *testing.T) {
+	bg := context.Background()
+	assert := xtesting.Assert(t)
+
+	//          Subnet Host listener conn
+	// NewHost     x
+	// Dial        x     x      x
+	// Listen      x     x
+	// Accept      x     x      x
+	// Read/Write  x     x             x
+
+	// ---- NewHost ----
+
+	// subnet.NewHost vs subnet.Close
+	testClose(t, "subnet", func(t *testNet) {
+		h, err := t.net.NewHost(bg, "γ")
+		assert.Eq(h, (*Host)(nil))
+		assert.Eq(errors.Cause(err), ErrNetDown)
+		assert.Eq(err.Error(), "virtnet \"pipet\": new host \"γ\": network is down")
+	}, serialOnly)
+
+	// ---- Dial ----
+
+	// host.Dial vs subnet.Close
+	testClose(t, "subnet", func(t *testNet) {
+		c, err := t.hα.Dial(bg, "β:1")
+		assert.Eq(c, nil)
+		assert.Eq(err, xneterr("dial", "α:3->β:1", ErrNetDown))
+	})
+
+	// host1.Dial vs host1.Close
+	testClose(t, "hα", func(t *testNet) {
+		c, err := t.hα.Dial(bg, "β:1")
+		assert.Eq(c, nil)
+		assert.Eq(err, xneterr("dial", "α:3->β:1", ErrHostDown))
+	})
+
+	// host1.Dial vs host2.Close
+	testClose(t, "hβ", func(t *testNet) {
+		c, err := t.hα.Dial(bg, "β:1")
+		assert.Eq(c, nil)
+		assert.Eq(err, xneterr("dial", "α:3->β:1", ErrConnRefused))
+	})
+
+	// host1.Dial vs host2.listener.Close
+	testClose(t, "lβ", func(t *testNet) {
+		c, err := t.hα.Dial(bg, "β:1")
+		assert.Eq(c, nil)
+		assert.Eq(err, xneterr("dial", "α:3->β:1", ErrConnRefused))
+	})
+
+	// ---- Listen ----
+
+	// host.Listen vs subnet.Close
+	testClose(t, "subnet", func(t *testNet) {
+		l, err := t.hα.Listen("")
+		assert.Eq(l, nil)
+		assert.Eq(err, xneterr("listen", "α:0", ErrNetDown))
+	}, serialOnly)
+
+	// host.Listen vs host.Close
+	testClose(t, "hα", func(t *testNet) {
+		l, err := t.hα.Listen("")
+		assert.Eq(l, nil)
+		assert.Eq(err, xneterr("listen", "α:0", ErrHostDown))
+	}, serialOnly)
+
+	// ---- Accept ----
+
+	// listener.Accept vs subnet.Close
+	testClose(t, "subnet", func(t *testNet) {
+		c, err := t.lα.Accept()
+		assert.Eq(c, nil)
+		assert.Eq(err, xneterr("accept", "α:1", ErrNetDown))
+	})
+
+	// listener.Accept vs host.Close
+	testClose(t, "hα", func(t *testNet) {
+		c, err := t.lα.Accept()
+		assert.Eq(c, nil)
+		assert.Eq(err, xneterr("accept", "α:1", ErrHostDown))
+	})
+
+	// listener.Accept vs listener.Close
+	testClose(t, "lα", func(t *testNet) {
+		c, err := t.lα.Accept()
+		assert.Eq(c, nil)
+		assert.Eq(err, xneterr("accept", "α:1", ErrSockDown))
+	})
+
+	// ---- Read/Write ----
+
+	buf := []byte("hello world!")
+
+	// conn.{Read,Write} vs subnet.Close
+	testClose(t, "subnet", func(t *testNet) {
+		n, err := t.cαβ.Read(buf)
+		assert.Eq(n, 0)
+		// err can be also EOF because subnet.Close closes cβα too and
+		// depending on scheduling we might first get EOF on our end.
+		if err != io.EOF {
+			assert.Eq(err, xneterr("read", "β:2->α:2", ErrNetDown))
+		}
+	})
+	testClose(t, "subnet", func(t *testNet) {
+		n, err := t.cαβ.Write(buf)
+		assert.Eq(n, 0)
+		assert.Eq(err, xneterr("write", "α:2->β:2", ErrNetDown))
+	})
+
+	// conn1.{Read,Write} vs host1.Close
+	testClose(t, "hα", func(t *testNet) {
+		n, err := t.cαβ.Read(buf)
+		assert.Eq(n, 0)
+		assert.Eq(err, xneterr("read", "β:2->α:2", ErrHostDown))
+	})
+	testClose(t, "hα", func(t *testNet) {
+		n, err := t.cαβ.Write(buf)
+		assert.Eq(n, 0)
+		assert.Eq(err, xneterr("write", "α:2->β:2", ErrHostDown))
+	})
+
+	// conn1.{Read,Write} vs host2.Close
+	testClose(t, "hβ", func(t *testNet) {
+		n, err := t.cαβ.Read(buf)
+		assert.Eq(n, 0)
+		assert.Eq(err, io.EOF)
+	})
+	testClose(t, "hβ", func(t *testNet) {
+		n, err := t.cαβ.Write(buf)
+		assert.Eq(n, 0)
+		assert.Eq(err, xneterr("write", "α:2->β:2", io.ErrClosedPipe))
+	})
+
+	// conn1.{Read,Write} vs conn1.Close
+	testClose(t, "cαβ", func(t *testNet) {
+		n, err := t.cαβ.Read(buf)
+		assert.Eq(n, 0)
+		assert.Eq(err, xneterr("read", "β:2->α:2", ErrSockDown))
+	})
+	testClose(t, "cαβ", func(t *testNet) {
+		n, err := t.cαβ.Write(buf)
+		assert.Eq(n, 0)
+		assert.Eq(err, xneterr("write", "α:2->β:2", ErrSockDown))
+	})
+
+	// conn1.{Read,Write} vs conn2.Close
+	testClose(t, "cβα", func(t *testNet) {
+		n, err := t.cαβ.Read(buf)
+		assert.Eq(n, 0)
+		assert.Eq(err, io.EOF)
+	})
+	testClose(t, "cβα", func(t *testNet) {
+		n, err := t.cαβ.Write(buf)
+		assert.Eq(n, 0)
+		assert.Eq(err, xneterr("write", "α:2->β:2", io.ErrClosedPipe))
+	})
+}
+
+// TestVNetDown verifies that engine shutdown error signal is properly handled.
+func TestVNetDown(t0 *testing.T) {
+	assert := xtesting.Assert(t0)
+
+	t := newTestNet(t0)
+	errSomeProblem := errors.New("some problem")
+	SubnetShutdown(t.net, errSomeProblem) // notifier.VNetDown does this
+
+	// SubNetwork.Close = shutdown(nil) and all that interactions were
+	// verified in TestClose. Here lets check only that we get proper Close error.
+	err := t.net.Close()
+	assert.Eq(errors.Cause(err), errSomeProblem)
+	assert.Eq(err.Error(), "virtnet \"pipet\": close: some problem")
+}