See merge request nexedi/slapos!865

This is to match Theia file hierarchy

See merge request nexedi/slapos!866

https://github.com/eclipse-theia/theia/blob/v1.8.0/CHANGELOG.md#v180---26112020

Until now, standalone subsystem was started as a daemon first time a terminal
was openned and since it was running as daemon, stopping the theia instance
did not stop any of the services running in the embedded slapos.

Before nexedi/slapos.core!265 there was
two supervisor running as daemon:
 - ~/srv/slapos/etc/supervisord.conf which runs slapos proxy etc
 - ~/srv/slapos/inst/etc/supervisord.conf with runs instances in the embedded
   slapos
After, the second one runs as a service in the first one, but the first one
was still running as daemon.

This changes so that the first supervisor runs as a service managed by the
Theia instance, so stopping Theia instance will effectively stop the services.

When upgrading, running instances should continue to run as detached. To
attach them to the new service, procedure could be something like this

Stop supervisors inside Theia instance:

  supervisorctl -c ~/srv/slapos/etc/supervisord.conf shutdown
  supervisorctl -c ~/srv/slapos/inst/etc/supervisord.conf shutdown

Restart slappartX:slapos-standalone-instance-XXX-on-watch from host slapos

We are observing some segmentation fault with python curses applications
that might be because the TERMINFO are different from the system one.
I did not debugged, but since I set this environment variable I did not
observe any segmentation fault.

By comparing strace, the invocations seems same with or without $TERMINFO.
In both cases the termcaps from the correct ncurses are selected, this
just seem to workaround for some reason.

 * switch software/slapos-testing (`SlapOS.Eggs.UnitTest-*`) to use nxdtest and update README to describe how to run tests locally with nxdtest's runTestSuite. Switch tests to `python -m unittest` (or slight variations) instead of the deprecated `python setup.py`, except a few tests which only pass when invoked with `setup.py`.
 * switch software/slapos-sr-testing (`SlapOS.SoftwareReleases.IntegrationTest-*`) to use nxdtest and update README to describe how to run tests locally with nxdtest's runTestSuite. Switch all tests to `python -m unittest discover` intead of the deprecated `python setup.py` - which was especially problematic in these tests, because it sets `$PYTHONPATH` which affect subprocesses ran by the tests. In follow up commits we'll drop the workarounds for `$PYTHONPATH`.

`EggTestSuite` becomes unused and will be removed in follow up changes.

See merge request !862

 * use python3 (newest builds requires python3)
 * correctly exclude release directories during rsync so that resiliency
works
 * use correct names for project/software/instances

We also try to switch the tests to use python -m unittest as a way of
invoking tests, instead of the deprecated python setup.py test

When use this interpreter to run buildout, buildout will pre-compile eggs with
different level of optimisations, for this it spawns sys.executable with -O
option (see [1]). Since pymain does not support -O argument and python only
support changing optimization level at startup, we can not easily handle -O
argument, execpt by re-executing ourselves with PYTHONOPTIMIZE environment
variable, which has same effect than setting -O flag

1: https://lab.nexedi.com/nexedi/slapos.buildout/blob/488b2dfd3088cba7b322827c904fe1f93fba73e6/src/zc/buildout/easy_install.py#L1716-1722

Change to run all tests with python -m unittest, which will allow us to
remove workaround for the $PYTHONPATH set by setup.py test.

For this we use `[python-interpreter]` which uses pygolang.main to behave
like a "normal" python interpreter with eggs available, but without the
usual problems of interpreters generated by zc.recipe.eggs - it supports
all command line flags and have current directory in sys.path.

Tests now have an explicit name, that we set to the software name as before,
and the special test dynamically checking all profiles json schemas (part
of slapos.cookbook egg, at the root of slapos repository),is named
json-schemas. It is also changed to run as default tests in python2 and
python3, not only in extras for python2.

fixes "component/trafficserver: Version up to 8.1.0"

Jérome reports that Pygolang build on Python3 fails because geventmp turned out to be not pinned:

nexedi/slapos!862 (comment 121359)

Geventmp is indeed used by pygolang only on Python3

nexedi/pygolang@cd67996e

and with Python2-only testing I missed to handle it.

-> Fix it and add test for pygolang/py3.

/reviewed-on nexedi/slapos!864

See merge request nexedi/slapos!856

Globbing can be unreliable after incremental profile update and rebuild because
buildout does not remove old eggs. Also it is much more clean to be precise.

-> Use [python-interpreter] to build bin/python with sys.path to wendelin.core
and other needed eggs.

Without referencing component/numpy/ numpy will be installed from just PyPI.

Go1.15 is incremental improvement over Go1.14 with better compiler and runtime:

https://blog.golang.org/go1.15
https://golang.org/doc/go1.15

Don't drop support for Go1.13 yet, as that (no longer supported) Go
release is still being used by replication-manager and restic
components.

Remain default at Go1.14 yet.

Switch helloworld to Go1.15 and test this patch on that
software-release.

Going Go1.14.10 -> Go1.14.12 brings in runtime and security fixes:

https://golang.org/doc/devel/release.html#go1.14

Tested on helloworld SR.

Tests added:
- connexion parameters
- monitor url
- slaves instanciation

Adds connexion parameters:
- ipv6
- port
- ipv4

Upgrade dnspython from 1.15.0 to 1.16.0

Removes unused parameters in instance-powerdns-replicate.cfg.jinja:
- private-ipv4
- public-ipv4
- domain

See merge request nexedi/slapos!860

When invoking tests with setup.py test, missing eggs are installed before
running tests, using their latest available version, but we want to always
install eggs using buildout and using pinned versions.

The section template-slapuser-script creates the "bin/slapos" wrapper
which is needed inside runner-import.sh script.

Otherwise it's almost impossible to check for which node statistics are shown.

Fixes 8d5910dc

Limit stats to frontend entries only, as they are the most important for the
frontend operator, and also having thousands of entries makes the stats page
unusable.

fixes 8d5910dc

Despite the introspection has replaced old style Caddy-based log-access, some
bits were not removed, so remove then now.

fixes 0c830c4c

It was done in frontend partition, but missed in kedifa.

fixes 3469e864

This reverts commit 5d46a499.

Changes to "fix argument parsing" (to allow passing argument containing space)
had a regression that backslash characters are escaped.

Let's go back to slapos.toolbox 0.112 for now.