See merge request nexedi/slapos!837

Just running test.py with ip and port allows to expose the internal testing
backend. IPv4 and IPv6 are supported.

By default there is no sense to play with software type, since it's fixed in
slapos.testing.testcase.

slapos.cookbook:softwaretype tends to lowercase keys in each section, which
has been undetected due to using lowercase references of slaves in the tests.

By restructuring information in the sections, and putting slave references
inside of dumped part of information, now the slave reference case is kept.

Also real care was taken to stabilise published lists by sorting them, as it
also slipped, that they could be unstable.

Tests has been updated to catch this issue, also other tests were fixed, as
they had wrong assertions.

See commit e83edfc9.

See merge request !850

This reverts commit cc1713c3.

Unfortunately a lot of installation OS ISOs (like Debian 9 and 10) do not
support such device during installation process, because of missing drivers
on the OS CD.

Also note the ISOs limitation in the advanced field.

Make the value and its changes easier to read.

Revert f8f72a17 ([erp5] don't use caucase generated certificate for now, 2019-03-12) since nothing prevents us drom using caucase certificate now.
 
Use [managed resources](nexedi/slapos.core!259) to simplify existing tests and introduce tests for:

## Access Log

 - [x] balancer partition should produce logs in apache "combined" log format with microsecond timing of requests.
 - [x] these logs should be rotated daily
 - [x] an [apachedex](https://lab.nexedi.com/nexedi/apachedex) report is ran on these logs daily.

## Balancing

 - [x] requests are balanced to multiple backends using round-robin algorithm
 - [x] if backend is down it is excluded
 - [x] a "sticky cookie" is used so that clients are associated to the same backend
    - [x] the cookie is set by balancer
    - [x] when client comes with a cookie it "sticks" on the associated backend
    - [x] if "sticked" backend is down, another backend will be used

## Content-Encoding

 - [x] balancer encodes responses in gzip for some configured content types.

## HTTP

 - [x] Server uses HTTP/1.1 or more and keep connection with clients

## TLS (server certificate)

In this MR we also change apache to use a caucase managed certificate and add test coverage for:

 - [x] balancer listen on https with a certificate that can be verified using the CA from caucase.
 - [x] balancer uses the new certificate when its own certificate is renewed.

But we don't add support for:
 -  ~~balancer can be instantiated with a certificate and key passed as SlapOS request parameters (code [here](https://lab.nexedi.com/nexedi/slapos/blob/757c1a4ddee93659d5e2649e4252d87bf9494566/stack/erp5/instance-balancer.cfg.in#L208-213))~~ this use case is the job of caucase, so we no longer support this.

## TLS (client certificate)
 - [x] balancer verifies frontend certificates from frontend caucases ( also tested in "Forwarded-For" section )
 - [x] if frontend provided a verified certificate, balancer set `remote-user` header
 - [x] balancer updates CRL from caucases ( `caucase-updater-housekeeper` )
 - (NOT TESTED) balancer updates CA certificate from caucase ( `caucase-updater-housekeeper` ). Since this is would be complex to test and basic functionality of `caucase-updater-housekeeper` for frontend caucases is covered by CRL test, we don't test this for simplicity.

## "Forwarded-For" header

This was also covered by existing tests:  

 - [x] balancer set `X-Forwarded-For` header when frontend certificate can be verified
 - [x] balancer strips existing `X-Forwarded-For`

## Integration with the rest of ERP5 software release

This was also covered by existing tests:  

- [x] The https URL of each Zope family is published and replies properly
- [x] Some https URLs are generated for `runUnitTest`, so that test run with an https certificate. This is also covered by regular ERP5 functional tests.

See merge request nexedi/slapos!840

See merge request nexedi/slapos!839

https://github.com/eclipse-theia/theia/blob/v1.7.0/CHANGELOG.md#v170---29102020

The only change to support python3 in software/plantuml is the move
from hexagonit to slapos download recipe in component/tomcat.
We also move the test to python3 by using plantuml egg v0.3.0. As this
egg doesn't support python2, the test is not done anymore in Python2.

Future Cython+ work will use it.

Fix for #20200514-218C705 - \[testnode\] frontend for log access

Depends on nexedi/erp5!1304

See merge request nexedi/slapos!848

This fixes the nextcloud SR, which was broken by
commit 92779bf4
(mariadb is not a part anymore).

This fixes commit a62e5e7b.
See also commit 491e6e28.

And set it as log_frontend_url in testnode config

Just add the following 2 lines in a SR:

[mariadb]
location = ${mariadb-10.4:location}

It does not build with GCC 8.2

Going Go1.14.9 -> Go1.14.10 brings in compiler and runtime fixes
including fix for crash in garbage-collector due to race condition:

https://github.com/golang/go/issues/40642
https://github.com/golang/go/issues/40641

Tested on helloworld SR.

See merge request nexedi/slapos!844

validators.url is enough, even for Caddy, to check that URL is correct, and
caddy_backend_url_validator was introduced before validators.

Also calling an external command for each slave takes a lot of time.

Thanks to this other sections can directly reference them, and so they are
correctly created as needed, so linking section does not need update-command

The password is anyway present in the section itself, so it's eventual change
will result with reinstalling the section.

Also change a bit existing frontend_url to manage it the same way.

This way buildout can reuse egg caches and it's a bit faster:

To run a simple instance buildout, from 2.837s it goes down to 1.875s.
To run slapos node instance 10 times just after requesting an ERP5 instance, it goes from ~112s to 98s

before:

    hyperfine "/srv/slapgrid/slappart4/srv/slapos/inst/slappart0/tmp/shared/python2.7/60364a13cc977dd5a894e0239ac889b9/bin/python2.7 /srv/slapgrid/slappart4/srv/slapos/inst/slappart0/tmp/soft/c63ba7265399450b28f9ea6d5667a5e7/bin/buildout -U"
    Benchmark #1: /srv/slapgrid/slappart4/srv/slapos/inst/slappart0/tmp/shared/python2.7/60364a13cc977dd5a894e0239ac889b9/bin/python2.7 /srv/slapgrid/slappart4/srv/slapos/inst/slappart0/tmp/soft/c63ba7265399450b28f9ea6d5667a5e7/bin/buildout -U
      Time (mean ± σ):      2.837 s ±  0.275 s    [User: 2.481 s, System: 0.285 s]
      Range (min … max):    2.482 s …  3.222 s    10 runs

after:

    hyperfine "/srv/slapgrid/slappart4/srv/slapos/inst/slappart0/tmp/shared/python2.7/60364a13cc977dd5a894e0239ac889b9/bin/python2.7 /srv/slapgrid/slappart4/srv/slapos/inst/slappart0/tmp/soft/c63ba7265399450b28f9ea6d5667a5e7/bin/buildout -U"
    Benchmark #1: /srv/slapgrid/slappart4/srv/slapos/inst/slappart0/tmp/shared/python2.7/60364a13cc977dd5a894e0239ac889b9/bin/python2.7 /srv/slapgrid/slappart4/srv/slapos/inst/slappart0/tmp/soft/c63ba7265399450b28f9ea6d5667a5e7/bin/buildout -U
      Time (mean ± σ):      1.875 s ±  0.067 s    [User: 1.660 s, System: 0.148 s]
      Range (min … max):    1.816 s …  2.038 s    10 runs

See merge request nexedi/slapos!846

Changes configuration files to run repman tests in python3.