See merge request !860

When invoking tests with setup.py test, missing eggs are installed before
running tests, using their latest available version, but we want to always
install eggs using buildout and using pinned versions.

The section template-slapuser-script creates the "bin/slapos" wrapper
which is needed inside runner-import.sh script.

Otherwise it's almost impossible to check for which node statistics are shown.

Fixes 8d5910dc

Limit stats to frontend entries only, as they are the most important for the
frontend operator, and also having thousands of entries makes the stats page
unusable.

fixes 8d5910dc

Despite the introspection has replaced old style Caddy-based log-access, some
bits were not removed, so remove then now.

fixes 0c830c4c

It was done in frontend partition, but missed in kedifa.

fixes 3469e864

This reverts commit 5d46a499.

Changes to "fix argument parsing" (to allow passing argument containing space)
had a regression that backslash characters are escaped.

Let's go back to slapos.toolbox 0.112 for now.

When updating existing instance, running slapos node instance failed with:

    ln: failed to create symbolic link '/srv/slapgrid/slappart9/srv/frontend-static/logo.png': File exists

This fixes by doing nothing if logo.png already exists

In 74d18b9d I switched ERP5 to use caucase certificate, but caucase is not yet usable for this and this caused issues with upgrading existing instances.

We'll maybe try caucase again here, in that case 620c9332 can be reverted if we want to reuse some parts of what was made.

See merge request !857

Revert "software/erp5: use a caucase managed certificate for balancer"

This reverts commit 74d18b9d and also follow
up fixup ( 555b26a2 ).

We are not ready to use caucase here, there are still too many problems with
caucase (keys are lost at each SR update etc) and design might still evolve,
so let's go back to self signed certificate for now.

Also remove the promise and the updater, since they are also not working and
causing problems on instances that have been updated (and where the key no
longer match the certificate)

fix balancer CSR generation:

Caucase rerequest uses a CSR *template* and use it to generate
a new CSR with a new key, so we should not use the actual key to
generate this CSR, because it is caucase rerequest job to generate
the key.
Also, we should be careful not to generate a new CSR every time this
command run, otherwise a new key will be generated and a new CSR will
be sent to caucase, but caucase will not sign it automatically (since
we configure it to sign only one certificate).

This means that the case of IP address changes is currently not
supported automatically. To support it we would need to:
  - force generation of a new CSR template
  - force caucase rerequester to request a new certificate (by removing
  existing certificate)
  - force caucased to sign the new certificate

This commit also fix indentation and remove simplefile macro that is no longer used

See merge request nexedi/slapos!837

Just running test.py with ip and port allows to expose the internal testing
backend. IPv4 and IPv6 are supported.

By default there is no sense to play with software type, since it's fixed in
slapos.testing.testcase.

slapos.cookbook:softwaretype tends to lowercase keys in each section, which
has been undetected due to using lowercase references of slaves in the tests.

By restructuring information in the sections, and putting slave references
inside of dumped part of information, now the slave reference case is kept.

Also real care was taken to stabilise published lists by sorting them, as it
also slipped, that they could be unstable.

Tests has been updated to catch this issue, also other tests were fixed, as
they had wrong assertions.

See commit e83edfc9.

See merge request nexedi/slapos!850

This reverts commit cc1713c3.

Unfortunately a lot of installation OS ISOs (like Debian 9 and 10) do not
support such device during installation process, because of missing drivers
on the OS CD.

Also note the ISOs limitation in the advanced field.

Make the value and its changes easier to read.

Revert f8f72a17 ([erp5] don't use caucase generated certificate for now, 2019-03-12) since nothing prevents us drom using caucase certificate now.
 
Use [managed resources](nexedi/slapos.core!259) to simplify existing tests and introduce tests for:

## Access Log

 - [x] balancer partition should produce logs in apache "combined" log format with microsecond timing of requests.
 - [x] these logs should be rotated daily
 - [x] an [apachedex](https://lab.nexedi.com/nexedi/apachedex) report is ran on these logs daily.

## Balancing

 - [x] requests are balanced to multiple backends using round-robin algorithm
 - [x] if backend is down it is excluded
 - [x] a "sticky cookie" is used so that clients are associated to the same backend
    - [x] the cookie is set by balancer
    - [x] when client comes with a cookie it "sticks" on the associated backend
    - [x] if "sticked" backend is down, another backend will be used

## Content-Encoding

 - [x] balancer encodes responses in gzip for some configured content types.

## HTTP

 - [x] Server uses HTTP/1.1 or more and keep connection with clients

## TLS (server certificate)

In this MR we also change apache to use a caucase managed certificate and add test coverage for:

 - [x] balancer listen on https with a certificate that can be verified using the CA from caucase.
 - [x] balancer uses the new certificate when its own certificate is renewed.

But we don't add support for:
 -  ~~balancer can be instantiated with a certificate and key passed as SlapOS request parameters (code [here](https://lab.nexedi.com/nexedi/slapos/blob/757c1a4ddee93659d5e2649e4252d87bf9494566/stack/erp5/instance-balancer.cfg.in#L208-213))~~ this use case is the job of caucase, so we no longer support this.

## TLS (client certificate)
 - [x] balancer verifies frontend certificates from frontend caucases ( also tested in "Forwarded-For" section )
 - [x] if frontend provided a verified certificate, balancer set `remote-user` header
 - [x] balancer updates CRL from caucases ( `caucase-updater-housekeeper` )
 - (NOT TESTED) balancer updates CA certificate from caucase ( `caucase-updater-housekeeper` ). Since this is would be complex to test and basic functionality of `caucase-updater-housekeeper` for frontend caucases is covered by CRL test, we don't test this for simplicity.

## "Forwarded-For" header

This was also covered by existing tests:  

 - [x] balancer set `X-Forwarded-For` header when frontend certificate can be verified
 - [x] balancer strips existing `X-Forwarded-For`

## Integration with the rest of ERP5 software release

This was also covered by existing tests:  

- [x] The https URL of each Zope family is published and replies properly
- [x] Some https URLs are generated for `runUnitTest`, so that test run with an https certificate. This is also covered by regular ERP5 functional tests.

See merge request nexedi/slapos!840

See merge request nexedi/slapos!839

https://github.com/eclipse-theia/theia/blob/v1.7.0/CHANGELOG.md#v170---29102020

The only change to support python3 in software/plantuml is the move
from hexagonit to slapos download recipe in component/tomcat.
We also move the test to python3 by using plantuml egg v0.3.0. As this
egg doesn't support python2, the test is not done anymore in Python2.

Future Cython+ work will use it.