We don't want runUnitTest is to listen on 127.0.0.1, as it is not
allowed for SlapOS instances, runUnitTest wrapper is extended to include
`--zserver` with a pre-assigned ip:port.

runTestSuite starts several test in parrallel (controlled by
`--node_quantity` argument, which is passed by erp5testnode), so we need
to make sure that we provide it with enough ip:port. For this, we
extended runTestSuite with a `--zserver_address_list` argument and we
generate a wrapper with a list of `testrunner.node-count` pre-assigned
ip:ports.

Because zelenium tests needs to access this zserver over a secure origin
(otherwise modern browser features such as service worker are not
available), use an https proxy in the apache from the balancer
partition.

runUnitTest and runTestSuite have been extended with resp.
`--zserver_frontend_url` and `--zserver_frontend_url_list` arguments and
the URLs published by the balancer paritions are set in the wrappers.
For compatibility reasons, runTestSuite pass parameters as environment
variables.

Implementation notes:

This introduces a circular depencency, balancer partition needs to know
the address of the testrunners and zope partitions needs to know the
URLs of the corresponding http proxies on the apache. This is is handled
by `slapos.recipe:publish-early`:

 1. request zope family with an empty `test-runner-apache-url-list`.
    zope is instanciated a first time.
    zope returns `test-runner-address-list` ( a list of (host, port)
    tuples )
 2. request balancer with `test-runner-address-list`
    balancer is instanciated.
    balancer returns `{{ family_name }}-test-runner-url-list` ( a list
    of apache URLs ), which are published in the root partition.
 3. zope family is re-requested with
    updated`test-runner-apache-url-list` information

instance-erp5.cfg.in template was also reorganised to move
`[publish-early]` next to `[publish]` at the bottom of the file because
these sections are semantically related.
Also test runner generation is moved after zope generation, because we
want to allocate test runners ports after zopes, otherwise existing
zopes would get new ports when existing instances are upgraded.

`enable_authentication` parameter of `backend-list`
was not documented.

We have new parameters "how many parallel tests this instance can run"
( test-runner.node-count ) and "how many supplementary databases each
test needs" ( test-runner.extra-database-count ).

Instead of configuring "I need a total of 15 databases", we configure "I
need to be able to run 3 tests in parallel, each test requiring 4 extra
databases" which generates 3 * (4 +1) databases.

`mariadb.test-database-amount` becomes deprecated in favour of these new
arguments, but compatibility is preserved.

Slapos compatible wrapper is no longer installed in `parts/firefox/firefox-slapos` , but directly as `firefox` in the `${buildout:bin-directory}` of the software. `geckodriver` is also in the same directory.

Firefox version up to 52.9.0esr

Geckodriver version up to 0.17.0


/reviewed-on nexedi/slapos!387

  Allow the KVM software release request a disk via plugin,
  if the plugin is enabled the chown will be set to give the disk to the partition.

Not released yet functionality for regular expression cookie rewriting
is available: https://github.com/mholt/caddy/pull/2144

Add plugin assertions in TestSlave.test_master_partition_state and drop
check-free-disk-space from promise directory.

/reviewed-on nexedi/slapos!380

Introduce a simple tool to format JSON for software instance definitions.
This tool is inspired by `python -m json.tool`, but has the following differences:
 * keep the order of keys instead of sorting alphabetically (because we sometimes put the most important options at the top)
 * indent with 2 spaces

Run the tool on all the repository and commit the changes, with this command: 
```
for x in $(find . -name '*.json') ; do  echo $x; ./format-json $x $x; done
```

/reviewed-on nexedi/slapos!392

As check-free-disk-space promise is implemented in plugin directory, add it
to list of assertions and drop it from promise list assertions.

Also already existing monitoring plugins are asserted.

Before we move this code to a more suitable place, start by agreeing with small differences in the implementation.

This takes all the nice changes from caddy-frontend/tests/ but not what was specific to caddy or to testing slave instances.

/reviewed-on nexedi/slapos!389

This release should be now stable enough for release

/reviewed-on nexedi/slapos!391

In preparation for moving this to a better place, use the same code in
all the places where this code was duplicated.

Reconcile all implementations, document a little bit more the methods
and the future plans.

It is combination of requestoptional and serialised.

In order to avoid needless inheritance create JSONCodec and use it, also
with renaming Serialised to RequestJSONEncoded.

Also provide a test asserting that important log is seen in case if no expected
return is set.

Add testfixtures module to catch logging.

Current ESR version

Keep version 51 for now

This reverts 8f35ea4d
It should no longer be needed because erp5.util should now have the fix
from
nexedi/erp5@97f3310e

so that we don't have to escape % as %%

Major change is that slapos compatible wrapper is no longer installed in
parts/firefox/firefox-slapos , but directly as "firefox" in the
buildout:bin-directory of the software profile.
geckodriver is also in the same buildout:bin-directory.
This way, softwares using this just need to add
buildout:bin-directory to $PATH

erp5testnode knows since
erp5@59ee7970
that it should look for firefox in buildout:bin-directory of the
installed SR, so we should not need backward compatibility here.

Adjust seleniumrunner and jstestnode for the changes

Also drop unused firefox 45

Jérome changes from original 8cf6908d :
 - name sections [firefox-wrapper] and [firefox] because including the
version in section name means we have to also update section name when
we update version. Users have to be careful of installing
${firefox-wrapper:} and not ${firefox:}
 - introduce macros for implementation and simple section using the
macros, to make it easy to had more versions or to increase versions,
for both firefox and geckodriver. They are slightly different because
for firefox we need a wrapper, but geckodriver is usable directly.
 - use same version, the "versions up" will be separate commits.
 - now that seleniumrunner and jstestnode are using buildout.hash.cfg,
md5sums are in separate files.
Co-authored-by: Jérome Perrin <jerome@nexedi.com>