See merge request nexedi/slapos!981

This fixes:

  Installing jupyter.
  While:
    Installing jupyter.
    Base installation request: ...
    Getting distribution for 'argon2-cffi'.
  Error: Picked: argon2-cffi = 20.1.0

Addition of argon2-cffi in commit 7d1ea024
was a last-minute change to fix jupyter SR.

Accessing http with path starting with /HTTPS/ would result with 503 responses
from ATS itself.

Test is covering the problem and also asserting that fix does not damage https
access with path starting with /HTTP/

This defers the rendering of `instance-theia.cfg.in` to the instance
creation time instead of the software installation time, which will
allow buildout sections to be added based on the instance parameters.

See merge request !981

Edggetest monitoring got capability of region definition in each cluster.

In order to stabilise parameter serialisation the json-in-xml has been used.

Slave reference can be dangerous for buildout validity or file names.

Request and publication is switched to fully serialised approach, so that
complex data structures like lists and objects can be safely transmitted.

The requests are backward compatible, so for current and simple usage
automatic region is setup.

Much more information are published to main and shared instances, like
lists of available and assigned regions. Regions can be added and removed,
which will be reflected automatically for the slaves.

Information send to the bot node are minimised to only needed ones.

check-frontend-ip-list can be configured according to the list, and each
element overrides previous one:

 * globally on cluster
 * default per region
 * globally per slave
 * specific per region on slave

Note: It's known that:

 {%-     set base_slave_dict = json_module.loads(slave.pop('_')) %} {#- XXX: Unsafe! #}

      is really unsafe, but for most of usage of monitoring slaves it's
      considered good enough.

Minimize amount of parameters passed from master partition to the slaves, as
slave configuration shall be maximally self contained.

Note: This will allow further development (like regionalization) much simpler.

Since json-in-xml is used, check-http-header-dict shall be simply the JSON
object, no need to express it as a string.

Note: This is request backward incompatible change.

This work results with VM being restarted on parameter change.

The VM will be stopped in case of parameter change, then the system will try
to start it back. For external resources (like downloadable images) it will be
waited until such images are correctly present before starting back the VM.

Image download is done asynchronously with image-download-controller system.

The instantiation will fail, which is expected.

This will minimise amount of needless restarts.

Special care is taken to work with files appearing later, by preparing them as
empty ones.

Covered assets for parameter change are:
 * software release
 * boot-image-url-list
 * boot-image-url-select
 * virtual-hard-drive-url

For sanity virtual-hard-drive-md5sum became mandatory.

virtual-hard-drive-gzipped case is covered.

In older theia versions `~/srv/frontend-static/logo.png` was a symlink
to the actual logo file. Now the logo is copied into `frontend-static`
but a simple `cp -f` does not behave as expected when the destination
already exists and is a symbolic link:
- if the symlink is valid, the file is copied to the symlink target
- if the symlink is broken, the copy command fails

So the destination must be removed before copying.