Recent fixes for proftpd were still not correct, we still have issues with rpath for mod_auth_web and proftpd itself which uses libcap if available, error was:
    
    ======================================================================
    ERROR: setUpModule (test)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/srv/slappart11/cqg/soft/1ae6b62cded47f49a2e77af2f372cf2a/parts/slapos.core-repository/slapos/testing/testcase.py", line 154, in setUpModule
        installSoftwareUrlList(cls, [software_url], debug=debug)
      File "/srv/slappart11/cqg/soft/1ae6b62cded47f49a2e77af2f372cf2a/parts/slapos.core-repository/slapos/testing/testcase.py", line 348, in installSoftwareUrlList
        raise e
    RuntimeError: /srv/slappart11/cqg/inst/test0-0/tmp/soft/728168be59c5353c6e3c6a6bc4cad052/parts/proftpd/bin/ftpdctl uses system library /lib/x86_64-linux-gnu/libcap.so.2 for libcap.so.2 /srv/slappart11/cqg/inst/test0-0/tmp/soft/728168be59c5353c6e3c6a6bc4cad052/parts/proftpd/libexec/mod_auth_web.so has some not found libraries:
    /srv/slappart11/cqg/inst/test0-0/tmp/soft/728168be59c5353c6e3c6a6bc4cad052/parts/proftpd/libexec/mod_auth_web.so: /usr/lib/x86_64-linux-gnu/libssl.so.1.1: version `OPENSSL_1_1_1' not found (required by /srv/slappart11/cqg/inst/test0-0/tmp/shared/curl/da798e87b4f10345c33fbae0a4593114/lib/libcurl.so.4)
    /srv/slappart11/cqg/inst/test0-0/tmp/soft/728168be59c5353c6e3c6a6bc4cad052/parts/proftpd/sbin/proftpd uses system library /lib/x86_64-linux-gnu/libcap.so.2 for libcap.so.2
    /srv/slappart11/cqg/inst/test0-0/tmp/soft/728168be59c5353c6e3c6a6bc4cad052/parts/proftpd/sbin/in.proftpd uses system library /lib/x86_64-linux-gnu/libcap.so.2 for libcap.so.2
    Ran 0 tests in 530.067s
    ----------------------------------------------------------------------

/reviewed-on nexedi/slapos!685

Because we were simply using "setupClass", all classes where storing
snapshots in the same folder, so if more than one class failed we also
had an error in storing snapshot.

/reviewed-on !683

libcap is a system library that some system may have and some may not,
but we don't want slapos software to depend on the base system.

See also d0a88346

This build command never used the LDFLAGS that we set, so instead
rewrite this to inject path as a -R option that will be passed to
libtool.

Now that we check that we don't link with system libraries ( slapos.core!172 ) we discovered missing rpaths in our executables.

/reviewed-on !682

1.3.6a had a fix for building mod_sftp using OpenSSL 1.1.x, so we can
use default openssl again.
This should fix problems with proftpd-mod_auth_web that was linked
against curl which already used openssl 1.1

We don't seem to need it and we don't want configure script to pickup
system one.

Our software using sshd were sometimes failing in tests, because the way they publish key fingerprint was racy.

It is based on `collective.recipe.shelloutput`, which as we can see in the [recipe code](https://github.com/collective/collective.recipe.shelloutput/blob/78e15c19/collective/recipe/shelloutput/__init__.py) operates on `__init__`.

We are using `collective.recipe.shelloutput` to capture the output of `ssh-keygen -lf $KEY` and this must run after the file `$KEY` is generated ( it is generated by another `plone.recipe.command` version). We were trying to run the `collective.recipe.shelloutput` after the `plone.recipe.command`, but that was incorrect anyway, because `collective.recipe.shelloutput` reads the file at `__init__` step, where `plone.recipe.command` creates the file at `install` step.
As we could see in test suite, it was sometimes working, when `slapos node instance` ran only once, but it sometimes  working, when `slapos node instance` ran more than once, for example because a promise failed and `slapos node instance` was retried.

Since `collective.recipe.shelloutput` does not take into account the exit code of the command but simply capture with `"Error ..."` whatever the command might output on stderr, we add another step checking that the captured output  is not `"Error ..."` and if it is cause a buildout error so that `slapos node instance` is retried and then succeed.

What should happen now is:
 1. `collective.recipe.shelloutput` reads the key fingerprint, the file is not present so it captures `"Error ..."``
 2. a `plone.recipe.command` creates the key
 3. another `plone.recipe.command` checks that the captured fingerprint is not `"Error ..."` it fails
 4. buildout restarts
 5. `collective.recipe.shelloutput` reads key fingerprint correctly.

Slaprunner has been heavily modified, because it was using a `sshkeys_authority` which was incompatible with this as it uses symlinks for keys. Since we don't know what is the purpose of `sshkeys_authority`, we rewrote that software to use simple commands instead of that "ssh keys authority".

/reviewed-on !681

slapos.core to 1.5.6 and slapos.toolbox to 0.104:

 * cleanup stale monitor files and stale promises
 * support download-from-binary-cache-force-url-list option

/reviewed-on nexedi/slapos!679

These keys are not managed by trust of a certificate authority, just by
"trust of first use" so it does not make sense to use a key authority.

This also cause difficulties to publish the key fingerprint as a
parameter, because we can't get the key fingerprint until the authority
service is started.

Also enable ecdsa key.

This fixes random failures with slaprunner tests, because the published
fingerprint was never correct on first buildout run.

Existing webrunners will have a new ssh host key after this.

Adjust `self._cleanup` to new signature from nexedi/slapos.core!156

/reviewed-on nexedi/slapos!677

Now that component/glib/buildout.cfg is no longer included by pkgconfig,
qemu no longer build with:

    While:
	Installing.
	Getting section template.
	Initializing section template.
	Getting section template-nbd.
	Initializing section template-nbd.
	Getting section kvm.
	Initializing section kvm.
	Getting option kvm:configure-options.
	Getting section glib.
	Error: The referenced section, 'glib', was not defined.

Also explictly add pcre to prevent same problem later.

The OS may not have Python (slapos-node package does not depend
on any version of Python). Another possible issue was found on Suse:

  make[3]: Entering directory '.../parts/glib__compile__/glib-2.56.4/gio'
    GEN      gdbus-daemon-generated.c
  Traceback (most recent call last):
    File "./gdbus-2.0/codegen/gdbus-codegen.in", line 53, in <module>
      from codegen import codegen_main
    File ".../parts/glib__compile__/glib-2.56.4/gio/gdbus-2.0/codegen/codegen_main.py", line 30, in <module>
      from . import parser
    File ".../parts/glib__compile__/glib-2.56.4/gio/gdbus-2.0/codegen/parser.py", line 23, in <module>
      import xml.parsers.expat
  ImportError: No module named xml.parsers.expat

Building normal glib is a nightmare, at least as long as we
haven't stopped using Python 2, so let's try to avoid it.

This makes bin/pkg-config bigger (GCC 6.3.0):
 old size:  47880
 new size: 573992

With Link Time Optimization, GCC 8.3.0 produces a binary of 154656 bytes.
However, GCC 6.3.0 fails to link so we have to wait.

Improvements in check_surykatka_promise:

 * SSL certificate checked for expiration (15 days before expiration)
 * messages from the promise are more readable

/reviewed-on !675

Supports SSL checks in check_surykatka_json.

This version exposes ssl_certificate information for hosts.

Caddy by itself does not raise soft limit of open files, so it has to be set
by the wrapper.

As slapos.cookbook:wrapper can't be used for such case, the
slapos.recipe.template:jinja2 recipe inline style is used to have full
control over the created wrapper.

/reviewed-on nexedi/slapos!678

Version 3.0.0 requires requests>=2.22.0

Errors were raised in ERP5 with classes inheriting from ExtensionClass.

This promise check if nextcloud app was correctly deployed, it ensure that slapos instance will deploy until nextcloud install succeed.

/reviewed-on nexedi/slapos!676

The called python program is python3 program, and provided
PYTHONPATH is from python2, resulting in bad behaviour.

On testnodes, python can be too long for linux shebang (#!) line.

We had for example:
    
    /srv/slapgrid/slappart15/srv/testnode/cqc/inst/test0-0/tmp/shared/python2.7/338ccc0e387d64185920266f8a97c87b/bin/python2.7
  

that is 123 characters long.

With this, the generated glib-genmarshall script will have:
    
    #!/usr/bin/env  /srv/slapgrid/slappart15/srv/testnode/cqc/inst/test0-0/tmp/shared/python2.7/338ccc0e387d64185920266f8a97c87b/bin/python2.7

which exceeds the linux kernel limit.

Instead, we generate scripts that will use
    
    #!/usr/bin/env python

and it will be the caller responsibility to set $PATH to use the expected python.


This fixes the issues we had on SlapOS.SoftwareReleases.IntegrationTest tests since we updated glib to 2.56.4 in a7a86db9

/reviewed-on nexedi/slapos!670