Fix collective.recipe.shelloutput running "too early"

Our software using sshd were sometimes failing in tests, because the way they publish key fingerprint was racy.

It is based on `collective.recipe.shelloutput`, which as we can see in the [recipe code](https://github.com/collective/collective.recipe.shelloutput/blob/78e15c19/collective/recipe/shelloutput/__init__.py) operates on `__init__`.

We are using `collective.recipe.shelloutput` to capture the output of `ssh-keygen -lf $KEY` and this must run after the file `$KEY` is generated ( it is generated by another `plone.recipe.command` version). We were trying to run the `collective.recipe.shelloutput` after the `plone.recipe.command`, but that was incorrect anyway, because `collective.recipe.shelloutput` reads the file at `__init__` step, where `plone.recipe.command` creates the file at `install` step.
As we could see in test suite, it was sometimes working, when `slapos node instance` ran only once, but it sometimes  working, when `slapos node instance` ran more than once, for example because a promise failed and `slapos node instance` was retried.

Since `collective.recipe.shelloutput` does not take into account the exit code of the command but simply capture with `"Error ..."` whatever the command might output on stderr, we add another step checking that the captured output  is not `"Error ..."` and if it is cause a buildout error so that `slapos node instance` is retried and then succeed.

What should happen now is:
 1. `collective.recipe.shelloutput` reads the key fingerprint, the file is not present so it captures `"Error ..."``
 2. a `plone.recipe.command` creates the key
 3. another `plone.recipe.command` checks that the captured fingerprint is not `"Error ..."` it fails
 4. buildout restarts
 5. `collective.recipe.shelloutput` reads key fingerprint correctly.

Slaprunner has been heavily modified, because it was using a `sshkeys_authority` which was incompatible with this as it uses symlinks for keys. Since we don't know what is the purpose of `sshkeys_authority`, we rewrote that software to use simple commands instead of that "ssh keys authority".

/reviewed-on nexedi/slapos!681

software/seleniumserver/buildout.hash.cfg

@@ -19,4 +19,4 @@ md5sum = c4ac5de141ae6a64848309af03e51d88
 
 [template-selenium]
 filename = instance-selenium.cfg.in
-md5sum = 4f557a7b3aa9b4df1ca1fa6a754ca657
+md5sum = 1f0b67d2a542e94380c35afc9cd1946b

software/seleniumserver/instance-selenium.cfg.in

@@ -184,18 +184,23 @@ extra-args=-t dsa
 <=ssh-keygen-base
 extra-args=-t ecdsa -b 521
 
-[ssh-key-fingerprint-command]
-recipe = plone.recipe.command
-# recent openssh client display ECDSA key's fingerprint as SHA256
-command = ${openssh-output:keygen} -lf $${ssh-host-ecdsa-key:output}
-
-[ssh-key-fingerprint]
+[ssh-key-fingerprint-shelloutput]
 recipe = collective.recipe.shelloutput
-# XXX because collective.recipe.shelloutput ignore errors, we run the same
-# command in a plone.recipe.command so that if fails if something goes wrong.
+# recent openssh client display ECDSA key's fingerprint as SHA256
 commands =
-  fingerprint = $${ssh-key-fingerprint-command:command}
+  fingerprint = ${openssh-output:keygen} -lf $${ssh-host-ecdsa-key:output}
 
+[ssh-key-fingerprint]
+recipe = plone.recipe.command
+stop-on-error = true
+# XXX because collective.recipe.shelloutput ignore errors and capture output
+# "Error ...", we use a plone.recipe.command to check that this command did
+# not fail.
+# This command will always fail on first buildout run, because
+# collective.recipe.shelloutput is evaluated at buildout recipes __init__ step,
+# but the key file is created later at install step.
+command = echo "$${:fingerprint}" | ( grep ^Error || exit 0 && exit 1 )
+fingerprint = $${ssh-key-fingerprint-shelloutput:fingerprint}
 
 [sshd-config]
 recipe = slapos.recipe.template:jinja2

software/slaprunner/buildout.hash.cfg

@@ -18,7 +18,7 @@ md5sum = 8b78e32b877d591400746ec7fd68ed4c
 
 [template-runner]
 filename = instance-runner.cfg
-md5sum = 87545b1f9f3865c8cb1347edeb340678
+md5sum = 1216494c03752f0a3c1755e190eed3dc
 
 [template-runner-import-script]
 filename = template/runner-import.sh.jinja2
@@ -26,7 +26,7 @@ md5sum = fc22e2d2f03ce58631f157a5b4943e15
 
 [instance-runner-import]
 filename = instance-runner-import.cfg.in
-md5sum = b450c474464a326f3d0b98728460ac97
+md5sum = 918fb2984cb2ed7afba9200167f98a0f
 
 [instance-runner-export]
 filename = instance-runner-export.cfg.in
@@ -50,7 +50,7 @@ md5sum = 525e37ea8b2acf6209869999b15071a6
 
 [template-slapos-cfg]
 filename = template/slapos.cfg.in
-md5sum = da113b3e3e7bac9cc215fede7c4911a5
+md5sum = e6a3ca1604ae5458248135cd6de0f3e6
 
 [template-parameters]
 filename = parameters.xml.in

software/slaprunner/instance-runner-import.cfg.in

@@ -14,12 +14,8 @@ parts +=
   slaprunner-promise
   slaprunner-supervisord-wrapper
   runner-sshd-add-authorized-key
-  runner-sshd-graceful
   runner-sshd-promise
-  runner-sshkeys-authority
-  runner-sshkeys-authority-service
-  runner-sshkeys-sshd
-  runner-sshkeys-sshd-service
+  runner-sshd-service
   runtestsuite
   shellinabox
   shellinabox-service

software/slaprunner/instance-runner.cfg

@@ -15,12 +15,8 @@ common-runner-parts =
   apache-httpd-promise
   slaprunner-supervisord-wrapper
   runner-sshd-add-authorized-key
-  runner-sshd-graceful
   runner-sshd-promise
-  runner-sshkeys-authority
-  runner-sshkeys-authority-service
-  runner-sshkeys-sshd
-  runner-sshkeys-sshd-service
+  runner-sshd-service
   runtestsuite
   symlinks
   shellinabox
@@ -177,9 +173,7 @@ shared_root = $${runnerdirectory:shared-root}
 buildout-shared-part-list-dump = ${template-buildout-shared-part-list:output}
 pidfile-software = $${directory:run}/slapgrid-cp.pid
 pidfile-instance = $${directory:run}/slapgrid-sr.pid
-ssh_client = ${openssh:location}/bin/ssh
-public_key = $${runner-sshd-raw-server:rsa-keyfile}.pub
-private_key = $${runner-sshd-raw-server:rsa-keyfile}
+public_key = $${runner-sshd-ssh-host-rsa-key:output}
 instance-monitor-url = https://[$${:ipv6}]:$${slap-parameter:monitor-httpd-port}
 etc_dir = $${directory:etc}
 log_dir =  $${directory:log}
@@ -256,106 +250,72 @@ ip = $${slap-network-information:global-ipv6}
 recipe = slapos.recipe.template:jinja2
 rendered = $${directory:etc}/runner-sshd.conf
 path_pid = $${directory:run}/runner-sshd.pid
-host_key = $${directory:ssh}/runner_server_key.rsa
 template = inline:
   PidFile $${:path_pid}
   Port $${runner-sshd-port:port}
   ListenAddress $${slap-network-information:global-ipv6}
   Protocol 2
   UsePrivilegeSeparation no
-  HostKey $${:host_key}
+  HostKey $${runner-sshd-ssh-host-rsa-key:output}
+  HostKey $${runner-sshd-ssh-host-ecdsa-key:output}
   PasswordAuthentication no
   PubkeyAuthentication yes
   AuthorizedKeysFile $${buildout:directory}/.ssh/authorized_keys
   ForceCommand cd $${directory:home}; if [ -z "$SSH_ORIGINAL_COMMAND" ]; then HOME=$${directory:home} $${shell-environment:shell} -l; else HOME=$${directory:home} SHELL=$${shell-environment:shell} PATH=$${shell-environment:path} eval "$SSH_ORIGINAL_COMMAND"; fi
   Subsystem sftp ${openssh:location}/libexec/sftp-server
 
-[runner-sshd-raw-server]
+[runner-sshd-service]
 recipe = slapos.cookbook:wrapper
-host = $${slap-network-information:global-ipv6}
-rsa-keyfile = $${runner-sshd-config:host_key}
-home = $${directory:ssh}
 command-line = ${openssh:location}/sbin/sshd -D -e -f $${runner-sshd-config:rendered}
-wrapper-path = $${directory:bin}/runner_raw_sshd
-
-[runner-sshd-authorized-key]
-<= runner-sshd-raw-server
-recipe = slapos.cookbook:dropbear.add_authorized_key
-key = $${slap-parameter:user-authorized-key}
-
-[runner-sshd-server]
-recipe = collective.recipe.template
-log = $${directory:log}/runner-sshd.log
-input = inline:#!/bin/sh
-    exec $${runner-sshd-raw-server:wrapper-path} >> $${:log} 2>&1
-
-output = $${directory:bin}/runner_raw_sshd_log
-mode = 700
-
-[runner-sshd-graceful]
-recipe = slapos.cookbook:wrapper
-command-line = $${directory:bin}/killpidfromfile $${runner-sshd-config:path_pid} SIGHUP
-wrapper-path = $${directory:scripts}/runner-sshd-graceful
-
-[runner-sshkeys-directory]
-recipe = slapos.cookbook:mkdirectory
-requests = $${directory:sshkeys}/runner-requests/
-keys = $${directory:sshkeys}/runner-keys/
-
-[runner-sshkeys-authority]
-recipe = slapos.cookbook:sshkeys_authority
-request-directory = $${runner-sshkeys-directory:requests}
-keys-directory = $${runner-sshkeys-directory:keys}
-wrapper = $${directory:bin}/runner_sshkeys_authority
-keygen-binary = ${openssh:location}/bin/ssh-keygen
-
-[runner-sshkeys-authority-service]
-recipe = slapos.cookbook:wrapper
-command-line = $${runner-sshkeys-authority:wrapper}
-wrapper-path = $${directory:services}/runner-sshkeys-authority
 hash-existing-files = $${buildout:directory}/software_release/buildout.cfg
-
-[runner-sshkeys-sshd]
-<= runner-sshkeys-authority
-recipe = slapos.cookbook:sshkeys_authority.request
-name = sshd
-type = rsa
-executable = $${runner-sshd-server:output}
-public-key = $${runner-sshd-raw-server:rsa-keyfile}.pub
-private-key = $${runner-sshd-raw-server:rsa-keyfile}
-wrapper = $${directory:bin}/runner-sshd
-
-[runner-sshkeys-sshd-service]
-recipe = slapos.cookbook:wrapper
-command-line = $${runner-sshkeys-sshd:wrapper}
 wrapper-path = $${directory:services}/runner-sshd
-hash-existing-files = $${buildout:directory}/software_release/buildout.cfg
 
 [runner-sshd-add-authorized-key]
 recipe = slapos.cookbook:dropbear.add_authorized_key
 home = $${buildout:directory}
 key = $${slap-parameter:user-authorized-key}
 
-[runner-sshkeys-publickey-fingerprint-cmd]
+[runner-sshd-ssh-keygen-base]
 recipe = plone.recipe.command
-command = bash -o pipefail -c "$${runner-sshkeys-authority:keygen-binary} -lf $${runner-sshkeys-sshd:public-key} | cut  -f 2 -d\ | sed 's/+/%2B/g' | sed 's/\//%2F/g' | sed 's/SHA256://'"
+output = $${directory:etc}/$${:_buildout_section_name_}
+command = ${openssh-output:keygen} -f $${:output} -N '' $${:extra-args}
 
-[runner-sshkeys-publickey-fingerprint-shelloutput]
+[runner-sshd-ssh-host-rsa-key]
+<=runner-sshd-ssh-keygen-base
+extra-args=-t rsa
+[runner-sshd-ssh-host-ecdsa-key]
+<=runner-sshd-ssh-keygen-base
+extra-args=-t ecdsa -b 521
+
+[runner-sshd-publickey-fingerprint-shelloutput]
 recipe = collective.recipe.shelloutput
 # XXX because collective.recipe.shelloutput ignore errors, we run the same
 # command in a plone.recipe.command so that if fails if something goes wrong.
 commands =
-  fingerprint = $${runner-sshkeys-publickey-fingerprint-cmd:command}
+  fingerprint = bash -o pipefail -c "${openssh-output:keygen} -lf $${runner-sshd-ssh-host-ecdsa-key:output} | cut  -f 2 -d\ | sed 's/+/%2B/g' | sed 's/\//%2F/g' | sed 's/SHA256://'"
 
-[runner-sshkeys-publickey-fingerprint]
+[runner-sshd-publickey-fingerprint]
 # fingerprint for ssh url, see
 # https://tools.ietf.org/id/draft-salowey-secsh-uri-00.html#connparam
 # https://winscp.net/eng/docs/session_url#hostkey
 
+_fingerprint = $${runner-sshd-publickey-fingerprint-shelloutput:fingerprint}
+
 # format is host-key-alg-fingerprint, but we know that
 # $${runner-sshkeys-sshd:public-key} is rsa so for host-key-alg
 # we just use use rsa.
-fingerprint = ssh-rsa-$${runner-sshkeys-publickey-fingerprint-shelloutput:fingerprint}
+fingerprint = ssh-rsa-$${:_fingerprint}
+
+# XXX because collective.recipe.shelloutput ignore errors and capture output
+# "Error ...", we use a plone.recipe.command to check that this command did
+# not fail.
+# This command will always fail on first buildout run, because
+# collective.recipe.shelloutput is evaluated at buildout recipes __init__ step,
+# but the key file is created later at install step.
+recipe = plone.recipe.command
+stop-on-error = true
+command = echo "$${:_fingerprint}" | ( grep ^Error || exit 0 && exit 1 )
+
 
 #---------------------------
 #--
@@ -640,7 +600,7 @@ backend-url = $${slaprunner:access-url}
 init-user = $${runner-htpasswd:user}
 init-password = $${runner-htpasswd:password}
 ssh-command = ssh $${user-info:pw-name}@$${slap-network-information:global-ipv6} -p $${runner-sshd-port:port}
-ssh-url = ssh://$${user-info:pw-name};fingerprint=$${runner-sshkeys-publickey-fingerprint:fingerprint}@[$${slap-network-information:global-ipv6}]:$${runner-sshd-port:port}
+ssh-url = ssh://$${user-info:pw-name};fingerprint=$${runner-sshd-publickey-fingerprint:fingerprint}@[$${slap-network-information:global-ipv6}]:$${runner-sshd-port:port}
 git-public-url = https://[$${httpd-parameters:global_ip}]:$${httpd-parameters:global_port}/git-public/
 git-private-url = https://[$${httpd-parameters:global_ip}]:$${httpd-parameters:global_port}/git/
 monitor-base-url = $${monitor-publish-parameters:monitor-base-url}

software/slaprunner/template/slapos.cfg.in

@@ -58,10 +58,8 @@ host = {{ slaprunner['ipv4'] }}
 port = {{ slaprunner['proxy_port'] }}
 database_uri = {{ slaprunner['proxy_database'] }}
 
-[sshkeys_authority]
-ssh_client = {{ slaprunner['ssh_client'] }}
-public_key = {{ slaprunner['public_key'] }}
-private_key = {{ slaprunner['private_key'] }}
-
 [gitclient]
 git = {{ slaprunner['git-binary'] }}
+
+[sshkeys_authority]
+public_key = {{ slaprunner['public_key'] }}

software/slaprunner/test/test.py

@@ -246,7 +246,6 @@ class ServicesTestCase(SlaprunnerTestCase):
     ]
     expected_process_names = [
       'slaprunner-supervisord-{hash}-on-watch',
-      'runner-sshkeys-authority-{hash}-on-watch',
       'runner-sshd-{hash}-on-watch',
       'slaprunner-httpd-{hash}-on-watch',
       'gunicorn-{hash}-on-watch',