caddy-frontend: Implement KeDiFa SSL information

Use KeDiFa to store keys, and transmit the url to the requester for master
and slave partitions.

Download keys on the slave partitions level.

Use caucase to fetch main caucase CA.

kedifa-caucase-url is published in order to have access to it.

Note: caucase is prepended with kedifa, as this is that one.

Use kedifa-csr tool to generate CSR and use caucase-updater macro.

Switch to KeDiFa with SSL Auth and updated goodies.

KeDiFa endpoint URLs are randomised.

Only one (first) user certificate is going to be automatically accepted. This
one shall be operated by the cluster owner, the requester of frontend master
partition.

Then he will be able to sign certificates for other users and also for
services - so each node in the cluster.

Special trick from https://security.stackexchange.com/questions/74345/provide-subjectaltname-to-openssl-directly-on-command-line
is used for one command generation of extensions in the certificate.
Note: We could upgrade to openssl 1.1.1 in order to have it really
simplified (see https://security.stackexchange.com/a/183973 )

Improve CSR readability by creating cluster-identification, which is master
partition title, and use it as Organization of the CSR.

Reserve slots for data exchange in KeDiFa.

software/caddy-frontend/README.caddy_frontend.rst

@@ -83,13 +83,70 @@ Those slave instances will be redirected to the "master" instance, and you will
 
 Finally, the slave instance will be accessible from: https://someidentifier.moulefrite.org.
 
-About SSL
-=========
-
-``default`` and ``custom-personl`` software type can handle specific ssl for one slave instance.
+About SSL and SlapOS Master Zero Knowledge
+==========================================
 
 **IMPORTANT**: One Caddy can not serve more than one specific SSL site and be compatible with obsolete browser (i.e.: IE8). See http://wiki.apache.org/httpd/NameBasedSSLVHostsWithSNI
 
+SSL keys and certificates are directly send to the frontend cluster in order to follow zero knowledge principle of SlapOS Master.
+
+Master partition
+----------------
+
+After requesting master partition it will return ``master-key-generate-auth-url`` and ``master-key-upload-url``.
+
+Doing HTTP GET on ``master-key-generate-auth-url`` will return authentication token, which is used to communicate with ``master-key-upload-url``. This token shall be stored securely.
+
+By doing HTTP PUT to ``master-key-upload-url`` with appended authentication token it is possible to upload PEM bundle of certificate, key and any accompanying CA certificates to the master.
+
+Example sessions is::
+
+  request(...)
+
+  curl -X GET master-key-generate-auth-url
+  > authtoken
+
+  cat certificate.pem key.pem ca-bundle.pem > master.pem
+
+  curl -X PUT --data-binary @master.pem master-key-upload-url+authtoken
+
+This replaces old request parameters:
+
+ * ``apache-certificate``
+ * ``apache-key``
+ * ``apache-ca-certificate``
+
+(*Note*: They are still supported for backward compatibility, but any value send to the ``master-key-upload-url`` will supersede information from SlapOS Master.)
+
+Slave partition
+---------------
+
+After requesting slave partition it will return ``key-generate-auth-url`` and ``key-upload-url``.
+
+Doing HTTP GET on ``key-generate-auth-url`` will return authentication token, which is used to communicate with ``key-upload-url``. This token shall be stored securely.
+
+By doing HTTP PUT to ``key-upload-url`` with appended authentication token it is possible to upload PEM bundle of certificate, key and any accompanying CA certificates to the master.
+
+Example sessions is::
+
+  request(...)
+
+  curl -X GET key-generate-auth-url
+  > authtoken
+
+  cat certificate.pem key.pem ca-bundle.pem > master.pem
+
+  curl -X PUT --data-binary @master.pem key-upload-url+authtoken
+
+This replaces old request parameters:
+
+ * ``ssl_crt``
+ * ``ssl_key``
+ * ``ssl_ca_crt``
+
+(*Note*: They are still supported for backward compatibility, but any value send to the ``key-upload-url`` will supersede information from SlapOS Master.)
+
+
 How to have custom configuration in frontend server - XXX - to be written
 =========================================================================
 
@@ -195,13 +252,6 @@ Necessary to activate cache.
 
 ``enable_cache`` is an optional parameter.
 
-ssl_key, ssl_crt, ssl_ca_crt
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-SSL certificates of the slave.
-
-They are optional.
-
 Functionalities for Caddy configuration
 ---------------------------------------
 
@@ -210,7 +260,7 @@ In the slave Caddy configuration you can use parameters that will be replaced du
   * ``cache_access`` : url of the cache. Should replace backend url in configuration to use the cache
   * ``access_log`` : path of the slave error log in order to log in a file.
   * ``error_log`` : path of the slave access log in order to log in a file.
-  * ``ssl_key``, ``ssl_crt``, ``ssl_ca_crt``, ``ssl_crs`` : paths of the certificates given in slave instance parameters
+  * ``certificate`` : path to the certificate
 
 
 Examples
@@ -293,7 +343,7 @@ Request slave frontend instance so that https://[1:2:3:4:5:6:7:8]:1234 will be::
         "caddy_custom_https":'
   https://www.example.com:%(https_port)s, https://example.com:%(https_port)s {
     bind %(local_ipv4)s
-    tls %(ssl_crt)s %(ssl_key)s
+    tls %%(certificate)s %%(certificate)s
 
     log / %(access_log)s {combined}
     errors %(error_log)s
@@ -479,6 +529,13 @@ Note that in some cases promises will fail:
 
 This is known issue and shall be tackled soon.
 
+KeDiFa
+======
+
+Additional partition with KeDiFa (Key Distribution Facility) is by default requested on the same computer as master frontend partition.
+
+By adding to the request keys like ``-sla-kedifa-<key>`` it is possible to provide SLA information for kedifa partition. Eg to put it on computer ``couscous`` it shall be ``-sla-kedifa-computer_guid: couscous``.
+
 Notes
 =====
 

software/caddy-frontend/TODO.rst

@@ -3,13 +3,11 @@ Generally things to be done with ``caddy-frontend``:
  * tests: add assertion with results of promises in etc/promise for each partition
  * README: cleanup the documentation, explain various specifics
  * check the whole frontend slave snippet with ``caddy -validate`` during buildout run, and reject if does not pass validation
- * BUG?? check that changing ``apache-certificate`` on master partition results in reloading slave partition
  * (new) ``type:websocket`` slave
  * ``type:eventsource``:
 
    * **Jérome Perrin**: *For event source, if I understand https://github.com/mholt/caddy/issues/1355 correctly, we could use caddy as a proxy in front of nginx-push-stream . If we have a "central shared" caddy instance, can it handle keeping connections opens for many clients ?*
  * ``check-error-on-caddy-log`` like ``check-error-on-apache-log``
-
  * move out ``test/utils.py`` and use it from shared python distribution
  * reduce the time of configuration validation (in ``instance-apache-frontend.cfg.in`` sections ``[configtest]``, ``[caddy-configuration]``, ``[nginx-configuration]``), as it is not scalable on frontend with 2000+ slaves (takes few minutes instead of few, < 5, seconds), issue posted `upstream <https://github.com/mholt/caddy/issues/2220>`_
  * drop ``6tunnel`` and use ``bind`` in Caddy configuration, as soon as multiple binds will be possible, tracked in upstream `bind: support multiple values <https://github.com/mholt/caddy/pull/2128>`_ and `ipv6: does not bind on ipv4 and ipv6 for sites that resolve to both <https://github.com/mholt/caddy/issues/864>`_

software/caddy-frontend/buildout.hash.cfg

@@ -14,7 +14,7 @@
 # not need these here).
 [template]
 filename = instance.cfg.in
-md5sum = 2747f9125c8dffa0c27b79a6902a55cb
+md5sum = 733ef269151e9884e44174bb4dc9c6ea
 
 [template-common]
 filename = instance-common.cfg.in
@@ -22,15 +22,15 @@ md5sum = c801b7f9f11f0965677c22e6bbe9281b
 
 [template-apache-frontend]
 filename = instance-apache-frontend.cfg.in
-md5sum = 6fd023f0d29421d8579f0b3351473bb0
+md5sum = cb6406e0b8fe6b6decd587416ddbb882
 
 [template-apache-replicate]
 filename = instance-apache-replicate.cfg.in
-md5sum = 5c5462ccc327fe109e0c102f0d3e7e53
+md5sum = e4d6f2df21a60f5a68b3a78b01a6868c
 
 [template-slave-list]
 filename = templates/apache-custom-slave-list.cfg.in
-md5sum = 028ca41fdf7a758ba701ccc5e449419a
+md5sum = 7ddc510084c73cee910d9e0aa546d99a
 
 [template-slave-configuration]
 filename = templates/custom-virtualhost.conf.in
@@ -38,11 +38,11 @@ md5sum = 54ae95597a126ae552c3a913ddf29e5e
 
 [template-replicate-publish-slave-information]
 filename = templates/replicate-publish-slave-information.cfg.in
-md5sum = 696ef7690f51a521cc6f6c9d9d5d844e
+md5sum = 125181ce0065b2acddb0e3ee38f81864
 
 [template-caddy-frontend-configuration]
 filename = templates/Caddyfile.in
-md5sum = 0134a1586f15cd5665069d6d81a505be
+md5sum = 1814950fe926b4a5f730d81d52cc827f
 
 [caddy-backend-url-validator]
 filename = templates/caddy-backend-url-validator.in
@@ -58,7 +58,7 @@ md5sum = f20d6c3d2d94fb685f8d26dfca1e822b
 
 [template-default-slave-virtualhost]
 filename = templates/default-virtualhost.conf.in
-md5sum = e57b9ae012f777482295698a23b7e850
+md5sum = ce86264b0c4cc1d447edf40db34e01c4
 
 [template-cached-slave-virtualhost]
 filename = templates/cached-virtualhost.conf.in
@@ -66,7 +66,7 @@ md5sum = 907372828d1ceb05c41240078196f439
 
 [template-log-access]
 filename = templates/template-log-access.conf.in
-md5sum = 122b05829ecc4c0ad4e47e7d1c21166b
+md5sum = 59644acda51eb1a51eab448e6fc6aa74
 
 [template-empty]
 filename = templates/empty.in
@@ -86,15 +86,15 @@ md5sum = 117238225b3fc3c5b5be381815f44c67
 
 [template-nginx-configuration]
 filename = templates/nginx.cfg.in
-md5sum = fadb2fcaf0f2b4fe735617fac222f7ed
+md5sum = e5b65b424d01fa539c92f4ba10e847eb
 
 [template-nginx-eventsource-slave-virtualhost]
 filename = templates/nginx-eventsource-slave.conf.in
-md5sum = 176cbca2070734a185a7ae5a4d1181c5
+md5sum = 217a6c801b8330b0b825f7b8b4c77184
 
 [template-nginx-notebook-slave-virtualhost]
 filename = templates/nginx-notebook-slave.conf.in
-md5sum = ee3b5c23f1c81aa43ce7cd8f8e327f70
+md5sum = 3fd1fefaec2d630eee54f75e7a873c27
 
 [template-apache-lazy-script-call]
 filename = templates/apache-lazy-script-call.sh.in
@@ -115,3 +115,7 @@ md5sum = d9b6476bb0b36cf463fddb00d41dfbaa
 [caddyprofiledeps-dummy]
 filename = caddyprofiledummy.py
 md5sum = 38792c2dceae38ab411592ec36fff6a8
+
+[template-kedifa]
+filename = instance-kedifa.cfg.in
+md5sum = ee58402bbf374e3a3522ce59002880e3

software/caddy-frontend/common.cfg

@@ -30,6 +30,25 @@ parts +=
   http-proxy
   caddyprofiledeps
 
+  kedifa-develop
+  kedifa
+
+[kedifa-repository]
+recipe = slapos.recipe.build:gitclone
+repository = https://lab.nexedi.com/nexedi/kedifa.git
+git-executable = ${git:location}/bin/git
+revision = 67bd60ea1bfb4fc6aafdfe4fa204f725731f20cf
+
+[kedifa-develop]
+recipe = zc.recipe.egg:develop
+setup = ${kedifa-repository:location}
+
+[kedifa]
+recipe = zc.recipe.egg
+eggs =
+  ${python-cryptography:egg}
+  kedifa
+
 [caddyprofiledeps-setup]
 recipe = slapos.recipe.build:download
 url = ${:_profile_base_location_}/setup.py
@@ -88,9 +107,13 @@ dash = ${dash:location}
 dcron = ${dcron:location}
 gzip = ${gzip:location}
 logrotate = ${logrotate:location}
-openssl = ${openssl:location}
+openssl = ${openssl:location}/bin/openssl
+openssl_cnf = ${openssl:location}/etc/ssl/openssl.cnf
 trafficserver = ${trafficserver:location}
 sha256sum = ${coreutils:location}/bin/sha256sum
+kedifa = ${:bin_directory}/kedifa
+kedifa-getter = ${:bin_directory}/kedifa-getter
+kedifa-csr = ${:bin_directory}/kedifa-csr
 
 monitor_template = ${monitor-template:output}
 template_cached_slave_virtualhost = ${template-cached-slave-virtualhost:target}
@@ -125,11 +148,13 @@ context =
   key monitor2_template monitor2-template:rendered
   key template_caddy_frontend template-caddy-frontend:target
   key template_caddy_replicate template-caddy-replicate:target
+  key template_kedifa template-kedifa:target
   key template_replicate_publish_slave_information template-replicate-publish-slave-information:target
   key caddy_backend_url_validator caddy-backend-url-validator:output
   key caddy_custom_http_validator caddy-custom-http-validator:output
 
   section template_frontend_parameter_dict template-frontend-parameter-section
+  key caucase_jinja2_library caucase-jinja2-library:target
 
 [template-caddy-frontend]
 recipe = slapos.recipe.build:download
@@ -155,6 +180,11 @@ recipe = slapos.recipe.build:download
 url = ${:_profile_base_location_}/instance-apache-replicate.cfg.in
 mode = 0644
 
+[template-kedifa]
+recipe = slapos.recipe.build:download
+url = ${:_profile_base_location_}/instance-kedifa.cfg.in
+mode = 0644
+
 [download-template]
 recipe = slapos.recipe.build:download
 url = ${:_profile_base_location_}/templates/${:filename}

software/caddy-frontend/instance-apache-frontend.cfg.in

 {%- if slap_software_type == software_type -%}
+{% import "caucase" as caucase with context %}
 {%- set TRUE_VALUES = ['y', 'yes', '1', 'true'] -%}
 [buildout]
 extends =
@@ -11,14 +12,11 @@ parts =
   logrotate
   cron
   cron-entry-logrotate
-  ca-frontend
-  ca-frontend-service
-  certificate-authority
-  certificate-authority-service
   logrotate-entry-caddy
   logrotate-entry-nginx
   caddy-frontend
   switch-caddy-softwaretype
+  caucase-updater
   frontend-caddy-graceful
   frontend-nginx-graceful
   not-found-html
@@ -136,6 +134,7 @@ slapparameter_dict = {{ dumps(instance_parameter['configuration']) }}
 slap_software_type = {{ dumps(instance_parameter['slap-software-type']) }}
 context =
     import json_module json
+    import os_module os
     raw common_profile {{ parameter_dict['common_profile'] }}
     key slap_software_type :slap_software_type
     key slapparameter_dict :slapparameter_dict
@@ -151,6 +150,48 @@ caddy-location = {{ parameter_dict['caddy_location'] }}
 template-nginx-eventsource-slave-virtualhost = {{ parameter_dict['template_nginx_eventsource_slave_virtualhost'] }}
 template-nginx-notebook-slave-virtualhost = {{ parameter_dict['template_nginx_notebook_slave_virtualhost'] }}
 
+[kedifa-login-config]
+d = ${directory:ca-dir}
+csr = ${:d}/kedifa-login-csr.pem
+key = ${:d}/kedifa-login-certificate.pem
+certificate = ${:key}
+ca-certificate = ${:d}/kedifa-caucase-ca.pem
+cas-ca-certificate = ${:d}/kedifa-cas-caucase-ca.pem
+crl = ${:d}/kedifa-login-crl.pem
+
+[kedifa-login-csr]
+recipe = plone.recipe.command
+organization = {{ slapparameter_dict['cluster-identification'] }}
+organizational_unit = {{ instance_parameter['configuration.frontend-name'] }}
+command =
+{% if slapparameter_dict['kedifa-caucase-url'] %}
+  if [ ! -f ${:csr} ] && [ ! -f ${:key} ]  ; then
+    {{ parameter_dict['openssl'] }} req -new -sha256 \
+      -newkey rsa:2048 -nodes -keyout ${:key} \
+      -subj "/O=${:organization}/OU=${:organizational_unit}" \
+      -out ${:csr}
+  fi
+{% endif %}
+  test -f ${:key} && test -f ${:csr}
+update-command = ${:command}
+csr = ${kedifa-login-config:csr}
+key = ${kedifa-login-config:key}
+stop-on-error = True
+
+{{ caucase.updater(
+     prefix='caucase-updater',
+     buildout_bin_directory=parameter_dict['bin_directory'],
+     updater_path='${directory:service}/kedifa-login-certificate-caucase-updater',
+     url=slapparameter_dict['kedifa-caucase-url'],
+     data_dir='${directory:srv}/caucase-updater',
+     crt_path='${kedifa-login-config:certificate}',
+     ca_path='${kedifa-login-config:ca-certificate}',
+     crl_path='${kedifa-login-config:crl}',
+     key_path='${kedifa-login-csr:key}',
+     template_csr='${kedifa-login-csr:csr}',
+     openssl=parameter_dict['openssl'] ~ '/bin/openssl',
+)}}
+
 [dynamic-custom-personal-template-slave-list]
 < = jinja2-template-base
 template = {{ parameter_dict['template_slave_list'] }}
@@ -158,17 +199,25 @@ filename = custom-personal-instance-slave-list.cfg
 extensions = jinja2.ext.do
 slave_instance_list = {{ dumps(instance_parameter['slave-instance-list']) }}
 extra_slave_instance_list = {{ dumps(instance_parameter.get('configuration.extra_slave_instance_list')) }}
+master_key_download_url = {{ dumps(slapparameter_dict['master-key-download-url']) }}
+slave_kedifa_information = {{ dumps(slapparameter_dict['slave-kedifa-information']) }}
 local_ipv4 = {{ dumps(instance_parameter['ipv4-random']) }}
 local_ipv6 = {{ dumps(instance_parameter['ipv6-random']) }}
 software_type = single-custom-personal
 bin_directory = {{ parameter_dict['bin_directory'] }}
 sixtunnel_executable = {{ parameter_dict['sixtunnel'] }}/bin/6tunnel
+kedifa-getter = {{ parameter_dict['kedifa-getter'] }}
+kedifa-csr = {{ parameter_dict['kedifa-csr'] }}
 service_directory = ${directory:service}
 extra-context =
+    key kedifa_caucase_ca_certificate kedifa-login-config:ca-certificate
+    key kedifa_login_certificate kedifa-login-config:certificate
     key caddy_configuration_directory caddy-directory:slave-configuration
     key nginx_configuration_directory caddy-directory:nginx-slave-configuration
     key caddy_cached_configuration_directory caddy-directory:slave-with-cache-configuration
     key slave_with_cache_configuration_directory caddy-directory:slave-with-cache-configuration
+    key kedifa_getter :kedifa-getter
+    key kedifa_csr :kedifa-csr
     key http_port configuration:plain_http_port
     key https_port configuration:port
     key nginx_http_port configuration:plain_nginx_port
@@ -176,7 +225,10 @@ extra-context =
     key public_ipv4 configuration:public-ipv4
     key slave_instance_list :slave_instance_list
     key extra_slave_instance_list :extra_slave_instance_list
-    key custom_ssl_directory caddy-directory:vh-ssl
+    key master_key_download_url :master_key_download_url
+    key slave_kedifa_information :slave_kedifa_information
+    key autocert caddy-directory:autocert
+    key master_certificate caddy-configuration:master-certificate
     key caddy_log_directory caddy-directory:slave-log
     key local_ipv4 :local_ipv4
     key local_ipv6 :local_ipv6
@@ -199,9 +251,6 @@ extra-context =
     key promise_directory directory:promises
     key report_directory directory:bin
     key bin_directory :bin_directory
-    key login_certificate ca-frontend:cert-file
-    key login_key ca-frontend:key-file
-    key login_ca_crt ca-custom-frontend:rendered
     key enable_http2_by_default configuration:enable-http2-by-default
     key global_disable_http2 configuration:global-disable-http2
     key proxy_try_duration configuration:proxy-try-duration
@@ -210,6 +259,7 @@ extra-context =
     key error_log caddy-configuration:error-log
     key sixtunnel_executable :sixtunnel_executable
     key service_directory directory:service
+    key run_directory directory:etc-run
     key not_found_file caddy-configuration:not-found-file
 
 [dynamic-virtualhost-template-slave]
@@ -236,11 +286,7 @@ extra-context =
     key httpd_home software-release-path:caddy-location
     key httpd_mod_ssl_cache_directory caddy-directory:mod-ssl
     key instance_home buildout:directory
-    key login_certificate ca-frontend:cert-file
-    key login_key ca-frontend:key-file
-    key login_ca_crt ca-custom-frontend:rendered
-    key ca_dir  certificate-authority:ca-dir
-    key ca_crl certificate-authority:ca-crl
+    key master_certificate caddy-configuration:master-certificate
     key access_log caddy-configuration:access-log
     key slave_configuration_directory caddy-directory:slave-configuration
     key cached_port caddy-configuration:cache-through-port
@@ -280,9 +326,6 @@ wrapper-path = ${directory:bin}/caddy-wrapper
 recipe = slapos.cookbook:wrapper
 command-line = ${caddy-wrapper:wrapper-path} -pidfile ${caddy-configuration:pid-file}
 wrapper-path = ${directory:service}/frontend_caddy
-wait-for-files =
-	       ${ca-frontend:cert-file}
-	       ${ca-frontend:key-file}
 hash-files = ${buildout:directory}/software_release/buildout.cfg
 
 [not-found-html]
@@ -298,9 +341,10 @@ slave-configuration = ${directory:etc}/caddy-slave-conf.d/
 slave-with-cache-configuration = ${directory:etc}/caddy-slave-with-cache-conf.d/
 cache = ${directory:var}/cache
 mod-ssl = ${:cache}/httpd_mod_ssl
-vh-ssl = ${:slave-configuration}/ssl
 slave-log = ${directory:log}/httpd
 nginx-slave-configuration = ${directory:etc}/nginx-slave-conf.d/
+autocert = ${directory:srv}/autocert
+master-autocert-dir = ${:autocert}/master-autocert
 
 [caddy-configuration]
 frontend-configuration = ${directory:etc}/Caddyfile
@@ -308,8 +352,9 @@ access-log = ${directory:log}/frontend-access.log
 error-log = ${directory:log}/frontend-error.log
 pid-file = ${directory:run}/httpd.pid
 frontend-configuration-verification = ${frontend-caddy-validate:rendered}
-frontend-graceful-command = ${:frontend-configuration-verification}; if [ $? -eq 0 ]; then kill -USR1 $(cat ${:pid-file}); fi
+frontend-graceful-command = ${:frontend-configuration-verification} && kill -USR1 $(cat ${:pid-file})
 not-found-file = ${caddy-directory:document-root}/notfound.html
+master-certificate = ${caddy-directory:master-autocert-dir}/master.pem
 # Communication with ATS
 cache-port = ${trafficserver-variable:input-port}
 cache-through-port = 26011
@@ -320,57 +365,6 @@ recipe = slapos.cookbook:wrapper
 command-line = ${frontend-caddy-validate:rendered}
 wrapper-path = ${directory:bin}/caddy-configtest
 
-[certificate-authority]
-recipe = slapos.cookbook:certificate_authority
-openssl-binary = {{ parameter_dict['openssl'] }}/bin/openssl
-ca-dir = ${directory:ca-dir}
-requests-directory = ${cadirectory:requests}
-wrapper = ${directory:bin}/certificate_authority
-ca-private = ${cadirectory:private}
-ca-certs = ${cadirectory:certs}
-ca-newcerts = ${cadirectory:newcerts}
-ca-crl = ${cadirectory:crl}
-
-[certificate-authority-service]
-recipe = slapos.cookbook:wrapper
-command-line = ${certificate-authority:wrapper}
-wrapper-path = ${directory:service}/certificate_authority
-hash-files = ${buildout:directory}/software_release/buildout.cfg
-
-[cadirectory]
-recipe = slapos.cookbook:mkdirectory
-requests = ${directory:ca-dir}/requests/
-private = ${directory:ca-dir}/private/
-certs = ${directory:ca-dir}/certs/
-newcerts = ${directory:ca-dir}/newcerts/
-crl = ${directory:ca-dir}/crl/
-
-[ca-frontend]
-<= certificate-authority
-recipe = slapos.cookbook:certificate_authority.request
-key-file = ${cadirectory:certs}/frontend.key
-cert-file = ${cadirectory:certs}/frontend.crt
-executable = ${directory:service}/frontend_caddy
-wrapper = ${directory:bin}/frontend_caddy
-key-content = ${configuration:apache-key}
-cert-content = ${configuration:apache-certificate}
-# Put domain name
-name = ${configuration:domain}
-
-[ca-frontend-service]
-recipe = slapos.cookbook:wrapper
-command-line = ${ca-frontend:wrapper}
-wrapper-path = ${directory:service}/frontend_caddy
-hash-files = ${buildout:directory}/software_release/buildout.cfg
-
-[ca-custom-frontend]
-< = jinja2-template-base
-template = {{ parameter_dict['template_empty'] }}
-rendered = ${cadirectory:certs}/frontend.ca.crt
-apache-ca-certificate = ${configuration:apache-ca-certificate}
-extra-context =
-    key content :apache-ca-certificate
-
 [cron]
 recipe = slapos.cookbook:cron
 dcrond-binary = {{ parameter_dict['dcron'] }}/sbin/crond
@@ -548,7 +542,8 @@ config-wrapper-path = ${trafficserver-line:wrapper-path}
 template = {{ parameter_dict['template_graceful_script'] }}
 rendered = ${directory:etc-run}/frontend-caddy-safe-graceful
 mode = 0700
-path_list = ${caddy-configuration:frontend-configuration} ${frontend-configuration:log-access-configuration} ${caddy-directory:slave-configuration}/*.conf ${caddy-directory:slave-with-cache-configuration}/*.conf ${caddy-directory:vh-ssl}/*.key ${caddy-directory:vh-ssl}/*.crt ${caddy-directory:vh-ssl}/*.proxy_ca_crt
+# XXX: Fix path_list for direct ssl ${caddy-directory:vh-ssl}/*.proxy_ca_crt
+path_list = ${caddy-configuration:frontend-configuration} ${frontend-configuration:log-access-configuration} ${caddy-directory:slave-configuration}/*.conf ${caddy-directory:slave-with-cache-configuration}/*.conf ${caddy-directory:master-autocert-dir}/*.key ${caddy-directory:master-autocert-dir}/*.crt ${caddy-directory:master-autocert-dir}/*.pem ${caddy-directory:autocert}/*/*.pem
 sha256sum = {{ parameter_dict['sha256sum'] }}
 signature_file = ${directory:run}/caddy_graceful_signature
 extra-context =
@@ -562,7 +557,8 @@ extra-context =
 template = {{ parameter_dict['template_graceful_script'] }}
 rendered = ${directory:etc-run}/frontend-nginx-safe-graceful
 mode = 0700
-path_list = ${dynamic-nginx-frontend-template:rendered} ${caddy-directory:nginx-slave-configuration}/*.conf
+# XXX: Fix path_list for direct ssl ${caddy-directory:vh-ssl}/*.proxy_ca_crt
+path_list = ${dynamic-nginx-frontend-template:rendered} ${caddy-directory:nginx-slave-configuration}/*.conf ${caddy-directory:master-autocert-dir}/*.key ${caddy-directory:master-autocert-dir}/*.crt ${caddy-directory:master-autocert-dir}/*.pem ${caddy-directory:autocert}/*/*.pem
 sha256sum = {{ parameter_dict['sha256sum'] }}
 signature_file = ${directory:run}/nginx_graceful_signature
 extra-context =
@@ -756,14 +752,13 @@ rendered = ${directory:etc}/nginx.cfg
 mode = 0600
 extra-context =
   key port nginx-configuration:port
-  key ssl_certificate nginx-configuration:ssl_certificate
-  key ssl_key nginx-configuration:ssl_key
   key local_ip nginx-configuration:local_ip
   key plain_port nginx-configuration:plain_port
   key slave_configuration_directory nginx-configuration:slave-configuration-directory
   key error_log nginx-configuration:error_log
   key access_log nginx-configuration:access_log
   key not_found_file caddy-configuration:not-found-file
+  key master_certificate caddy-configuration:master-certificate
 
 [nginx-configuration]
 access_log = ${directory:log}/nginx-access.log
@@ -776,10 +771,8 @@ worker_processes = 4
 worker_connections = 1024
 slave-configuration-directory = ${caddy-directory:nginx-slave-configuration}
 pid-file = ${directory:run}/nginx.pid 
-nginx-graceful-command = ${:nginx-configuration-verification}; if [ $? -eq 0 ]; then kill -USR1 $(cat ${:pid-file}); fi
+nginx-graceful-command = ${:nginx-configuration-verification} && kill -USR1 $(cat ${:pid-file})
 nginx-configuration-verification = ${frontend-nginx-validate:rendered}
-ssl_certificate = ${ca-frontend:cert-file}
-ssl_key = ${ca-frontend:key-file}
 
 [promise-nginx-configuration]
 <= promise-plugin-base

software/caddy-frontend/instance-apache-replicate.cfg.in

@@ -30,6 +30,10 @@ context =
 {% if not sla_computer_caddy_1_key in slapparameter_dict %}
 {%   do slapparameter_dict.__setitem__(sla_computer_caddy_1_key, '${slap-connection:computer-id}') %}
 {% endif %}
+{% set sla_computer_kedifa_key = '-sla-kedifa-computer_guid' %}
+{% if not sla_computer_kedifa_key in slapparameter_dict %}
+{%   do slapparameter_dict.__setitem__(sla_computer_kedifa_key, '${slap-connection:computer-id}') %}
+{% endif %}
 
 # Here we request individually each frontend.
 # The presence of sla parameters is checked and added if found
@@ -122,18 +126,6 @@ context =
 {%       do slave_error_list.append('slave https-url %r invalid' % (slave['https-url'],)) %}
 {%     endif %}
 {%   endif %}
-{%   if slave.get('ssl_ca_crt') and not (slave.get('ssl_crt') and slave.get('ssl_key')) %}
-{%     do slave_error_list.append('ssl_ca_crt is present, so ssl_crt and ssl_key are required')  %}
-{%   endif %}
-{%   if slave.get('ssl_key') and slave.get('ssl_crt') %}
-{%     set key_popen = popen([openssl, 'rsa', '-noout', '-modulus']) %}
-{%     set crt_popen = popen([openssl, 'x509', '-noout', '-modulus']) %}
-{%     set key_modulus = key_popen.communicate('' ~ slave['ssl_key'])[0] | trim %}
-{%     set crt_modulus = crt_popen.communicate('' ~ slave['ssl_crt'])[0] | trim %}
-{%     if not key_modulus or key_modulus != crt_modulus  %}
-{%       do slave_error_list.append('slave ssl_key and ssl_crt does not match')  %}
-{%     endif %}
-{%   endif %}
 {%   if slave.get('custom_domain') %}
 {%     set slave_custom_domain = '' ~ slave['custom_domain'] %}
 {%     if slave_custom_domain.startswith('*.') %}
@@ -178,6 +170,10 @@ name = {{ frontend_request.get('name') }}
 {%   if frontend_request.get('state') %}
 state = {{ frontend_request.get('state') }}
 {%   endif%}
+config-slave-kedifa-information = ${request-kedifa:connection-slave-kedifa-information}
+config-kedifa-caucase-url = ${request-kedifa:connection-caucase-url}
+config-master-key-download-url = ${request-kedifa:connection-master-key-download-url}
+config-cluster-identification = {{ cluster_identification }}
 {%   set slave_configuration_dict = slapparameter_dict %}
 {%   do slave_configuration_dict.update(frontend_request.get('config')) %}
 {%   do slave_configuration_dict.__setitem__(slave_list_name, json_module.dumps(authorized_slave_list)) %}
@@ -201,6 +197,9 @@ slave-amount = {{ slave_instance_list | length }}
 accepted-slave-amount = {{ authorized_slave_list | length }}
 rejected-slave-amount = {{ rejected_slave_dict | length }}
 rejected-slave-dict = {{ dumps(json_module.dumps(rejected_slave_dict)) }}
+master-key-upload-url = ${request-kedifa:connection-master-key-upload-url}
+master-key-generate-auth-url = ${request-kedifa:connection-master-key-generate-auth-url}
+kedifa-caucase-url = ${request-kedifa:connection-caucase-url}
 
 #----------------------------
 #--
@@ -213,6 +212,37 @@ replicate = ${dynamic-publish-slave-information:rendered}
 custom-personal = ${dynamic-publish-slave-information:rendered}
 custom-group = ${dynamic-publish-slave-information:rendered}
 
+[request-kedifa]
+<= slap-connection
+recipe = slapos.cookbook:requestoptional.serialised
+config-monitor-cors-domains = {{ slapparameter_dict.get('monitor-cors-domains', 'monitor.app.officejs.com') }}
+config-monitor-username = ${monitor-instance-parameter:username}
+config-monitor-password = ${monitor-htpasswd:passwd}
+{% for key in ['kedifa_port', 'caucase_port'] -%}
+{%-   if key in slapparameter_dict %}
+config-{{ key }} = {{ dumps(slapparameter_dict[key]) }}
+{%-   endif %}
+{%- endfor %}
+config-slave-list = {{ dumps(slave_instance_list) }}
+config-cluster-identification = {{ cluster_identification }}
+
+{% set frontend_software_url_key = "-frontend-software-release-url" %}
+{% if slapparameter_dict.has_key(frontend_software_url_key) %}
+software-url = {{ slapparameter_dict.pop(frontend_software_url_key) }}
+{% else %}
+software-url = ${slap-connection:software-release-url}
+{% endif %}
+software-type = kedifa
+name = kedifa
+return = slave-kedifa-information master-key-generate-auth-url master-key-upload-url master-key-download-url caucase-url
+{% set sla_kedifa_key = "-sla-kedifa-" %}
+{% set sla_kedifa_key_length = sla_kedifa_key | length %}
+{% for key in slapparameter_dict.keys() %}
+{%   if key.startswith(sla_kedifa_key) %}
+sla-{{ key[sla_kedifa_key_length:] }} = {{ slapparameter_dict.pop(key) }}
+{%   endif %}
+{% endfor %}
+
 [rejected-slave-information]
 {% for slave_id, rejected_list in rejected_slave_dict.iteritems() %}
 {{ slave_id }} = {{ dumps(json_module.dumps(rejected_list)) }}
@@ -241,6 +271,7 @@ extra-context =
     section slave_information slave-information
     section rejected_slave_information rejected-slave-information
     section active_slave_instance_dict active-slave-instance
+    key slave_kedifa_information request-kedifa:connection-slave-kedifa-information
 
 [monitor-conf-parameters]
 monitor-url-list +=
@@ -256,6 +287,7 @@ parts =
   monitor-base
   publish-slave-information
   publish-information
+  request-kedifa
 {% for part in part_list %}
 {{ '  %s' % part }}
 {% endfor %}

software/caddy-frontend/instance-caddy-input-schema.json

@@ -11,24 +11,6 @@
       "title": "Frontend Replication Quantity",
       "type": "integer"
     },
-    "apache-ca-certificate": {
-      "description": "[NOT IMPLEMENTED] SSL CA Certificate used by the server. You can append it to 'apache-certificate'.",
-      "textarea": true,
-      "title": "[NOT IMPLEMENTED] SSL CA Certificate",
-      "type": "string"
-    },
-    "apache-certificate": {
-      "description": "SSL Certificate used by the server.",
-      "textarea": true,
-      "title": "SSL Certificate",
-      "type": "string"
-    },
-    "apache-key": {
-      "description": "SSL Key used by the server.",
-      "textarea": true,
-      "title": "SSL Key",
-      "type": "string"
-    },
     "domain": {
       "description": "Base Domain for create subdomains (ie.: example.com).",
       "pattern": "^([a-zA-Z0-9]([a-zA-Z0-9\\-]{0,61}[a-zA-Z0-9])?\\.)+[a-zA-Z]{2,6}$",

software/caddy-frontend/instance-kedifa.cfg.in

+{%- if slap_software_type == software_type -%}
+{% import "caucase" as caucase with context %}
+# KeDiFa instance profile
+[buildout]
+extends =
+  {{ parameter_dict['common_profile'] }}
+  {{ parameter_dict['monitor_template'] }}
+
+parts =
+  directory
+  kedifa
+  slave-kedifa-information
+  caucased
+  caucased-promise
+  caucase-updater
+
+[caucased]
+hash-files = ${buildout:directory}/software_release/buildout.cfg
+
+{%  set caucase_host = '[' ~ instance_parameter['ipv6-random'] ~ ']' %}
+{%  set caucase_netloc = caucase_host ~ ':' ~ instance_parameter['configuration.caucase_port'] -%}
+{%  set caucase_url = 'http://' ~ caucase_netloc -%}
+{{  caucase.caucased(
+      prefix='caucased',
+      buildout_bin_directory=parameter_dict['bin_directory'],
+      caucased_path='${directory:service}/caucased',
+      backup_dir='${directory:backup-caucased}',
+      data_dir='${directory:caucased}',
+      netloc=caucase_netloc,
+      service_auto_approve_count=0,
+      user_auto_approve_count=1,
+      key_len=2048,
+      promise='${directory:promise}/caucased',
+)}}
+
+# Create all needed directories
+[directory]
+recipe = slapos.cookbook:mkdirectory
+
+bin = ${buildout:directory}/bin/
+etc = ${buildout:directory}/etc/
+srv = ${buildout:directory}/srv/
+var = ${buildout:directory}/var/
+
+backup = ${:srv}/backup
+log = ${:var}/log
+run = ${:var}/run
+service = ${:etc}/service
+etc-run = ${:etc}/run
+promise = ${:etc}/promise
+
+logrotate-backup = ${:backup}/logrotate
+logrotate-entries = ${:etc}/logrotate.d
+
+cron-entries = ${:etc}/cron.d
+crontabs = ${:etc}/crontabs
+cronstamps = ${:etc}/cronstamps
+
+# KeDiFa directories
+kedifa = ${:srv}/kedifa
+etc-kedifa = ${:etc}/kedifa
+
+# CAUCASE directories
+caucased = ${:srv}/caucased
+backup-caucased = ${:backup}/caucased
+
+# reservation
+reservation = ${:srv}/reservation
+
+[kedifa-csr]
+recipe = plone.recipe.command
+organization = {{ slapparameter_dict['cluster-identification'] }}
+organizational_unit = Kedifa Partition
+command =
+  if [ ! -f ${:csr} ] && [ ! -f ${:key} ]  ; then
+    /bin/bash -c '{{ parameter_dict['openssl'] }} req -new -sha256 \
+      -newkey rsa:2048 -nodes -keyout ${:key} \
+      -subj "/O=${:organization}/OU=${:organizational_unit}" \
+      -reqexts SAN \
+      -config <(cat {{ parameter_dict['openssl_cnf'] }} \
+      <(printf "\n[SAN]\nsubjectAltName=IP:${kedifa-config:ip}")) \
+      -out ${:csr}'
+  fi
+update-command = ${:command}
+csr = ${kedifa-config:csr}
+key = ${kedifa-config:key}
+stop-on-error = True
+
+{{ caucase.updater(
+     prefix='caucase-updater',
+     buildout_bin_directory=parameter_dict['bin_directory'],
+     updater_path='${directory:service}/caucase-updater',
+     url=caucase_url,
+     data_dir='${directory:srv}/caucase-updater',
+     crt_path='${kedifa-config:certificate}',
+     ca_path='${kedifa-config:ca-certificate}',
+     crl_path='${kedifa-config:crl}',
+     key_path='${kedifa-csr:key}',
+     on_renew='${kedifa-reloader:wrapper-path}',
+     template_csr='${kedifa-csr:csr}',
+     openssl=parameter_dict['openssl'] ~ '/bin/openssl',
+)}}
+
+[jinja2-template-base]
+recipe = slapos.recipe.template:jinja2
+rendered = ${buildout:directory}/${:filename}
+extra-context =
+slapparameter_dict = {{ dumps(instance_parameter['configuration']) }}
+slap_software_type = {{ dumps(instance_parameter['slap-software-type']) }}
+context =
+    import json_module json
+    raw common_profile {{ parameter_dict['common_profile'] }}
+    key slap_software_type :slap_software_type
+    key slapparameter_dict :slapparameter_dict
+    section directory directory
+    ${:extra-context}
+
+[kedifa-config]
+ip = {{ instance_parameter['ipv6-random'] }}
+port = {{ instance_parameter['configuration.kedifa_port'] }}
+db = ${directory:kedifa}/kedifa.sqlite
+certificate = ${directory:etc-kedifa}/certificate.pem
+key = ${:certificate}
+ca-certificate = ${directory:etc-kedifa}/ca-certificate.pem
+cas-ca-certificate = ${directory:etc-kedifa}/cas-ca-certificate.pem
+crl = ${directory:etc-kedifa}/crl.pem
+csr = ${directory:etc-kedifa}/csr.pem
+pidfile = ${directory:run}/kedifa.pid
+
+[kedifa-reloader]
+recipe = slapos.cookbook:wrapper
+command-line = kill -SIGHUP `cat ${kedifa-config:pidfile}`
+wrapper-path = ${directory:etc-run}/kedifa-reloader
+
+[kedifa]
+recipe = slapos.cookbook:wrapper
+command-line = {{ parameter_dict['kedifa'] }}
+  --ip ${kedifa-config:ip}
+  --port ${kedifa-config:port}
+  --db ${kedifa-config:db}
+  --certificate ${kedifa-config:certificate}
+  --ca-certificate ${kedifa-config:ca-certificate}
+  --crl ${kedifa-config:crl}
+  --pidfile ${kedifa-config:pidfile}
+
+wrapper-path = ${directory:service}/kedifa
+hash-files = ${buildout:directory}/software_release/buildout.cfg
+
+# Publish KeDiFa configuration for upload and download for each slave
+{%- set slave_kedifa_information = {} -%}
+{%- for slave in slapparameter_dict['slave-list'] -%}
+{%-   set slave_reference = slave['slave_reference'] -%}
+{%-   set slave_dict = {} -%}
+{%-   do slave_dict.__setitem__('key-generate-auth-url', 'https://[${kedifa-config:ip}]:${kedifa-config:port}/${%s-auth-random:passwd}/generateauth' % (slave_reference,)) -%}
+{%-   do slave_dict.__setitem__('key-upload-url', 'https://[${kedifa-config:ip}]:${kedifa-config:port}/${%s-auth-random:passwd}?auth=' % (slave_reference,)) -%}
+{%-   do slave_dict.__setitem__('key-download-url', 'https://[${kedifa-config:ip}]:${kedifa-config:port}/${%s-auth-random:passwd}' % (slave_reference,)) -%}
+{%-   do slave_kedifa_information.__setitem__(slave_reference, slave_dict) %}
+[{{ slave_reference }}-auth-random-generate]
+recipe = plone.recipe.command
+file = ${directory:reservation}/${:_buildout_section_name_}
+command =
+  [ ! -f ${:file} ] && {{ parameter_dict['curl'] }}/bin/curl -s -g -X POST https://[${kedifa-config:ip}]:${kedifa-config:port}/reserve-id --cert ${kedifa-config:certificate} --cacert ${kedifa-config:ca-certificate} > ${:file}.tmp && mv ${:file}.tmp ${:file}
+update-command = ${:command}
+
+[{{ slave_reference }}-auth-random]
+recipe = collective.recipe.shelloutput
+file = {{ '${' + slave_reference }}-auth-random-generate:file}
+commands =
+  passwd = cat ${:file} 2>/dev/null || echo "NotReadyYet"
+
+{% endfor %}
+
+[master-auth-random-generate]
+recipe = plone.recipe.command
+file = ${directory:reservation}/${:_buildout_section_name_}
+command =
+  [ ! -f ${:file} ] && {{ parameter_dict['curl'] }}/bin/curl -s -g -X POST https://[${kedifa-config:ip}]:${kedifa-config:port}/reserve-id --cert ${kedifa-config:certificate} --cacert ${kedifa-config:ca-certificate} > ${:file}.tmp && mv ${:file}.tmp ${:file}
+update-command = ${:command}
+
+[master-auth-random]
+recipe = collective.recipe.shelloutput
+file = ${master-auth-random-generate:file}
+commands =
+  passwd = cat ${:file} 2>/dev/null || echo "NotReadyYet"
+
+[slave-kedifa-information]
+recipe = slapos.cookbook:publish.serialised
+slave-kedifa-information = {{ json_module.dumps(slave_kedifa_information) }}
+caucase-url = {{ caucase_url }}
+master-key-generate-auth-url = https://[${kedifa-config:ip}]:${kedifa-config:port}/${master-auth-random:passwd}/generateauth
+master-key-upload-url = https://[${kedifa-config:ip}]:${kedifa-config:port}/${master-auth-random:passwd}?auth=
+master-key-download-url = https://[${kedifa-config:ip}]:${kedifa-config:port}/${master-auth-random:passwd}
+{%- endif -%} {# if slap_software_type in software_type #}

software/caddy-frontend/instance-output-schema.json

@@ -10,6 +10,14 @@
       "description": "Base domain used by the instance",
       "type": "string"
     },
+    "master-key-generate-auth-url": {
+      "description": "URL to GET once auth for master-key-upload-url",
+      "type": "string"
+    },
+    "master-key-upload-url": {
+      "description": "URL to PUT PEM bundle of main certificate and key",
+      "type": "string"
+    },
     "monitor-base-url": {
       "description": "Base url for monitor",
       "type": "string"

software/caddy-frontend/instance-slave-caddy-input-schema.json

@@ -146,20 +146,6 @@
       "title": "SSL Certificate Authority's Certificate",
       "type": "string"
     },
-    "ssl_crt": {
-      "default": "",
-      "description": "Content of the SSL Certificate file",
-      "textarea": true,
-      "title": "SSL Certificate",
-      "type": "string"
-    },
-    "ssl_key": {
-      "default": "",
-      "description": "Content of the SSL Key file",
-      "textarea": true,
-      "title": "SSL Key",
-      "type": "string"
-    },
     "ssl_proxy_ca_crt": {
       "default": "",
       "description": "Content of the SSL Certificate Authority file of the backend (to be used with ssl-proxy-verify)",

software/caddy-frontend/instance-slave-output-schema.json

@@ -6,6 +6,14 @@
       "description": "Base domain used by the instance",
       "type": "string"
     },
+    "key-generate-auth-url": {
+      "description": "URL to GET once auth for key-upload-url",
+      "type": "array"
+    },
+    "key-upload-url": {
+      "description": "URL to PUT PEM bundle of certificate and key",
+      "type": "array"
+    },
     "log-access-url": {
       "description": "List of URLs to access logs",
       "type": "array"

software/caddy-frontend/instance.cfg.in

@@ -28,6 +28,7 @@ custom-personal = ${dynamic-template-caddy-replicate:rendered}
 single-default = ${dynamic-template-caddy-frontend:rendered}
 single-custom-personal = ${dynamic-template-caddy-frontend:rendered}
 replicate = ${dynamic-template-caddy-replicate:rendered}
+kedifa = ${dynamic-template-kedifa:rendered}
 
 [dynamic-template-caddy-frontend-parameters]
 {% for key,value in template_frontend_parameter_dict.iteritems() %}
@@ -42,6 +43,9 @@ extensions = jinja2.ext.do
 extra-context =
   section parameter_dict dynamic-template-caddy-frontend-parameters
   raw software_type single-custom-personal
+caucase-jinja2-library = {{ caucase_jinja2_library }}
+import-list =
+  file caucase :caucase-jinja2-library
 
 [dynamic-template-caddy-replicate]
 < = jinja2-template-base
@@ -49,12 +53,11 @@ depends = ${caddyprofiledeps:recipe}
 template = {{ template_caddy_replicate }}
 filename = instance-caddy-replicate.cfg
 extensions = jinja2.ext.do
-openssl = {{ template_frontend_parameter_dict['openssl'] ~ '/bin/openssl' }}
 extra-context =
     import subprocess_module subprocess
     import functools_module functools
     import validators validators
-    key openssl :openssl
+    key cluster_identification instance-parameter:root-instance-title
     raw caddy_backend_url_validator {{ caddy_backend_url_validator }}
     raw caddy_custom_http_validator {{ caddy_custom_http_validator }}
     raw template_publish_slave_information {{ template_replicate_publish_slave_information }}
@@ -63,6 +66,18 @@ extra-context =
     raw template_monitor {{ monitor2_template }}
     raw common_profile {{ common_profile }}
 
+[dynamic-template-kedifa]
+< = jinja2-template-base
+template = {{ template_kedifa }}
+filename = instance-kedifa.cfg
+extensions = jinja2.ext.do
+extra-context =
+  section parameter_dict dynamic-template-caddy-frontend-parameters
+  raw software_type kedifa
+caucase-jinja2-library = {{ caucase_jinja2_library }}
+import-list =
+  file caucase :caucase-jinja2-library
+
 [instance-parameter]
 # Fetches parameters defined in SlapOS Master for this instance.
 # Always the same.
@@ -82,14 +97,14 @@ configuration.port = 4443
 configuration.plain_http_port = 8080
 configuration.plain_nginx_port = 8081
 configuration.nginx_port = 9443
+configuration.kedifa_port = 7879
+# Warning: Caucase takes also cacuase_port+1
+configuration.caucase_port = 8890
 # BBB: apache_custom_https and apache_custom_http
 configuration.apache_custom_https = ""
 configuration.apache_custom_http = ""
 configuration.caddy_custom_https = ""
 configuration.caddy_custom_http = ""
-configuration.apache-key =
-configuration.apache-certificate =
-configuration.apache-ca-certificate =
 configuration.open-port = 80 443
 configuration.disk-cache-size = 8G
 configuration.ram-cache-size = 1G

software/caddy-frontend/software.cfg

@@ -2,6 +2,16 @@
 extends = common.cfg
 
 [versions]
+# Versions pinned for kedifa need urllib3 >= 1.18
+urllib3 = 1.24
+requests = 2.20.0
+certifi = 2018.10.15
+idna = 2.7
+chardet = 3.0.4
+# ipaddress is patching IPAddress so IPv6 match works
+ipaddress = 1.0.22
+# Versions pinned for kedifa need urllib3 >= 1.18
+
 validators = 0.12.2
 PyRSS2Gen = 1.1
 cns.recipe.symlink = 0.2.3

software/caddy-frontend/templates/Caddyfile.in

@@ -5,8 +5,9 @@ import {{ slave_configuration_directory }}/*.conf
 import {{ slave_with_cache_configuration_directory }}/*.conf
 
 # Catch-all and 404 for not configured instances
+{% if os_module.path.exists(master_certificate) %}
 :{{ https_port }} {
-  tls {{ login_certificate }} {{ login_key }}
+  tls {{ master_certificate }} {{ master_certificate }}
   bind {{ local_ipv4 }}
   # Compress the output
   gzip
@@ -16,6 +17,7 @@ import {{ slave_with_cache_configuration_directory }}/*.conf
     * {{ not_found_file }}
   }
 }
+{% endif %}
 
 :{{ http_port }} {
   bind {{ local_ipv4 }}

software/caddy-frontend/templates/apache-custom-slave-list.cfg.in

@@ -20,6 +20,7 @@ recipe = slapos.recipe.template:jinja2
 extensions = jinja2.ext.do
 extra-context =
 context =
+    import os_module os
     raw common_profile {{ common_profile }}
     ${:extra-context}
 
@@ -36,13 +37,20 @@ sharedscripts = true
 notifempty = true
 create = true
 
-[cadirectory]
-recipe = slapos.cookbook:mkdirectory
-requests = {{ custom_ssl_directory }}/requests/
-private = {{ custom_ssl_directory }}/private/
-certs = {{ custom_ssl_directory }}/certs/
-newcerts = {{ custom_ssl_directory }}/newcerts/
-crl = {{ custom_ssl_directory }}/crl/
+{% if master_key_download_url %}
+{%   do part_list.append('master-key-download') %}
+[master-key-download]
+recipe = plone.recipe.command
+destination = {{ master_certificate }}
+command = {{ kedifa_getter }} --out ${:destination} --server-ca-certificate {{ kedifa_caucase_ca_certificate }} --identity {{ kedifa_login_certificate }} {{ master_key_download_url }}
+update-command = ${:command}
+{% endif %}
+
+{% if slave_kedifa_information %}
+{%   set slave_kedifa_information = json_module.loads(slave_kedifa_information) %}
+{% else %}
+{%   set slave_kedifa_information = {} %}
+{% endif %}
 
 [promise-plugin-base]
 recipe = slapos.cookbook:promise.plugin
@@ -55,6 +63,12 @@ output = {{ plugin_directory }}/${:name}
 {# Loop thought slave list to set up slaves #}
 {% for slave_instance in slave_instance_list %}
 {%   set slave_reference = slave_instance.get('slave_reference') %}
+{%   set slave_kedifa = slave_kedifa_information.get(slave_reference) %}
+{%   if slave_kedifa %}
+{%     set key_download_url = slave_kedifa.get('key-download-url') %}
+{%   else %}
+{%     set key_download_url = '' %}
+{%   endif %}
 {%   set slave_type = slave_instance.get('type', '') %}
 {%   set slave_section_title = 'dynamic-template-slave-instance-%s' % slave_reference %}
 {%   set slave_parameter_dict = generic_instance_parameter_dict.copy() %}
@@ -167,66 +181,37 @@ bytes = 8
 
 {# ################################################## #}
 {# Set Slave Certificates if needed                   #}
+{%   set cert_dirname = slave_reference.replace('-','.') %}
+{%   set autocert_dir = '/'.join([autocert, cert_dirname]) %}
+[{{ slave_reference }}-path]
+recipe = slapos.cookbook:mkdirectory
+cert = {{ autocert_dir }}
+{# Set certificate key for custom configuration #}
+{% set certificate = '%s/certificate.pem' % (autocert_dir, ) %}
+{% do slave_parameter_dict.__setitem__('certificate', certificate )%}
 
-{# Set ssl certificates for each slave #}
-{%   for cert_name in ('ssl_csr', 'ssl_proxy_ca_crt')%}
-{%     if slave_instance.get(cert_name) %}
-{%       set cert_title = '%s-%s' % (slave_reference, cert_name.replace('ssl_', '')) %}
-{%       set cert_file = '/'.join([custom_ssl_directory, cert_title.replace('-','.')]) %}
-{%       do part_list.append(cert_title) %}
-{%       do slave_parameter_dict.__setitem__(cert_name, cert_file) %}
-{%       do slave_instance.__setitem__('path_to_' + cert_name, cert_file) %}
-{# Store certificates on fs #}
-[{{ cert_title }}]
-< = jinja2-template-base
-template = {{ empty_template }}
-rendered = {{ cert_file }}
-extra-context =
-    key content {{ cert_title + '-config:value' }}
-# Store certificate in config
-[{{ cert_title + '-config' }}]
-value = {{ dumps(slave_instance.get(cert_name)) }}
-{%     endif %}
-{%   endfor %}
-
-{#-   Set Up Certs #}
-{%   do slave_instance.__setitem__('login_certificate', login_certificate) %}
-{%   do slave_instance.__setitem__('login_key', login_key) %}
-{%   do slave_instance.__setitem__('login_ca_crt', login_ca_crt) %}
-{%   do slave_parameter_dict.__setitem__('ssl_crt', login_certificate) %}
-{%   do slave_parameter_dict.__setitem__('ssl_key', login_key) %}
-{%   if slave_instance.get('ssl_key') and slave_instance.get('ssl_crt') %}
-{%     set cert_title = '%s-crt' % (slave_reference) %}
-{%     set key_title = '%s-key' % (slave_reference) %}
-{%     set cert_file = '/'.join([custom_ssl_directory, cert_title.replace('-','.')]) %}
-{%     set key_file = '/'.join([custom_ssl_directory, key_title.replace('-','.')]) %}
-{%     do part_list.append(cert_title) %}
-{%     do part_list.append(key_title) %}
-{%     do slave_parameter_dict.__setitem__("ssl_crt", cert_file) %}
-{%     do slave_parameter_dict.__setitem__("ssl_key", key_file) %}
-{%     do slave_instance.__setitem__('path_to_ssl_crt', cert_file) %}
-{%     do slave_instance.__setitem__('path_to_ssl_key', key_file) %}
-
-[{{key_title}}]
-< = jinja2-template-base
-template = {{ empty_template }}
-rendered = {{ key_file }}
-key-content = {{ dumps(slave_instance.get('ssl_key')) }}
-extra-context =
-    key content :key-content
-
-[{{cert_title}}]
-< = jinja2-template-base
-template = {{ empty_template }}
-rendered = {{ cert_file }}
-cert-content = {{ dumps('%s\n%s' % (slave_instance.get('ssl_crt'), slave_instance.get('ssl_ca_crt', '') or '')) }}
-extra-context =
-    key content :cert-content
-{%     endif %}
+[{{ slave_reference }}-key-download]
+recipe = plone.recipe.command
+destination = {{ '${' + slave_reference + '-path:cert}/downloaded.pem' }}
+used = {{ '${' + slave_reference + '-path:cert}/certificate.pem' }}
+source-master = ${master-key-download:destination}
+command =
+  {{ kedifa_getter }} --out ${:destination} --server-ca-certificate {{ kedifa_caucase_ca_certificate }} --identity {{ kedifa_login_certificate }} {{ key_download_url }}
+  if [ -f ${:destination} ] ; then
+    # if the slave specific certificate is available, use it
+    ln -sf ${:destination} ${:used}
+  elif [ -f ${:source-master} ] ; then
+    # if the master provided certificate is available, use it
+    ln -sf ${:source-master} ${:used}
+  else
+    rm -f ${:used}
+  fi
+update-command = ${:command}
 
 {# ########################################## #}
 {# Set Slave Configuration                    #}
 [{{ slave_configuration_section_name }}]
+certificate = {{ '${' + slave_reference + '-key-download:used}' }}
 https_port = {{ dumps('' ~ https_port) }}
 http_port = {{ dumps('' ~ http_port) }}
 local_ipv4 = {{ dumps('' ~ local_ipv4) }}

software/caddy-frontend/templates/default-virtualhost.conf.in

@@ -26,11 +26,12 @@
 {%- set default_path = slave_parameter.get('default-path', '').strip('/') | urlencode %}
 
 # SSL enabled hosts
+{% if os_module.path.exists(slave_parameter['certificate']) %}
 {{ https_host_list|join(', ') }} {
   bind {{ slave_parameter['local_ipv4'] }}
   # Compress the output
   gzip
-  tls {{ slave_parameter.get('path_to_ssl_crt', slave_parameter.get('login_certificate')) }} {{ slave_parameter.get('path_to_ssl_key', slave_parameter.get('login_key')) }} {
+  tls {{ slave_parameter['certificate'] }} {{ slave_parameter['certificate'] }} {
 {%- if enable_h2 %}
     # Allow HTTP2
     alpn h2 http/1.1
@@ -175,6 +176,7 @@
 {%-   endif %} {#-   if backend_url #}
 {%- endif %} {#- if slave_type ==  'zope' and backend_url #}
 }  {# https_host_list|join(', ') #}
+{% endif %}
 
 # SSL-disabled hosts
 {{ http_host_list|join(', ') }} {

software/caddy-frontend/templates/nginx-eventsource-slave.conf.in

@@ -7,9 +7,6 @@
 {%   set https_protocol = https_url.split("/")[0] %}
 {%   set proxy_pass = '%s//%s' % (protocol, slave_parameter.get('slave_reference')) %}
 {%   set https_proxy_pass = '%s//https_%s' % (protocol, slave_parameter.get('slave_reference')) %}
-{% set ssl_configuration_list = [('ssl_certificate', 'path_to_ssl_crt'),
-                                 ('ssl_certificate_key', 'path_to_ssl_key')] %}
-
 
 # TODO-Caddy upstream {{ slave_parameter.get('slave_reference') }} {
 # TODO-Caddy  server {{ upstream }};
@@ -75,12 +72,6 @@
 # TODO-Caddy  ssl_prefer_server_ciphers on;
 # TODO-Caddy  ssl_session_cache shared:SSL:10m;
 
-{% for key, value in ssl_configuration_list %}
-{%   if value in slave_parameter %}
-# TODO-Caddy {{ '  %s' % key }} {{ slave_parameter.get(value) }};
-{% endif %}
-{% endfor %}
-
 # TODO-Caddy  location /pub {
 # TODO-Caddy    push_stream_publisher;
 # TODO-Caddy    push_stream_channels_path $arg_id;

software/caddy-frontend/templates/nginx-notebook-slave.conf.in

@@ -5,6 +5,7 @@
 {%-   set https_upstream = https_url.split("/")[2] %}
 
 # SSL-enabled
+{% if os_module.path.exists(slave_parameter['certificate']) %}
 https://{{ slave_parameter.get('custom_domain') }}:{{ slave_parameter['nginx_https_port'] }} {
   bind {{ slave_parameter['local_ipv4'] }}
   # Compress the output
@@ -12,7 +13,7 @@ https://{{ slave_parameter.get('custom_domain') }}:{{ slave_parameter['nginx_htt
   log / {{ slave_parameter.get('access_log') }} "{remote} {>REMOTE_USER} [{when}] \"{method} {uri} {proto}\" {status} {size} \"{>Referer}\" \"{>User-Agent}\" {latency_ms}"
   errors {{ slave_parameter.get('error_log') }}
 
-  tls {{ slave_parameter.get('path_to_ssl_crt', slave_parameter.get('login_certificate')) }} {{ slave_parameter.get('path_to_ssl_key', slave_parameter.get('login_key')) }} {
+  tls {{ slave_parameter['certificate'] }} {{ slave_parameter['certificate'] }} {
     alpn http/1.1
   }
 
@@ -36,6 +37,7 @@ https://{{ slave_parameter.get('custom_domain') }}:{{ slave_parameter['nginx_htt
     insecure_skip_verify
   }
 }
+{% endif %}
 
 # SSL-disabled
 http://{{ slave_parameter.get('custom_domain') }}:{{ slave_parameter['nginx_http_port'] }} {

software/caddy-frontend/templates/nginx.cfg.in

@@ -58,8 +58,9 @@
 import {{ slave_configuration_directory }}/*.conf
 
 # Catch-all and 404 for not configured instances
+{% if os_module.path.exists(master_certificate) %}
 :{{ port }} {
-  tls {{ ssl_certificate }} {{ ssl_key }}
+  tls {{ master_certificate }} {{ master_certificate }}
   bind {{ local_ip }}
   # Serve an error 204 (No Content) for favicon.ico
   status 204 /favicon.ico
@@ -69,6 +70,7 @@ import {{ slave_configuration_directory }}/*.conf
     * {{ not_found_file }}
   }
 }
+{% endif %}
 
 :{{ plain_port }} {
   bind {{ local_ip }}

software/caddy-frontend/templates/replicate-publish-slave-information.cfg.in

@@ -34,6 +34,16 @@
 {%   do slave_information_dict[slave_reference].__setitem__('request-error-list', rejected_info_list) %}
 {% endfor %}
 
+{% for slave_reference, kedifa_dict in json_module.loads(slave_kedifa_information).iteritems() %}
+{%   if slave_reference not in rejected_slave_information %}
+{%     if slave_reference not in slave_information_dict %}
+{%       do slave_information_dict.__setitem__(slave_reference, {}) %}
+{%     endif %}
+{%     do slave_information_dict[slave_reference].__setitem__('key-generate-auth-url', kedifa_dict['key-generate-auth-url']) %}
+{%     do slave_information_dict[slave_reference].__setitem__('key-upload-url', kedifa_dict['key-upload-url']) %}
+{%   endif %}
+{% endfor %}
+
 # Publish information for each slave
 {% set active_slave_instance_list = json_module.loads(active_slave_instance_dict['active-slave-instance-list']) %}
 {% for slave_reference, slave_information in slave_information_dict.iteritems() %}

software/caddy-frontend/templates/template-log-access.conf.in

+# Access log configuration
 {% for slave, directory in slave_log_directory.iteritems() %}
 https://[{{ parameter_dict['global_ipv6'] }}]:{{ parameter_dict['https_port'] }}/{{ slave }}, https://{{ parameter_dict['local_ipv4'] }}:{{ parameter_dict['https_port'] }}/{{ slave }} {
   bind {{ parameter_dict['local_ipv4'] }}

software/caddy-frontend/test/test.py

@@ -66,6 +66,8 @@ NGINX_HTTPS_PORT = '12443'
 MONITOR_HTTPD_PORT = '13000'
 MONITOR_F1_HTTPD_PORT = '13001'
 MONITOR_F2_HTTPD_PORT = '13002'
+CAUCASE_PORT = '15090'
+KEDIFA_PORT = '15080'
 
 
 # for development: debugging logs and install Ctrl+C handler