caddy-frontend: Implement caddy_custom*

caddy_custom_http and caddy_custom_https are implemented and exposed
instead of apache_custom_http and apache_custom_https, but with backward
compatbility for the latter form from apache-frontend.

In TODO mark missing usage of custom http found during work on this commit.

software/caddy-frontend/README.caddy_frontend.rst

@@ -157,12 +157,12 @@ Will append the specified path to the "VirtualHostRoot" of the zope's VirtualHos
 "path" is an optional parameter, ignored if not specified.
 Example of value: "/erp5/web_site_module/hosting/"
 
-apache_custom_https
-~~~~~~~~~~~~~~~~~~~
+caddy_custom_https
+~~~~~~~~~~~~~~~~~~
 Raw Caddy configuration in python template format (i.e. write "%%" for one "%") for the slave listening to the https port. Its content will be templatified in order to access functionalities such as cache access, ssl certificates... The list is available above.
 
-apache_custom_http
-~~~~~~~~~~~~~~~~~~
+caddy_custom_http
+~~~~~~~~~~~~~~~~~
 Raw Caddy configuration in python template format (i.e. write "%%" for one "%") for the slave listening to the http port. Its content will be templatified in order to access functionalities such as cache access, ssl certificates... The list is available above
 
 url
@@ -286,7 +286,7 @@ Request slave frontend instance so that https://[1:2:3:4:5:6:7:8]:1234 will be::
     partition_parameter_kw={
         "url":"https://[1:2:3:4:5:6:7:8]:1234",
 
-        "apache_custom_https":'
+        "caddy_custom_https":'
   https://www.example.com:%(https_port)s, https://example.com:%(https_port)s {
     bind %(local_ipv4)s
     tls %(ssl_crt)s %(ssl_key)s
@@ -300,7 +300,7 @@ Request slave frontend instance so that https://[1:2:3:4:5:6:7:8]:1234 will be::
       insecure_skip_verify
     }
   }
-        "apache_custom_http":'
+        "caddy_custom_http":'
   http://www.example.com:%(http_port)s, http://example.com:%(http_port)s {
     bind %(local_ipv4)s
     log / %(access_log)s {combined}
@@ -329,7 +329,7 @@ Request slave frontend instance so that https://[1:2:3:4:5:6:7:8]:1234 will be::
 	"domain": "www.example.org",
 	"enable_cache": "True",
 
-        "apache_custom_https":'
+        "caddy_custom_https":'
   ServerName www.example.org
   ServerAlias www.example.org
   ServerAlias example.org
@@ -343,7 +343,7 @@ Request slave frontend instance so that https://[1:2:3:4:5:6:7:8]:1234 will be::
   RewriteEngine On
   RewriteRule ^/(.*) %(cache_access)s/$1 [L,P]',
 
-        "apache_custom_http":'
+        "caddy_custom_http":'
   ServerName www.example.org
   ServerAlias www.example.org
   ServerAlias example.org
@@ -384,7 +384,7 @@ the proxy::
         "path":"/erp5",
         "domain":"example.org",
 
-  	"apache_custom_https":'
+  	"caddy_custom_https":'
   ServerName www.example.org
   ServerAlias www.example.org
   ServerAdmin example.org
@@ -413,7 +413,7 @@ the proxy::
   # Use cache
   RewriteRule ^/(.*) %(cache_access)s/VirtualHostBase/https/www.example.org:443/erp5/VirtualHostRoot/$1 [L,P]',
 
-    "apache_custom_http":'
+    "caddy_custom_http":'
   ServerName www.example.org
   ServerAlias www.example.org
   ServerAlias example.org

software/caddy-frontend/TODO.rst

@@ -17,7 +17,6 @@ Generally things to be done with ``caddy-frontend``:
  * cover test suite like resilient tests for KVM and prove it works the same way as Caddy
  * have ``caddy-frontend`` specific parameters, with backward compatibility to ``apache-frontend`` ones:
 
-  * ``caddy_custom_http`` and ``caddy_custom_https``
   * ``apache-ca-certificate``
   * ``apache-certificate`` and ``apache-key``
 
@@ -60,6 +59,10 @@ Generally things to be done with ``caddy-frontend``:
  * drop ``6tunnel`` and use ``bind`` in Caddy configuration, as soon as multiple binds will be possible, tracked in upstream `bind: support multiple values <https://github.com/mholt/caddy/pull/2128>`_ and `ipv6: does not bind on ipv4 and ipv6 for sites that resolve to both <https://github.com/mholt/caddy/issues/864>`_
  * use caddy-frontend in `standalone style playbooks <https://lab.nexedi.com/nexedi/slapos.package/tree/master/playbook/roles/standalone-shared>`_
  * ensure `QUIC <https://en.wikipedia.org/wiki/QUIC>`_ is used by caddy
+ * fix missing ``caddy_custom_https`` in:
+
+    * instance-apache-replicate.cfg.in:``{%   if not ((slave.has_key('caddy_custom_http') or slave.has_key('apache_custom_http')) and not slave.get('slave_reference') in authorized_slave_string) %}``
+    * templates/apache-custom-slave-list.cfg.in:``{%   if caddy_custom_http %}``
 
 Things which can't be implemented:
 

software/caddy-frontend/buildout.hash.cfg

@@ -18,19 +18,19 @@ md5sum = 906e5bd66b1265b8109a86b6ab46e91f
 
 [template-apache-frontend]
 filename = instance-apache-frontend.cfg
-md5sum = 5602b4635e3da27ea6ff491824b0d8a5
+md5sum = a0e1e10f154342e8cc9936066e855b8f
 
 [template-apache-replicate]
 filename = instance-apache-replicate.cfg.in
-md5sum = cd83f92b43904e1f3826072013cd682b
+md5sum = 3650a34afe39cdb9fdb46d58fb2acd2e
 
 [template-slave-list]
 filename = templates/apache-custom-slave-list.cfg.in
-md5sum = 3993419eea72ad4b62c0d479860f3c17
+md5sum = ded5050d32d234ab13c843051bfac102
 
 [template-slave-configuration]
 filename = templates/custom-virtualhost.conf.in
-md5sum = 74275ad73b03114c69f80c8f8ae73374
+md5sum = 54ae95597a126ae552c3a913ddf29e5e
 
 [template-replicate-publish-slave-information]
 filename = templates/replicate-publish-slave-information.cfg.in
@@ -42,7 +42,7 @@ md5sum = 6689d96fc18d9aad78d77fe87770d4da
 
 [template-custom-slave-list]
 filename = templates/apache-custom-slave-list.cfg.in
-md5sum = 3993419eea72ad4b62c0d479860f3c17
+md5sum = ded5050d32d234ab13c843051bfac102
 
 [template-not-found-html]
 filename = templates/notfound.html

software/caddy-frontend/instance-apache-frontend.cfg

@@ -110,8 +110,11 @@ configuration.plain_http_port = 8080
 configuration.plain_nginx_port = 8081
 configuration.nginx_port = 9443
 configuration.server-admin = admin@example.com
+# BBB: apache_custom_https and apache_custom_http
 configuration.apache_custom_https = ""
 configuration.apache_custom_http = ""
+configuration.caddy_custom_https = ""
+configuration.caddy_custom_http = ""
 configuration.apache-key =
 configuration.apache-certificate =
 configuration.apache-ca-certificate =
@@ -210,11 +213,14 @@ extra-context =
 template = ${template-slave-configuration:target}
 rendered = $${directory:template}/slave-virtualhost.conf.in
 extensions = jinja2.ext.do
+# BBB: apache_custom_https and apache_custom_http
 extra-context =
     key https_port instance-parameter:configuration.port
     key http_port instance-parameter:configuration.plain_http_port
     key apache_custom_https instance-parameter:configuration.apache_custom_https
     key apache_custom_http instance-parameter:configuration.apache_custom_http
+    key caddy_custom_https instance-parameter:configuration.caddy_custom_https
+    key caddy_custom_http instance-parameter:configuration.caddy_custom_http
 
 # Deploy Caddy Frontend with Jinja power
 [dynamic-caddy-frontend-template]
@@ -606,8 +612,11 @@ public-ipv4 =
 port = 4443
 plain_http_port = 8080
 server-admin = admin@example.com
+# BBB: apache_custom_https and apache_custom_http
 apache_custom_https = ""
 apache_custom_http = ""
+caddy_custom_https = ""
+caddy_custom_http = ""
 apache-key =
 apache-certificate =
 open-port = 80 443

software/caddy-frontend/instance-apache-replicate.cfg.in

@@ -70,7 +70,8 @@ context =
 {% set authorized_slave_list = [] %}
 {% set rejected_slave_list = [] %}
 {% for slave in slave_instance_list %}
-{%   if not (slave.has_key('apache_custom_http') and not slave.get('slave_reference') in authorized_slave_string) %}
+{# BBB: apache_custom_http #}
+{%   if not ((slave.has_key('caddy_custom_http') or slave.has_key('apache_custom_http')) and not slave.get('slave_reference') in authorized_slave_string) %}
 {%     do authorized_slave_list.append(slave) %}
 {%   else %}
 {%     do rejected_slave_list.append(slave.get('slave_reference')) %}

software/caddy-frontend/instance-slave-caddy-input-schema.json

 {
     "$schema": "http://json-schema.org/draft-04/schema",
     "properties": {
-        "apache_custom_http": {
+        "caddy_custom_http": {
             "default": "",
             "description": "Raw http configuration in python template format. Your site will be rejected if you use it without notification and approval of frontend administrators",
             "textarea": true,
             "title": "HTTP configuration",
             "type": "string"
         },
-        "apache_custom_https": {
+        "caddy_custom_https": {
             "default": "",
             "description": "Raw https configuration in python template format. Your site will be rejected if you use it without notification and approval of frontend administrators",
             "textarea": true,

software/caddy-frontend/templates/apache-custom-slave-list.cfg.in

@@ -99,7 +99,8 @@ crl = {{ custom_ssl_directory }}/crl/
 {%     do cached_server_dict.__setitem__(slave_reference, slave_configuration_section_name) %}
 {%   endif %}
 
-{%   if not slave_instance.has_key('apache_custom_http') and not slave_instance.has_key('apache_custom_https') %}
+{# BBB: apache_custom_https and apache_custom_http #}
+{%   if not slave_instance.has_key('caddy_custom_http') and not slave_instance.has_key('caddy_custom_https') and not slave_instance.has_key('apache_custom_http') and not slave_instance.has_key('apache_custom_https') %}
 {%     do slave_publish_dict.__setitem__('domain', slave_instance.get('custom_domain')) %}
 {%     do slave_publish_dict.__setitem__('url', "http://%s" % slave_instance.get('custom_domain')) %}
 {%     do slave_publish_dict.__setitem__('site_url', "http://%s" % slave_instance.get('custom_domain')) %}
@@ -193,10 +194,11 @@ cert-content = {{ dumps(slave_instance.get('ssl_crt')) }}
 {# ########################################## #}
 {# Set Slave Configuration                    #}
 [{{ slave_configuration_section_name }}]
-{%     set apache_custom_http = ((slave_instance.pop('apache_custom_http', '')) % slave_parameter_dict) %}
-{%     set apache_custom_https = ((slave_instance.pop('apache_custom_https', '')) % slave_parameter_dict) %}
-apache_custom_http = {{ dumps(apache_custom_http) }}
-apache_custom_https = {{ dumps(apache_custom_https) }}
+{# BBB: apache_custom_https and apache_custom_http #}
+{%     set caddy_custom_http = ((slave_instance.pop('caddy_custom_http', slave_instance.pop('apache_custom_http', ''))) % slave_parameter_dict) %}
+{%     set caddy_custom_https = ((slave_instance.pop('caddy_custom_https', slave_instance.pop('apache_custom_https', ''))) % slave_parameter_dict) %}
+caddy_custom_http = {{ dumps(caddy_custom_http) }}
+caddy_custom_https = {{ dumps(caddy_custom_https) }}
 {{ '\n' }}
 {%     for key, value in slave_instance.iteritems() %}
 {{ key }} = {{ dumps(value) }}
@@ -211,7 +213,7 @@ rendered = {{ caddy_configuration_directory }}/${:filename}
 {%   endif %}
 
 
-{%   if apache_custom_http %}
+{%   if caddy_custom_http %}
 template = {{ template_custom_slave_configuration }}
 {%   elif slave_type == 'eventsource' %}
 template = {{ template_eventsource_slave_configuration }}

software/caddy-frontend/templates/custom-virtualhost.conf.in

-{{ slave_parameter.get('apache_custom_https', '') }}
-{{ slave_parameter.get('apache_custom_http', '') }}
+{{ slave_parameter.get('caddy_custom_https', '') }}
+{{ slave_parameter.get('caddy_custom_http', '') }}
\ No newline at end of file

software/caddy-frontend/test/test.py

@@ -70,6 +70,35 @@ if IS_CADDY:
 else:
   no_backend_response_code = 502
 
+caddy_custom_https = '''# caddy_custom_https_filled_in_accepted
+https://caddycustomhttpsaccepted.example.com:%%(https_port)s {
+  bind %%(local_ipv4)s
+  tls %%(ssl_crt)s %%(ssl_key)s
+
+  log / %%(access_log)s {combined}
+  errors %%(error_log)s
+
+  proxy / %(url)s {
+    transparent
+    timeout 600s
+    insecure_skip_verify
+  }
+}
+'''
+caddy_custom_http = '''# caddy_custom_http_filled_in_accepted
+http://caddycustomhttpsaccepted.example.com:%%(http_port)s {
+  bind %%(local_ipv4)s
+  log / %%(access_log)s {combined}
+  errors %%(error_log)s
+
+  proxy / %(url)s {
+    transparent
+    timeout 600s
+    insecure_skip_verify
+  }
+}
+'''
+
 # apache_custom_http[s] difference
 if IS_CADDY:
   LOG_REGEXP = '^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3} SOME_REMOTE_USER ' \
@@ -495,7 +524,8 @@ class TestSlave(SlaveHttpFrontendTestCase, TestDataMixin):
       'public-ipv4': utils.LOCAL_IPV4,
       'apache-certificate': open('wildcard.example.com.crt').read(),
       'apache-key': open('wildcard.example.com.key').read(),
-      '-frontend-authorized-slave-string': '_apache_custom_http_s-accepted',
+      '-frontend-authorized-slave-string':
+      '_apache_custom_http_s-accepted _caddy_custom_http_s-accepted',
       'port': HTTPS_PORT,
       'plain_http_port': HTTP_PORT,
       'nginx_port': NGINX_HTTPS_PORT,
@@ -632,6 +662,16 @@ class TestSlave(SlaveHttpFrontendTestCase, TestDataMixin):
         'apache_custom_https': apache_custom_https % dict(url=cls.backend_url),
         'apache_custom_http': apache_custom_http % dict(url=cls.backend_url),
       },
+      'caddy_custom_http_s-rejected': {
+        'url': cls.backend_url,
+        'caddy_custom_https': '# caddy_custom_https_filled_in_rejected',
+        'caddy_custom_http': '# caddy_custom_http_filled_in_rejected',
+      },
+      'caddy_custom_http_s-accepted': {
+        'url': cls.backend_url,
+        'caddy_custom_https': caddy_custom_https % dict(url=cls.backend_url),
+        'caddy_custom_http': caddy_custom_http % dict(url=cls.backend_url),
+      },
       'prefer-gzip-encoding-to-backend': {
         'url': cls.backend_url,
         'prefer-gzip-encoding-to-backend': 'true',
@@ -672,10 +712,11 @@ class TestSlave(SlaveHttpFrontendTestCase, TestDataMixin):
       {
         'monitor-base-url': None,
         'domain': 'example.com',
-        'accepted-slave-amount': '32',
-        'rejected-slave-amount': '1',
-        'slave-amount': '33',
-        'rejected-slave-list': '["_apache_custom_http_s-rejected"]'},
+        'accepted-slave-amount': '33',
+        'rejected-slave-amount': '2',
+        'slave-amount': '35',
+        'rejected-slave-list':
+        '["_caddy_custom_http_s-rejected", "_apache_custom_http_s-rejected"]'},
       parameter_dict
     )
 
@@ -2068,10 +2109,6 @@ class TestSlave(SlaveHttpFrontendTestCase, TestDataMixin):
     self.assertEqual(
       result.json()['Incoming Headers']['cookie'], 'Coffee=present')
 
-  @skip('Feature postponed')
-  def test_caddy_custom_http_s_rejected(self):
-    raise NotImplementedError
-
   def test_apache_custom_http_s_rejected(self):
     parameter_dict = self.slave_connection_parameter_dict_dict[
       'apache_custom_http_s-rejected']
@@ -2143,7 +2180,81 @@ class TestSlave(SlaveHttpFrontendTestCase, TestDataMixin):
 
     configuration_file_with_custom_http_list = [
       q for q in slave_configuration_file_list
-      if 'apache_custom_https_filled_in_accepted' in open(q).read()]
+      if 'apache_custom_http_filled_in_accepted' in open(q).read()]
+    self.assertEqual(1, len(configuration_file_with_custom_http_list))
+
+  def test_caddy_custom_http_s_rejected(self):
+    parameter_dict = self.slave_connection_parameter_dict_dict[
+      'caddy_custom_http_s-rejected']
+    self.assertEqual({}, parameter_dict)
+    slave_configuration_file_list = glob.glob(os.path.join(
+      self.instance_path, '*', 'etc', '*slave-conf.d', '*.conf'))
+    # no configuration file contains provided custom http
+    configuration_file_with_custom_https_list = [
+      q for q in slave_configuration_file_list
+      if 'caddy_custom_https_filled_in_rejected' in open(q).read()]
+    self.assertEqual([], configuration_file_with_custom_https_list)
+
+    configuration_file_with_custom_http_list = [
+      q for q in slave_configuration_file_list
+      if 'caddy_custom_http_filled_in_rejected' in open(q).read()]
+    self.assertEqual([], configuration_file_with_custom_http_list)
+
+  def test_caddy_custom_http_s_accepted(self):
+    parameter_dict = self.slave_connection_parameter_dict_dict[
+      'caddy_custom_http_s-accepted']
+    self.assertLogAccessUrlWithPop(
+      parameter_dict, 'caddy_custom_http_s-accepted')
+    self.assertEqual(
+      parameter_dict,
+      {'replication_number': '1', 'public-ipv4': utils.LOCAL_IPV4}
+    )
+
+    result = self.fakeHTTPSResult(
+      'caddycustomhttpsaccepted.example.com',
+      parameter_dict['public-ipv4'], 'test-path')
+
+    self.assertEqual(
+      utils.der2pem(result.peercert),
+      open('wildcard.example.com.crt').read())
+
+    self.assertEqualResultJson(result, 'Path', '/test-path')
+
+    headers = result.headers.copy()
+
+    self.assertKeyWithPop('Server', headers)
+    self.assertKeyWithPop('Date', headers)
+
+    # drop vary-keys
+    headers.pop('Content-Length', None)
+    headers.pop('Transfer-Encoding', None)
+    headers.pop('Connection', None)
+    headers.pop('Keep-Alive', None)
+
+    self.assertEqual(
+      headers,
+      {
+        'Content-type': 'text/json',
+        'Set-Cookie': 'secured=value;secure, nonsecured=value'
+      }
+    )
+
+    result_http = self.fakeHTTPResult(
+      'caddycustomhttpsaccepted.example.com',
+      parameter_dict['public-ipv4'], 'test-path')
+    self.assertEqualResultJson(result_http, 'Path', '/test-path')
+
+    slave_configuration_file_list = glob.glob(os.path.join(
+      self.instance_path, '*', 'etc', '*slave-conf.d', '*.conf'))
+    # no configuration file contains provided custom http
+    configuration_file_with_custom_https_list = [
+      q for q in slave_configuration_file_list
+      if 'caddy_custom_https_filled_in_accepted' in open(q).read()]
+    self.assertEqual(1, len(configuration_file_with_custom_https_list))
+
+    configuration_file_with_custom_http_list = [
+      q for q in slave_configuration_file_list
+      if 'caddy_custom_http_filled_in_accepted' in open(q).read()]
     self.assertEqual(1, len(configuration_file_with_custom_http_list))
 
   def test_https_url(self):

software/caddy-frontend/test/test_data/test.TestSlave.test_file_list_log-CADDY.txt

@@ -3,6 +3,8 @@ TestSlave-1/var/log/frontend-access.log
 TestSlave-1/var/log/frontend-error.log
 TestSlave-1/var/log/httpd/_apache_custom_http_s-accepted_access_log
 TestSlave-1/var/log/httpd/_apache_custom_http_s-accepted_error_log
+TestSlave-1/var/log/httpd/_caddy_custom_http_s-accepted_access_log
+TestSlave-1/var/log/httpd/_caddy_custom_http_s-accepted_error_log
 TestSlave-1/var/log/httpd/_custom_domain_access_log
 TestSlave-1/var/log/httpd/_custom_domain_error_log
 TestSlave-1/var/log/httpd/_custom_domain_ssl_crt_ssl_key_access_log

software/caddy-frontend/test/test_data/test.TestSlave.test_monitor_promise_list-CADDY.txt

 TestSlave-1/etc/monitor-promise/check-_apache_custom_http_s-accepted-error-log-last-day
 TestSlave-1/etc/monitor-promise/check-_apache_custom_http_s-accepted-error-log-last-hour
+TestSlave-1/etc/monitor-promise/check-_caddy_custom_http_s-accepted-error-log-last-day
+TestSlave-1/etc/monitor-promise/check-_caddy_custom_http_s-accepted-error-log-last-hour
 TestSlave-1/etc/monitor-promise/check-_custom_domain-error-log-last-day
 TestSlave-1/etc/monitor-promise/check-_custom_domain-error-log-last-hour
 TestSlave-1/etc/monitor-promise/check-_custom_domain_ssl_crt_ssl_key-error-log-last-day