Skip to content

slapos
slapos
Commit 78f3c519 authored by Łukasz Nowak's avatar Łukasz Nowak Committed by Rafael Monnerat
Browse files
Options

caddy-frontend: Drop whitespace control in Jinja2 

Therte is no need to control whitespace adding by Jijna2 and dropping it
simplifies the templates.

It will allow to take better control over generated configuration files.
parent ac9c1973
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 272 additions and 273 deletions
software/caddy-frontend/TODO.rst
View file @ 78f3c519
......@@ -9,7 +9,6 @@ Generally things to be done with ``caddy-frontend``:
* ``ssl_proxy_ca_crt`` for ``ssl_proxy_verify``, this is related to bug https://github.com/mholt/caddy/issues/1550, proposed solution `just adding your CA to the system's trust store`
* ``check-error-on-caddy-log`` like ``check-error-on-apache-log``
* cover test suite like resilient tests for KVM and prove it works the same way as Caddy
* simplify Jijna2 syntax and drop whitespace control, as it is not needed
* make beautiful (eg. with whitespaces and nice comments) generated files (mostly Jinja2)
* have ``caddy-frontend`` specific parameters, with backward compatibility to ``apache-frontend`` ones (like ``apache_custom_http`` --> ``caddy_custom_http``)
* change ``switch-softwaretype`` to way how ``software/erp5`` does, which will help with dropping jinja2 template for ``caddy-wrapper``, which is workaround for current situation https://lab.nexedi.com/nexedi/slapos/merge_requests/312#note_62678
......
software/caddy-frontend/buildout.hash.cfg
View file @ 78f3c519
......@@ -22,11 +22,11 @@ md5sum = ba6b8d3b11f4d9809bc0620a05946a16
[template-apache-replicate]
filename = instance-apache-replicate.cfg.in
md5sum = e2d7a5bc47e1c2e79d9ba84684277c6e
md5sum = cd83f92b43904e1f3826072013cd682b
[template-slave-list]
filename = templates/apache-custom-slave-list.cfg.in
md5sum = acfe2c5530ddd46803f99e7f5249aa5e
md5sum = 3993419eea72ad4b62c0d479860f3c17
[template-slave-configuration]
filename = templates/custom-virtualhost.conf.in
......@@ -34,7 +34,7 @@ md5sum = 74275ad73b03114c69f80c8f8ae73374
[template-replicate-publish-slave-information]
filename = templates/replicate-publish-slave-information.cfg.in
md5sum = 665e83d660c9b779249b2179d7ce4b4e
md5sum = 8d318af17da5631d4242c0d6d1531066
[template-caddy-frontend-configuration]
filename = templates/Caddyfile.in
......@@ -42,7 +42,7 @@ md5sum = 5c048a80c2e9374adea0cd647d14b192
[template-custom-slave-list]
filename = templates/apache-custom-slave-list.cfg.in
md5sum = acfe2c5530ddd46803f99e7f5249aa5e
md5sum = 3993419eea72ad4b62c0d479860f3c17
[template-not-found-html]
filename = templates/notfound.html
......@@ -50,11 +50,11 @@ md5sum = f20d6c3d2d94fb685f8d26dfca1e822b
[template-default-slave-virtualhost]
filename = templates/default-virtualhost.conf.in
md5sum = 9568465d1c1423343f7b043c8345f917
md5sum = c291422b417a906ee944de17321c0d12
[template-cached-slave-virtualhost]
filename = templates/cached-virtualhost.conf.in
md5sum = 811b4fca0668b84655372687f234ee81
md5sum = 194b1c41ae2edd1528e85bd01eb1f95b
[template-log-access]
filename = templates/template-log-access.conf.in
......@@ -86,7 +86,7 @@ md5sum = b1d6bac767db77ad1662edd06aabdf49
[template-nginx-eventsource-slave-virtualhost]
filename = templates/nginx-eventsource-slave.conf.in
md5sum = a0c5c376753da042c5f8444a33066acf
md5sum = 69d65e461cd7cd5ef5b1ccd0098b50c8
[template-nginx-notebook-slave-virtualhost]
filename = templates/nginx-notebook-slave.conf.in
......
software/caddy-frontend/instance-apache-replicate.cfg.in
View file @ 78f3c519
{% if slap_software_type in software_type -%}
{% if slap_software_type in software_type %}
[jinja2-template-base]
recipe = slapos.recipe.template:jinja2
......@@ -12,33 +12,33 @@ context =
key slave_instance_list slap-parameter:slave_instance_list
${:extra-context}
{% set part_list = [] -%}
{% set part_list = [] %}
{% set single_type_key = 'single-' %}
{% if slap_software_type == "replicate" %}
{% set frontend_type = slapparameter_dict.pop('-frontend-type', 'single-default') -%}
{% elif slap_software_type in ['default', 'RootSoftwareInstance'] -%}
{% set frontend_type = "%s%s" % (single_type_key, 'custom-personal') -%}
{% else -%}
{% set frontend_type = "%s%s" % (single_type_key, slap_software_type) -%}
{% endif -%}
{% set frontend_quantity = slapparameter_dict.pop('-frontend-quantity', '1') | int -%}
{% set slave_list_name = 'extra_slave_instance_list' -%}
{% set frontend_type = slapparameter_dict.pop('-frontend-type', 'single-default') %}
{% elif slap_software_type in ['default', 'RootSoftwareInstance'] %}
{% set frontend_type = "%s%s" % (single_type_key, 'custom-personal') %}
{% else %}
{% set frontend_type = "%s%s" % (single_type_key, slap_software_type) %}
{% endif %}
{% set frontend_quantity = slapparameter_dict.pop('-frontend-quantity', '1') | int %}
{% set slave_list_name = 'extra_slave_instance_list' %}
{% set frontend_list = [] %}
{% set frontend_section_list = [] %}
{% set request_dict = {} %}
{% set namebase = 'caddy-frontend' -%}
{% set namebase = 'caddy-frontend' %}
# XXX Dirty hack, not possible to define default value before
{% set sla_computer_caddy_1_key = '-sla-1-computer_guid' -%}
{% if not sla_computer_caddy_1_key in slapparameter_dict -%}
{% do slapparameter_dict.__setitem__(sla_computer_caddy_1_key, '${slap-connection:computer-id}') -%}
{% endif -%}
{% set sla_computer_caddy_1_key = '-sla-1-computer_guid' %}
{% if not sla_computer_caddy_1_key in slapparameter_dict %}
{% do slapparameter_dict.__setitem__(sla_computer_caddy_1_key, '${slap-connection:computer-id}') %}
{% endif %}
# Here we request individually each frontend.
# The presence of sla parameters is checked and added if found
{% for i in range(1, frontend_quantity + 1) -%}
{% set frontend_name = "%s-%s" % (namebase, i) -%}
{% set request_section_title = 'request-%s' % frontend_name -%}
{% set sla_key = "-sla-%s-" % i -%}
{% for i in range(1, frontend_quantity + 1) %}
{% set frontend_name = "%s-%s" % (namebase, i) %}
{% set request_section_title = 'request-%s' % frontend_name %}
{% set sla_key = "-sla-%s-" % i %}
{% set sla_key_length = sla_key | length %}
{% set sla_dict = {} %}
{% set config_key = "-frontend-config-%s-" % i %}
......@@ -48,13 +48,13 @@ context =
{% if key.startswith(sla_key) %}
{% do sla_dict.__setitem__(key[sla_key_length:], slapparameter_dict.pop(key)) %}
# We check for specific configuration regarding the frontend
{% elif key.startswith(config_key) -%}
{% elif key.startswith(config_key) %}
{% do config_dict.__setitem__(key[config_key_length:], slapparameter_dict.pop(key)) %}
{% endif -%}
{% endfor -%}
{% do frontend_list.append(frontend_name) -%}
{% do frontend_section_list.append(request_section_title) -%}
{% do part_list.append(request_section_title) -%}
{% endif %}
{% endfor %}
{% do frontend_list.append(frontend_name) %}
{% do frontend_section_list.append(request_section_title) %}
{% do part_list.append(request_section_title) %}
# Filling request dict for slave
{% set state_key = "-frontend-%s-state" % i %}
{% do request_dict.__setitem__(request_section_title,
......@@ -64,9 +64,9 @@ context =
'sla': sla_dict,
'state': slapparameter_dict.pop(state_key, None)
}) %}
{% endfor -%}
{% endfor %}
{% set authorized_slave_string = slapparameter_dict.pop('-frontend-authorized-slave-string', '') -%}
{% set authorized_slave_string = slapparameter_dict.pop('-frontend-authorized-slave-string', '') %}
{% set authorized_slave_list = [] %}
{% set rejected_slave_list = [] %}
{% for slave in slave_instance_list %}
......@@ -75,7 +75,7 @@ context =
{% else %}
{% do rejected_slave_list.append(slave.get('slave_reference')) %}
{% endif %}
{% endfor -%}
{% endfor %}
[monitor-instance-parameter]
monitor-httpd-port = {{ slapparameter_dict.get('monitor-httpd-port', '8196') }}
......@@ -110,11 +110,11 @@ state = {{ frontend_request.get('state') }}
{% do slave_configuration_dict.__setitem__("frontend-name", frontend_request.get('name')) %}
config-_ = {{ json_module.dumps(slave_configuration_dict) }}
{% if frontend_request.get('sla') %}
{% for parameter, value in frontend_request.get('sla').iteritems() -%}
{% for parameter, value in frontend_request.get('sla').iteritems() %}
sla-{{ parameter }} = {{ value }}
{% endfor -%}
{% endif -%}
{% endfor -%}
{% endfor %}
{% endif %}
{% endfor %}
[publish-information]
......@@ -138,9 +138,9 @@ custom-personal = ${dynamic-publish-slave-information:rendered}
custom-group = ${dynamic-publish-slave-information:rendered}
[slave-information]
{% for frontend_section in frontend_section_list -%}
{% for frontend_section in frontend_section_list %}
{{ frontend_section }} = {{ "${%s:connection-slave-instance-information-list}" % frontend_section }}
{% endfor -%}
{% endfor %}
[dynamic-publish-slave-information]
< = jinja2-template-base
......@@ -154,7 +154,7 @@ extra-context =
monitor-url-list +=
{% for frontend in frontend_section_list %}
{{ ' ${' + frontend + ':connection-monitor-base-url}' }}
{% endfor -%}
{% endfor %}
[buildout]
extends = {{ template_monitor }}
......@@ -162,9 +162,9 @@ parts =
monitor-base
publish-slave-information
publish-information
{% for part in part_list -%}
{% for part in part_list %}
{{ ' %s' % part }}
{% endfor -%}
{% endfor %}
# publish-information
eggs-directory = {{ eggs_directory }}
......@@ -185,4 +185,4 @@ cert_file = ${slap-connection:cert-file}
slave_instance_list =
-frontend-quantity = 1
-frontend-type = single-default
{%- endif %}
{% endif %}
software/caddy-frontend/templates/cached-virtualhost.conf.in
View file @ 78f3c519
{% set TRUE_VALUES = ['y', 'yes', '1', 'true'] -%}
{% set TRUE_VALUES = ['y', 'yes', '1', 'true'] %}
{% set server_alias_list = slave_parameter.get('server-alias', '').split() %}
{% set ssl_proxy_verify = ('' ~ slave_parameter.get('ssl-proxy-verify', '')).lower() in TRUE_VALUES -%}
{%- set host_list = [slave_parameter.get('custom_domain')] + server_alias_list -%}
{%- set http_backend_host_list = [] %}
{%- set https_backend_host_list = [] %}
{%- for host in host_list %}
{%- do http_backend_host_list.append('http://%s:%s' % (host, cached_port)) %}
{%- do https_backend_host_list.append('http://%s:%s' % (host, ssl_cached_port)) %}
{%- endfor %}
{% set ssl_proxy_verify = ('' ~ slave_parameter.get('ssl-proxy-verify', '')).lower() in TRUE_VALUES %}
{% set host_list = [slave_parameter.get('custom_domain')] + server_alias_list %}
{% set http_backend_host_list = [] %}
{% set https_backend_host_list = [] %}
{% for host in host_list %}
{% do http_backend_host_list.append('http://%s:%s' % (host, cached_port)) %}
{% do https_backend_host_list.append('http://%s:%s' % (host, ssl_cached_port)) %}
{% endfor %}
# Only accept generic (i.e not Zope) backends on http
{{ http_backend_host_list|join(', ') }} {
bind {{ local_ipv4 }}
{%- if ssl_proxy_verify and 'ssl_proxy_ca_crt' in slave_parameter %}
{% if ssl_proxy_verify and 'ssl_proxy_ca_crt' in slave_parameter %}
status 501 /
{%- endif %}
{% endif %}
# Rewrite part
proxy / {{ slave_parameter.get('backend_url', '') }} {
# As backend is trusting REMOTE_USER header unset it always
......@@ -22,34 +22,34 @@
transparent
timeout 600s
{%- if ssl_proxy_verify %}
{%- if 'ssl_proxy_ca_crt' in slave_parameter %}
{% if ssl_proxy_verify %}
{% if 'ssl_proxy_ca_crt' in slave_parameter %}
# TODO-Caddy SSLProxyCACertificateFile {{ slave_parameter.get('path_to_ssl_proxy_ca_crt', '') }}
# Requires https://github.com/mholt/caddy/issues/1550 or "just adding your CA to the system's trust store"
{%- endif %}
{%- else %}
{% endif %}
{% else %}
insecure_skip_verify
{%- endif %}
{% endif %}
}
}
{{ https_backend_host_list|join(', ') }} {
bind {{ local_ipv4 }}
{%- if ssl_proxy_verify and 'ssl_proxy_ca_crt' in slave_parameter %}
{% if ssl_proxy_verify and 'ssl_proxy_ca_crt' in slave_parameter %}
status 501 /
{%- endif %}
{% endif %}
proxy / {{ slave_parameter.get('https_backend_url', '') }} {
# As backend is trusting REMOTE_USER header unset it always
header_upstream -REMOTE_USER
transparent
timeout 600s
{%- if ssl_proxy_verify %}
{%- if 'ssl_proxy_ca_crt' in slave_parameter %}
{% if ssl_proxy_verify %}
{% if 'ssl_proxy_ca_crt' in slave_parameter %}
# TODO-Caddy SSLProxyCACertificateFile {{ slave_parameter.get('path_to_ssl_proxy_ca_crt', '') }}
# Requires https://github.com/mholt/caddy/issues/1550 or "just adding your CA to the system's trust store"
{%- endif %}
{%- else %}
{% endif %}
{% else %}
insecure_skip_verify
{%- endif %}
{% endif %}
}
}
software/caddy-frontend/templates/nginx-eventsource-slave.conf.in
View file @ 78f3c519
{% set url = slave_parameter.get('url') -%}
{% set https_url = slave_parameter.get('https-url', url) -%}
{% if url.startswith("http://") or url.startswith("https://") -%}
{% set upstream = url.split("/")[2] -%}
{% set https_upstream = https_url.split("/")[2] -%}
{% set protocol = url.split("/")[0] -%}
{% set https_protocol = https_url.split("/")[0] -%}
{% set proxy_pass = '%s//%s' % (protocol, slave_parameter.get('slave_reference')) -%}
{% set https_proxy_pass = '%s//https_%s' % (protocol, slave_parameter.get('slave_reference')) -%}
{%- set ssl_configuration_list = [('ssl_certificate', 'path_to_ssl_crt'),
('ssl_certificate_key', 'path_to_ssl_key')] -%}
{% set url = slave_parameter.get('url') %}
{% set https_url = slave_parameter.get('https-url', url) %}
{% if url.startswith("http://") or url.startswith("https://") %}
{% set upstream = url.split("/")[2] %}
{% set https_upstream = https_url.split("/")[2] %}
{% set protocol = url.split("/")[0] %}
{% set https_protocol = https_url.split("/")[0] %}
{% set proxy_pass = '%s//%s' % (protocol, slave_parameter.get('slave_reference')) %}
{% set https_proxy_pass = '%s//https_%s' % (protocol, slave_parameter.get('slave_reference')) %}
{% set ssl_configuration_list = [('ssl_certificate', 'path_to_ssl_crt'),
('ssl_certificate_key', 'path_to_ssl_key')] %}
# TODO-Caddy upstream {{ slave_parameter.get('slave_reference') }} {
......@@ -75,10 +75,10 @@
# TODO-Caddy ssl_prefer_server_ciphers on;
# TODO-Caddy ssl_session_cache shared:SSL:10m;
{% for key, value in ssl_configuration_list -%}
{% if value in slave_parameter -%}
{% for key, value in ssl_configuration_list %}
{% if value in slave_parameter %}
# TODO-Caddy {{ ' %s' % key }} {{ slave_parameter.get(value) }};
{% endif -%}
{% endif %}
{% endfor %}
# TODO-Caddy location /pub {
......@@ -110,4 +110,4 @@
# TODO-Caddy default_type "text/event-stream; charset=utf-8";
# TODO-Caddy }
# TODO-Caddy}
{% endif -%}
{% endif %}
software/caddy-frontend/templates/replicate-publish-slave-information.cfg.in
View file @ 78f3c519
{% set part_list = [] -%}
{% set slave_information_dict = {} -%}
{% set part_list = [] %}
{% set slave_information_dict = {} %}
# regroup slave information from all frontends
{%- for frontend, slave_list_raw in slave_information.iteritems() -%}
{% for frontend, slave_list_raw in slave_information.iteritems() %}
{% if slave_list_raw %}
{% set slave_list = json_module.loads(slave_list_raw) -%}
{% set slave_list = json_module.loads(slave_list_raw) %}
{% else %}
{% set slave_list = [] %}
{% endif %}
{% for slave_dict in slave_list -%}
{% for slave_dict in slave_list %}
{% set slave_reference = slave_dict.pop('slave-reference') %}
{% set log_access_url = slave_dict.pop('log-access', '') %}
{% set current_slave_dict = slave_information_dict.get(slave_reference, {}) %}
{% do current_slave_dict.update(slave_dict) -%}
{% do current_slave_dict.update(slave_dict) %}
{% set log_access_list = current_slave_dict.get('log-access-urls', []) %}
{% do log_access_list.append( frontend + ': ' + log_access_url) %}
{% do current_slave_dict.__setitem__(
......@@ -22,22 +22,22 @@
{% do current_slave_dict.__setitem__(
'replication_number',
current_slave_dict.get('replication_number', 0) + 1
) -%}
{% do slave_information_dict.__setitem__(slave_reference, current_slave_dict) -%}
{% endfor -%}
) %}
{% do slave_information_dict.__setitem__(slave_reference, current_slave_dict) %}
{% endfor %}
{% endfor %}
# Publish information for each slave
{% for slave_reference, slave_information in slave_information_dict.iteritems() %}
{% set publish_section_title = 'publish-%s' % slave_reference -%}
{% do part_list.append(publish_section_title) -%}
{% set publish_section_title = 'publish-%s' % slave_reference %}
{% do part_list.append(publish_section_title) %}
[{{ publish_section_title }}]
recipe = slapos.cookbook:publish
-slave-reference = {{ slave_reference }}
log-access-url = {{ json_module.dumps(slave_information.pop('log-access-urls', 1000)) }}
{% for key, value in slave_information.iteritems() -%}
{% for key, value in slave_information.iteritems() %}
{{ key }} = {{ value }}
{% endfor -%}
{% endfor %}
{% endfor %}
[buildout]
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7