In some cases domain can come from "outside" of the profile, and be filled
with "garbage", so if custom_domain is set, do not overwrite it.

…ration state calculation""

The states has been correctly propagated and used. Original author had no
idea what he was doing, possibly because of some incorrect assumptions while
developing this fix.

/reviewed-on nexedi/slapos!587

The last state value has been reverted by mistake, put back correct one.

Frontend operator shall have easy access to information about rejected
slaves, possibly the best in the JSON file.

Also the keys for the human readable information are slave's titles, not
references.

The information is published via hand crafted HTTPS endpoint.

Note: The SSL certificate is generated manually. Existing caucase is special
      for KeDiFa, this is another step to move all generated certificates (or
      otherwise self-signed) to internal, full automatic caucase.

Rename was missing in buildout.hash.cfg

/reviewed-on nexedi/slapos!575

Before normalize_ae_gzip was 0, and better keep it this way as it is
suspected root problem of incorrectly varianted cache depending on
client Accept-Encoding.

Instead of expensive and long checking of the configuration during promise run
read last stored state.

This decouples configuration validation calculation from promise check. The
validation information is updated often (on each configuration change, on each
reload, etc) and every 2 hours.

Validation happens on each configuration change, but for sure it is checked
each 2 hours.

State of configuration is calculated in separate script.

Move files from var/log/trafficserver/*old to
srv/backup/logrotate/trafficserver, xz them and clean files older than a
year.

As some tools are not configurable with logrotate provide a script to
"rotate" files generated by those tools.
This script more or less follows logrotate behaviour:

 * moves rotated files to logrotate directory
 * compresses them with xz
 * have to be configured to keep only files for some days

Roll logs each 24h, as it would be expected.

Stderr to null redirection happened in the wrong place.

Promises will fail with stderr filled, and it can be such if sha256sum emits
error messages, so just discard them.

Some arguments needs Caddy process restart, so implement it with
hash-files and also inform the master partition requester about parameters
which will result with process restart.

Kedifa partition was missing monitoring at all, so add it and monitor kedifa
and exposer ip and port.

Partition running caddy was missing monitoring for exposer, so add it.

Use unreal address to avoid any tries for network connectivity.

Kedifa requires some time to process new slave, and during that time the key
download URL is not available, but as it is required for proper mapping file,
use some url to mimic it.

During buildout run no network communication is required in order to prepare
fallback certificates, so call kedifa-updater with --prepare-only

It is needed by users to check certificate of KeDiFa while uploading
certificates.

Each time new slave appears the kedifa-updater has to be run immediately, in
order for certificates to be properly setup.

Otherwise caddy can be left in non-runnable state until next kedifa-updater
would run again.

caddy-frontend master partition does not implement any promise in etc/promise,
all is migrated to etc/plugin

caddy-frontend-is-running-actual-software-release promise is not needed
anymore, as hash-files are used.

By default whole slave makes websocket connection to the backend.
With websocket-path, only the path has websocket style connections,
the rest is standard HTTP.

There is no need anymore to have two processes for normal and nginx slaves,
as nginx ones are served by caddy anyway.

Also inform the requester that type:eventsource is not implemented.

Differences between tls and non-tls are minimal, so simplify the generation
as much as possible with simple tls switch.

It seems more readable than creating Jinja2 macros, which would be used only
twice.

Since Caddy v0.11.4 it is possible to disable log rotation, thus disable it
and rely purely on SlapOS defined log rotation.

See https://github.com/mholt/caddy/releases/tag/v0.11.4

When there are no shared instances, the file was empty, but caddy
refuses to start when using an import statement on an empty file, with
this error:

```
Error during parsing: Could not read tokens while importing .../etc/log-access.conf: EOF
```

/reviewed-on nexedi/slapos!545

This also means that caddy source is fetched directly from upstream, as all
required fixes has been incorporated into the upstream.

Since https://github.com/mholt/caddy/releases/tag/v0.11.4 TLS-SNI challenge
is replaced by ACME TLS-ALPN challenge, so switch has changed.

Drop direct usage of gowork for now, in order to have caddy built using go
module, support for gowork with go modules might come later.

/reviewed-on nexedi/slapos!544

This reverts commit 7993ff81.

Custom configuration checks are hard to be trusted, as they can impact too
many aspects of running frontend.

Frontend administrator knows the risks of custom configuration, and shall take
proper care.

/reviewed-on nexedi/slapos!543

Instead of complex architecture in the profiles, reuse kedifa-updater
capability to do backward compatibility certificate management thanks to its
fall-back mechanism.

kedifa-updater uses state file to know, if it ever succeed to download
certificate from KeDiFa, and so it really makes it that pushing at least once
certificate to KeDiFa, even if it is sometimes unresponsive, will switch to
it.

Fallback certificate is used, thus each slave listens immediately on HTTP and
HTTPS. Thanks to this, asynchronous updates do not need to communicate with
slapos node instance, and slapos node instance does not care about the
certificates anymore.

Instead of fetching certificates on each slapos node instance use new
kedifa-updater, which is a tool to asynchronously fetch certificates and
has a hook to reload the server in case if new certificate is available.

custom_ssl_directory is NOT BBB

This mostly useful during tests to have stable results, especially when
some slaves are rejected.

This change is expected to be no-op during normal run.

Note: The slave rejection system does not guarantee any ordering, as the sort
      order can change, because of parameters can reorder slaves. Thus, even
      if slave A was requested before slave B, and they conflict each other,
      slave A can be rejected instead of "expected" slave B.

This is consistent across usage in caddy-frontend and allow better reusage.