Skip to content

slapos
slapos
Commit 8eeb74fe authored by Kazuhiko Shiozaki's avatar Kazuhiko Shiozaki
Browse files
Options

software/erp5-zope2: revive Zope2 ERP5 software release.

parent 715c6b7c
Hide whitespace changes
Inline Side-by-side
Showing with 9903 additions and 0 deletions
software/erp5-zope2/README.rst 0 → 100644
View file @ 8eeb74fe
Available ``software-type`` values
==================================
- ``default``
Recommended for development and production use. Automatic creation of
erp5-site.
Notes
=====
This software release is not intended to be accessed directly, but through a
front-end instance which is expected to contains the RewriteRules_ (or
equivalent) needed to relocate Zope's urls via its VirtualHostMonster_. See the
``frontend`` erp5 instance parameter.
ERP5 defaults connect to the public cloudooo on https://cloudooo.erp5.net/.
See the ``cloudooo`` Software Release to setup a cloudooo cluster if necessary.
Replication
===========
Replication allows setting up an ERP5 instance whose data follows another
instance.
Relations between ERP5 instances in a replication graph depend in what is
supported by individual data managers (ex: a neo cluster can replicate from a
neo cluster which itself replicates from a 3rd).
Replication lag constraints (aka sync/async replication) depends on individual
data managers (ex: neo replication between clusters is always asynchronous).
Ignoring replication lag, replicated data can be strictly identical (ex:
replicating ZODB or SQL database will contain the same data as upstream), or
may imply some remaping (ex: replicating Zope logs from an instance with 2 zope
families with 2 partition of 2 zopes each to an instance with a single zope
total).
Data whose replication is supported
-----------------------------------
- neo database
Data whose replication will eventually be supported
---------------------------------------------------
- mariadb database
- zope ``zope-*-access.log`` and ``zope-*-Z2.log``
- ``mariadb-slow.log``
Data whose replication is not planned
-------------------------------------
- zeo: use neo instead
Setting up replication
----------------------
In addition to your usual parameter set, you needs to provide the following parameters::
{
"zope-partition-dict": {}, So no zope is instantiated
"zodb": [
{
"storage-dict": {
"upstream-masters": ..., As published by to-become upstream ERP5 instance as "neo-masters"
},
"type": "neo", The only ZODB type supporting replication
...
}
...
]
...
}
Port ranges
===========
This software release assigns the following port ranges by default:
==================== ==========
Partition type Port range
==================== ==========
memcached-persistent 2000-2009
memcached-volatile 2010-2019
smtp 2025-2029
neo (admin, master) 2050-2052
mariadb 2099
zeo 2100-2149
balancer 2150-2199
zope 2200-*
jupyter 8888
caucase 8890,8891
==================== ==========
Non-zope partitions are unique in an ERP5 cluster, so you shouldn't have to
care about them as a user (but a Software Release developer needs to know
them).
Zope partitions should be assigned port ranges starting at 2200, incrementing
by some value which depends on how many zope process you want per partition
(see the ``port-base`` parameter in ``zope-partition-dict``).
Notes to the Software Release developer: These ranges are not strictly
defined. Not each port is actually used so one may reduce alread-assigned
ranges if needed (ex: memcached partitions use actually fewer ports). There
should be enough room for evolution (as between smtp and mariadb types). It is
important to not allocate any port after 2200 as user may have assigned ports
to his zope processes.
.. _RewriteRules: http://httpd.apache.org/docs/current/en/mod/mod_rewrite.html#rewriterule
.. _VirtualHostMonster: http://docs.zope.org/zope2/zope2book/VirtualHosting.html
\ No newline at end of file
software/erp5-zope2/instance-erp5-input-schema.json 0 → 100644
View file @ 8eeb74fe
{
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Parameters to instantiate ERP5",
"type": "object",
"additionalProperties": false,
"definitions": {
"routing-rule-list": {
"description": "Maps the path received in requests to given zope path. Rules are applied in the order they are given. This requires the path received from the outside world (typically: frontend) to have its root correspond to Zope's root (for frontend: 'path' parameter must be empty), with the customary VirtualHostMonster construct (for frontend: 'type' must be 'zope').",
"type": "array",
"default": [
[
"/",
"/"
]
],
"items": {
"type": "array",
"minItems": 2,
"maxItems": 2,
"items": [
{
"title": "External path",
"description": "Path as received from the outside world, based on VirtualHostRoot element.",
"type": "string"
},
{
"title": "Internal path",
"description": "Zope path, based on Zope root object, the external path should correspond to. '%(site-id)s' is replaced by the site-id value, and '%%' replaced by '%'.",
"type": "string"
}
]
}
},
"tcpv4port": {
"$ref": "./schemas-definitions.json#/tcpv4port"
}
},
"properties": {
"sla-dict": {
"description": "Where to request instances. Each key is a query string for criterions (e.g. \"computer_guid=foo\"), and each value is a list of partition references (note: Zope partitions reference must be prefixed with \"zope-\").",
"additionalProperties": {
"type": "array",
"items": {
"type": "string"
},
"uniqueItems": true
},
"type": "object"
},
"site-id": {
"description": "ERP5Site object's id. An empty value disables automatic site creation.",
"default": "erp5",
"type": "string"
},
"bt5": {
"description": "Business Template to install at automatic site creation. By default, all configurators are installed.",
"type": "string"
},
"id-store-interval": {
"description": "Set Store Interval of default SQL Non Continuous Increasing Id Generator at automatic site creation. If unset, the value from the erp5_core Business Template is not touched.",
"type": "integer"
},
"timezone": {
"description": "Zope's timezone. Possible values are determined by host's libc, and typically come from a separate package (tzdata, ...)",
"default": "UTC",
"type": "string"
},
"deadlock-debugger-password": {
"description": "Password for /manage_debug_threads",
"type": "string"
},
"inituser-login": {
"description": "Login of the initial/rescue user",
"default": "zope",
"type": "string"
},
"inituser-password": {
"description": "Password of the initial/rescue user",
"type": "string"
},
"developer-list": {
"description": "List of logins which should get the Developer role (required to modify portal_components' content), defaulting to inituser-login's value",
"items": {
"pattern": "^\\S+$",
"type": "string"
},
"uniqueItems": true,
"type": "array"
},
"activity-timeout": {
"description": "How long a CMFActivity-initiated transaction may last, in seconds",
"default": null,
"type": [
"number",
"null"
]
},
"publisher-timeout": {
"description": "How long a publisher-initiated transaction may last, in seconds",
"default": 300,
"type": [
"number",
"null"
]
},
"family-override": {
"description": "Family-wide options, possibly overriding global options",
"default": {},
"patternProperties": {
".*": {
"default": {},
"properties": {
"webdav": {
"description": "Serve webdav queries, implies timerserver-interval=0 (disabled)",
"default": false,
"type": "boolean"
},
"activity-timeout": {
"description": "Override global activity timeout",
"type": [
"number",
"null"
]
},
"publisher-timeout": {
"description": "Override global publisher timeout",
"type": [
"number",
"null"
]
}
},
"type": "object"
}
},
"type": "object"
},
"hostalias-dict": {
"description": "Hostname-to-domain-name mapping",
"default": {},
"additionalProperties": {
"description": "A hostname to which current entry will resolve",
"type": "string"
},
"type": "object"
},
"hosts-dict": {
"description": "Host entries to be used in addition to and/or overriding auto-generated ones (erp5-catalog-0, erp5-memcached-persistent, erp5-memcached-volatile and erp5-smtp)",
"patternProperties": {
".*": {
"description": "An IP or domain name to which current entry will resolve",
"type": "string"
}
},
"type": "object"
},
"frontend": {
"description": "Front-end slave instance request parameters",
"properties": {
"software-url": {
"description": "Front-end's software type. If this parameter is empty, no front-end instance is requested. Else, sla-dict must specify 'frontend' which is a special value matching all frontends (e.g. {\"instance_guid=bar\": [\"frontend\"]}).",
"default": "",
"type": "string",
"format": "uri"
},
"domain": {
"description": "The domain name to request front-end to respond as.",
"default": "",
"type": "string"
},
"software-type": {
"description": "Request a front-end slave instance of this software type.",
"default": "RootSoftwareInstance",
"type": "string"
},
"virtualhostroot-http-port": {
"description": "Front-end slave http port. Port where http requests to frontend will be redirected.",
"default": 80,
"type": "integer"
},
"virtualhostroot-https-port": {
"description": "Front-end slave https port. Port where https requests to frontend will be redirected.",
"default": 443,
"type": "integer"
}
},
"type": "object"
},
"wsgi": {
"description": "If set to true, Zope is run as a WSGI application, instead of using the Medusa HTTP server.",
"type": "boolean",
"default": true
},
"zope-partition-dict": {
"description": "Zope layout definition",
"default": {
"1": {}
},
"patternProperties": {
".*": {
"additionalProperties": false,
"properties": {
"family": {
"description": "The family this partition is part of. For example: 'public', 'admin', 'backoffice', 'web-service'... Each family gets its own balancer entry. It has no special meaning for the system.",
"default": "default",
"type": "string"
},
"instance-count": {
"description": "Number of Zopes to setup on this partition",
"default": 1,
"type": "integer"
},
"thread-amount": {
"description": "Number of worker threads for each created Zope process",
"default": 4,
"type": "integer"
},
"timerserver-interval": {
"description": "Timerserver tick period, in seconds, or 0 to disable",
"default": 1,
"type": "number"
},
"private-dev-shm": {
"description": "Size of private /dev/shm for wendelin.core. If sysctl kernel.unprivileged_userns_clone exists, it must be set to 1.",
"type": "string"
},
"ssl-authentication": {
"title": "Enable SSL Client authentication on this zope instance.",
"description": "If set to true, will set SSL Client verification to required on apache VirtualHost which allow to access this zope instance.",
"type": "boolean",
"default": false
},
"longrequest-logger-interval": {
"description": "Period, in seconds, with which LongRequestLogger polls worker thread stack traces, or -1 to disable",
"default": -1,
"type": "integer"
},
"longrequest-logger-timeout": {
"description": "Transaction duration after which LongRequestLogger will start logging its stack trace, in seconds",
"default": 1,
"type": "integer"
},
"large-file-threshold": {
"description": "Requests bigger than this size get saved into a temporary file instead of being read completely into memory, in bytes",
"default": "10MB",
"type": "string"
},
"port-base": {
"allOf": [
{
"$ref": "#/definitions/tcpv4port"
},
{
"description": "Start allocating ports at this value. Useful if one needs to make several partitions share the same port range (ie, several partitions bound to a single address)",
"default": 2200
}
]
}
},
"type": "object"
}
},
"type": "object"
},
"kumofs": {
"description": "Persistent memcached service",
"allOf": [
{
"$ref": "./instance-kumofs-schema.json"
},
{
"properties": {
"tcpv4-port": {
"default": 2000
}
}
}
],
"type": "object"
},
"memcached": {
"description": "Volatile memcached service",
"allOf": [
{
"$ref": "./instance-kumofs-schema.json"
},
{
"properties": {
"tcpv4-port": {
"default": 2010
}
}
}
],
"type": "object"
},
"cloudooo-url-list": {
"description": "Format conversion service URLs",
"type": "array",
"items": {
"pattern": "^https?://",
"type": "string",
"format": "uri"
}
},
"cloudooo-retry-count": {
"description": "Define retry count for cloudooo in network error case in test",
"type": "integer",
"default": 2
},
"smtp": {
"description": "Mail queuing and relay service",
"allOf": [
{
"$ref": "./instance-smtp-schema.json"
},
{
"properties": {
"tcpv4-port": {
"default": 2010
}
}
}
],
"type": "object"
},
"mariadb": {
"description": "Relational database service",
"allOf": [
{
"$ref": "./instance-mariadb-schema.json"
},
{
"properties": {
"tcpv4-port": {
"default": 2099
}
}
}
],
"type": "object"
},
"zodb-zeo": {
"description": "Common settings ZEO servers",
"additionalProperties": false,
"properties": {
"tcpv4-port": {
"allOf": [
{
"$ref": "#/definitions/tcpv4port"
},
{
"description": "Start allocating ports at this value, going upward"
}
]
},
"backup-periodicity": {
"description": "When to backup, specified in the same format as for systemd.time(7) calendar events (years & seconds not supported, DoW & DoM can not be combined). Enter 'never' to disable backups.",
"default": "daily",
"type": "string"
},
"tidstorage-repozo-path": {
"description": "Directory for backup timestamp and tidstorage status files.",
"default": "~/srv/backup/tidstorage",
"type": "string"
}
},
"type": "object"
},
"zodb": {
"description": "Zope Object DataBase mountpoints. See https://github.com/zopefoundation/ZODB/blob/4/src/ZODB/component.xml for extra options.",
"items": {
"required": [
"type"
],
"properties": {
"name": {
"description": "Database name",
"default": "main",
"type": "string"
},
"mount-point": {
"description": "Mount point",
"default": "/",
"type": "string"
},
"storage-dict": {
"description": "Storage configuration. For NEO, 'logfile' is automatically set (see https://lab.nexedi.com/nexedi/neoppod/blob/master/neo/client/component.xml for other settings).",
"properties": {
"ssl": {
"description": "For external NEO. Pass false if you want to disable SSL or pass custom values for ca/cert/key.",
"default": true,
"type": "boolean"
}
},
"patternProperties": {
".!$": {
"$ref": "#/properties/zodb/items/patternProperties/.!$"
}
},
"additionalProperties": {
"$ref": "#/properties/zodb/items/additionalProperties"
},
"type": "object"
},
"type": true,
"server": true
},
"oneOf": [
{
"title": "zeo",
"properties": {
"type": {
"description": "Storage type",
"const": "zeo"
},
"server": {
"description": "Instantiate a server. If missing, 'storage-dict' must contain the necessary properties to mount the ZODB. The partition reference is 'zodb'.",
"$ref": "./instance-zeo-schema.json"
}
}
},
{
"title": "neo",
"properties": {
"type": {
"description": "Storage type",
"const": "neo"
},
"server": {
"description": "Instantiate a server. If missing, 'storage-dict' must contain the necessary properties to mount the ZODB. Partitions references are 'neo-0', 'neo-1', ...",
"$ref": "../neoppod/instance-neo-input-schema.json#/definitions/neo-cluster"
}
}
}
],
"patternProperties": {
".!$": {
"description": "Override with the value of the first item whose zope id matches against the pattern.",
"items": {
"items": [
{
"description": "Override pattern (Python regular expression).",
"type": "string"
},
{
"description": "Override value (parameter for matching nodes).",
"type": [
"integer",
"string"
]
}
],
"type": "array"
},
"type": "array"
}
},
"additionalProperties": {
"type": [
"integer",
"string"
]
},
"type": "object"
},
"type": "array"
},
"jupyter": {
"description": "Jupyter subinstance parameters",
"additionalProperties": false,
"properties": {
"enable": {
"description": "Whether to enable creation of associated Jupyter subinstance",
"default": false,
"type": "boolean"
},
"zope-family": {
"description": "Zope family to connect Jupyter to by default",
"default": "<first instantiated Zope family>",
"type": "string"
}
},
"type": "object"
},
"wcfs": {
"description": "Parameters for wendelin.core filesystem",
"additionalProperties": false,
"properties": {
"enable": {
"description": "Whether to enable WCFS filesystem and use it to access ZBigArray/ZBigFile data. In WCFS mode wendelin.core clients (Zope/ERP5 processes) share in-RAM cache for in-ZODB data without duplicating it for every client. This cache sharing does not affect correctness as isolation property is continued to be provided to every client.",
"default": false,
"type": "boolean"
}
}
},
"wendelin-core-zblk-fmt": {
"description": "In wendelin.core there are 2 formats for storing data, so called ZBlk0 and ZBlk1. See https://lab.nexedi.com/nexedi/wendelin.core/blob/2e5e1d3d/bigfile/file_zodb.py#L19 for more details.",
"default": "",
"type": "string"
},
"caucase": {
"description": "Caucase certificate authority parameters",
"allOf": [
{
"properties": {
"url": {
"title": "Caucase URL",
"description": "URL of existing caucase instance to use. If empty, a new caucase instance will be deployed. If not empty, other properties in this section will be ignored.",
"default": "",
"type": "string",
"format": "uri"
}
}
},
{
"$ref": "../caucase/instance-caucase-input-schema.json"
}
],
"type": "object"
},
"test-runner": {
"description": "Test runner parameters.",
"additionalProperties": false,
"properties": {
"enabled": {
"description": "Generate helper scripts to run test suite.",
"default": true,
"type": "boolean"
},
"coverage": {
"type": "object",
"title": "Coverage",
"description": "Coverage configuration",
"additionalProperties": false,
"properties": {
"enabled": {
"description": "Collect python coverage data during test run.",
"default": false,
"type": "boolean"
},
"include": {
"description": "File name patterns to include in coverage data, relative to software buildout's directory. Default to all repositories defined in software by ${erp5_repository_list:repository_id_list}.",
"type": "array",
"items": {
"type": "string"
},
"examples": [
[
"parts/erp5/*",
"parts/custom-repository/*",
"develop-eggs/custom-egg/*"
]
]
},
"branch": {
"description": "Enable branch coverage",
"type": "boolean",
"default": false
},
"upload-url": {
"description": "URL to upload coverage data. This is interpreted as a RFC 6570 URI Template, with the following parameters: test_name, test_result_id and test_result_revision. The request will be a PUT request with the coverage file content as body, suitable for WebDav servers. If the URL contains user and password, they will be used to attempt authentication using Digest and Basic authentication schemes.",
"type": "string",
"format": "uri",
"examples": [
"https://user:password@example.com/{test_result_id}/{test_name}.coverage.sqlite3"
]
}
}
},
"node-count": {
"description": "Number of tests this instance can execute in parallel. This must be at least equal to the number of nodes configured on testnode running the test",
"default": 3,
"type": "integer"
},
"extra-database-count": {
"description": "Number of extra databases this instance tests will need.",
"default": 3,
"type": "integer"
},
"selenium": {
"default": {
"target": "firefox"
},
"examples": [
{
"target": "selenium-server",
"server-url": "https://selenium.example.com",
"desired-capabilities": {
"browserName": "firefox",
"version": "68.0.2esr",
"acceptInsecureCerts": true
}
},
{
"target": "selenium-server",
"server-url": "https://selenium.example.com",
"desired-capabilities": {
"browserName": "chrome",
"version": "91.0.4472.101"
}
}
],
"oneOf": [
{
"type": "object",
"title": "Selenium Server",
"description": "Configuration for Selenium server",
"additionalProperties": false,
"required": [
"desired-capabilities",
"server-url",
"target"
],
"properties": {
"target": {
"description": "Target system",
"type": "string",
"const": "selenium-server",
"default": "selenium-server"
},
"server-url": {
"description": "URL of the selenium server",
"type": "string",
"format": "uri"
},
"verify-server-certificate": {
"description": "Verify the SSL/TLS certificate of the selenium server when using HTTPS",
"type": "boolean",
"default": true
},
"server-ca-certificate": {
"description": "PEM encoded bundle of CA certificates to verify the SSL/TLS certificate of the selenium server when using HTTPS",
"type": "string",
"default": "Root certificates from http://certifi.io/en/latest/"
},
"desired-capabilities": {
"description": "Desired browser capabilities",
"required": [
"browserName"
],
"type": "object",
"properties": {
"browserName": {
"description": "Name of the browser being used",
"type": "string",
"examples": [
"firefox",
"chrome",
"safari"
]
},
"version": {
"description": "The browser version",
"type": "string"
}
}
}
}
},
{
"type": "object",
"title": "Firefox",
"description": "Configuration for using firefox running as a sub-process",
"additionalProperties": false,
"properties": {
"target": {
"description": "Target system",
"const": "firefox",
"type": "string",
"default": "firefox"
}
}
}
]
},
"random-activity-priority": {
"type": "string",
"title": "Random Activity Priority",
"description": "Control `random_activity_priority` argument of test runner. Can be set to an empty string to automatically generate a seed for each test."
}
},
"type": "object"
},
"balancer": {
"description": "HTTP(S) load balancer proxy parameters",
"properties": {
"path-routing-list": {
"$ref": "#/definitions/routing-rule-list",
"title": "Global path routing rules"
},
"family-path-routing-dict": {
"type": "object",
"title": "Family-specific path routing rules",
"description": "Applied, only for the eponymous family, before global path routing rules.",
"patternProperties": {
".+": {
"$ref": "#/definitions/routing-rule-list"
}
}
},
"ssl": {
"description": "HTTPS certificate generation parameters",
"additionalProperties": false,
"properties": {
"frontend-caucase-url-list": {
"title": "Frontend Caucase URL List",
"description": "List of URLs of caucase service of frontend groups to authenticate access from them.",
"type": "array",
"items": {
"type": "string",
"format": "uri"
},
"uniqueItems": true
},
"caucase-url": {
"title": "Caucase URL",
"description": "URL of caucase service to use. If not set, global setting will be used.",
"type": "string",
"format": "uri"
},
"csr": {
"title": "csr",
"description": "PEM-encoded certificate signature request to request server certificate with. If not provided, HTTPS will be disabled.",
"type": "string"
},
"max-crl-update-delay": {
"title": "Periodicity of CRL update (days)",
"description": "CRL will be updated from caucase at least this often.",
"type": "number",
"default": 1.0
}
},
"type": "object"
}
},
"type": "object"
}
}
}
software/erp5-zope2/instance-erp5-output-schema.json 0 → 100644
View file @ 8eeb74fe
{
"$schema": "http://json-schema.org/draft-04/schema#",
"description": "Values returned by ERP5 instantiation",
"additionalProperties": false,
"properties": {
"hosts-dict": {
"description": "Hosts mapping, including auto-generated entries",
"patternProperties": {
".*": {
"description": "IP current entry resolves to",
"type": "string"
}
},
"type": "object"
},
"site-id": {
"description": "Chosen ERP5Site object identifier",
"type": "string"
},
"inituser-login": {
"description": "Initial user login",
"type": "string"
},
"inituser-password": {
"description": "Initial user password",
"type": "string"
},
"deadlock-debugger-password": {
"description": "Deadlock debugger password",
"type": "string"
},
"memcached-persistent-url": {
"description": "Persistent memcached access information",
"pattern": "^memcached://",
"type": "string"
},
"memcached-volatile-url": {
"description": "Volatile memcached access information",
"pattern": "^memcached://",
"type": "string"
},
"mariadb-database-list": {
"description": "Relational database access information",
"items": {
"pattern": "^mysql://",
"type": "string"
},
"uniqueItems": true,
"type": "array"
},
"mariadb-test-database-list": {
"description": "Relational database access information",
"items": {
"pattern": "^mysql://",
"type": "string"
},
"uniqueItems": true,
"type": "array"
},
"neo-masters": {
"$ref": "../neoppod/instance-neo-output-schema.json#/properties/masters"
},
"neo-admins": {
"$ref": "../neoppod/instance-neo-output-schema.json#/properties/admins"
},
"jupyter-url": {
"description": "Jupyter notebook web UI access information",
"pattern": "^https://",
"type": "string"
},
"caucase-http-url": {
"description": "Caucase url on HTTP. For HTTPS URL, uses https scheme, if port is explicitely specified in http URL, take that port and add 1 and use it as https port. If it is not specified.",
"pattern": "^http://",
"type": "string"
}
},
"patternProperties": {
"family-.*": {
"description": "Zope family access information",
"pattern": "^https://",
"type": "string"
}
},
"type": "object"
}
software/erp5-zope2/instance-kumofs-schema.json 0 → 100644
View file @ 8eeb74fe
{
"$schema": "http://json-schema.org/draft-04/schema#",
"type": "object",
"additionalProperties": false,
"properties": {
"tcpv4-port": {
"allOf": [
{
"$ref": "./schemas-definitions.json#/tcpv4port"
},
{
"description": "Start allocating ports at this value, going upward"
}
]
},
"ram-storage-size": {
"description": "If 0 use disk storage, otherwise use ram and limit data size to this many megabytes",
"default": 0,
"type": "integer"
}
}
}
software/erp5-zope2/instance-mariadb-schema.json 0 → 100644
View file @ 8eeb74fe
{
"$schema": "http://json-schema.org/draft-04/schema#",
"type": "object",
"additionalProperties": false,
"properties": {
"tcpv4-port": {
"allOf": [
{
"$ref": "./schemas-definitions.json#/tcpv4port"
},
{
"description": "Start allocating ports at this value, going downward"
}
]
},
"database-list": {
"description": "Databases to create and respective user credentials getting all privileges on it",
"default": [
{
"name": "erp5",
"user": "user",
"password": "insecure"
}
],
"minItems": 1,
"items": {
"required": [
"name",
"user",
"password"
],
"properties": {
"name": {
"description": "Database name",
"type": "string"
},
"user": {
"description": "User name",
"type": "string"
},
"password": {
"description": "User password",
"type": "string"
}
},
"type": "object"
},
"type": "array"
},
"catalog-backup": {
"description": "Backup control knobs",
"properties": {
"full-retention-days": {
"description": "How many days full backups must be retained, -1 meaning full backups are disabled and 0 meaning no expiration",
"default": 7,
"minimum": -1,
"type": "integer"
},
"incremental-retention-days": {
"description": "How many days incremental backups (binlogs) must be retained, -1 meaning incremental backups are disabled and 0 meaning no expiration, defaulting to full-retention-days' value",
"minimum": -1,
"type": "integer"
}
},
"type": "object"
},
"backup-periodicity": {
"description": "When to backup, specified in the same format as for systemd.time(7) calendar events (years & seconds not supported, DoW & DoM can not be combined).",
"default": "daily",
"type": "string"
},
"innodb-buffer-pool-size": {
"description": "See MariaDB documentation on innodb_buffer_pool_size",
"minimum": 0,
"type": "integer"
},
"innodb-buffer-pool-instances": {
"description": "See MariaDB documentation on innodb_buffer_pool_instances",
"minimum": 1,
"type": "integer"
},
"innodb-log-file-size": {
"description": "See MariaDB documentation on innodb_log_file_size",
"minimum": 0,
"type": "integer"
},
"innodb-log-buffer-size": {
"description": "See MariaDB documentation on innodb_log_buffer_size",
"minimum": 0,
"type": "integer"
},
"innodb-file-per-table": {
"description": "See MariaDB documentation on innodb_file_per_table",
"minimum": 0,
"maximum": 1,
"default": 0,
"type": "integer"
},
"long-query-time": {
"description": "Number of seconds above which long queries are logged",
"minimum": 0,
"default": 1,
"type": "number"
},
"max-connection-count": {
"description": "See MariaDB documentation on max_connections. If not provided, a value suitable for the number of request Zope processes is chosen.",
"minimum": 0,
"type": "integer"
},
"relaxed-writes": {
"description": "When enabled, sets innodb_flush_log_at_trx_commit = 0, innodb_flush_method = nosync, innodb_doublewrite = 0 and sync_frm = 0 - RTFM, those options are dangerous",
"default": false,
"type": "boolean"
},
"character-set-server": {
"description": "The server default character set",
"default": "utf8mb4",
"type": "string"
},
"collation-server": {
"description": "The server default collation",
"default": "utf8mb4_general_ci",
"type": "string"
},
"ssl": {
"description": "Enable and define SSL support for network connections",
"default": {},
"properties": {
"ca-crt": {
"description": "Certificate Authority's certificate, in PEM format",
"type": "string"
},
"crt": {
"description": "Server's certificate, in PEM format (mandatory to enable SSL support)",
"type": "string"
},
"key": {
"description": "Server's key, in PEM format (mandatory to enable SSL support)",
"type": "string"
},
"crl": {
"description": "Server's certificate revocation list, in PEM format",
"type": "string"
},
"cipher": {
"description": "Permissible cipher specifications, separated by colons",
"type": "string"
}
},
"type": "object"
},
"odbc-ini": {
"description": "Contents of odbc.ini file, see unixodbc document",
"default": "",
"type": "string"
},
"environment-variables": {
"description": "Extra environment variables for mysqld may be required to use third party ODBC libraries for CONNECT storage engine.",
"items": {
"type": "string"
},
"type": "array"
}
}
}
software/erp5-zope2/instance-smtp-schema.json 0 → 100644
View file @ 8eeb74fe
{
"$schema": "http://json-schema.org/draft-04/schema#",
"type": "object",
"additionalProperties": false,
"properties": {
"tcpv4-port": {
"allOf": [
{
"$ref": "./schemas-definitions.json#/tcpv4port"
},
{
"description": "Start allocating ports at this value, going upward"
}
]
},
"postmaster": {
"description": "Mail address to send technical mails to. Non-empty value required for smptd relay service to be deployed. Values will be put in alias-dict as 'postmaster' key (alias-dict takes precedence)",
"default": "",
"type": "string"
},
"alias-dict": {
"description": "Mail alias support",
"default": {},
"patternProperties": {
".*": {
"description": "List of addresses alias expands to",
"type": "array"
}
},
"type": "object"
},
"relay": {
"description": "Forward outgoing mails to a specific relay. If enabled, relay must support TLS-encrypted SASL authentication.",
"dependencies": {
"host": [
"sasl-credential"
]
},
"properties": {
"host": {
"description": "Host name or address of relay, with optional port (ex: '[example.com]:submission'). Enclosing hostname with [] prevents MX lookup.",
"type": "string"
},
"sasl-credential": {
"description": "SASL credential, in the login:password form",
"type": "string"
}
},
"default": {},
"type": "object"
},
"divert": {
"description": "Intercept all mails and send them to given addresses instead of original recipient",
"type": "array",
"items": {
"type": "string"
},
"uniqueItems": true
}
}
}
software/erp5-zope2/instance-zeo-schema.json 0 → 100644
View file @ 8eeb74fe
{
"$schema": "http://json-schema.org/draft-04/schema#",
"additionalProperties": false,
"properties": {
"backup": {
"description": "'%(backup)s' is expanded to partition's ZODB backup path (typically 'srv/backup/zodb'), and %(name)s with the export id",
"default": "%(backup)s/%(name)s",
"type": "string"
},
"family": {
"description": "Opaque name used to regroup/separate mountpoints under different ZEO processes (must be valid as a file name and as a ConfigParser section name)",
"default": "default",
"pattern": "^[^<>:\"/\\|?*\\]\\[ ]*$",
"type": "string"
},
"path": {
"description": "FileStorage file path, '%(zodb)s' occurrences are replaced with the path to partition's srv/zodb directory, and %(name)s with the export id",
"default": "%(zodb)s/%(name)s.fs",
"type": "string"
}
},
"type": "object"
}
software/erp5-zope2/schemas-definitions.json 0 → 100644
View file @ 8eeb74fe
{
"$schema": "http://json-schema.org/draft-07/schema#",
"tcpv4port": {
"minimum": 0,
"maximum": 65535,
"type": "integer"
}
}
software/erp5-zope2/software-shared.cfg 0 → 100644
View file @ 8eeb74fe
[buildout]
extends = software.cfg
shared-parts = /opt/slapgrid/shared-parts
eggs-directory = /opt/slapgrid/shared-eggs
abi-tag-eggs = true
software/erp5-zope2/software.cfg 0 → 100644
View file @ 8eeb74fe
[buildout]
extends =
../../stack/erp5-zope2/buildout.cfg
software/erp5-zope2/software.cfg.json 0 → 100644
View file @ 8eeb74fe
{
"name": "ERP5",
"description": "ERP5, Open-Source ERP",
"serialisation": "json-in-xml",
"software-type": {
"default": {
"title": "Default",
"software-type": "default",
"request": "instance-erp5-input-schema.json",
"response": "instance-erp5-output-schema.json",
"index": 0
}
}
}
software/erp5-zope2/test/README.md 0 → 100644
View file @ 8eeb74fe
Tests for ERP5 software release
software/erp5-zope2/test/setup.py 0 → 100644
View file @ 8eeb74fe
##############################################################################
#
# Copyright (c) 2018 Nexedi SA and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
from setuptools import setup, find_packages
version = '0.0.1.dev0'
name = 'slapos.test.erp5'
with open("README.md") as f:
long_description = f.read()
setup(name=name,
version=version,
description="Test for SlapOS' ERP5 software release",
long_description=long_description,
long_description_content_type='text/markdown',
maintainer="Nexedi",
maintainer_email="info@nexedi.com",
url="https://lab.nexedi.com/nexedi/slapos",
packages=find_packages(),
install_requires=[
'slapos.core',
'supervisor',
'slapos.libnetworkcache',
'erp5.util',
'psutil',
'requests',
'mysqlclient',
'cryptography',
'pexpect',
'pyOpenSSL',
],
test_suite='test',
)
software/erp5-zope2/test/test/__init__.py 0 → 100644
View file @ 8eeb74fe
##############################################################################
#
# Copyright (c) 2022 Nexedi SA and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import itertools
import json
import os
import sys
from slapos.testing.testcase import makeModuleSetUpAndTestCaseClass
_setUpModule, SlapOSInstanceTestCase = makeModuleSetUpAndTestCaseClass(
os.path.abspath(
os.path.join(os.path.dirname(__file__), '..', '..', 'software.cfg')))
setup_module_executed = False
def setUpModule():
# slapos.testing.testcase's only need to be executed once
global setup_module_executed
if not setup_module_executed:
_setUpModule()
setup_module_executed = True
# Metaclass to parameterize our tests.
# This is a rough adaption of the parameterized package:
# https://github.com/wolever/parameterized
# Consult following note for rationale why we don't use parameterized:
# https://lab.nexedi.com/nexedi/slapos/merge_requests/1306
class ERP5InstanceTestMeta(type):
"""Adjust ERP5InstanceTestCase instances to be run in several flavours (e.g. NEO/ZEO)
Adjustments can be declared via setting the '__test_matrix__' attribute
of a test case.
A test matrix is a dict which maps the flavoured class name suffix to
a tuple of parameters.
A parameter is a function which receives the instance_parameter_dict
and modifies it in place (therefore no return value is needed).
You can use the 'matrix' helper function to construct a test matrix.
If .__test_matrix__ is 'None' the test case is ignored.
If the test case should be run without any adaptions, you can set
.__test_matrix__ to 'matrix((default,))'.
"""
def __new__(cls, name, bases, attrs):
base_class = super().__new__(cls, name, bases, attrs)
if base_class._isParameterized():
cls._parameterize(base_class)
return base_class
# _isParameterized tells whether class is parameterized.
# All classes with 'metaclass=ERP5InstanceTestMeta' are parameterized
# except from a class which has been automatically instantiated from
# such user class. This exception prevents infinite recursion due to
# a parameterized class which tries to parameterize itself again.
def _isParameterized(self):
return not getattr(self, '.created_by_parametrize', False)
# Create multiple test classes from single definition.
@classmethod
def _parameterize(cls, base_class):
mod_dict = sys.modules[base_class.__module__].__dict__
for class_name_suffix, parameter_tuple in (base_class.__test_matrix__ or {}).items():
parameterized_cls_dict = dict(
base_class.__dict__,
**{
# Avoid infinite loop by a parameterized class which
# parameterize itself again and again and..
".created_by_parametrize": True,
# Switch
#
# .getInstanceParameterDict to ._test_getInstanceParameterDict
# ._base_getInstanceParameterDict to .getInstanceParameterDict
#
# so that we could inject base implementation to be called above
# user-defined getInstanceParameterDict.
"_test_getInstanceParameterDict": base_class.getInstanceParameterDict,
"getInstanceParameterDict": cls._getParameterizedInstanceParameterDict(parameter_tuple)
}
)
name = f"{base_class.__name__}_{class_name_suffix}"
mod_dict[name] = type(name, (base_class,), parameterized_cls_dict)
# _getParameterizedInstanceParameterDict returns a modified version of
# a test cases original 'getInstanceParameterDict'. The modified version
# applies parameters on the default instance parameters.
@staticmethod
def _getParameterizedInstanceParameterDict(parameter_tuple):
@classmethod
def getInstanceParameterDict(cls):
instance_parameter_dict = json.loads(
cls._test_getInstanceParameterDict().get("_", r"{}")
)
[p(instance_parameter_dict) for p in parameter_tuple]
return {"_": json.dumps(instance_parameter_dict)}
return getInstanceParameterDict
# Hide tests in unpatched base class: It doesn't make sense to run tests
# in original class, because parameters have not been assigned yet.
#
# We can't simply call 'delattr', because this wouldn't remove
# inherited tests. Overriding dir is sufficient, because this is
# the way how unittest discovers tests:
# https://github.com/python/cpython/blob/3.11/Lib/unittest/loader.py#L237
def __dir__(self):
if self._isParameterized():
return [attr for attr in super().__dir__() if not attr.startswith('test')]
return super().__dir__()
def matrix(*parameter_tuple):
"""matrix creates a mapping of test_name -> parameter_tuple.
Each provided parameter_tuple won't be combined within itself,
but with any other provided parameter_tuple, for instance
>>> parameter_tuple0 = (param0, param1)
>>> parameter_tuple1 = (param2, param3)
>>> matrix(parameter_tuple0, parameter_tuple1)
will return all options of (param0 | param1) & (param2 | param3):
- param0_param2
- param0_param3
- param1_param2
- param1_param3
"""
return {
"_".join([p.__name__ for p in params]): params
for params in itertools.product(*parameter_tuple)
}
# Define parameters (function which receives instance params + modifies them).
#
# default runs tests without any adaption
def default(instance_parameter_dict): ...
def zeo(instance_parameter_dict):
instance_parameter_dict['zodb'] = [{"type": "zeo", "server": {}}]
def neo(instance_parameter_dict):
# We don't provide encryption certificates in test runs for the sake
# of simplicity. By default SSL is turned on, we need to explicitly
# deactivate it:
# https://lab.nexedi.com/nexedi/slapos/blob/a8150a1ac/software/neoppod/instance-neo-input-schema.json#L61-65
instance_parameter_dict['zodb'] = [{"type": "neo", "server": {"ssl": False}}]
class ERP5InstanceTestCase(SlapOSInstanceTestCase, metaclass=ERP5InstanceTestMeta):
"""ERP5 base test case
"""
__test_matrix__ = matrix((zeo, neo)) # switch between NEO and ZEO mode
@classmethod
def getRootPartitionConnectionParameterDict(cls):
"""Return the output parameters from the root partition"""
return json.loads(
cls.computer_partition.getConnectionParameterDict()['_'])
@classmethod
def getComputerPartition(cls, partition_reference):
for computer_partition in cls.slap.computer.getComputerPartitionList():
if partition_reference == computer_partition.getInstanceParameter(
'instance_title'):
return computer_partition
@classmethod
def getComputerPartitionPath(cls, partition_reference):
partition_id = cls.getComputerPartition(partition_reference).getId()
return os.path.join(cls.slap._instance_root, partition_id)
software/erp5-zope2/test/test/benchmarks.py 0 → 100644
View file @ 8eeb74fe
##############################################################################
#
# Copyright (c) 2022 Nexedi SA and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import datetime
import json
import pathlib
import subprocess
import time
import typing
import urllib.parse
import psutil
import requests
from . import ERP5InstanceTestCase, default, matrix, setUpModule
from .test_erp5 import ZopeSkinsMixin
class TestOrderBuildPackingListSimulation(
ZopeSkinsMixin,
ERP5InstanceTestCase,
):
"""Create orders and build packing lists.
"""
__partition_reference__ = 's'
__test_matrix__ = matrix((default, ))
_start: datetime.datetime
_previous: datetime.datetime
@classmethod
def getInstanceParameterDict(cls) -> dict:
return {
'_':
json.dumps(
{
"bt5":
" ".join(
[
"erp5_full_text_mroonga_catalog",
"erp5_configurator_standard",
"erp5_scalability_test",
]),
"mariadb": {
# We use a large innodb-buffer-pool-size because the simulation
# select method used for sale packing list does not use index and
# cause slpow queries
"innodb-buffer-pool-size": 32 * 1024 * 1024 * 1024, # 32Go
},
"zope-partition-dict": {
"activities": {
"instance-count": 32,
"family": "activities",
"thread-amount": 2,
"port-base": 2300
},
"default": {
"instance-count": 1,
"family": "default",
"port-base": 2200
},
},
})
}
@classmethod
def _setUpClass(cls) -> None:
super()._setUpClass()
cls.zope_base_url = cls._getAuthenticatedZopeUrl('')
cls.create_sale_order_batch_url = urllib.parse.urljoin(
cls.zope_base_url, 'ERP5Site_createScalabilityTestSaleOrderBatch')
def setUp(self) -> None:
super().setUp()
self.measurement_file = open(f'measures{self.id()}.jsonl', 'w')
self.addCleanup(self.measurement_file.close)
def write_measurement(
self, measurement: dict[str, typing.Union[str, float]]) -> None:
json.dump(
measurement,
self.measurement_file,
)
self.measurement_file.write('\n')
self.measurement_file.flush()
def take_measurements(self, step: str) -> None:
# Time for this iteration
now = datetime.datetime.now()
elapsed = now - self._previous
self._previous = now
# Memory usage of all zopes
with self.slap.instance_supervisor_rpc as supervisor:
zope_memory_info_list = [
psutil.Process(process['pid']).memory_info()
for process in supervisor.getAllProcessInfo()
if process['name'].startswith('zope-') and process['pid']
]
zope_total_rss = sum(mem.rss for mem in zope_memory_info_list)
zope_count = len(zope_memory_info_list)
# Database size
root_fs = pathlib.Path(
self.getComputerPartitionPath('zodb')) / 'srv' / 'zodb' / 'root.fs'
root_fs_size = root_fs.stat().st_size
self.logger.info(
"Measurements for %s (after %s): "
"elapsed=%s zope_total_rss=%s / %s root_fs_size=%s",
step,
now - self._start,
elapsed,
zope_total_rss,
zope_count,
root_fs_size,
)
self.write_measurement(
{
'step': step,
'step_duration_seconds': elapsed.total_seconds(),
'step_duration': str(elapsed),
'zope_total_rss': zope_total_rss,
'zope_count': zope_count,
'root_fs_size': root_fs_size,
'now': str(now),
})
def test(self) -> None:
self._start = self._previous = datetime.datetime.now()
with requests.Session() as session:
ret = session.get(
urllib.parse.urljoin(
self.zope_base_url, 'ERP5Site_bootstrapScalabilityTest'),
verify=False,
params={'user_quantity:int': 1})
if not ret.ok:
self.logger.error(ret.text)
ret.raise_for_status()
self._waitForActivities(
timeout=datetime.timedelta(hours=2).total_seconds())
# XXX default reference generator for sale packing list cause
# many conflict errors, disable it.
self._addPythonScript(
script_id='Delivery_generateReference',
params='*args, **kw',
body='context.setReference("no reference for benchmark")',
)
self.take_measurements("setup")
# XXX now that we have installed business templates,
# restart all zopes to workaround a bug with accessors not
# working after some time (packing_list_line.getStartDate no longer
# acquire from parent's sale packing list)
with self.slap.instance_supervisor_rpc as supervisor:
supervisor.stopAllProcesses()
supervisor.startAllProcesses()
self.slap.waitForInstance()
self.take_measurements("restart")
with requests.Session() as session:
for i in range(100):
for j in range(5):
ret = session.get(
self.create_sale_order_batch_url,
verify=False,
params={
'random_seed': f'{i}.{j}',
'order_count:int': '50',
},
)
if not ret.ok:
self.logger.error(ret.text)
ret.raise_for_status()
self._waitForActivities(
timeout=datetime.timedelta(hours=2).total_seconds())
self.take_measurements(f"iteration_{i+1:03}")
# final measurements, take a "zodb analyze" snapshot
zodb_cmd = pathlib.Path(
self.computer_partition_root_path
) / 'software_release' / 'bin' / 'zodb'
root_fs = pathlib.Path(
self.getComputerPartitionPath('zodb')) / 'srv' / 'zodb' / 'root.fs'
self.write_measurement(
{
'zodb analyze':
subprocess.check_output((zodb_cmd, 'analyze', root_fs), text=True)
})
# and a pt-query-digest for slow log
pt_query_digest = pathlib.Path(
self.computer_partition_root_path
) / 'software_release' / 'parts' / 'percona-toolkit' / 'bin' / 'pt-query-digest'
mariadb_slowquery_log = pathlib.Path(
self.getComputerPartitionPath(
'mariadb')) / 'var' / 'log' / 'mariadb_slowquery.log'
self.write_measurement(
{
'pt-query-digest':
subprocess.check_output(
(pt_query_digest, mariadb_slowquery_log), text=True)
})
breakpoint()
software/erp5-zope2/test/test/test_balancer.py 0 → 100644
View file @ 8eeb74fe
import glob
import hashlib
import json
import logging
import os
import re
import shutil
import subprocess
import tempfile
import time
import urllib.parse
from http.server import BaseHTTPRequestHandler
from unittest import mock
import OpenSSL.SSL
import pexpect
import psutil
import requests
from cryptography import x509
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.x509.oid import NameOID
from slapos.testing.testcase import ManagedResource
from slapos.testing.utils import (CrontabMixin, ManagedHTTPServer,
findFreeTCPPort)
from . import ERP5InstanceTestCase, setUpModule, matrix, default
setUpModule # pyflakes
class EchoHTTPServer(ManagedHTTPServer):
"""An HTTP Server responding with the request path and incoming headers,
encoded in json.
"""
class RequestHandler(BaseHTTPRequestHandler):
def do_GET(self):
# type: () -> None
self.send_response(200)
self.send_header("Content-Type", "application/json")
response = json.dumps(
{
'Path': self.path,
'Incoming Headers': dict(self.headers.items()),
},
indent=2,
).encode('utf-8')
self.end_headers()
self.wfile.write(response)
log_message = logging.getLogger(__name__ + '.EchoHTTPServer').info
class EchoHTTP11Server(ManagedHTTPServer):
"""An HTTP/1.1 Server responding with the request path and incoming headers,
encoded in json.
"""
class RequestHandler(BaseHTTPRequestHandler):
protocol_version = 'HTTP/1.1'
def do_GET(self):
# type: () -> None
self.send_response(200)
self.send_header("Content-Type", "application/json")
response = json.dumps(
{
'Path': self.path,
'Incoming Headers': dict(self.headers.items()),
},
indent=2,
).encode('utf-8')
self.send_header("Content-Length", str(len(response)))
self.end_headers()
self.wfile.write(response)
log_message = logging.getLogger(__name__ + '.EchoHTTP11Server').info
class CaucaseService(ManagedResource):
"""A caucase service.
"""
url = None # type: str
directory = None # type: str
_caucased_process = None # type: subprocess.Popen
def open(self):
# type: () -> None
# start a caucased and server certificate.
software_release_root_path = os.path.join(
self._cls.slap._software_root,
hashlib.md5(self._cls.getSoftwareURL().encode()).hexdigest(),
)
caucased_path = os.path.join(software_release_root_path, 'bin', 'caucased')
self.directory = tempfile.mkdtemp()
caucased_dir = os.path.join(self.directory, 'caucased')
os.mkdir(caucased_dir)
os.mkdir(os.path.join(caucased_dir, 'user'))
os.mkdir(os.path.join(caucased_dir, 'service'))
backend_caucased_netloc = f'{self._cls._ipv4_address}:{findFreeTCPPort(self._cls._ipv4_address)}'
self.url = 'http://' + backend_caucased_netloc
self._caucased_process = subprocess.Popen(
[
caucased_path,
'--db', os.path.join(caucased_dir, 'caucase.sqlite'),
'--server-key', os.path.join(caucased_dir, 'server.key.pem'),
'--netloc', backend_caucased_netloc,
'--service-auto-approve-count', '1',
],
# capture subprocess output not to pollute test's own stdout
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
)
for _ in range(30):
try:
if requests.get(self.url).status_code == 200:
break
except Exception:
pass
time.sleep(1)
else:
raise RuntimeError('caucased failed to start.')
def close(self):
# type: () -> None
self._caucased_process.terminate()
self._caucased_process.wait()
self._caucased_process.stdout.close()
shutil.rmtree(self.directory)
class BalancerTestCase(ERP5InstanceTestCase):
# We explicitly specify 'balancer' as our software type here,
# therefore we don't request ZODB. We therefore don't
# need to run these tests with both NEO and ZEO mode,
# it wouldn't make any difference.
# https://lab.nexedi.com/nexedi/slapos/blob/273037c8/stack/erp5/instance.cfg.in#L216-230
__test_matrix__ = matrix((default,))
@classmethod
def getInstanceSoftwareType(cls):
return 'balancer'
@classmethod
def _getInstanceParameterDict(cls):
# type: () -> dict
return {
'tcpv4-port': 8000,
'computer-memory-percent-threshold': 100,
# XXX what is this ? should probably not be needed here
'name': cls.__name__,
'monitor-passwd': 'secret',
'apachedex-configuration': [
'--logformat', '%h %l %u %t "%r" %>s %O "%{Referer}i" "%{User-Agent}i" %{ms}T',
'--erp5-base', '+erp5', '.*/VirtualHostRoot/erp5(/|\\?|$)',
'--base', '+other', '/',
'--skip-user-agent', 'Zabbix',
'--error-detail',
'--js-embed',
'--quiet',
],
'apachedex-promise-threshold': 100,
'haproxy-server-check-path': '/',
'zope-family-dict': {
'default': ['dummy_http_server'],
},
'dummy_http_server': [[cls.getManagedResource("backend_web_server", EchoHTTPServer).netloc, 1, False]],
'backend-path-dict': {
'default': '',
},
'ssl-authentication-dict': {'default': False},
'ssl': {
'caucase-url': cls.getManagedResource("caucase", CaucaseService).url,
},
'timeout-dict': {'default': None},
'family-path-routing-dict': {},
'path-routing-list': [],
}
@classmethod
def getInstanceParameterDict(cls):
# type: () -> dict
return {'_': json.dumps(cls._getInstanceParameterDict())}
def setUp(self):
# type: () -> None
self.default_balancer_url = json.loads(
self.computer_partition.getConnectionParameterDict()['_'])['default']
class SlowHTTPServer(ManagedHTTPServer):
"""An HTTP Server which reply after a timeout.
Timeout is 2 seconds by default, and can be specified in the path of the URL
"""
class RequestHandler(BaseHTTPRequestHandler):
def do_GET(self):
# type: () -> None
self.send_response(200)
self.send_header("Content-Type", "text/plain")
timeout = 2
try:
timeout = int(self.path[1:])
except ValueError:
pass
time.sleep(timeout)
self.end_headers()
self.wfile.write(b"OK\n")
log_message = logging.getLogger(__name__ + '.SlowHTTPServer').info
class TestTimeout(BalancerTestCase, CrontabMixin):
__partition_reference__ = 't'
@classmethod
def _getInstanceParameterDict(cls):
# type: () -> dict
parameter_dict = super()._getInstanceParameterDict()
# use a slow server instead
parameter_dict['dummy_http_server'] = [[cls.getManagedResource("slow_web_server", SlowHTTPServer).netloc, 1, False]]
# and set timeout of 1 second
parameter_dict['timeout-dict'] = {'default': 1}
return parameter_dict
def test_timeout(self):
# type: () -> None
self.assertEqual(
requests.get(
urllib.parse.urljoin(self.default_balancer_url, '/1'),
verify=False).status_code,
requests.codes.ok)
self.assertEqual(
requests.get(
urllib.parse.urljoin(self.default_balancer_url, '/5'),
verify=False).status_code,
requests.codes.gateway_timeout)
class TestLog(BalancerTestCase, CrontabMixin):
"""Check logs emitted by balancer
"""
__partition_reference__ = 'l'
@classmethod
def _getInstanceParameterDict(cls):
# type: () -> dict
parameter_dict = super()._getInstanceParameterDict()
# use a slow server instead
parameter_dict['dummy_http_server'] = [[cls.getManagedResource("slow_web_server", SlowHTTPServer).netloc, 1, False]]
return parameter_dict
def test_access_log_format(self):
# type: () -> None
requests.get(
urllib.parse.urljoin(self.default_balancer_url, '/url_path'),
verify=False,
)
time.sleep(.5) # wait a bit more until access is logged
with open(os.path.join(self.computer_partition_root_path, 'var', 'log', 'apache-access.log')) as access_log_file:
access_line = access_log_file.read().splitlines()[-1]
self.assertIn('/url_path', access_line)
# last \d is the request time in milli seconds, since this SlowHTTPServer
# sleeps for 2 seconds, it should take between 2 and 3 seconds to process
# the request - but our test machines can be slow sometimes, so we tolerate
# it can take up to 20 seconds.
match = re.match(
r'([(\d\.)]+) - - \[(.*?)\] "(.*?)" (\d+) (\d+) "(.*?)" "(.*?)" (\d+)',
access_line
)
self.assertTrue(match)
assert match
request_time = int(match.groups()[-1])
self.assertGreater(request_time, 2 * 1000)
self.assertLess(request_time, 20 * 1000)
def test_access_log_apachedex_report(self):
# type: () -> None
# make a request so that we have something in the logs
requests.get(self.default_balancer_url, verify=False)
# crontab for apachedex is executed
self._executeCrontabAtDate('generate-apachedex-report', '23:59')
# it creates a report for the day
apachedex_report, = glob.glob(
os.path.join(
self.computer_partition_root_path,
'srv',
'monitor',
'private',
'apachedex',
'ApacheDex-*.html',
))
with open(apachedex_report) as f:
report_text = f.read()
self.assertIn('APacheDEX', report_text)
# having this table means that apachedex could parse some lines.
self.assertIn('<h2>Hits per status code</h2>', report_text)
def test_access_log_rotation(self):
# type: () -> None
# run logrotate a first time so that it create state files
self._executeCrontabAtDate('logrotate', '2000-01-01')
# make a request so that we have something in the logs
requests.get(self.default_balancer_url, verify=False).raise_for_status()
# slow query crontab depends on crontab for log rotation
# to be executed first.
self._executeCrontabAtDate('logrotate', '2050-01-01')
# this logrotate leaves the log for the day as non compressed
rotated_log_file = os.path.join(
self.computer_partition_root_path,
'srv',
'backup',
'logrotate',
'apache-access.log-20500101',
)
self.assertTrue(os.path.exists(rotated_log_file))
requests.get(self.default_balancer_url, verify=False).raise_for_status()
# on next day execution of logrotate, log files are compressed
self._executeCrontabAtDate('logrotate', '2050-01-02')
self.assertTrue(os.path.exists(rotated_log_file + '.xz'))
self.assertFalse(os.path.exists(rotated_log_file))
def test_error_log(self):
# type: () -> None
# stop backend server
backend_server = self.getManagedResource("slow_web_server", SlowHTTPServer)
self.addCleanup(backend_server.open)
backend_server.close()
# after a while, balancer should detect and log this event in error log
time.sleep(5)
self.assertEqual(
requests.get(self.default_balancer_url, verify=False).status_code,
requests.codes.service_unavailable)
with open(os.path.join(self.computer_partition_root_path, 'var', 'log', 'apache-error.log')) as error_log_file:
error_line = error_log_file.read().splitlines()[-1]
self.assertIn('proxy family_default has no server available!', error_line)
# this log also include a timestamp
self.assertRegex(error_line, r'\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}')
class BalancerCookieHTTPServer(ManagedHTTPServer):
"""An HTTP Server which can set balancer cookie.
This server set cookie when requested /set-cookie path.
The reply body is the name used when registering this resource
using getManagedResource. This way we can assert which
backend replied.
"""
@property
def RequestHandler(self):
server = self
class RequestHandler(BaseHTTPRequestHandler):
def do_GET(self):
# type: () -> None
self.send_response(200)
self.send_header("Content-Type", "text/plain")
if self.path == '/set_cookie':
# the balancer tells the backend what's the name of the balancer cookie with
# the X-Balancer-Current-Cookie header.
self.send_header('Set-Cookie', '%s=anything' % self.headers['X-Balancer-Current-Cookie'])
# The name of this cookie is SERVERID
assert self.headers['X-Balancer-Current-Cookie'] == 'SERVERID'
self.end_headers()
self.wfile.write(server._name.encode('utf-8'))
log_message = logging.getLogger(__name__ + '.BalancerCookieHTTPServer').info
return RequestHandler
class TestBalancer(BalancerTestCase):
"""Check balancing capabilities
"""
__partition_reference__ = 'b'
@classmethod
def _getInstanceParameterDict(cls):
# type: () -> dict
parameter_dict = super()._getInstanceParameterDict()
# use two backend servers
parameter_dict['dummy_http_server'] = [
[cls.getManagedResource("backend_web_server1", BalancerCookieHTTPServer).netloc, 1, False],
[cls.getManagedResource("backend_web_server2", BalancerCookieHTTPServer).netloc, 1, False],
]
return parameter_dict
def test_balancer_round_robin(self):
# type: () -> None
# requests are by default balanced to both servers
self.assertEqual(
{requests.get(self.default_balancer_url, verify=False).text for _ in range(10)},
{'backend_web_server1', 'backend_web_server2'}
)
def test_balancer_server_down(self):
# type: () -> None
# if one backend is down, it is excluded from balancer
self.getManagedResource("backend_web_server2", BalancerCookieHTTPServer).close()
self.addCleanup(self.getManagedResource("backend_web_server2", BalancerCookieHTTPServer).open)
self.assertEqual(
{requests.get(self.default_balancer_url, verify=False).text for _ in range(10)},
{'backend_web_server1',}
)
def test_balancer_set_cookie(self):
# type: () -> None
# if backend provides a "SERVERID" cookie, balancer will overwrite it with the
# backend selected by balancing algorithm
self.assertIn(
requests.get(urllib.parse.urljoin(self.default_balancer_url, '/set_cookie'), verify=False).cookies['SERVERID'],
('default-0', 'default-1'),
)
def test_balancer_respects_sticky_cookie(self):
# type: () -> None
# if request is made with the sticky cookie, the client stick on one balancer
cookies = dict(SERVERID='default-1')
self.assertEqual(
{requests.get(self.default_balancer_url, verify=False, cookies=cookies).text for _ in range(10)},
{'backend_web_server2',}
)
# if that backend becomes down, requests are balanced to another server
self.getManagedResource("backend_web_server2", BalancerCookieHTTPServer).close()
self.addCleanup(self.getManagedResource("backend_web_server2", BalancerCookieHTTPServer).open)
self.assertEqual(
requests.get(self.default_balancer_url, verify=False, cookies=cookies).text,
'backend_web_server1')
def test_balancer_stats_socket(self):
# type: () -> None
# real time statistics can be obtained by using the stats socket and there
# is a wrapper which makes this a bit easier.
socat_process = subprocess.Popen(
[self.computer_partition_root_path + '/bin/haproxy-socat-stats'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT
)
try:
output, _ = socat_process.communicate(b"show stat\n")
except:
socat_process.kill()
socat_process.wait()
raise
self.assertEqual(socat_process.poll(), 0)
# output is a csv
self.assertIn(b'family_default,FRONTEND,', output)
class TestTestRunnerEntryPoints(BalancerTestCase):
"""Check balancer has some entries for test runner.
"""
__partition_reference__ = 't'
@classmethod
def _getInstanceParameterDict(cls):
# type: () -> dict
parameter_dict = super()._getInstanceParameterDict()
parameter_dict['dummy_http_server-test-runner-address-list'] = [
[
cls.getManagedResource("backend_0", EchoHTTPServer).hostname,
cls.getManagedResource("backend_0", EchoHTTPServer).port,
],
[
cls.getManagedResource("backend_1", EchoHTTPServer).hostname,
cls.getManagedResource("backend_1", EchoHTTPServer).port,
],
[
cls.getManagedResource("backend_2", EchoHTTPServer).hostname,
cls.getManagedResource("backend_2", EchoHTTPServer).port,
],
]
return parameter_dict
def test_use_proper_backend(self):
# type: () -> None
# requests are directed to proper backend based on URL path
test_runner_url_list = self.getRootPartitionConnectionParameterDict(
)['default-test-runner-url-list']
url_0, url_1, url_2 = test_runner_url_list
self.assertEqual(
urllib.parse.urlparse(url_0).netloc,
urllib.parse.urlparse(url_1).netloc)
self.assertEqual(
urllib.parse.urlparse(url_0).netloc,
urllib.parse.urlparse(url_2).netloc)
path_0 = '/VirtualHostBase/https/{netloc}/VirtualHostRoot/_vh_unit_test_0/something'.format(
netloc=urllib.parse.urlparse(url_0).netloc)
path_1 = '/VirtualHostBase/https/{netloc}/VirtualHostRoot/_vh_unit_test_1/something'.format(
netloc=urllib.parse.urlparse(url_0).netloc)
path_2 = '/VirtualHostBase/https/{netloc}/VirtualHostRoot/_vh_unit_test_2/something'.format(
netloc=urllib.parse.urlparse(url_0).netloc)
self.assertEqual(
{
requests.get(url_0 + 'something', verify=False).json()['Path']
for _ in range(10)
}, {path_0})
self.assertEqual(
{
requests.get(url_1 + 'something', verify=False).json()['Path']
for _ in range(10)
}, {path_1})
self.assertEqual(
{
requests.get(url_2 + 'something', verify=False).json()['Path']
for _ in range(10)
}, {path_2})
# If a test runner backend is down, others can be accessed.
self.getManagedResource("backend_0", EchoHTTPServer).close()
self.assertEqual(
{
requests.get(url_0 + 'something', verify=False).status_code
for _ in range(5)
}, {503})
self.assertEqual(
{
requests.get(url_1 + 'something', verify=False).json()['Path']
for _ in range(10)
}, {path_1})
class TestHTTP(BalancerTestCase):
"""Check HTTP protocol with a HTTP/1.1 backend
"""
@classmethod
def _getInstanceParameterDict(cls):
# type: () -> dict
parameter_dict = super()._getInstanceParameterDict()
# use a HTTP/1.1 server instead
parameter_dict['dummy_http_server'] = [[cls.getManagedResource("HTTP/1.1 Server", EchoHTTP11Server).netloc, 1, False]]
return parameter_dict
__partition_reference__ = 'h'
def test_http_version(self):
# type: () -> None
self.assertEqual(
subprocess.check_output([
'curl',
'--silent',
'--show-error',
'--output',
'/dev/null',
'--insecure',
'--write-out',
'%{http_version}',
self.default_balancer_url,
]),
b'2',
)
def test_keep_alive(self):
# type: () -> None
# when doing two requests, connection is established only once
with requests.Session() as session:
session.verify = False
# do a first request, which establish a first connection
session.get(self.default_balancer_url).raise_for_status()
# "break" new connection method and check we can make another request
with mock.patch(
"requests.packages.urllib3.connectionpool.HTTPSConnectionPool._new_conn",
) as new_conn:
session.get(self.default_balancer_url).raise_for_status()
new_conn.assert_not_called()
parsed_url = urllib.parse.urlparse(self.default_balancer_url)
# check that we have an open file for the ip connection
self.assertTrue([
c for c in psutil.Process(os.getpid()).connections()
if c.status == 'ESTABLISHED' and c.raddr.ip == parsed_url.hostname
and c.raddr.port == parsed_url.port
])
class ContentTypeHTTPServer(ManagedHTTPServer):
"""An HTTP/1.1 Server which reply with content type from path.
For example when requested http://host/text/plain it will reply
with Content-Type: text/plain header.
The body is always "OK"
"""
class RequestHandler(BaseHTTPRequestHandler):
protocol_version = 'HTTP/1.1'
def do_GET(self):
# type: () -> None
self.send_response(200)
if self.path == '/':
self.send_header("Content-Length", '0')
return self.end_headers()
content_type = self.path[1:]
body = b"OK"
self.send_header("Content-Type", content_type)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
log_message = logging.getLogger(__name__ + '.ContentTypeHTTPServer').info
class TestContentEncoding(BalancerTestCase):
"""Test how responses are gzip encoded or not depending on content type header.
"""
__partition_reference__ = 'ce'
@classmethod
def _getInstanceParameterDict(cls):
# type: () -> dict
parameter_dict = super()._getInstanceParameterDict()
parameter_dict['dummy_http_server'] = [
[cls.getManagedResource("content_type_server", ContentTypeHTTPServer).netloc, 1, False],
]
return parameter_dict
def test_gzip_encoding(self):
# type: () -> None
for content_type in (
'text/cache-manifest',
'text/html',
'text/plain',
'text/css',
'application/hal+json',
'application/json',
'application/x-javascript',
'text/xml',
'application/xml',
'application/rss+xml',
'text/javascript',
'application/javascript',
'image/svg+xml',
'application/x-font-ttf',
'application/font-woff',
'application/font-woff2',
'application/x-font-opentype',
'application/wasm',):
resp = requests.get(urllib.parse.urljoin(self.default_balancer_url, content_type), verify=False)
self.assertEqual(resp.headers['Content-Type'], content_type)
self.assertEqual(
resp.headers.get('Content-Encoding'),
'gzip',
'{} uses wrong encoding: {}'.format(content_type, resp.headers.get('Content-Encoding')))
self.assertEqual(resp.text, 'OK')
def test_no_gzip_encoding(self):
# type: () -> None
resp = requests.get(urllib.parse.urljoin(self.default_balancer_url, '/image/png'), verify=False)
self.assertNotIn('Content-Encoding', resp.headers)
self.assertEqual(resp.text, 'OK')
class CaucaseCertificate(ManagedResource):
"""A certificate signed by a caucase service.
"""
ca_crt_file = None # type: str
crl_file = None # type: str
csr_file = None # type: str
cert_file = None # type: str
key_file = None # type: str
def open(self):
# type: () -> None
self.tmpdir = tempfile.mkdtemp()
self.ca_crt_file = os.path.join(self.tmpdir, 'ca-crt.pem')
self.crl_file = os.path.join(self.tmpdir, 'ca-crl.pem')
self.csr_file = os.path.join(self.tmpdir, 'csr.pem')
self.cert_file = os.path.join(self.tmpdir, 'crt.pem')
self.key_file = os.path.join(self.tmpdir, 'key.pem')
def close(self):
# type: () -> None
shutil.rmtree(self.tmpdir)
@property
def _caucase_path(self):
# type: () -> str
"""path of caucase executable.
"""
software_release_root_path = os.path.join(
self._cls.slap._software_root,
hashlib.md5(self._cls.getSoftwareURL().encode()).hexdigest(),
)
return os.path.join(software_release_root_path, 'bin', 'caucase')
def request(self, common_name, caucase):
# type: (str, CaucaseService) -> None
"""Generate certificate and request signature to the caucase service.
This overwrite any previously requested certificate for this instance.
"""
cas_args = [
self._caucase_path,
'--ca-url', caucase.url,
'--ca-crt', self.ca_crt_file,
'--crl', self.crl_file,
]
key = rsa.generate_private_key(
public_exponent=65537,
key_size=2048,
backend=default_backend()
)
with open(self.key_file, 'wb') as f:
f.write(
key.private_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PrivateFormat.TraditionalOpenSSL,
encryption_algorithm=serialization.NoEncryption(),
))
csr = x509.CertificateSigningRequestBuilder().subject_name(
x509.Name([
x509.NameAttribute(
NameOID.COMMON_NAME,
common_name,
),
])).sign(
key,
hashes.SHA256(),
default_backend(),
)
with open(self.csr_file, 'wb') as f:
f.write(csr.public_bytes(serialization.Encoding.PEM))
csr_id = subprocess.check_output(
cas_args + [
'--send-csr', self.csr_file,
],
).split()[0].decode()
assert csr_id
for _ in range(30):
if not subprocess.call(
cas_args + [
'--get-crt', csr_id, self.cert_file,
],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
) == 0:
break
else:
time.sleep(1)
else:
raise RuntimeError('getting service certificate failed.')
with open(self.cert_file) as cert_file:
assert 'BEGIN CERTIFICATE' in cert_file.read()
def revoke(self, caucase):
# type: (CaucaseService) -> None
"""Revoke the client certificate on this caucase instance.
"""
subprocess.check_call([
self._caucase_path,
'--ca-url', caucase.url,
'--ca-crt', self.ca_crt_file,
'--crl', self.crl_file,
'--revoke-crt', self.cert_file, self.key_file,
])
class TestFrontendXForwardedFor(BalancerTestCase):
__partition_reference__ = 'xff'
@classmethod
def _getInstanceParameterDict(cls):
# type: () -> dict
frontend_caucase = cls.getManagedResource('frontend_caucase', CaucaseService)
certificate = cls.getManagedResource('client_certificate', CaucaseCertificate)
certificate.request('shared frontend', frontend_caucase)
parameter_dict = super()._getInstanceParameterDict()
# add another "-auth" backend, that will have ssl-authentication enabled
parameter_dict['zope-family-dict']['default-auth'] = ['dummy_http_server']
parameter_dict['backend-path-dict']['default-auth'] = '/'
parameter_dict['ssl-authentication-dict'] = {
'default': False,
'default-auth': True,
}
parameter_dict['timeout-dict']['default-auth'] = None
parameter_dict['ssl']['frontend-caucase-url-list'] = [frontend_caucase.url]
return parameter_dict
def test_x_forwarded_for_added_when_verified_connection(self):
# type: () -> None
client_certificate = self.getManagedResource('client_certificate', CaucaseCertificate)
for backend in ('default', 'default-auth'):
balancer_url = json.loads(self.computer_partition.getConnectionParameterDict()['_'])[backend]
result = requests.get(
balancer_url,
headers={'X-Forwarded-For': '1.2.3.4'},
cert=(client_certificate.cert_file, client_certificate.key_file),
verify=False,
).json()
self.assertEqual(result['Incoming Headers'].get('x-forwarded-for', '').split(', ')[0], '1.2.3.4')
def test_x_forwarded_for_stripped_when_not_verified_connection(self):
# type: () -> None
balancer_url = json.loads(self.computer_partition.getConnectionParameterDict()['_'])['default']
result = requests.get(
balancer_url,
headers={'X-Forwarded-For': '1.2.3.4'},
verify=False,
).json()
self.assertNotEqual(result['Incoming Headers'].get('x-forwarded-for', '').split(', ')[0], '1.2.3.4')
balancer_url = json.loads(self.computer_partition.getConnectionParameterDict()['_'])['default-auth']
with self.assertRaisesRegex(Exception, "certificate required"):
requests.get(
balancer_url,
headers={'X-Forwarded-For': '1.2.3.4'},
verify=False,
)
class TestServerTLSProvidedCertificate(BalancerTestCase):
"""Check that certificate and key can be provided as instance parameters.
"""
__partition_reference__ = 's'
@classmethod
def _getInstanceParameterDict(cls):
# type: () -> dict
server_caucase = cls.getManagedResource('server_caucase', CaucaseService)
server_certificate = cls.getManagedResource('server_certificate', CaucaseCertificate)
server_certificate.request(cls._ipv4_address, server_caucase)
parameter_dict = super()._getInstanceParameterDict()
with open(server_certificate.cert_file) as f:
parameter_dict['ssl']['cert'] = f.read()
with open(server_certificate.key_file) as f:
parameter_dict['ssl']['key'] = f.read()
return parameter_dict
def test_certificate_validates_with_provided_ca(self):
# type: () -> None
server_certificate = self.getManagedResource("server_certificate", CaucaseCertificate)
requests.get(self.default_balancer_url, verify=server_certificate.ca_crt_file)
class TestClientTLS(BalancerTestCase):
__partition_reference__ = 'c'
@classmethod
def _getInstanceParameterDict(cls):
# type: () -> dict
frontend_caucase1 = cls.getManagedResource('frontend_caucase1', CaucaseService)
certificate1 = cls.getManagedResource('client_certificate1', CaucaseCertificate)
certificate1.request('client_certificate1', frontend_caucase1)
frontend_caucase2 = cls.getManagedResource('frontend_caucase2', CaucaseService)
certificate2 = cls.getManagedResource('client_certificate2', CaucaseCertificate)
certificate2.request('client_certificate2', frontend_caucase2)
parameter_dict = super()._getInstanceParameterDict()
parameter_dict['ssl-authentication-dict'] = {
'default': True,
}
parameter_dict['ssl']['frontend-caucase-url-list'] = [
frontend_caucase1.url,
frontend_caucase2.url,
]
return parameter_dict
def test_refresh_crl(self):
# type: () -> None
logger = self.logger
class DebugLogFile:
def write(self, msg):
logger.info("output from caucase_updater: %s", msg)
def flush(self):
pass
for client_certificate_name, caucase_name in (
('client_certificate1', 'frontend_caucase1'),
('client_certificate2', 'frontend_caucase2'),
):
client_certificate = self.getManagedResource(client_certificate_name,
CaucaseCertificate)
# when client certificate can be authenticated, backend receive the CN of
# the client certificate in "remote-user" header
def _make_request():
# type: () -> dict
return requests.get(
self.default_balancer_url,
cert=(client_certificate.cert_file, client_certificate.key_file),
verify=False,
).json()
self.assertEqual(_make_request()['Incoming Headers'].get('remote-user'),
client_certificate_name)
# when certificate is revoked, updater service should update the CRL
# used by balancer from the caucase service used for client certificates
# (ie. the one used by frontend).
caucase = self.getManagedResource(caucase_name, CaucaseService)
client_certificate.revoke(caucase)
# until the CRL is updated, the client certificate is still accepted.
self.assertEqual(_make_request()['Incoming Headers'].get('remote-user'),
client_certificate_name)
# We have two services, in charge of updating CRL and CA certificates for
# each frontend CA
caucase_updater_list = glob.glob(
os.path.join(
self.computer_partition_root_path,
'etc',
'service',
'caucase-updater-*',
))
self.assertEqual(len(caucase_updater_list), 2)
# find the one corresponding to this caucase
for caucase_updater_candidate in caucase_updater_list:
with open(caucase_updater_candidate) as f:
if caucase.url in f.read():
caucase_updater = caucase_updater_candidate
break
else:
self.fail("Could not find caucase updater script for %s" % caucase.url)
# simulate running updater service in the future, to confirm that it fetches
# the new CRL and make sure balancer uses that new CRL.
process = pexpect.spawnu("faketime +1day %s" % caucase_updater)
process.logfile = DebugLogFile()
process.expect("Got new CRL.*Next wake-up at.*")
process.terminate()
process.wait()
with self.assertRaisesRegex(Exception, 'certificate revoked'):
_make_request()
class TestPathBasedRouting(BalancerTestCase):
"""Check path-based routing rewrites URLs as expected.
"""
__partition_reference__ = 'pbr'
@classmethod
def _getInstanceParameterDict(cls):
# type: () -> dict
parameter_dict = super()._getInstanceParameterDict()
parameter_dict['zope-family-dict'][
'second'
] = parameter_dict['zope-family-dict'][
'default'
]
parameter_dict['timeout-dict']['second'] = None
parameter_dict['ssl-authentication-dict']['second'] = False
# Routing rules outermost slashes mean nothing. They are internally
# stripped and rebuilt in order to correctly represent the request's URL.
parameter_dict['family-path-routing-dict'] = {
'default': [
['foo/bar', 'erp5/boo/far/faz'], # no outermost slashes
['/foo', '/erp5/somewhere'],
['/foo/shadowed', '/foo_shadowed'], # unreachable
['/next', '/erp5/web_site_module/another_next_website'],
],
}
parameter_dict['path-routing-list'] = [
['/next', '/erp5/web_site_module/the_next_website'],
['/next2', '/erp5/web_site_module/the_next2_website'],
['//', '//erp5/web_site_module/123//'], # extraneous slashes
]
return parameter_dict
def test_routing(self):
# type: () -> None
published_dict = json.loads(self.computer_partition.getConnectionParameterDict()['_'])
scheme = 'scheme'
netloc = 'example.com:8080'
prefix = '/VirtualHostBase/' + scheme + '//' + urllib.parse.quote(
netloc,
safe='',
)
# For easier reading of test data, visually separating the virtual host
# base from the virtual host root
vhr = '/VirtualHostRoot'
def assertRoutingEqual(family, path, expected_path):
# type: (str, str, str) -> None
# sanity check: unlike the rules, this test is sensitive to outermost
# slashes, and paths must be absolute-ish for code simplicity.
assert path.startswith('/')
# Frontend is expected to provide URLs with the following path structure:
# /VirtualHostBase/<scheme>//<netloc>/VirtualHostRoot<path>
# where:
# - scheme is the user-input scheme
# - netloc is the user-input netloc
# - path is the user-input path
# Someday, frontends will instead propagate scheme and netloc via other
# means (likely: HTTP headers), in which case this test and the SR will
# need to be amended to reconstruct Virtual Host urls itself, and this
# test will need to be updated accordingly.
self.assertEqual(
requests.get(
urllib.parse.urljoin(published_dict[family], prefix + vhr + path),
verify=False,
).json()['Path'],
expected_path,
)
# Trailing slash presence is preserved.
assertRoutingEqual('default', '/foo/bar', prefix + '/erp5/boo/far/faz' + vhr + '/_vh_foo/_vh_bar')
assertRoutingEqual('default', '/foo/bar/', prefix + '/erp5/boo/far/faz' + vhr + '/_vh_foo/_vh_bar/')
# Subpaths are preserved.
assertRoutingEqual('default', '/foo/bar/hey', prefix + '/erp5/boo/far/faz' + vhr + '/_vh_foo/_vh_bar/hey')
# Rule precedence: later less-specific rules are applied.
assertRoutingEqual('default', '/foo', prefix + '/erp5/somewhere' + vhr + '/_vh_foo')
assertRoutingEqual('default', '/foo/', prefix + '/erp5/somewhere' + vhr + '/_vh_foo/')
assertRoutingEqual('default', '/foo/baz', prefix + '/erp5/somewhere' + vhr + '/_vh_foo/baz')
# Rule precedence: later more-specific rules are meaningless.
assertRoutingEqual('default', '/foo/shadowed', prefix + '/erp5/somewhere' + vhr + '/_vh_foo/shadowed')
# Rule precedence: family rules applied before general rules.
assertRoutingEqual('default', '/next', prefix + '/erp5/web_site_module/another_next_website' + vhr + '/_vh_next')
# Fallback on general rules when no family-specific rule matches
# Note: the root is special in that there is always a trailing slash in the
# produced URL.
assertRoutingEqual('default', '/', prefix + '/erp5/web_site_module/123' + vhr + '/')
# Rule-less family reach general rules.
assertRoutingEqual('second', '/foo/bar', prefix + '/erp5/web_site_module/123' + vhr + '/foo/bar') # Rules match whole-elements, so the rule order does not matter to
# elements which share a common prefix.
assertRoutingEqual('second', '/next', prefix + '/erp5/web_site_module/the_next_website' + vhr + '/_vh_next')
assertRoutingEqual('second', '/next2', prefix + '/erp5/web_site_module/the_next2_website' + vhr + '/_vh_next2')
software/erp5-zope2/test/test/test_erp5.py 0 → 100644
View file @ 8eeb74fe
##############################################################################
#
# Copyright (c) 2022 Nexedi SA and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import contextlib
import datetime
import glob
import http.client
import json
import os
import shutil
import socket
import ssl
import subprocess
import sys
import tempfile
import time
import unittest
import psutil
import requests
import urllib.parse
import xmlrpc.client
import urllib3
from slapos.testing.utils import CrontabMixin
from . import ERP5InstanceTestCase, setUpModule, matrix, default
setUpModule # pyflakes
class TestPublishedURLIsReachableMixin:
"""Mixin that checks that default page of ERP5 is reachable.
"""
def _checkERP5IsReachable(self, base_url, site_id, verify):
# We access ERP5 trough a "virtual host", which should make
# ERP5 produce URLs using https://virtual-host-name:1234/virtual_host_root
# as base.
virtual_host_url = urllib.parse.urljoin(
base_url,
'/VirtualHostBase/https/virtual-host-name:1234/{}/VirtualHostRoot/_vh_virtual_host_root/'
.format(site_id))
# What happens is that instantiation just create the services, but does not
# wait for ERP5 to be initialized. When this test run ERP5 instance is
# instantiated, but zope is still busy creating the site and haproxy replies
# with 503 Service Unavailable when zope is not started yet, with 404 when
# erp5 site is not created, with 500 when mysql is not yet reachable, so we
# configure this requests session to retry.
# XXX we should probably add a promise instead
with requests.Session() as session:
session.mount(
base_url,
requests.adapters.HTTPAdapter(
max_retries=urllib3.util.retry.Retry(
total=20,
backoff_factor=.5,
status_forcelist=(404, 500, 503))))
r = session.get(virtual_host_url, verify=verify, allow_redirects=False)
self.assertEqual(r.status_code, requests.codes.found)
# access on / are redirected to login form, with virtual host preserved
self.assertEqual(r.headers.get('location'), 'https://virtual-host-name:1234/virtual_host_root/login_form')
# login page can be rendered and contain the text "ERP5"
r = session.get(
urllib.parse.urljoin(base_url, f'{site_id}/login_form'),
verify=verify,
allow_redirects=False,
)
self.assertEqual(r.status_code, requests.codes.ok)
self.assertIn("ERP5", r.text)
def test_published_family_default_v6_is_reachable(self):
"""Tests the IPv6 URL published by the root partition is reachable.
"""
param_dict = self.getRootPartitionConnectionParameterDict()
self._checkERP5IsReachable(
param_dict['family-default-v6'],
param_dict['site-id'],
verify=False,
)
def test_published_family_default_v4_is_reachable(self):
"""Tests the IPv4 URL published by the root partition is reachable.
"""
param_dict = self.getRootPartitionConnectionParameterDict()
self._checkERP5IsReachable(
param_dict['family-default'],
param_dict['site-id'],
verify=False,
)
class TestDefaultParameters(ERP5InstanceTestCase, TestPublishedURLIsReachableMixin):
"""Test ERP5 can be instantiated with no parameters
"""
__partition_reference__ = 'defp'
__test_matrix__ = matrix((default,))
class TestMedusa(ERP5InstanceTestCase, TestPublishedURLIsReachableMixin):
"""Test ERP5 Medusa server
"""
__partition_reference__ = 'medusa'
@classmethod
def getInstanceParameterDict(cls):
return {'_': json.dumps({'wsgi': False})}
class TestJupyter(ERP5InstanceTestCase, TestPublishedURLIsReachableMixin):
"""Test ERP5 Jupyter notebook
"""
__partition_reference__ = 'jupyter'
@classmethod
def getInstanceParameterDict(cls):
return {'_': json.dumps({'jupyter': {'enable': True}})}
def test_jupyter_notebook_is_reachable(self):
param_dict = self.getRootPartitionConnectionParameterDict()
self.assertEqual(
'https://[%s]:8888/tree' % self._ipv6_address,
param_dict['jupyter-url']
)
result = requests.get(
param_dict['jupyter-url'], verify=False, allow_redirects=False)
self.assertEqual(
[requests.codes.found, True, '/login?next=%2Ftree'],
[result.status_code, result.is_redirect, result.headers['Location']]
)
class TestBalancerPorts(ERP5InstanceTestCase):
"""Instantiate with two zope families, this should create for each family:
- a balancer entry point with corresponding haproxy
- a balancer entry point for test runner
"""
__partition_reference__ = 'ap'
@classmethod
def getInstanceParameterDict(cls):
return {
'_':
json.dumps({
"zope-partition-dict": {
"family1": {
"instance-count": 3,
"family": "family1"
},
"family2": {
"instance-count": 5,
"family": "family2"
},
},
})
}
def checkValidHTTPSURL(self, url):
parsed = urllib.parse.urlparse(url)
self.assertEqual(parsed.scheme, 'https')
self.assertTrue(parsed.hostname)
self.assertTrue(parsed.port)
def test_published_family_parameters(self):
# when we request two families, we have two published family-{family_name} URLs
param_dict = self.getRootPartitionConnectionParameterDict()
for family_name in ('family1', 'family2'):
self.checkValidHTTPSURL(
param_dict[f'family-{family_name}'])
self.checkValidHTTPSURL(
param_dict[f'family-{family_name}-v6'])
def test_published_test_runner_url(self):
# each family's also a list of test test runner URLs, by default 3 per family
param_dict = self.getRootPartitionConnectionParameterDict()
for family_name in ('family1', 'family2'):
family_test_runner_url_list = param_dict[
f'{family_name}-test-runner-url-list']
self.assertEqual(3, len(family_test_runner_url_list))
for url in family_test_runner_url_list:
self.checkValidHTTPSURL(url)
def test_zope_listen(self):
# we requested 3 zope in family1 and 5 zopes in family2, we should have 8 zope running.
with self.slap.instance_supervisor_rpc as supervisor:
all_process_info = supervisor.getAllProcessInfo()
self.assertEqual(
3 + 5,
len([p for p in all_process_info if p['name'].startswith('zope-')]))
def test_haproxy_listen(self):
# We have 2 families, haproxy should listen to a total of 3 ports per family
# normal access on ipv4 and ipv6 and test runner access on ipv4 only
with self.slap.instance_supervisor_rpc as supervisor:
all_process_info = supervisor.getAllProcessInfo()
process_info, = (p for p in all_process_info if p['name'].startswith('haproxy-'))
haproxy_master_process = psutil.Process(process_info['pid'])
haproxy_worker_process, = haproxy_master_process.children()
self.assertEqual(
sorted([socket.AF_INET] * 4 + [socket.AF_INET6] * 2),
sorted(
c.family
for c in haproxy_worker_process.connections()
if c.status == 'LISTEN'
))
class TestSeleniumTestRunner(ERP5InstanceTestCase, TestPublishedURLIsReachableMixin):
"""Test ERP5 can be instantiated with selenium server for test runner.
"""
__partition_reference__ = 'sel'
@classmethod
def getInstanceParameterDict(cls):
return {
'_':
json.dumps({
'test-runner': {
'selenium': {
"target": "selenium-server",
"server-url": "https://example.com",
"verify-server-certificate": False,
"desired-capabilities": {
"browserName": "firefox",
"version": "68.0.2esr",
}
}
}
})
}
def test_test_runner_configuration_json_file(self):
runUnitTest_script, = glob.glob(
self.computer_partition_root_path + "/../*/bin/runUnitTest.real")
config_file = None
with open(runUnitTest_script) as f:
for line in f:
if 'ERP5_TEST_RUNNER_CONFIGURATION' in line:
_, config_file = line.split('=')
assert config_file
with open(config_file.strip()) as f:
self.assertEqual(
f.read(),
json.dumps(json.loads(self.getInstanceParameterDict()['_'])['test-runner'], sort_keys=True))
class TestDisableTestRunner(ERP5InstanceTestCase, TestPublishedURLIsReachableMixin):
"""Test ERP5 can be instantiated without test runner.
"""
__partition_reference__ = 'distr'
@classmethod
def getInstanceParameterDict(cls):
return {'_': json.dumps({'test-runner': {'enabled': False}})}
def test_no_runUnitTestScript(self):
"""No runUnitTest script should be generated in any partition.
"""
# self.computer_partition_root_path is the path of root partition.
# we want to assert that no scripts exist in any partition.
bin_programs = list(map(os.path.basename,
glob.glob(self.computer_partition_root_path + "/../*/bin/*")))
self.assertTrue(bin_programs) # just to check the glob was correct.
self.assertNotIn('runUnitTest', bin_programs)
self.assertNotIn('runTestSuite', bin_programs)
def test_no_haproxy_testrunner_port(self):
# Haproxy only listen on two ports, there is no haproxy ports allocated for test runner
with self.slap.instance_supervisor_rpc as supervisor:
all_process_info = supervisor.getAllProcessInfo()
process_info, = (p for p in all_process_info if p['name'].startswith('haproxy'))
haproxy_master_process = psutil.Process(process_info['pid'])
haproxy_worker_process, = haproxy_master_process.children()
self.assertEqual(
sorted([socket.AF_INET, socket.AF_INET6]),
sorted(
c.family
for c in haproxy_worker_process.connections()
if c.status == 'LISTEN'
))
class TestZopeNodeParameterOverride(ERP5InstanceTestCase, TestPublishedURLIsReachableMixin):
"""Test override zope node parameters
"""
__partition_reference__ = 'override'
__test_matrix__ = matrix((default,))
@classmethod
def getInstanceParameterDict(cls):
# The following example includes the most commonly used options,
# but not necessarily in a meaningful way.
return {'_': json.dumps({
"zodb": [{
"type": "zeo",
"server": {},
"cache-size-bytes": "20MB",
"cache-size-bytes!": [
("bb-0", 1<<20),
("bb-.*", "500MB"),
],
"pool-timeout": "10m",
"storage-dict": {
"cache-size!": [
("a-.*", "50MB"),
],
},
}],
"zope-partition-dict": {
"a": {
"instance-count": 3,
},
"bb": {
"instance-count": 5,
"port-base": 2300,
},
},
})}
def test_zope_conf(self):
zeo_addr = json.loads(
self.getComputerPartition('zodb').getConnectionParameter('_')
)["storage-dict"]["root"]["server"]
def checkParameter(line, kw):
k, v = line.split()
self.assertFalse(k.endswith('!'), k)
try:
expected = kw.pop(k)
except KeyError:
if k == 'server':
return
self.assertIsNotNone(expected)
self.assertEqual(str(expected), v)
def checkConf(zodb, storage):
zodb["mount-point"] = "/"
zodb["pool-size"] = 4
zodb["pool-timeout"] = "10m"
storage["storage"] = "root"
storage["server"] = zeo_addr
with open(f'{partition}/etc/zope-{zope}.conf') as f:
conf = list(map(str.strip, f.readlines()))
i = conf.index("<zodb_db root>") + 1
conf = iter(conf[i:conf.index("</zodb_db>", i)])
for line in conf:
if line == '<zeoclient>':
for line in conf:
if line == '</zeoclient>':
break
checkParameter(line, storage)
for k, v in storage.items():
self.assertIsNone(v, k)
del storage
else:
checkParameter(line, zodb)
for k, v in zodb.items():
self.assertIsNone(v, k)
partition = self.getComputerPartitionPath('zope-a')
for zope in range(3):
checkConf({
"cache-size-bytes": "20MB",
}, {
"cache-size": "50MB",
})
partition = self.getComputerPartitionPath('zope-bb')
for zope in range(5):
checkConf({
"cache-size-bytes": "500MB" if zope else 1<<20,
}, {
"cache-size": None,
})
class TestWatchActivities(ERP5InstanceTestCase):
"""Tests for bin/watch_activities scripts in zope partitions.
"""
__partition_reference__ = 'wa'
def test(self):
# "watch_activities" scripts use watch command. We'll fake a watch command
# that executes the actual command only once to check the output.
tmpdir = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, tmpdir)
with open(os.path.join(tmpdir, 'watch'), 'w') as f:
f.write("""#!/bin/sh
if [ "$1" != "-n" ] || [ "$2" != "5" ]
then
echo unexpected arguments: "$1" "$2"
exit 1
fi
shift
shift
exec bash -c "$@"
""")
os.fchmod(f.fileno(), 0o700)
try:
output = subprocess.check_output(
[
os.path.join(
self.getComputerPartitionPath('zope-1'),
'bin',
'watch_activities',
)
],
env=dict(os.environ,
PATH=os.pathsep.join([tmpdir, os.environ['PATH']])),
stderr=subprocess.STDOUT,
text=True,
)
except subprocess.CalledProcessError as e:
self.fail(e.output)
self.assertIn(' dict ', output)
class ZopeSkinsMixin:
"""Mixins with utility methods to test zope behaviors.
"""
@classmethod
def _setUpClass(cls):
super()._setUpClass()
cls._waitForActivities()
@classmethod
def _waitForActivities(cls, timeout=datetime.timedelta(minutes=10).total_seconds()):
"""Wait for ERP5 to be ready and have processed all activities.
"""
for _ in range(int(timeout / 5)):
with cls.getXMLRPCClient() as erp5_xmlrpc_client:
try:
if erp5_xmlrpc_client.portal_activities.countMessage() == 0:
break
except (xmlrpc.client.ProtocolError,
xmlrpc.client.Fault,
http.client.HTTPException):
pass
time.sleep(5)
else:
raise AssertionError("Timeout waiting for activities")
@classmethod
def _getAuthenticatedZopeUrl(cls, path, family_name='default'):
"""Returns a URL to access a zope family through balancer,
with credentials in the URL.
path is joined with urllib.parse.urljoin to the URL of the portal.
"""
param_dict = cls.getRootPartitionConnectionParameterDict()
parsed = urllib.parse.urlparse(param_dict['family-' + family_name])
base_url = parsed._replace(
netloc='{}:{}@{}:{}'.format(
param_dict['inituser-login'],
param_dict['inituser-password'],
parsed.hostname,
parsed.port,
),
path=param_dict['site-id'] + '/',
).geturl()
return urllib.parse.urljoin(base_url, path)
@classmethod
@contextlib.contextmanager
def getXMLRPCClient(cls):
# don't verify certificate
ssl_context = ssl.create_default_context()
ssl_context.check_hostname = False
ssl_context.verify_mode = ssl.CERT_NONE
erp5_xmlrpc_client = xmlrpc.client.ServerProxy(
cls._getAuthenticatedZopeUrl(''),
context=ssl_context,
)
with erp5_xmlrpc_client:
yield erp5_xmlrpc_client
@classmethod
def _addPythonScript(cls, script_id, params, body):
with cls.getXMLRPCClient() as erp5_xmlrpc_client:
custom = erp5_xmlrpc_client.portal_skins.custom
try:
custom.manage_addProduct.PythonScripts.manage_addPythonScript(
script_id)
except xmlrpc.client.ProtocolError as e:
if e.errcode != 302:
raise
getattr(custom, script_id).ZPythonScriptHTML_editAction(
'',
'',
params,
body,
)
class ZopeTestMixin(ZopeSkinsMixin, CrontabMixin):
"""Mixin class for zope features.
"""
wsgi = NotImplemented # type: bool
__partition_reference__ = 'z'
@classmethod
def getInstanceParameterDict(cls):
return {
'_':
json.dumps({
"zope-partition-dict": {
"default": {
"longrequest-logger-interval": 1,
"longrequest-logger-timeout": 1,
},
"multiple": {
"family": "multiple",
"instance-count": 3,
"port-base": 2210,
},
},
"wsgi": cls.wsgi,
})
}
@classmethod
def _setUpClass(cls):
super()._setUpClass()
cls.zope_base_url = cls._getAuthenticatedZopeUrl('')
param_dict = cls.getRootPartitionConnectionParameterDict()
cls.zope_deadlock_debugger_url = cls._getAuthenticatedZopeUrl(
'/manage_debug_threads?{deadlock-debugger-password}'.format(
**param_dict))
# a python script to verify activity processing
cls._addPythonScript(
script_id='ERP5Site_verifyActivityProcessing',
params='mode',
body='''if 1:
import json
portal = context.getPortalObject()
if mode == "count":
return json.dumps(dict(count=len(portal.portal_activities.getMessageList())))
if mode == "activate":
for _ in range(10):
portal.portal_templates.activate(activity="SQLQueue").getTitle()
return "activated"
raise ValueError("Unknown mode: %s" % mode)
''',
)
cls.zope_verify_activity_processing_url = urllib.parse.urljoin(
cls.zope_base_url,
'ERP5Site_verifyActivityProcessing',
)
# a python script logging to event log
cls._addPythonScript(
script_id='ERP5Site_logMessage',
params='name',
body='''if 1:
from erp5.component.module.Log import log
return log("hello %s" % name)
''',
)
cls.zope_log_message_url = urllib.parse.urljoin(
cls.zope_base_url,
'ERP5Site_logMessage',
)
# a python script issuing a long request
cls._addPythonScript(
script_id='ERP5Site_executeLongRequest',
params='',
body='''if 1:
import time
for _ in range(5):
time.sleep(1)
return "done"
''',
)
cls.zope_long_request_url = urllib.parse.urljoin(
cls.zope_base_url,
'ERP5Site_executeLongRequest',
)
def setUp(self):
super().setUp()
# run logrotate a first time so that it create state files
self._executeCrontabAtDate('logrotate', '2000-01-01')
def tearDown(self):
super().tearDown()
# reset logrotate status
logrotate_status = os.path.join(
self.getComputerPartitionPath('zope-default'),
'srv',
'logrotate.status',
)
if os.path.exists(logrotate_status):
os.unlink(logrotate_status)
for logfile in glob.glob(
os.path.join(
self.getComputerPartitionPath('zope-default'),
'srv',
'backup',
'logrotate',
'*',
)):
os.unlink(logfile)
for logfile in glob.glob(
os.path.join(
self.getComputerPartitionPath('zope-default'),
'srv',
'monitor',
'private',
'documents',
'*',
)):
os.unlink(logfile)
def _getCrontabCommand(self, crontab_name):
# type: (str) -> str
"""Read a crontab and return the command that is executed.
overloaded to use crontab from zope partition
"""
with open(
os.path.join(
self.getComputerPartitionPath('zope-default'),
'etc',
'cron.d',
crontab_name,
)) as f:
crontab_spec, = f.readlines()
self.assertNotEqual(crontab_spec[0], '@', crontab_spec)
return crontab_spec.split(None, 5)[-1]
def test_event_log_rotation(self):
requests.get(
self.zope_log_message_url,
params={
"name": "world"
},
verify=False,
).raise_for_status()
zope_event_log_path = os.path.join(
self.getComputerPartitionPath('zope-default'),
'var',
'log',
'zope-0-event.log',
)
with open(zope_event_log_path) as f:
self.assertIn('hello world', f.read())
self._executeCrontabAtDate('logrotate', '2050-01-01')
# this logrotate leaves the log for the day as non compressed
rotated_log_file = os.path.join(
self.getComputerPartitionPath('zope-default'),
'srv',
'backup',
'logrotate',
'zope-0-event.log-20500101',
)
with open(rotated_log_file) as f:
self.assertIn('hello world', f.read())
requests.get(
self.zope_log_message_url,
params={
"name": "le monde"
},
verify=False,
).raise_for_status()
with open(zope_event_log_path) as f:
self.assertNotIn('hello world', f.read())
with open(zope_event_log_path) as f:
self.assertIn('hello le monde', f.read())
# on next day execution of logrotate, log files are compressed
self._executeCrontabAtDate('logrotate', '2050-01-02')
self.assertTrue(os.path.exists(rotated_log_file + '.xz'))
self.assertFalse(os.path.exists(rotated_log_file))
def test_access_log_rotation(self):
requests.get(
self.zope_base_url,
verify=False,
headers={
'User-Agent': 'before rotation'
},
).raise_for_status()
zope_access_log_path = os.path.join(
self.getComputerPartitionPath('zope-default'),
'var',
'log',
'zope-0-Z2.log',
)
with open(zope_access_log_path) as f:
self.assertIn('before rotation', f.read())
self._executeCrontabAtDate('logrotate', '2050-01-01')
# this logrotate leaves the log for the day as non compressed
rotated_log_file = os.path.join(
self.getComputerPartitionPath('zope-default'),
'srv',
'backup',
'logrotate',
'zope-0-Z2.log-20500101',
)
with open(rotated_log_file) as f:
self.assertIn('before rotation', f.read())
requests.get(
self.zope_base_url,
verify=False,
headers={
'User-Agent': 'after rotation'
},
).raise_for_status()
with open(zope_access_log_path) as f:
self.assertNotIn('before rotation', f.read())
with open(zope_access_log_path) as f:
self.assertIn('after rotation', f.read())
# on next day execution of logrotate, log files are compressed
self._executeCrontabAtDate('logrotate', '2050-01-02')
self.assertTrue(os.path.exists(rotated_log_file + '.xz'))
self.assertFalse(os.path.exists(rotated_log_file))
def test_long_request_log_rotation(self):
requests.get(self.zope_long_request_url,
verify=False,
params={
'when': 'before rotation'
}).raise_for_status()
zope_long_request_log_path = os.path.join(
self.getComputerPartitionPath('zope-default'),
'var',
'log',
'longrequest_logger_zope-0.log',
)
with open(zope_long_request_log_path) as f:
self.assertIn('before rotation', f.read())
self._executeCrontabAtDate('logrotate', '2050-01-01')
# this logrotate leaves the log for the day as non compressed
rotated_log_file = os.path.join(
self.getComputerPartitionPath('zope-default'),
'srv',
'backup',
'logrotate',
'longrequest_logger_zope-0.log-20500101',
)
with open(rotated_log_file) as f:
self.assertIn('before rotation', f.read())
requests.get(
self.zope_long_request_url,
verify=False,
params={
'when': 'after rotation'
},
).raise_for_status()
with open(zope_long_request_log_path) as f:
self.assertNotIn('before rotation', f.read())
with open(zope_long_request_log_path) as f:
self.assertIn('after rotation', f.read())
# on next day execution of logrotate, log files are compressed
self._executeCrontabAtDate('logrotate', '2050-01-02')
self.assertTrue(os.path.exists(rotated_log_file + '.xz'))
self.assertFalse(os.path.exists(rotated_log_file))
def test_basic_authentication_user_in_access_log(self):
param_dict = self.getRootPartitionConnectionParameterDict()
requests.get(self.zope_base_url,
verify=False,
auth=requests.auth.HTTPBasicAuth(
param_dict['inituser-login'],
param_dict['inituser-password'],
)).raise_for_status()
zope_access_log_path = os.path.join(
self.getComputerPartitionPath('zope-default'),
'var',
'log',
'zope-0-Z2.log',
)
with open(zope_access_log_path) as f:
self.assertIn(param_dict['inituser-login'], f.read())
def test_deadlock_debugger(self):
dump_response = requests.get(
self.zope_deadlock_debugger_url,
verify=False,
)
dump_response.raise_for_status()
self.assertIn('Thread ', dump_response.text)
def test_activity_processing(self):
def wait_for_activities(max_retries):
for retry in range(max_retries):
time.sleep(10)
resp = requests.get(
self.zope_verify_activity_processing_url,
params={
'mode': 'count',
'retry': retry,
},
verify=False,
)
if not resp.ok:
# XXX we start by flushing existing activities from site creation
# and initial upgrader run. During this time it may happen that
# ERP5 replies with site errors, we tolerate these errors and only
# check the final state.
continue
count = resp.json()['count']
if not count:
break
else:
self.assertEqual(count, 0)
wait_for_activities(60)
requests.get(
self.zope_verify_activity_processing_url,
params={
'mode': 'activate'
},
verify=False,
).raise_for_status()
wait_for_activities(10)
def test_multiple_zope_family_log_files(self):
logfiles = [
os.path.basename(p) for p in glob.glob(
os.path.join(
self.getComputerPartitionPath('zope-multiple'), 'var', 'log', '*'))
]
self.assertEqual(
sorted([l for l in logfiles if l.startswith('zope')]), [
'zope-0-Z2.log',
'zope-0-event.log',
'zope-0-neo-root.log',
'zope-1-Z2.log',
'zope-1-event.log',
'zope-1-neo-root.log',
'zope-2-Z2.log',
'zope-2-event.log',
'zope-2-neo-root.log',
] if '_neo' in self.__class__.__name__ else [
'zope-0-Z2.log',
'zope-0-event.log',
'zope-1-Z2.log',
'zope-1-event.log',
'zope-2-Z2.log',
'zope-2-event.log',
])
class TestZopeMedusa(ZopeTestMixin, ERP5InstanceTestCase):
wsgi = False
class TestZopeWSGI(ZopeTestMixin, ERP5InstanceTestCase):
wsgi = True
@unittest.expectedFailure
def test_long_request_log_rotation(self):
super().test_long_request_log_rotation()
@unittest.expectedFailure
def test_basic_authentication_user_in_access_log(self):
super().test_basic_authentication_user_in_access_log()
class TestZopePublisherTimeout(ZopeSkinsMixin, ERP5InstanceTestCase):
__partition_reference__ = 't'
@classmethod
def getInstanceParameterDict(cls):
return {
'_':
json.dumps({
# a default timeout of 3
"publisher-timeout": 3,
# and a family without timeout
"family-override": {
"no-timeout": {
"publisher-timeout": None,
},
},
"zope-partition-dict": {
# a family to process activities, so that our test
# does not hit a zope node processing activities
"activity": {
"family": "activity",
},
"default": {
"family": "default",
"port-base": 2210,
},
"no-timeout": {
"family": "no-timeout",
"port-base": 22220,
},
},
})
}
@classmethod
def _setUpClass(cls):
super()._setUpClass()
cls._addPythonScript(
'ERP5Site_doSlowRequest',
'',
'''if 1:
import time
def recurse(o):
time.sleep(0.1)
for sub in o.objectValues():
recurse(sub)
recurse(context.getPortalObject())
'''
)
def test_long_request_interupted_on_default_family(self):
ret = requests.get(self._getAuthenticatedZopeUrl(
'ERP5Site_doSlowRequest', family_name='default'), verify=False)
self.assertIn('TimeoutReachedError', ret.text)
self.assertEqual(ret.status_code, requests.codes.server_error)
def test_long_request_not_interupted_on_no_timeout_family(self):
with self.assertRaises(requests.exceptions.Timeout):
requests.get(
self._getAuthenticatedZopeUrl('ERP5Site_doSlowRequest', family_name='no-timeout'),
verify=False,
timeout=6)
class TestCloudooo(ZopeSkinsMixin, ERP5InstanceTestCase):
"""Test ERP5 can be instantiated with cloudooo parameters
"""
__partition_reference__ = 'c'
@classmethod
def getInstanceParameterDict(cls):
return {
'_':
json.dumps({
'cloudooo-url-list': [
'https://cloudooo1.example.com/',
'https://cloudooo2.example.com/',
],
'cloudooo-retry-count': 123,
})
}
def test_cloudooo_url_list_preference(self):
self.assertEqual(
requests.get(
self._getAuthenticatedZopeUrl(
'portal_preferences/getPreferredDocumentConversionServerUrlList'),
verify=False).text,
"['https://cloudooo1.example.com/', 'https://cloudooo2.example.com/']")
@unittest.expectedFailure # setting "retry" is not implemented
def test_cloudooo_retry_count_preference(self):
self.assertEqual(
requests.get(
self._getAuthenticatedZopeUrl(
'portal_preferences/getPreferredDocumentConversionServerRetry'),
verify=False).text,
"123")
class TestCloudoooDefaultParameter(ZopeSkinsMixin, ERP5InstanceTestCase):
"""Test default ERP5 cloudooo parameters
"""
__partition_reference__ = 'cd'
def test_cloudooo_url_list_preference(self):
self.assertIn(
requests.get(
self._getAuthenticatedZopeUrl(
'portal_preferences/getPreferredDocumentConversionServerUrlList'),
verify=False).text,
[
"['https://cloudooo1.erp5.net/', 'https://cloudooo.erp5.net/']",
"['https://cloudooo.erp5.net/', 'https://cloudooo1.erp5.net/']",
])
@unittest.expectedFailure # default value of "retry" does not match schema
def test_cloudooo_retry_count_preference(self):
self.assertEqual(
requests.get(
self._getAuthenticatedZopeUrl(
'portal_preferences/getPreferredDocumentConversionServerRetry'),
verify=False).text,
"2")
software/erp5-zope2/test/test/test_mariadb.py 0 → 100644
View file @ 8eeb74fe
##############################################################################
#
# Copyright (c) 2018 Nexedi SA and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import os
import json
import glob
import urllib.parse
import socket
import sys
import time
import contextlib
import datetime
import subprocess
import gzip
import lzma
import MySQLdb
from slapos.testing.utils import CrontabMixin
from slapos.testing.utils import getPromisePluginParameterDict
from . import ERP5InstanceTestCase
from . import setUpModule
from . import matrix
from . import default
setUpModule # pyflakes
class MariaDBTestCase(ERP5InstanceTestCase):
"""Base test case for mariadb tests.
"""
__partition_reference__ = 'm'
# We explicitly specify 'mariadb' as our software type here,
# therefore we don't request ZODB. We therefore don't
# need to run these tests with both NEO and ZEO mode,
# it wouldn't make any difference.
# https://lab.nexedi.com/nexedi/slapos/blob/273037c8/stack/erp5/instance.cfg.in#L216-230
__test_matrix__ = matrix((default,))
@classmethod
def getInstanceSoftwareType(cls):
return "mariadb"
@classmethod
def _getInstanceParameterDict(cls):
# type: () -> dict
return {
'tcpv4-port': 3306,
'max-connection-count': 5,
'long-query-time': 3,
'max-slowqueries-threshold': 1,
'slowest-query-threshold': 0.1,
# XXX what is this ? should probably not be needed here
'name': cls.__name__,
'monitor-passwd': 'secret',
# XXX should probably not be needed here
'computer-memory-percent-threshold': 100,
}
@classmethod
def getInstanceParameterDict(cls):
# type: () -> dict
return {'_': json.dumps(cls._getInstanceParameterDict())}
def getDatabaseConnection(self):
# type: () -> MySQLdb.connections.Connection
connection_parameter_dict = json.loads(
self.computer_partition.getConnectionParameterDict()['_'])
db_url = urllib.parse.urlparse(connection_parameter_dict['database-list'][0])
self.assertEqual('mysql', db_url.scheme)
self.assertTrue(db_url.path.startswith('/'))
database_name = db_url.path[1:]
return MySQLdb.connect(
user=db_url.username,
passwd=db_url.password,
host=db_url.hostname,
port=db_url.port,
db=database_name,
use_unicode=True,
charset='utf8mb4'
)
class TestCrontabs(MariaDBTestCase, CrontabMixin):
_save_instance_file_pattern_list = \
MariaDBTestCase._save_instance_file_pattern_list + (
'*/srv/backup/*',
)
def test_full_backup(self):
# type: () -> None
self._executeCrontabAtDate('mariadb-backup', '2050-01-01')
full_backup_file, = glob.glob(
os.path.join(
self.computer_partition_root_path,
'srv',
'backup',
'mariadb-full',
'205001010000??.sql.gz',
))
with gzip.open(full_backup_file, 'rt') as dump:
self.assertIn('CREATE TABLE', dump.read())
def test_logrotate_and_slow_query_digest(self):
# type: () -> None
# slow query digest needs to run after logrotate, since it operates on the rotated
# file, so this tests both logrotate and slow query digest.
# run logrotate a first time so that it create state files
self._executeCrontabAtDate('logrotate', '2000-01-01')
# make two slow queries. We are using long-query-time=3, so the queries
# must take more than 3 seconds to be logged.
cnx = self.getDatabaseConnection()
with contextlib.closing(cnx):
cnx.query("SELECT SLEEP(3.1)")
cnx.store_result()
cnx.query("SELECT SLEEP(3.2)")
# slow query crontab depends on crontab for log rotation
# to be executed first.
self._executeCrontabAtDate('logrotate', '2050-01-01')
# this logrotate leaves the log for the day as non compressed
rotated_log_file = os.path.join(
self.computer_partition_root_path,
'srv',
'backup',
'logrotate',
'mariadb_slowquery.log-20500101',
)
self.assertTrue(os.path.exists(rotated_log_file))
# then crontab to generate slow query report is executed
self._executeCrontabAtDate('generate-mariadb-slow-query-report', '2050-01-01')
# and it creates a report for the day
slow_query_report = os.path.join(
self.computer_partition_root_path,
'srv',
'monitor',
'private',
'slowquery_digest',
'slowquery_digest.txt-2050-01-01.xz',
)
with lzma.open(slow_query_report, 'rt') as f:
# this is the hash for our "select sleep(n)" slow query
self.assertIn("ID 0xF9A57DD5A41825CA", f.read())
# on next day execution of logrotate, log files are compressed
self._executeCrontabAtDate('logrotate', '2050-01-02')
self.assertTrue(os.path.exists(rotated_log_file + '.xz'))
self.assertFalse(os.path.exists(rotated_log_file))
# there's a promise checking that the threshold is not exceeded
# and it reports a problem since we set a threshold of 1 slow query
check_slow_query_promise_plugin = getPromisePluginParameterDict(
os.path.join(
self.computer_partition_root_path,
'etc',
'plugin',
'check-slow-query-pt-digest-result.py',
))
with self.assertRaises(subprocess.CalledProcessError) as error_context:
subprocess.check_output(
'faketime 2050-01-01 %s' % check_slow_query_promise_plugin['command'],
text=True,
shell=True)
self.assertEqual(
error_context.exception.output,
"Threshold is lower than expected: \n"
"Expected total queries : 1.0 and current is: 2\n"
"Expected slowest query : 0.1 and current is: 3\n",
)
class TestMariaDB(MariaDBTestCase):
def test_utf8_collation(self):
# type: () -> None
cnx = self.getDatabaseConnection()
with contextlib.closing(cnx):
cnx.query(
"""
CREATE TABLE test_utf8_collation (
col1 CHAR(10)
)
""")
cnx.store_result()
cnx.query(
"""
insert into test_utf8_collation values ("à"), ("あ")
""")
cnx.store_result()
cnx.query(
"""
select * from test_utf8_collation where col1 = "a"
""")
self.assertEqual((('à',),), cnx.store_result().fetch_row(maxrows=2))
class TestMroonga(MariaDBTestCase):
def test_mroonga_plugin_loaded(self):
# type: () -> None
cnx = self.getDatabaseConnection()
with contextlib.closing(cnx):
cnx.query("show plugins")
plugins = cnx.store_result().fetch_row(maxrows=1000)
self.assertIn(
('Mroonga', 'ACTIVE', 'STORAGE ENGINE', 'ha_mroonga.so', 'GPL'),
plugins)
def test_mroonga_normalize_udf(self):
# type: () -> None
# example from https://mroonga.org/docs/reference/udf/mroonga_normalize.html#usage
cnx = self.getDatabaseConnection()
with contextlib.closing(cnx):
cnx.query(
"""
SELECT mroonga_normalize("ABCDあぃうぇ㍑")
""")
# XXX this is returned as bytes by mroonga/mariadb (this might be a bug)
self.assertEqual((('abcdあぃうぇリットル'.encode(),),),
cnx.store_result().fetch_row(maxrows=2))
if 0:
# this example fail with:
# OperationalError: (1123, "Can't initialize function 'mroonga_normalize'; mroonga_normalize(): nonexistent normalizer NormalizerMySQLUnicodeCIExceptKanaCI")
# same error on mroonga "official" docker images using mysql
# https://hub.docker.com/layers/groonga/mroonga/latest/images/sha256-e5a979801c95544ca3a1228d2c4d819820850e0162649553f2e94850e5e1c988?context=explore
# so it's probably OK to ignore
cnx.query(
"""
SELECT mroonga_normalize("aBｃＤあぃウェ㍑", "NormalizerMySQLUnicodeCIExceptKanaCIKanaWithVoicedSoundMark")
""")
self.assertEqual((('ABCDあぃうぇ㍑'.encode(),),),
cnx.store_result().fetch_row(maxrows=2))
def test_mroonga_full_text_normalizer(self):
# type: () -> None
# example from https://mroonga.org//docs/tutorial/storage.html#how-to-specify-the-normalizer
cnx = self.getDatabaseConnection()
with contextlib.closing(cnx):
cnx.query("SET NAMES utf8")
cnx.store_result()
cnx.query(
"""
CREATE TABLE diaries (
day DATE PRIMARY KEY,
content VARCHAR(64) NOT NULL,
FULLTEXT INDEX (content) COMMENT 'normalizer "NormalizerAuto"'
) Engine=Mroonga DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
""")
cnx.store_result()
cnx.query(
"""INSERT INTO diaries VALUES ("2013-04-23", "ブラックコーヒーを飲んだ。")""")
cnx.store_result()
cnx.query(
"""
SELECT *
FROM diaries
WHERE MATCH (content) AGAINST ("+ふらつく" IN BOOLEAN MODE)
""")
self.assertEqual((), cnx.store_result().fetch_row(maxrows=2))
cnx.query(
"""
SELECT *
FROM diaries
WHERE MATCH (content) AGAINST ("+ﾌﾞﾗｯｸ" IN BOOLEAN MODE)
""")
self.assertEqual(
((datetime.date(2013, 4, 23), 'ブラックコーヒーを飲んだ。'),),
cnx.store_result().fetch_row(maxrows=2),
)
def test_mroonga_full_text_normalizer_TokenBigramSplitSymbolAlphaDigit(self):
# type: () -> None
# Similar to as ERP5's testI18NSearch with erp5_full_text_mroonga_catalog
cnx = self.getDatabaseConnection()
with contextlib.closing(cnx):
cnx.query(
"""
CREATE TABLE `full_text` (
`uid` BIGINT UNSIGNED NOT NULL,
`SearchableText` MEDIUMTEXT,
PRIMARY KEY (`uid`),
FULLTEXT `SearchableText` (`SearchableText`) COMMENT 'parser "TokenBigramSplitSymbolAlphaDigit"'
) ENGINE=mroonga
""")
cnx.store_result()
cnx.query(
"""
INSERT INTO full_text VALUES
(1, "Gabriel Fauré Quick brown fox jumps over the lazy dog"),
(2, "武者小路 実篤 Slow white fox jumps over the diligent dog."),
(3, "( - + )")""")
cnx.store_result()
cnx.query(
"""
SELECT uid
FROM full_text
WHERE MATCH (`full_text`.`SearchableText`) AGAINST ('*D+ Faure' IN BOOLEAN MODE)
""")
self.assertEqual(((1,),), cnx.store_result().fetch_row(maxrows=2))
cnx.query(
"""
SELECT uid
FROM full_text
WHERE MATCH (`full_text`.`SearchableText`) AGAINST ('*D+ 武者' IN BOOLEAN MODE)
""")
self.assertEqual(((2,),), cnx.store_result().fetch_row(maxrows=2))
cnx.query(
"""
SELECT uid
FROM full_text
WHERE MATCH (`full_text`.`SearchableText`) AGAINST ('*D+ +quick +fox +dog' IN BOOLEAN MODE)
""")
self.assertEqual(((1,),), cnx.store_result().fetch_row(maxrows=2))
def test_mroonga_full_text_stem(self):
# type: () -> None
# example from https://mroonga.org//docs/tutorial/storage.html#how-to-specify-the-token-filters
cnx = self.getDatabaseConnection()
with contextlib.closing(cnx):
cnx.query("SELECT mroonga_command('register token_filters/stem')")
self.assertEqual(((b'true',),), cnx.store_result().fetch_row(maxrows=2))
cnx.query(
"""
CREATE TABLE memos (
id INT NOT NULL PRIMARY KEY,
content TEXT NOT NULL,
FULLTEXT INDEX (content) COMMENT 'normalizer "NormalizerAuto", token_filters "TokenFilterStem"'
) Engine=Mroonga DEFAULT CHARSET=utf8
""")
cnx.store_result()
cnx.query(
"""INSERT INTO memos VALUES (1, "I develop Groonga"), (2, "I'm developing Groonga"), (3, "I developed Groonga")"""
)
cnx.store_result()
cnx.query(
"""
SELECT *
FROM memos
WHERE MATCH (content) AGAINST ("+develops" IN BOOLEAN MODE)
""")
self.assertEqual([
(1, "I develop Groonga"),
(2, "I'm developing Groonga"),
(3, "I developed Groonga"),
], list(sorted(cnx.store_result().fetch_row(maxrows=4))))
software/erp5-zope2/test/test/test_wcfs.py 0 → 100644
View file @ 8eeb74fe
# Copyright (C) 2022 Nexedi SA and Contributors.
#
# This program is free software: you can Use, Study, Modify and Redistribute
# it under the terms of the GNU General Public License version 3, or (at your
# option) any later version, as published by the Free Software Foundation.
#
# You can also Link and Combine this program with other software covered by
# the terms of any of the Free Software licenses or any of the Open Source
# Initiative approved licenses and Convey the resulting work. Corresponding
# source of such a combination shall include the source code for all other
# software used.
#
# This program is distributed WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#
# See COPYING file for full licensing terms.
# See https://www.nexedi.com/licensing for rationale and options.
import json
import os.path
import unittest
from slapos.grid.utils import md5digest
from . import ERP5InstanceTestCase, setUpModule as _setUpModule
from .test_erp5 import TestPublishedURLIsReachableMixin
# skip tests when software release is built with wendelin.core 1.
def setUpModule():
_setUpModule()
cls = ERP5InstanceTestCase
if not os.path.exists(
os.path.join(
cls.slap.software_directory,
md5digest(cls.getSoftwareURL()),
'bin', 'wcfs')):
raise unittest.SkipTest("built with wendelin.core 1")
class TestWCFS(ERP5InstanceTestCase, TestPublishedURLIsReachableMixin):
"""Test Wendelin Core File System
"""
__partition_reference__ = 'wcfs'
# Only run in ZEO mode; don't run with NEO.
# Current NEO/py and NEO/go versions have interoperability
# issues. Once these issues are fixed the following
# lines have to be removed so that test case runs agains NEO.
# Please see the following MR for more context:
# https://lab.nexedi.com/nexedi/slapos/merge_requests/1283#note_174854
@classmethod
def setUpClass(cls):
if json.loads(cls.getInstanceParameterDict()["_"])['zodb'][0]["type"] == "neo":
raise unittest.SkipTest("Not yet fixed WCFS+NEO interoperability issue.")
super().setUpClass()
@classmethod
def getInstanceParameterDict(cls):
return {'_': json.dumps({'wcfs': {'enable': True}})}
def test_wcfs_accessible(self):
"""Verify that wcfs filesystem is basically accessible.
- we can read .wcfs/zurl
- its content is equal to published `serving-zurl`
"""
zurl = json.loads(
self.getComputerPartition('wcfs').getConnectionParameter('_')
)['serving-zurl']
mntpt = lookupMount(zurl)
zurl_ = readfile("%s/.wcfs/zurl" % mntpt)
self.assertEqual(zurl_, zurl)
# lookupMount returns /proc/mount entry for wcfs mounted to serve zurl.
def lookupMount(zurl):
for line in readfile('/proc/mounts').splitlines():
# <zurl> <mountpoint> fuse.wcfs ...
zurl_, mntpt, typ, _ = line.split(None, 3)
if typ != 'fuse.wcfs':
continue
if zurl_ == zurl:
return mntpt
raise KeyError("lookup mount %s: no /proc/mounts entry" % zurl)
# readfile returns content of file @path.
def readfile(path):
with open(path) as f:
return f.read()
software/erp5-zope2/upgrade_test/README.md 0 → 100644
View file @ 8eeb74fe
Upgrade tests for ERP5 software release
software/erp5-zope2/upgrade_test/setup.py 0 → 100644
View file @ 8eeb74fe
##############################################################################
#
# Copyright (c) 2020 Nexedi SA and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
from setuptools import setup, find_packages
version = '0.0.1.dev0'
name = 'slapos.test.upgrade_erp5'
with open("README.md") as f:
long_description = f.read()
setup(name=name,
version=version,
description="Upgrade test for SlapOS' ERP5 software release",
long_description=long_description,
long_description_content_type='text/markdown',
maintainer="Nexedi",
maintainer_email="info@nexedi.com",
url="https://lab.nexedi.com/nexedi/slapos",
packages=find_packages(),
install_requires=[
'slapos.core',
'supervisor',
'slapos.libnetworkcache',
],
test_suite='test',
)
software/erp5-zope2/upgrade_test/test.py 0 → 100644
View file @ 8eeb74fe
##############################################################################
#
# Copyright (c) 2020 Nexedi SA and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import contextlib
import glob
import json
import os
import ssl
import sys
import tempfile
import time
import requests
import urllib.parse
import xmlrpc.client
import urllib3
from slapos.grid.utils import md5digest
from slapos.testing.testcase import (
SlapOSNodeCommandError,
installSoftwareUrlList,
makeModuleSetUpAndTestCaseClass,
)
old_software_release_url = 'https://lab.nexedi.com/nexedi/slapos/raw/1.0.167.8/software/erp5/software.cfg'
new_software_release_url = os.path.abspath(
os.path.join(os.path.dirname(__file__), '..', 'software.cfg'))
_, SlapOSInstanceTestCase = makeModuleSetUpAndTestCaseClass(
old_software_release_url,
software_id="upgrade_erp5",
skip_software_check=True,
)
def setUpModule():
installSoftwareUrlList(
SlapOSInstanceTestCase,
[old_software_release_url, new_software_release_url],
debug=SlapOSInstanceTestCase._debug,
)
class ERP5UpgradeTestCase(SlapOSInstanceTestCase):
# use short partition names for unix sockets
__partition_reference__ = 'u'
@classmethod
def setUpOldInstance(cls):
"""setUp hook executed while to old instance is running, before update
"""
pass
_current_software_url = old_software_release_url
@classmethod
def getSoftwareURL(cls):
return cls._current_software_url
@classmethod
def setUpClass(cls):
# request and instantiate with old software url
super().setUpClass()
cls.setUpOldInstance()
# request instance on new software
cls._current_software_url = new_software_release_url
cls.logger.debug('requesting instance on new software')
cls.requestDefaultInstance()
# wait for slapos node instance
snapshot_name = "{}.{}.setUpClass new instance".format(
cls.__module__, cls.__name__)
try:
if cls._debug and cls.instance_max_retry:
try:
cls.slap.waitForInstance(max_retry=cls.instance_max_retry - 1)
except SlapOSNodeCommandError:
cls.slap.waitForInstance(debug=True)
else:
cls.slap.waitForInstance(
max_retry=cls.instance_max_retry, debug=cls._debug)
cls.logger.debug("instance on new software done")
except BaseException:
cls.logger.exception("Error during instance on new software")
cls._storeSystemSnapshot(snapshot_name)
cls._cleanup(snapshot_name)
cls.setUp = lambda self: self.fail('Setup Class failed.')
raise
else:
cls._storeSystemSnapshot(snapshot_name)
cls.computer_partition = cls.requestDefaultInstance()
class TestERP5Upgrade(ERP5UpgradeTestCase):
@classmethod
def tearDownClass(cls):
cls.session.close()
super().tearDownClass()
@classmethod
def setUpOldInstance(cls):
cls._default_instance_old_parameter_dict = param_dict = json.loads(
cls.computer_partition.getConnectionParameterDict()['_'])
# use a session to retry on failures, when ERP5 is not ready.
# (see also TestPublishedURLIsReachableMixin)
cls.session = requests.Session()
cls.session.mount(
param_dict['family-default-v6'],
requests.adapters.HTTPAdapter(
max_retries=urllib3.util.retry.Retry(
total=20,
backoff_factor=.1,
status_forcelist=(404, 500, 503),
)))
# rebuild an url with user and password
parsed = urllib.parse.urlparse(param_dict['family-default'])
cls.authenticated_zope_base_url = parsed._replace(
netloc='{}:{}@{}:{}'.format(
param_dict['inituser-login'],
param_dict['inituser-password'],
parsed.hostname,
parsed.port,
),
path=param_dict['site-id'] + '/',
).geturl()
cls.zope_base_url = '{family_default_v6}/{site_id}'.format(
family_default_v6=param_dict['family-default-v6'],
site_id=param_dict['site-id'],
)
# wait for old site creation
cls.session.get(
f'{cls.zope_base_url}/person_module',
auth=requests.auth.HTTPBasicAuth(
username=param_dict['inituser-login'],
password=param_dict['inituser-password'],
),
verify=False,
allow_redirects=False,
).raise_for_status()
# Create scripts to create test data and search catalog for test data.
@contextlib.contextmanager
def getXMLRPCClient():
# don't verify certificate
ssl_context = ssl.create_default_context()
ssl_context.check_hostname = False
ssl_context.verify_mode = ssl.CERT_NONE
erp5_xmlrpc_client = xmlrpc.client.ServerProxy(
cls.authenticated_zope_base_url,
context=ssl_context,
)
with erp5_xmlrpc_client:
yield erp5_xmlrpc_client
def addPythonScript(script_id, params, body):
with getXMLRPCClient() as erp5_xmlrpc_client:
custom = erp5_xmlrpc_client.portal_skins.custom
try:
custom.manage_addProduct.PythonScripts.manage_addPythonScript(
script_id)
except xmlrpc.client.ProtocolError as e:
if e.errcode != 302:
raise
getattr(custom, script_id).ZPythonScriptHTML_editAction(
'',
'',
params,
body,
)
# a python script to create a person with a name
addPythonScript(
script_id='ERP5Site_createTestPerson',
params='name',
body='''if 1:
portal = context.getPortalObject()
portal.person_module.newContent(
first_name=name,
)
return 'Done.'
''',
)
# a python script to search for persons by name
addPythonScript(
script_id='ERP5Site_searchTestPerson',
params='name',
body='''if 1:
import json
portal = context.getPortalObject()
result = [brain.getObject().getTitle() for brain in portal.portal_catalog(
portal_type='Person',
title=name,)]
assert result # raise so that we retry until indexed
return json.dumps(result)
''',
)
cls.session.post(
'{zope_base_url}/ERP5Site_createTestPerson'.format(
zope_base_url=cls.zope_base_url),
auth=requests.auth.HTTPBasicAuth(
username=param_dict['inituser-login'],
password=param_dict['inituser-password'],
),
data={
'name': 'before upgrade'
},
verify=False,
allow_redirects=False,
).raise_for_status()
assert cls.session.get(
'{zope_base_url}/ERP5Site_searchTestPerson'.format(
zope_base_url=cls.zope_base_url),
auth=requests.auth.HTTPBasicAuth(
username=param_dict['inituser-login'],
password=param_dict['inituser-password'],
),
params={
'name': 'before upgrade'
},
verify=False,
allow_redirects=False,
).json() == ['before upgrade']
def test_published_url_is_same(self):
default_instance_new_parameter_dict = json.loads(
self.computer_partition.getConnectionParameterDict()['_'])
self.assertEqual(
default_instance_new_parameter_dict['family-default-v6'],
self._default_instance_old_parameter_dict['family-default-v6'],
)
def test_published_url_is_reachable(self):
default_instance_new_parameter_dict = json.loads(
self.computer_partition.getConnectionParameterDict()['_'])
# get certificate from caucase
with tempfile.NamedTemporaryFile(
prefix="ca.crt.pem",
mode="w",
delete=False,
) as ca_cert:
ca_cert.write(
requests.get(
urllib.parse.urljoin(
default_instance_new_parameter_dict['caucase-http-url'],
'/cas/crt/ca.crt.pem',
)).text)
ca_cert.flush()
self.session.get(
'{}/{}/login_form'.format(
default_instance_new_parameter_dict['family-default-v6'],
default_instance_new_parameter_dict['site-id'],
),
verify=False,
# TODO: we don't use caucase yet here.
# verify=ca_cert.name,
).raise_for_status()
def test_all_instances_use_new_software_release(self):
self.assertEqual(
{
os.path.basename(os.readlink(sr))
for sr in glob.glob(
os.path.join(
self.slap.instance_directory,
'*',
'software_release',
))
},
{md5digest(self.getSoftwareURL())},
)
def test_catalog_available(self):
param_dict = json.loads(
self.computer_partition.getConnectionParameterDict()['_'])
# data created before upgrade is available
self.assertEqual(
self.session.get(
'{zope_base_url}/ERP5Site_searchTestPerson'.format(
zope_base_url=self.zope_base_url),
auth=requests.auth.HTTPBasicAuth(
username=param_dict['inituser-login'],
password=param_dict['inituser-password'],
),
params={
'name': 'before upgrade'
},
verify=False,
allow_redirects=False,
).json(), ['before upgrade'])
# create data after upgrade
self.session.post(
'{zope_base_url}/ERP5Site_createTestPerson'.format(
zope_base_url=self.zope_base_url),
auth=requests.auth.HTTPBasicAuth(
username=param_dict['inituser-login'],
password=param_dict['inituser-password'],
),
data={
'name': 'after upgrade'
},
verify=False,
allow_redirects=False,
).raise_for_status()
# new data can also be found
self.assertEqual(
self.session.get(
'{zope_base_url}/ERP5Site_searchTestPerson'.format(
zope_base_url=self.zope_base_url),
auth=requests.auth.HTTPBasicAuth(
username=param_dict['inituser-login'],
password=param_dict['inituser-password'],
),
params={
'name': 'after upgrade'
},
verify=False,
allow_redirects=False,
).json(), ['after upgrade'])
stack/erp5-zope2/TODO 0 → 100644
View file @ 8eeb74fe
(in no special order)
General:
- ipv6 support (besides frontend-backend apache connection)
requires important changes at ERP5 level
- resilience
- mariadb: make x509 mandatory (needs ZMySQLD*A support)
- make postfix log inside partition
- document postfix parameters (only once it actually works)
Backups:
- flush binlogs independently from full backups (in addition to anyway flushing them on full backup creation)
- rotate tidstorage consistency points
- make mysql backup path an instance parameter
- make srv/backup/zodb the default value for a parameter (zodb{ 'backup_root': ...} or so) to have a single value to modify to relocate zodb backups of a partition
- make srv/backup/logrotate customisable (per partition, otherwise files will overwrite each other)
stack/erp5-zope2/buildout.cfg 0 → 100644
View file @ 8eeb74fe
[buildout]
extends =
# Exact version of Zope
ztk-versions.cfg
zope-versions.cfg
buildout.hash.cfg
../../component/fonts/buildout.cfg
../../component/git/buildout.cfg
../../component/ghostscript/buildout.cfg
../../component/graphviz/buildout.cfg
../../component/gzip/buildout.cfg
../../component/xz-utils/buildout.cfg
../../component/haproxy/buildout.cfg
../../component/socat/buildout.cfg
../../component/rsyslogd/buildout.cfg
../../component/findutils/buildout.cfg
../../component/librsvg/buildout.cfg
../../component/imagemagick/buildout.cfg
../../component/jpegoptim/buildout.cfg
../../component/kumo/buildout.cfg
../../component/libdmtx/buildout.cfg
../../component/matplotlib/buildout.cfg
../../component/numpy/buildout.cfg
../../component/statsmodels/buildout.cfg
../../component/h5py/buildout.cfg
../../component/ocropy/buildout.cfg
../../component/optipng/buildout.cfg
../../component/pandas/buildout.cfg
../../component/percona-toolkit/buildout.cfg
../../component/patch/buildout.cfg
../../component/pillow/buildout.cfg
../../component/pycrypto-python/buildout.cfg
../../component/pysvn-python/buildout.cfg
../../component/python-ldap-python/buildout.cfg
../../component/scikit-learn/buildout.cfg
../../component/scikit-image/buildout.cfg
../../component/PyWavelets/buildout.cfg
../../component/subversion/buildout.cfg
../../component/tempstorage/buildout.cfg
../../component/tesseract/buildout.cfg
../../component/w3m/buildout.cfg
../../component/poppler/buildout.cfg
../../component/zabbix/buildout.cfg
../../component/sed/buildout.cfg
../../component/coreutils/buildout.cfg
../../component/grep/buildout.cfg
../../component/dash/buildout.cfg
../../component/bash/buildout.cfg
../../component/wget/buildout.cfg
../../component/aspell/buildout.cfg
../../component/jsl/buildout.cfg
../../component/6tunnel/buildout.cfg
../../component/userhosts/buildout.cfg
../../component/postfix/buildout.cfg
../../component/zbarlight/buildout.cfg
../../component/pylint/buildout.cfg
../../component/perl-Image-ExifTool/buildout.cfg
../../component/wendelin.core/buildout.cfg
../../component/jupyter-py2/buildout.cfg
../../component/pygolang/buildout.cfg
../../component/bcrypt/buildout.cfg
../../component/python-pynacl/buildout.cfg
../../component/python-xmlsec/buildout.cfg
../../component/selenium/buildout.cfg
../../stack/caucase/buildout.cfg
../../software/neoppod/software-common.cfg
# keep neoppod extends last
parts +=
erp5-util-develop
slapos-cookbook
mroonga-mariadb
tesseract
zabbix-agent
.coveragerc
# Buildoutish
eggs-all-scripts
testrunner
test-suite-runner
# get git repositories
genbt5list
# some additional utils
zodbpack
# Create instance template
template
# jupyter
jupyter-notebook-initialized-scripts
# override instance-jupyter-notebook not to render into default template.cfg
[instance-jupyter-notebook]
output = ${buildout:directory}/template-jupyter.cfg
[download-base]
<= download-base-neo
url = ${:_profile_base_location_}/${:filename}
[mariadb-start-clone-from-backup]
<= download-base
[mariadb-resiliency-after-import-script]
<= download-base
[mariadb-slow-query-report-script]
<= download-base
[template-mariadb]
<= download-base
link-binary =
${coreutils:location}/bin/basename
${coreutils:location}/bin/cat
${coreutils:location}/bin/cp
${coreutils:location}/bin/ls
${coreutils:location}/bin/tr
${coreutils:location}/bin/uname
${gettext:location}/lib/gettext/hostname
${grep:location}/bin/grep
${sed:location}/bin/sed
${mariadb:location}/bin/mysqlbinlog
[template-kumofs]
<= download-base
[template-zope-conf]
<= download-base
[site-zcml]
<= download-base
[template-my-cnf]
<= download-base
[template-mariadb-initial-setup]
<= download-base
[template-postfix]
< = download-base
[template-postfix-master-cf]
< = download-base
[template-postfix-main-cf]
< = download-base
[template-postfix-aliases]
< = download-base
[template-run-zelenium]
< = download-base
[template]
recipe = slapos.recipe.template:jinja2
# XXX: "template.cfg" is hardcoded in instanciation recipe
output = ${buildout:directory}/template.cfg
url = ${:_profile_base_location_}/${:filename}
context =
key mariadb_link_binary template-mariadb:link-binary
key zope_link_binary template-zope:link-binary
key zope_fonts template-zope:fonts
key zope_fontconfig_includes template-zope:fontconfig-includes
key apache_location apache:location
key bin_directory buildout:bin-directory
key buildout_bin_directory buildout:bin-directory
key caucase_jinja2_library caucase-jinja2-library:target
key coreutils_location coreutils:location
key curl_location curl:location
key cyrus_sasl_location cyrus-sasl:location
key dash_location dash:location
key bash_location bash:location
key dcron_location dcron:location
key default_cloudooo_url_list erp5-defaults:cloudooo-connection-url-list
key erp5_location erp5:location
key findutils_location findutils:location
key gzip_location gzip:location
key xz_utils_location xz-utils:location
key haproxy_location haproxy:location
key socat_location socat:location
key rsyslogd_location rsyslogd:location
key instance_common_cfg instance-common:output
key jupyter_enable_default erp5-defaults:jupyter-enable-default
key wcfs_enable_default erp5-defaults:wcfs-enable-default
key kumo_location kumo:location
key local_bt5_repository local-bt5-repository:list
key logrotate_location logrotate:location
key mariadb_location mariadb:location
key mariadb_resiliency_after_import_script mariadb-resiliency-after-import-script:target
key mariadb_slow_query_report_script mariadb-slow-query-report-script:target
key mariadb_start_clone_from_backup mariadb-start-clone-from-backup:target
key mroonga_mariadb_install_sql mroonga-mariadb:install-sql
key mroonga_mariadb_plugin_dir mroonga-mariadb:plugin-dir
key groonga_plugin_dir groonga:groonga-plugin-dir
key groonga_mysql_normalizer_plugin_dir groonga-normalizer-mysql:groonga-plugin-dir
key matplotlibrc_location matplotlibrc:location
key parts_directory buildout:parts-directory
key openssl_location openssl:location
key percona_toolkit_location percona-toolkit:location
key perl_dbd_mariadb_path perl-DBD-mariadb:perl-PATH
key postfix_location postfix:location
key root_common root-common:target
key site_zcml site-zcml:target
key sixtunnel_location 6tunnel:location
key template_run_zelenium template-run-zelenium:target
key egg_interpreter erp5-python-interpreter:interpreter
key template_apache_conf template-apache-backend-conf:target
key template_balancer template-balancer:target
key template_erp5 template-erp5:target
key template_haproxy_cfg template-haproxy-cfg:target
key template_rsyslogd_cfg template-rsyslogd-cfg:target
key template_jupyter_cfg instance-jupyter-notebook:output
key template_kumofs template-kumofs:target
key template_mariadb template-mariadb:target
key template_mariadb_initial_setup template-mariadb-initial-setup:target
key template_my_cnf template-my-cnf:target
key template_mysqld_wrapper template-mysqld-wrapper:output
key template_postfix template-postfix:target
key template_postfix_aliases template-postfix-aliases:target
key template_postfix_main_cf template-postfix-main-cf:target
key template_postfix_master_cf template-postfix-master-cf:target
key instance_wcfs_cfg_in instance-wcfs.cfg.in:target
key template_zeo template-zeo:target
key template_zodb_base template-zodb-base:target
key template_zope template-zope:target
key template_zope_conf template-zope-conf:target
key template_fonts_conf template-fonts-conf:output
key userhosts_location userhosts:location
key unixodbc_location unixodbc:location
key wget_location wget:location
key extra_path_list eggs:extra-paths
key python_executable_for_kernel erp5-python-interpreter-jupyter:interpreter_path
key erp5_kernel_location erp5-kernel:location
key erp5_kernel_filename erp5-kernel:filename
key kernel_json_location kernel-json:location
key kernel_json_filename kernel-json:filename
[template-erp5]
<= download-base
[template-zeo]
<= download-base
[template-zodb-base]
<= download-base
[template-zope]
<= download-base
link-binary =
${aspell-en-dictionary:bin-aspell}
${dmtx-utils:location}/bin/dmtxwrite
${git:location}/bin/git
${graphviz:location}/bin/dot
${grep:location}/bin/grep
${imagemagick:location}/bin/convert
${ghostscript:location}/bin/gs
${imagemagick:location}/bin/identify
${jpegoptim:location}/bin/jpegoptim
${jsl:location}/bin/jsl
${librsvg:location}/bin/rsvg-convert
${mariadb:location}/bin/mysql
${mariadb:location}/bin/mysqldump
${openssl:location}/bin/openssl
${optipng:location}/bin/optipng
${perl-Image-ExifTool:location}/bin/exiftool
${poppler:location}/bin/pdfinfo
${poppler:location}/bin/pdftohtml
${poppler:location}/bin/pdftotext
${python2.7:location}/bin/2to3
${sed:location}/bin/sed
${tesseract:location}/bin/tesseract
${w3m:location}/bin/w3m
fonts =
${liberation-fonts:location}
${ipaex-fonts:location}
${ipa-fonts:location}
${ocrb-fonts:location}
${android-fonts:location}
fontconfig-includes =
${fontconfig:location}/etc/fonts/conf.d
[template-balancer]
<= download-base
[template-haproxy-cfg]
<= download-base
[template-rsyslogd-cfg]
<= download-base
[instance-wcfs.cfg.in]
<= download-base
[erp5-bin]
<= erp5
repository = https://lab.nexedi.com/nexedi/erp5-bin.git
branch = master
[erp5-doc]
<= erp5
repository = https://lab.nexedi.com/nexedi/erp5-doc.git
branch = master
[bt5-repository]
# Format:
# <url or path> [...]
#
# Use absolute paths for local repositories, and URLs for non-local otherwise.
#
list = ${local-bt5-repository:list}
[local-bt5-repository]
# Same as bt5-repository, but only local repository.
# Used to generate bt5lists.
list = ${erp5:location}/bt5 ${erp5:location}/product/ERP5/bootstrap ${erp5-bin:location}/bt5 ${erp5-doc:location}/bt5
[genbt5list]
recipe = plone.recipe.command
stop-on-error = true
genbt5list = ${erp5:location}/product/ERP5/bin/genbt5list
command =
echo '${local-bt5-repository:list}' |xargs ${buildout:executable} ${:genbt5list}
update-command = ${:command}
[erp5_repository_list]
repository_id_list = erp5 erp5-bin erp5-doc
# ERP5 defaults, which can be overridden in inheriting recipes (e.g. wendelin)
[erp5-defaults]
# Default cloudooo is https://cloudooo.erp5.net/ and https://cloudooo1.erp5.net/ in random order.
# The random order is applied in instance profile
cloudooo-connection-url-list =
https://cloudooo.erp5.net/
https://cloudooo1.erp5.net/
# Jupyter is by default disabled in ERP5
jupyter-enable-default = false
# WCFS is by default disabled in ERP5
wcfs-enable-default = false
[erp5]
recipe = slapos.recipe.build:gitclone
repository = https://lab.nexedi.com/nexedi/erp5.git
branch = master
git-executable = ${git:location}/bin/git
[testrunner]
# XXX: Workaround for fact ERP5Type is not an distribution and does not
# expose entry point for test runner
recipe = zc.recipe.egg
eggs = ${eggs:eggs}
extra-paths = ${eggs:extra-paths}
entry-points =
runUnitTest=runUnitTest:main
scripts = runUnitTest
initialization =
import glob, os, sys, json
buildout_directory = '''${buildout:directory}'''
parts_directory = '''${buildout:parts-directory}'''
repository_id_list = \
'''${erp5_repository_list:repository_id_list}'''.split()[::-1]
# read testrunner configuration from slapos instance parameters to
# configure coverage if enabled.
with open(os.environ['ERP5_TEST_RUNNER_CONFIGURATION']) as f:
test_runner_configuration = json.load(f)
test_runner_configuration.setdefault('coverage', {})
test_runner_configuration['coverage'].setdefault('enabled', False)
coverage_process = None
if test_runner_configuration['coverage']['enabled']:
test_runner_configuration['coverage'].setdefault(
'include', [os.path.join('parts', repo, '*') for repo in repository_id_list])
assets_directory = ''
test_name = sys.argv[-1].replace(':', '_')
if os.environ.get('SLAPOS_TEST_LOG_DIRECTORY'):
assets_directory = os.path.join(os.environ['SLAPOS_TEST_LOG_DIRECTORY'], test_name)
if not os.path.exists(assets_directory):
os.makedirs(assets_directory)
coverage_data_file = os.path.abspath(
os.path.join(assets_directory, 'coverage.sqlite3'))
curdir = os.path.abspath(os.curdir)
# change current directory when importing coverage so that it considers paths
# relative to the root of the software
os.chdir(buildout_directory)
import coverage
coverage_process = coverage.Coverage(
include=test_runner_configuration['coverage']['include'],
data_file=coverage_data_file,
branch=test_runner_configuration['coverage'].get('branch'),
)
coverage_process.set_option('run:relative_files', 'true')
coverage_process.set_option('run:plugins', ['erp5_coverage_plugin'])
coverage_process.start()
os.chdir(curdir)
import Products
Products.__path__[:0] = filter(None,
os.getenv('INSERT_PRODUCTS_PATH', '').split(os.pathsep))
os.environ['ZOPE_SCRIPTS'] = ''
os.environ['erp5_tests_bt5_path'] = ','.join(
os.path.join(parts_directory, x, 'bt5') for x in repository_id_list)
extra_path_list = '''${:extra-paths}'''.split()
sys.path[:0] = sum((
glob.glob(os.path.join(x, 'tests'))
for x in extra_path_list), [])
sys.path[:0] = sum((
glob.glob(os.path.join(x, 'Products', '*', 'tests'))
for x in extra_path_list), [])
sys.path[:0] = sum((
glob.glob(os.path.join(x, 'Products', '*', 'tests'))
for x in os.getenv('INSERT_PRODUCTS_PATH', '').split(os.pathsep)), [])
import runUnitTest
try:
sys.exit(runUnitTest.main())
finally:
if coverage_process:
coverage_process.stop()
coverage_process.save()
# upload the coverage so that they can be combined from another machine
upload_url = test_runner_configuration['coverage'].get('upload-url')
if upload_url:
import requests
import time
import uritemplate
from six.moves.urllib.parse import urlparse
auth_list = (None, )
parsed_url = urlparse(upload_url)
if parsed_url.username:
# try Digest and Basic authentication and retry 5 times to tolerate transiant errors
auth_list = (
requests.auth.HTTPDigestAuth(parsed_url.username, parsed_url.password),
requests.auth.HTTPBasicAuth(parsed_url.username, parsed_url.password),
) * 5
url = uritemplate.URITemplate(upload_url).expand(
test_name=test_name,
# Environment variables are set in parts/erp5/product/ERP5Type/tests/runTestSuite.py
test_result_id=os.environ.get('ERP5_TEST_RESULT_ID', 'unknown_test_result_id'),
test_result_revision=os.environ.get('ERP5_TEST_RESULT_REVISION', 'unknown_test_result_revision'),
)
for auth in auth_list:
with open(coverage_data_file, 'rb') as f:
resp = requests.put(url, data=f, auth=auth)
if resp.ok:
# print just the hostname, not to include the auth part
print('Uploaded coverage data to {parsed_url.hostname}'.format(parsed_url=parsed_url))
break
print('Error {resp.status_code} uploading coverage data to {parsed_url.hostname} with {auth.__class__.__name__}'.format(
resp=resp, parsed_url=parsed_url, auth=auth))
time.sleep(1)
else:
sys.stderr.write('Error uploading coverage data to {parsed_url.hostname}\n'.format(parsed_url=parsed_url))
[.coveragerc]
recipe = slapos.recipe.template
output = ${buildout:directory}/${:_buildout_section_name_}
inline =
# coverage configuration file, useful when making html report
[run]
plugins =
erp5_coverage_plugin
relative_files = true
[test-suite-runner]
# XXX: Workaround for fact ERP5Type is not an distribution and does not
# expose entry point for test runner
recipe = zc.recipe.egg
eggs = ${eggs:eggs}
extra-paths = ${eggs:extra-paths}
entry-points =
runTestSuite=Products.ERP5Type.tests.runTestSuite:main
scripts = runTestSuite
initialization =
import os
import sys
import Products
[Products.__path__.insert(0, p) for p in reversed(os.environ.get('INSERT_PRODUCTS_PATH', '').split(':')) if p]
os.environ['ZOPE_SCRIPTS'] = ''
repository_id_list = list(reversed('''${erp5_repository_list:repository_id_list}'''.split()))
sys.path[0:0] = ['/'.join(['''${buildout:parts-directory}''', x]) for x in repository_id_list]
[erp5-python-interpreter]
<= python-interpreter
# a python interpreter with all eggs available, usable for the software release but also
# for external tools (such as python extension in theia).
eggs += ${eggs:eggs}
extra-paths += ${eggs:extra-paths}
[erp5-python-interpreter-jupyter]
<= erp5-python-interpreter
interpreter = pythonwitheggs_jupyter
interpreter_path = ${buildout:directory}/bin/${:interpreter}
eggs +=
jupyter_client
jupyter_core
ipython_genutils
ipykernel
ipywidgets
requests
[zope-product-with-eggtestinfo]
recipe = zc.recipe.egg:custom
setup-eggs =
eggtestinfo
egg = ${:_buildout_section_name_}
[Products.CMFUid]
<= zope-product-with-eggtestinfo
[Products.CMFActionIcons]
<= zope-product-with-eggtestinfo
[Products.CMFCalendar]
<= zope-product-with-eggtestinfo
[Products.CMFCore]
<= zope-product-with-eggtestinfo
[Products.CMFDefault]
<= zope-product-with-eggtestinfo
[Products.CMFTopic]
<= zope-product-with-eggtestinfo
[Products.DCWorkflow]
<= zope-product-with-eggtestinfo
Products.DCWorkflow-patches = ${:_profile_base_location_}/../../component/egg-patch/Products.DCWorkflow/workflow_method.patch#975b49e96bae33ac8563454fe5fa9899
Products.DCWorkflow-patch-options = -p1
[Products.GenericSetup]
<= zope-product-with-eggtestinfo
[eggs]
<= neoppod
eggs = ${neoppod:eggs}
${caucase-eggs:eggs}
${wendelin.core:egg}
${numpy:egg}
${matplotlib:egg}
${lxml-python:egg}
${ocropy:egg}
${pandas:egg}
${pillow-python:egg}
${python-ldap-python:egg}
${python-xmlsec:egg}
${pysvn-python:egg}
${pycrypto-python:egg}
${scipy:egg}
${scikit-learn:egg}
${scikit-image:egg}
sympy
more-itertools
${h5py:egg}
openpyxl
${statsmodels:egg}
${zbarlight:egg}
lock_file
astor
APacheDEX
PyStemmer
Pympler
SOAPpy
chardet
collective.recipe.template
erp5diff
interval
ipdb
Jinja2
jsonschema
mechanize
mock
oauthlib
objgraph
${python-pynacl:egg}
${bcrypt:egg}
paramiko
ply
pyflakes
PyPDF2
python-magic
python-memcached
pytz
requests
responses
urlnorm
uuid
xml_marshaller
xupdate_processor
feedparser
validictory
erp5.util
z3c.etestbrowser
huBarcode
qrcode
spyne
httplib2
suds
pprofile
pycountry
xfw
jsonschema
${selenium:egg}
pytesseract
decorator
networkx
# Needed for checking ZODB Components source code
${astroid:egg}
${pylint:egg}
jedi
yapf
typing
pytracemalloc
xlrd
# Zope
Zope2
${tempstorage:egg}
# Zope acquisition patch
Acquisition
# for runzeo
${ZEO:egg}
# Other Zope 2 packages
Products.PluggableAuthService
Products.PluginRegistry
# CMF 2.2
${Products.CMFActionIcons:egg}
${Products.CMFCalendar:egg}
${Products.CMFCore:egg}
${Products.CMFDefault:egg}
${Products.CMFTopic:egg}
${Products.CMFUid:egg}
${Products.DCWorkflow:egg}
${Products.GenericSetup:egg}
five.localsitemanager
# Other products
Products.MimetypesRegistry
Products.TIDStorage
Products.LongRequestLogger
# BBB: Temporarily keep zope.app.testing awaiting we use newer version of CMF
# (for tests like testCookieCrumbler).
zope.app.testing
# Currently forked in our repository
# Products.PortalTransforms
# Dependency for our fork of PortalTransforms
StructuredText
# Needed for parsing .po files from our Localizer subset
polib
# Needed for Google OAuth
google-api-python-client
# Need for Facebook OAuth
facebook-sdk
# Used by ERP5 Jupyter backend
ipykernel
# Used by DiffTool
deepdiff
unidiff
# WSGI server
zope.globalrequest
waitress
# OpenId Connect
oic
# json schema validation
strict-rfc3339
jsonschema[format]
entry-points =
runwsgi=Products.ERP5.bin.zopewsgi:runwsgi
scripts =
apachedex
performance_tester_erp5
runwsgi
runzope
runzeo
tidstoraged
tidstorage_repozo
wcfs
web_checker_utility
extra-paths =
${erp5:location}
# patches for eggs
patch-binary = ${patch:location}/bin/patch
PyPDF2-patches = ${:_profile_base_location_}/../../component/egg-patch/PyPDF2/0001-Custom-implementation-of-warnings.formatwarning-remo.patch#d25bb0f5dde7f3337a0a50c2f986f5c8
PyPDF2-patch-options = -p1
Acquisition-patches = ${:_profile_base_location_}/../../component/egg-patch/Acquisition/aq_dynamic-2.13.12.patch#1d9a56e9af4371f5b6951ebf217a15d7
Acquisition-patch-options = -p1
python-magic-patches = ${:_profile_base_location_}/../../component/egg-patch/python_magic/magic.patch#de0839bffac17801e39b60873a6c2068
python-magic-patch-options = -p1
# neoppod installs bin/coverage so we inject erp5 plugin here so that coverage script can use it in report
[neoppod]
eggs +=
erp5_coverage_plugin
[eggs-all-scripts]
recipe = zc.recipe.egg
eggs =
munnel
patch-binary = ${eggs:patch-binary}
# develop erp5.util from parts/erp5/
[erp5-util-develop]
recipe = zc.recipe.egg:develop
setup = ${erp5:location}
[zodbpack]
recipe = zc.recipe.egg
eggs =
slapos.toolbox[zodbpack]
scripts =
zodbpack
depends =
${slapos-toolbox-dependencies:eggs}
[versions]
# See ../../software/neoppod/software-common.cfg for versions common with NEO:
# neoppod, mysqlclient, slapos.recipe.template
# patched eggs
Acquisition = 2.13.12+SlapOSPatched001
Products.DCWorkflow = 2.2.4+SlapOSPatched001
ocropy = 1.0+SlapOSPatched001
pysvn = 1.9.15+SlapOSPatched001
python-ldap = 2.4.32+SlapOSPatched001
python-magic = 0.4.12+SlapOSPatched001
PyPDF2 = 1.26.0+SlapOSPatched001
## https://lab.nexedi.com/nexedi/slapos/merge_requests/648
pylint = 1.4.4+SlapOSPatched002
# astroid 1.4.1 breaks testDynamicClassGeneration
astroid = 1.3.8+SlapOSPatched001
# use newer version than specified in ZTK
PasteDeploy = 1.5.2
argparse = 1.4.0
zope.dottedname = 4.1.0
# modified version that works fine for buildout installation
SOAPpy = 0.12.0nxd001
# CMF 2.3 is not yet supported.
Products.CMFCalendar = 2.2.3
Products.CMFCore = 2.2.10
Products.CMFDefault = 2.2.4
Products.CMFTopic = 2.2.1
Products.CMFUid = 2.2.1
# newer version requires zope.traversing>=4.0.0a2.
zope.app.appsetup = 3.16.0
# newer version requires zope.i18n>=4.0.0a3
zope.app.publication = 3.14.0
# newer version requires zope.testbrowser>=4
zope.app.testing = 3.8.1
# Pinned versions
APacheDEX = 1.8
Pillow = 6.2.2
Products.CMFActionIcons = 2.1.3
Products.GenericSetup = 1.8.6
Products.LongRequestLogger = 2.1.0
# Products.MimetypesRegistry 2.1 requires AccessControl>=3.0.0Acquisition.
Products.MimetypesRegistry = 2.0.10
Products.PluggableAuthService = 1.10.0
Products.PluginRegistry = 1.4
Products.TIDStorage = 5.5.0
pyPdf = 1.13
PyStemmer = 1.3.0
Pympler = 0.4.3
StructuredText = 2.11.1
WSGIUtils = 0.7
erp5-coverage-plugin = 0.0.1
erp5diff = 0.8.1.8
five.formlib = 1.0.4
five.localsitemanager = 2.0.5
google-api-python-client = 1.6.1
httplib2 = 0.10.3
huBarcode = 1.0.0
interval = 1.0.0
ipdb = 0.10.2
logilab-common = 1.3.0
munnel = 0.3
nt-svcutils = 2.13.0
oauth2client = 4.0.0
oauthlib = 3.1.0
objgraph = 3.1.0
polib = 1.0.8
pprofile = 2.0.4
pyasn1-modules = 0.0.8
pycountry = 17.1.8
pycrypto = 2.6.1
pyflakes = 1.5.0
python-memcached = 1.58
pytracemalloc = 1.2
qrcode = 5.3
rsa = 3.4.2
spyne = 2.12.14
suds = 0.4
facebook-sdk = 2.0.0
urlnorm = 1.1.4
uuid = 1.30
validictory = 1.1.0
xfw = 0.10
xupdate-processor = 0.5
scikit-image = 0.14.0
PyWavelets = 0.5.2
networkx = 2.1
pytesseract = 0.2.2
zbarlight = 2.3
cloudpickle = 0.5.3
dask = 0.18.1
toolz = 0.9.0
zope.globalrequest = 1.5
waitress = 1.4.4
Products.ZSQLMethods = 2.13.5
fpconst = 0.7.2
graphviz = 0.5.2
olefile = 0.44
python-libmilter = 1.0.3
zope.app.debug = 3.4.1
zope.app.dependable = 3.5.1
zope.app.form = 4.0.2
mpmath = 0.19
openpyxl = 2.4.8
jdcal = 1.3
deepdiff = 3.3.0
unidiff = 0.5.5
jsonpickle = 0.9.6
responses = 0.10.6
cookies = 2.2.1
jedi = 0.15.1
parso = 0.5.1
yapf = 0.28.0
z3c.etestbrowser = 3.0.1
zope.testbrowser = 5.5.1
WSGIProxy2 = 0.4.6
WebTest = 2.0.33
beautifulsoup4 = 4.8.2
WebOb = 1.8.5
soupsieve = 1.9.5
eggtestinfo = 0.3
oic = 0.15.1
Beaker = 1.11.0
Mako = 1.1.4
pyjwkest = 1.4.2
alabaster = 0.7.12
future = 0.18.2
pycryptodomex = 3.10.1
strict-rfc3339 = 0.7
webcolors = 1.10
rfc3987 = 1.3.8
jsonpointer = 2.2
pandas = 0.19.2
numpy = 1.13.1
scipy = 0.19.0
ZConfig = 2.9.3
# Override configuration specified in component/ZODB
BTrees= 4.5.1
persistent= 4.6.4
zodbpickle= 2.0.0
# Override configuration specified in component/neoppod
zope.event= 3.5.2
zope.exceptions = 3.6.2
zope.testing = 3.9.7
[ZODB]
major = 4
[ZODB5]
egg-versions =
ZODB = 5.8.0
transaction = 2.4.0
stack/erp5-zope2/buildout.hash.cfg 0 → 100644
View file @ 8eeb74fe
# THIS IS NOT A BUILDOUT FILE, despite purposedly using a compatible syntax.
# The only allowed lines here are (regexes):
# - "^#" comments, copied verbatim
# - "^[" section beginings, copied verbatim
# - lines containing an "=" sign which must fit in the following categorie.
# - "^\s*filename\s*=\s*path\s*$" where "path" is relative to this file
# Copied verbatim.
# - "^\s*hashtype\s*=.*" where "hashtype" is one of the values supported
# by the re-generation script.
# Re-generated.
# - other lines are copied verbatim
# Substitution (${...:...}), extension ([buildout] extends = ...) and
# section inheritance (< = ...) are NOT supported (but you should really
# not need these here).
[mariadb-resiliency-after-import-script]
filename = instance-mariadb-resiliency-after-import-script.sh.in
md5sum = 85ce1e2f3d251aa435fef8118dca8a63
[mariadb-slow-query-report-script]
filename = mysql-querydigest.sh.in
md5sum = 6457ab192d709aa2c9014e9a3e91ca20
[mariadb-start-clone-from-backup]
filename = instance-mariadb-start-clone-from-backup.sh.in
md5sum = d10b8e35b02b5391cf46bf0c7dbb1196
[template-mariadb]
filename = instance-mariadb.cfg.in
md5sum = 93b2277185e4949a3d17be79d3710d2d
[template-kumofs]
filename = instance-kumofs.cfg.in
md5sum = 45cc45510b59ceb730b6e38448b5c0c3
[template-zope-conf]
filename = zope.conf.in
md5sum = ce8d03a1b4c1a9e5085ec54ea2744007
[site-zcml]
filename = site.zcml
md5sum = 43556e5bca8336dd543ae8068512aa6d
[template-my-cnf]
filename = my.cnf.in
md5sum = c0bde08ec6bd6d333315a15026266b65
[template-mariadb-initial-setup]
filename = mariadb_initial_setup.sql.in
md5sum = f928b9dc99f7f970caadfe7dd6f95d34
[template-postfix]
filename = instance-postfix.cfg.in
md5sum = 8f7bfca893a01c390df7a3dc9c2410e1
[template-postfix-master-cf]
filename = postfix_master.cf.in
md5sum = ef164517e3f7170d03499967d625c3bb
[template-postfix-main-cf]
filename = postfix_main.cf.in
md5sum = e9f03c66627beb4054d45123450162d2
[template-postfix-aliases]
filename = postfix_aliases.in
md5sum = 0969fbb25b05c02ef3c2d437b2f4e1a0
[template-run-zelenium]
filename = run-zelenium-test.py.in
md5sum = b95084ae9eed95a68eada45e28ef0c04
[template]
filename = instance.cfg.in
md5sum = 3f7b28085ceff321a3cb785db60f7c3e
[template-erp5]
filename = instance-erp5.cfg.in
md5sum = 30a1e738a8211887e75a5e75820e5872
[template-zeo]
filename = instance-zeo.cfg.in
md5sum = 0ba5735ab87ee53e2c203b1563b55ff0
[template-zodb-base]
filename = instance-zodb-base.cfg.in
md5sum = 0ac4b74436f554cd677f19275d18d880
[template-zope]
filename = instance-zope.cfg.in
md5sum = 0451190711157fc204418662126d5cf8
[template-balancer]
filename = instance-balancer.cfg.in
md5sum = b0751d3d12cfcc8934cb1027190f5e5e
[template-haproxy-cfg]
filename = haproxy.cfg.in
md5sum = 1645ef8990ab2b50f91a4c02f0cf8882
[template-rsyslogd-cfg]
filename = rsyslogd.cfg.in
md5sum = 5cf0316fdd17a940031e4083bbededd8
[instance-wcfs.cfg.in]
filename = instance-wcfs.cfg.in
md5sum = 0f921643a68e3a8de5529d653710ddca
stack/erp5-zope2/haproxy.cfg.in 0 → 100644
View file @ 8eeb74fe
{# This file configures haproxy to redirect requests from ports to specific urls.
# It provides TLS support for server and optionnaly for client.
#
# All parameters are given through the `parameter_dict` variable, see the
# list entries :
#
# parameter_dict = {
# # Path of the PID file. HAProxy will write its own PID to this file
# # Sending USR2 signal to this pid will cause haproxy to reload
# # its configuration.
# "pidfile": "<file_path>",
#
# # AF_UNIX socket for logs. Syslog must be listening on this socket.
# "log-socket": "<file_path>",
#
# # AF_UNIX socket for statistics and control.
# # Haproxy will listen on this socket.
# "stats-socket": "<file_path>",
#
# # IPv4 to listen on
# # All backends from `backend-dict` will listen on this IP.
# "ipv4": "0.0.0.0",
#
# # IPv6 to listen on
# # All backends from `backend-dict` will listen on this IP.
# "ipv6": "::1",
#
# # Certificate and key in PEM format. All ports will serve TLS using
# # this certificate.
# "cert": "<file_path>",
#
# # CA to verify client certificates in PEM format.
# # If set, client certificates will be verified with these CAs.
# # If not set, client certificates are not verified.
# "ca-cert": "<file_path>",
#
# # An optional CRL in PEM format (the file can contain multiple CRL)
# # This is required if ca-cert is passed.
# "crl": "<file_path>",
#
# # Path to use for HTTP health check on backends from `backend-dict`.
# "server-check-path": "/",
#
# # The mapping of backends, keyed by family name
# "backend-dict": {
# "family-secure": {
# ( 8000, # port int
# 'https', # proto str
# True, # ssl_required bool
# None, # timeout (in seconds) int | None
# [ # backends
# '10.0.0.10:8001', # netloc str
# 1, # max_connection_count int
# False, # is_web_dav bool
# ],
# ),
# },
# "family-default": {
# ( 8002, # port int
# 'https', # proto str
# False, # ssl_required bool
# None, # timeout (in seconds) int | None
# [ # backends
# '10.0.0.10:8003', # netloc str
# 1, # max_connection_count int
# False, # is_web_dav bool
# ],
# ),
# },
#
# # The mapping of zope paths.
# # This is a Zope specific feature.
# # `enable_authentication` has same meaning as for `backend-list`.
# "zope-virtualhost-monster-backend-dict": {
# # {(ip, port): ( enable_authentication, {frontend_path: ( internal_url ) }, ) }
# ('[::1]', 8004): (
# True, {
# 'zope-1': 'http://10.0.0.10:8001',
# 'zope-2': 'http://10.0.0.10:8002',
# },
# ),
# },
# }
#
# This sample of `parameter_dict` will make haproxy listening to :
# From to `backend-list`:
# For "family-secure":
# - 0.0.0.0:8000 redirecting internaly to http://10.0.0.10:8001 and
# - [::1]:8000 redirecting internaly to http://10.0.0.10:8001
# only accepting requests from clients providing a verified TLS certificate
# emitted by a CA from `ca-cert` and not revoked in `crl`.
# For "family-default":
# - 0.0.0.0:8002 redirecting internaly to http://10.0.0.10:8003
# - [::1]:8002 redirecting internaly to http://10.0.0.10:8003
# accepting requests from any client.
#
# For both families, X-Forwarded-For header will be stripped unless
# client presents a certificate that can be verified with `ca-cert` and `crl`.
#
# From zope-virtualhost-monster-backend-dict`:
# - [::1]:8004 with some path based rewrite-rules redirecting to:
# * http://10.0.0.10/8001 when path matches /zope-1(.*)
# * http://10.0.0.10/8002 when path matches /zope-2(.*)
# with some VirtualHostMonster rewrite rules so zope writes URLs with
# [::1]:8004 as server name.
# For more details, refer to
# https://docs.zope.org/zope2/zope2book/VirtualHosting.html#using-virtualhostroot-and-virtualhostbase-together
-#}
{% set server_check_path = parameter_dict['server-check-path'] -%}
global
maxconn 4096
master-worker
pidfile {{ parameter_dict['pidfile'] }}
# SSL configuration was generated with mozilla SSL Configuration Generator
# generated 2020-10-28, Mozilla Guideline v5.6, HAProxy 2.1, OpenSSL 1.1.1g, modern configuration
# https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=modern&openssl=1.1.1g&guideline=5.6
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-options prefer-client-ciphers no-sslv3 no-tlsv10 no-tlsv11 no-tlsv12 no-tls-tickets
ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tlsv12 no-tls-tickets
stats socket {{ parameter_dict['stats-socket'] }} level admin
defaults
mode http
retries 1
option redispatch
maxconn 2000
balance roundrobin
stats uri /haproxy
stats realm Global\ statistics
timeout connect 10s
timeout queue 60s
timeout client 305s
option http-server-close
# compress some content types
compression algo gzip
compression type application/font-woff application/font-woff2 application/hal+json application/javascript application/json application/rss+xml application/wasm application/x-font-opentype application/x-font-ttf application/x-javascript application/xml image/svg+xml text/cache-manifest text/css text/html text/javascript text/plain text/xml
log {{ parameter_dict['log-socket'] }} local0 info
{% set bind_ssl_crt = 'ssl crt ' ~ parameter_dict['cert'] ~ ' alpn h2,http/1.1' %}
{% set family_path_routing_dict = parameter_dict['family-path-routing-dict'] %}
{% set path_routing_list = parameter_dict['path-routing-list'] %}
{% for name, (port, _, certificate_authentication, timeout, backend_list) in sorted(six.iteritems(parameter_dict['backend-dict'])) -%}
listen family_{{ name }}
{%- if parameter_dict.get('ca-cert') -%}
{%- set ssl_auth = ' ca-file ' ~ parameter_dict['ca-cert'] ~ ' verify' ~ ( ' required' if certificate_authentication else ' optional' ) ~ ' crl-file ' ~ parameter_dict['crl'] %}
{%- else %}
{%- set ssl_auth = '' %}
{%- endif %}
bind {{ parameter_dict['ipv4'] }}:{{ port }} {{ bind_ssl_crt }} {{ ssl_auth }}
bind {{ parameter_dict['ipv6'] }}:{{ port }} {{ bind_ssl_crt }} {{ ssl_auth }}
cookie SERVERID rewrite
http-request set-header X-Balancer-Current-Cookie SERVERID
{% if timeout %}
{#
Apply a slightly longer timeout than the zope timeout so that clients can see the
TimeoutReachedError from zope, that is a bit more informative than the 504 error
page from haproxy.
#}
timeout server {{ timeout + 3 }}s
{%- endif %}
# remove X-Forwarded-For unless client presented a verified certificate
acl client_cert_verified ssl_c_used ssl_c_verify 0
http-request del-header X-Forwarded-For unless client_cert_verified
# set Remote-User if client presented a verified certificate
http-request del-header Remote-User
http-request set-header Remote-User %{+Q}[ssl_c_s_dn(cn)] if client_cert_verified
# logs
capture request header Referer len 512
capture request header User-Agent len 512
log-format "%{+Q}o %{-Q}ci - - [%trg] %r %ST %B %{+Q}[capture.req.hdr(0)] %{+Q}[capture.req.hdr(1)] %Ta"
{% for outer_prefix, inner_prefix in family_path_routing_dict.get(name, []) + path_routing_list %}
{% set outer_prefix = outer_prefix.strip('/') -%}
http-request replace-path ^(/+VirtualHostBase/+[^/]+/+[^/]+)/+VirtualHostRoot/+{% if outer_prefix %}{{ outer_prefix }}($|/.*){% else %}(.*){% endif %} \1/{{ inner_prefix.strip('/') }}/VirtualHostRoot/{% if outer_prefix %}_vh_{{ outer_prefix.replace('/', '/_vh_') }}{% endif %}\2
{% endfor %}
{% set has_webdav = [] -%}
{% for address, connection_count, webdav in backend_list -%}
{% if webdav %}{% do has_webdav.append(None) %}{% endif -%}
{% set server_name = name ~ '-' ~ loop.index0 %}
server {{ server_name }} {{ address }} cookie {{ server_name }} check inter 3s rise 1 fall 2 maxqueue 5 maxconn {{ connection_count }}
{%- endfor -%}
{%- if not has_webdav and server_check_path %}
option httpchk GET {{ server_check_path }}
{%- endif %}
{% endfor %}
{% for (ip, port), (_, backend_dict) in sorted(six.iteritems(parameter_dict['zope-virtualhost-monster-backend-dict'])) -%}
{% set group_name = 'testrunner_' ~ loop.index0 %}
frontend frontend_{{ group_name }}
bind {{ ip }}:{{ port }} {{ bind_ssl_crt }}
timeout client 8h
# logs
capture request header Referer len 512
capture request header User-Agent len 512
log-format "%{+Q}o %{-Q}ci - - [%trg] %r %ST %B %{+Q}[capture.req.hdr(0)] %{+Q}[capture.req.hdr(1)] %Tt"
{% for name in sorted(backend_dict.keys()) %}
use_backend backend_{{ group_name }}_{{ name }} if { path -m beg /{{ name }} }
{%- endfor %}
{% for name, url in sorted(backend_dict.items()) %}
backend backend_{{ group_name }}_{{ name }}
http-request replace-path ^/{{ name }}(.*) /VirtualHostBase/https/{{ ip }}:{{ port }}/VirtualHostRoot/_vh_{{ name }}\1
timeout server 8h
server {{ name }} {{ urllib_parse.urlparse(url).netloc }}
{%- endfor %}
{% endfor %}
stack/erp5-zope2/instance-balancer.cfg.in 0 → 100644
View file @ 8eeb74fe
{% import "caucase" as caucase with context %}
{% set part_list = [] -%}
{% macro section(name) %}{% do part_list.append(name) %}{{ name }}{% endmacro -%}
{% set ssl_parameter_dict = slapparameter_dict['ssl'] -%}
{% set frontend_caucase_url_list = ssl_parameter_dict.get('frontend-caucase-url-list', []) -%}
{#
XXX: This template only supports exactly one IPv4 and (if ipv6 is used) one IPv6
per partition. No more (undefined result), no less (IndexError).
-#}
{% set ipv4 = (ipv4_set | list)[0] -%}
{% if ipv6_set -%}
{% set ipv6 = (ipv6_set | list)[0] -%}
{% endif -%}
[jinja2-template-base]
recipe = slapos.recipe.template:jinja2
{{ caucase.updater(
prefix='caucase-updater',
buildout_bin_directory=parameter_dict['bin-directory'],
updater_path='${directory:services-on-watch}/caucase-updater',
url=ssl_parameter_dict['caucase-url'],
data_dir='${directory:srv}/caucase-updater',
crt_path='${apache-conf-ssl:caucase-cert}',
ca_path='${directory:srv}/caucase-updater/ca.crt',
crl_path='${directory:srv}/caucase-updater/crl.pem',
key_path='${apache-conf-ssl:caucase-key}',
on_renew='${haproxy-reload:output}',
max_sleep=ssl_parameter_dict.get('max-crl-update-delay', 1.0),
template_csr_pem=ssl_parameter_dict.get('csr'),
openssl=parameter_dict['openssl'] ~ '/bin/openssl',
)}}
{# XXX we don't use caucase yet.
{% do section('caucase-updater') -%}
{% do section('caucase-updater-promise') -%}
#}
{% set frontend_caucase_url_hash_list = [] -%}
{% for frontend_caucase_url in frontend_caucase_url_list -%}
{% set hash = hashlib.md5(six.ensure_binary(frontend_caucase_url)).hexdigest() -%}
{% do frontend_caucase_url_hash_list.append(hash) -%}
{% set data_dir = '${directory:client-cert-ca}/%s' % hash -%}
{{ caucase.updater(
prefix='caucase-updater-%s' % hash,
buildout_bin_directory=parameter_dict['bin-directory'],
updater_path='${directory:services-on-watch}/caucase-updater-%s' % hash,
url=frontend_caucase_url,
data_dir=data_dir,
ca_path='%s/ca.crt' % data_dir,
crl_path='%s/crl.pem' % data_dir,
on_renew='${caucase-updater-housekeeper:output}',
max_sleep=ssl_parameter_dict.get('max-crl-update-delay', 1.0),
openssl=parameter_dict['openssl'] ~ '/bin/openssl',
)}}
{% do section('caucase-updater-%s' % hash) -%}
{% endfor -%}
{% if frontend_caucase_url_hash_list -%}
[caucase-updater-housekeeper]
recipe = collective.recipe.template
output = ${directory:bin}/caucase-updater-housekeeper
mode = 700
input =
inline:
#!${buildout:executable}
import glob
import os
import subprocess
hash_list = {{ repr(frontend_caucase_url_hash_list) }}
crt_list = ['%s.crt' % e for e in hash_list]
for path in glob.glob('${haproxy-conf-ssl:ca-cert-dir}/*.crt'):
if os.path.basename(path) not in crt_list:
os.unlink(path)
crl_list = ['%s.crl' % e for e in hash_list]
for path in glob.glob('${haproxy-conf-ssl:crl-dir}/*.crl'):
if os.path.basename(path) not in crl_list:
os.unlink(path)
for hash in hash_list:
crt = '${directory:client-cert-ca}/%s/ca.crt' % hash
crt_link = '${haproxy-conf-ssl:ca-cert-dir}/%s.crt' % hash
crl = '${directory:client-cert-ca}/%s/crl.pem' % hash
crl_link = '${haproxy-conf-ssl:crl-dir}/%s.crl' % hash
if os.path.isfile(crt) and not os.path.islink(crt_link):
os.symlink(crt, crt_link)
if os.path.isfile(crl) and not os.path.islink(crl_link):
os.symlink(crl, crl_link)
subprocess.check_call(['{{ parameter_dict["openssl"] }}/bin/c_rehash', '${haproxy-conf-ssl:ca-cert-dir}'])
subprocess.check_call(['{{ parameter_dict["openssl"] }}/bin/c_rehash', '${haproxy-conf-ssl:crl-dir}'])
# assemble all CA and all CRLs in one file for haproxy
with open('${haproxy-conf-ssl:ca-cert}.tmp', 'w') as f:
for path in glob.glob('${haproxy-conf-ssl:ca-cert-dir}/*.crt'):
with open(path) as in_f:
f.write('#{}\n'.format(path))
f.write(in_f.read() + '\n')
with open('${haproxy-conf-ssl:crl}.tmp', 'w') as f:
for path in glob.glob('${haproxy-conf-ssl:crl-dir}/*.crl'):
with open(path) as in_f:
f.write('#{}\n'.format(path))
f.write(in_f.read() + '\n')
if os.path.exists('${haproxy-conf-ssl:ca-cert}'):
os.unlink('${haproxy-conf-ssl:ca-cert}')
if os.path.exists('${haproxy-conf-ssl:crl}'):
os.unlink('${haproxy-conf-ssl:crl}')
os.rename('${haproxy-conf-ssl:ca-cert}.tmp', '${haproxy-conf-ssl:ca-cert}')
os.rename('${haproxy-conf-ssl:crl}.tmp', '${haproxy-conf-ssl:crl}')
subprocess.check_call(['${haproxy-reload:output}'])
[caucase-updater-housekeeper-run]
recipe = plone.recipe.command
command = ${caucase-updater-housekeeper:output}
update-command = ${:command}
{% endif -%}
{% set haproxy_dict = {} -%}
{% set zope_virtualhost_monster_backend_dict = {} %}
{% set test_runner_url_dict = {} %} {# family_name => list of URLs #}
{% set next_port = functools.partial(next, itertools.count(slapparameter_dict['tcpv4-port'])) -%}
{% for family_name, parameter_id_list in sorted(
six.iteritems(slapparameter_dict['zope-family-dict'])) -%}
{% set zope_family_address_list = [] -%}
{% set ssl_authentication = slapparameter_dict['ssl-authentication-dict'][family_name] -%}
{% set has_webdav = [] -%}
{% for parameter_id in parameter_id_list -%}
{% set zope_address_list = slapparameter_dict[parameter_id] -%}
{% for zope_address, maxconn, webdav in zope_address_list -%}
{% if webdav -%}
{% do has_webdav.append(None) %}
{% endif -%}
{% set zope_effective_address = zope_address -%}
{% do zope_family_address_list.append((zope_effective_address, maxconn, webdav)) -%}
{% endfor -%}
{# # Generate entries with rewrite rule for test runnners #}
{% set test_runner_address_list = slapparameter_dict.get(parameter_id ~ '-test-runner-address-list', []) %}
{% if test_runner_address_list -%}
{% set test_runner_backend_mapping = {} %}
{% set test_runner_balancer_url_list = [] %}
{% set test_runner_external_port = next_port() %}
{% for i, (test_runner_internal_ip, test_runner_internal_port) in enumerate(test_runner_address_list) %}
{% do test_runner_backend_mapping.__setitem__(
'unit_test_' ~ i,
'http://' ~ test_runner_internal_ip ~ ':' ~ test_runner_internal_port ) %}
{% do test_runner_balancer_url_list.append(
'https://' ~ ipv4 ~ ':' ~ test_runner_external_port ~ '/unit_test_' ~ i ~ '/' ) %}
{% endfor %}
{% do zope_virtualhost_monster_backend_dict.__setitem__(
(ipv4, test_runner_external_port),
( ssl_authentication, test_runner_backend_mapping ) ) -%}
{% do test_runner_url_dict.__setitem__(family_name, test_runner_balancer_url_list) -%}
{% endif -%}
{% endfor -%}
{# Make rendering fail artificially if any family has no known backend.
# This is useful as haproxy's hot-reconfiguration mechanism is
# supervisord-incompatible.
# As jinja2 postpones KeyError until place-holder value is actually used,
# do a no-op getitem.
-#}
{% do zope_family_address_list[0][0] -%}
{#
# We use to have haproxy then apache, now haproxy is playing apache's role
# To keep port stable, we consume one port so that haproxy use the same port
# that apache was using before.
-#}
{% set _ = next_port() -%}
{% set haproxy_port = next_port() -%}
{% set backend_path = slapparameter_dict['backend-path-dict'][family_name] -%}
{% if has_webdav -%}
{% set external_scheme = 'webdavs' -%}
{% else %}
{% set external_scheme = 'https' -%}
{% endif -%}
{% do haproxy_dict.__setitem__(family_name, (haproxy_port, external_scheme, slapparameter_dict['ssl-authentication-dict'][family_name], slapparameter_dict['timeout-dict'][family_name], zope_family_address_list)) -%}
{% endfor -%}
[haproxy-cfg-parameter-dict]
ipv4 = {{ ipv4 }}
ipv6 = {{ ipv6 }}
cert = ${haproxy-conf-ssl:certificate}
{% if frontend_caucase_url_list -%}
ca-cert = ${haproxy-conf-ssl:ca-cert}
crl = ${haproxy-conf-ssl:crl}
{% endif %}
stats-socket = ${directory:run}/ha.sock
path-routing-list = {{ dumps(slapparameter_dict['path-routing-list']) }}
family-path-routing-dict = {{ dumps(slapparameter_dict['family-path-routing-dict']) }}
pidfile = ${directory:run}/haproxy.pid
log-socket = ${rsyslogd-cfg-parameter-dict:log-socket}
server-check-path = {{ dumps(slapparameter_dict['haproxy-server-check-path']) }}
backend-dict = {{ dumps(haproxy_dict) }}
zope-virtualhost-monster-backend-dict = {{ dumps(zope_virtualhost_monster_backend_dict) }}
[haproxy-cfg]
< = jinja2-template-base
url = {{ parameter_dict['template-haproxy-cfg'] }}
output = ${directory:etc}/haproxy.cfg
context =
section parameter_dict haproxy-cfg-parameter-dict
import urllib_parse six.moves.urllib.parse
extensions = jinja2.ext.do
[haproxy-reload]
recipe = collective.recipe.template
output = ${directory:bin}/${:_buildout_section_name_}
mode = 700
input =
inline:
#!/bin/sh
kill -USR2 $(cat "${haproxy-cfg-parameter-dict:pidfile}")
[{{ section('haproxy') }}]
recipe = slapos.cookbook:wrapper
wrapper-path = ${directory:services-on-watch}/haproxy
command-line = "{{ parameter_dict['haproxy'] }}/sbin/haproxy" -f "${haproxy-cfg:output}"
hash-files = ${haproxy-cfg:output}
[apache-conf-ssl]
# XXX caucase is/was buggy and this certificate does not match key for instances
# that were updated, so don't use it yet.
caucase-cert = ${directory:apache-conf}/apache-caucase.crt
caucase-key = ${directory:apache-conf}/apache-caucase.pem
[simplefile]
< = jinja2-template-base
inline = {{ '{{ content }}' }}
{% macro simplefile(section_name, file_path, content, mode='') -%}
{% set content_section_name = section_name ~ '-content' -%}
[{{ content_section_name }}]
content = {{ dumps(content) }}
[{{ section(section_name) }}]
< = simplefile
output = {{ file_path }}
context = key content {{content_section_name}}:content
mode = {{ mode }}
{%- endmacro %}
[{{ section('haproxy-socat-stats')}}]
recipe = slapos.cookbook:wrapper
wrapper-path = ${directory:bin}/${:_buildout_section_name_}
command-line = "{{ parameter_dict['socat'] }}/bin/socat" unix-connect:${haproxy-cfg-parameter-dict:stats-socket} stdio
[rsyslogd-cfg-parameter-dict]
log-socket = ${directory:run}/log.sock
access-log-file = ${directory:log}/apache-access.log
error-log-file = ${directory:log}/apache-error.log
pid-file = ${directory:run}/rsyslogd.pid
spool-directory = ${directory:rsyslogd-spool}
[rsyslogd-cfg]
<= jinja2-template-base
url = {{ parameter_dict['template-rsyslogd-cfg'] }}
output = ${directory:etc}/rsyslogd.conf
context = section parameter_dict rsyslogd-cfg-parameter-dict
[{{ section ('rsyslogd') }}]
recipe = slapos.cookbook:wrapper
command-line = {{ parameter_dict['rsyslogd'] }}/sbin/rsyslogd -i ${rsyslogd-cfg-parameter-dict:pid-file} -n -f ${rsyslogd-cfg:output}
wrapper-path = ${directory:services-on-watch}/rsyslogd
hash-existing-files = ${buildout:directory}/software_release/buildout.cfg
hash-files = ${rsyslogd-cfg:output}
[{{ section ('rsyslogd-listen-promise') }}]
<= monitor-promise-base
promise = check_command_execute
name = rsyslogd_listen_promise.py
config-command = test -S ${rsyslogd-cfg-parameter-dict:log-socket}
[haproxy-conf-ssl]
certificate = ${build-certificate-and-key:certificate-and-key}
{% if frontend_caucase_url_list -%}
ca-cert = ${directory:etc}/frontend-ca.pem
ca-cert-dir = ${directory:ca-cert}
crl = ${directory:etc}/frontend-crl.pem
crl-dir = ${directory:crl}
depends = ${caucase-updater-housekeeper-run:recipe}
{%- endif %}
[build-certificate-and-key]
{% if ssl_parameter_dict.get('key') -%}
certificate-and-key = ${tls-certificate-and-key-from-parameters:output}
{{ simplefile(
'tls-certificate-and-key-from-parameters',
'${directory:etc}/certificate-and-key-from-parameters.pem',
ssl_parameter_dict['cert'] ~ "\n" ~ ssl_parameter_dict['key']) }}
{% else %}
recipe = plone.recipe.command
command = "{{ parameter_dict['openssl'] }}/bin/openssl" req -newkey rsa -batch -new -x509 -days 3650 -nodes -keyout "${:certificate-and-key}" -out "${:certificate-and-key}"
certificate-and-key = ${directory:etc}/certificate-and-key-generated.pem
{%- endif %}
[{{ section('haproxy-promise') }}]
<= monitor-promise-base
# Check any haproxy port in ipv4, expect other ports and ipv6 to behave consistently
promise = check_socket_listening
name = haproxy.py
config-host = {{ ipv4 }}
config-port = {{ next(six.itervalues(haproxy_dict))[0] }}
[{{ section('publish') }}]
recipe = slapos.cookbook:publish.serialised
{% for family_name, (port, scheme, _, _, _) in haproxy_dict.items() -%}
{{ family_name ~ '-v6' }} = {% if ipv6_set %}{{ scheme ~ '://[' ~ ipv6 ~ ']:' ~ port }}{% endif %}
{{ family_name }} = {{ scheme ~ '://' ~ ipv4 ~ ':' ~ port }}
{% endfor -%}
{% for family_name, test_runner_url_list in test_runner_url_dict.items() -%}
{{ family_name ~ '-test-runner-url-list' }} = {{ dumps(test_runner_url_list) }}
{% endfor -%}
monitor-base-url = ${monitor-publish-parameters:monitor-base-url}
[{{ section('logrotate-rsyslogd') }}]
< = logrotate-entry-base
name = rsyslogd
log = ${rsyslogd-cfg-parameter-dict:access-log-file} ${rsyslogd-cfg-parameter-dict:error-log-file}
post = test ! -s ${rsyslogd-cfg-parameter-dict:pid-file} || kill -HUP $(cat ${rsyslogd-cfg-parameter-dict:pid-file})
[directory]
recipe = slapos.cookbook:mkdirectory
bin = ${buildout:directory}/bin
etc = ${buildout:directory}/etc
services = ${:etc}/run
services-on-watch = ${:etc}/service
var = ${buildout:directory}/var
run = ${:var}/run
log = ${:var}/log
srv = ${buildout:directory}/srv
apachedex = ${monitor-directory:private}/apachedex
rsyslogd-spool = ${:run}/rsyslogd-spool
{% if frontend_caucase_url_list -%}
ca-cert = ${:etc}/ssl.crt
crl = ${:etc}/ssl.crl
client-cert-ca = ${:srv}/client-cert-ca
{% endif -%}
[{{ section('resiliency-exclude-file') }}]
# Generate rdiff exclude file in case of resiliency
< = jinja2-template-base
inline = {{ '{{ "${directory:log}/**\\n" }}' }}
output = ${directory:srv}/exporter.exclude
[{{ section('monitor-generate-apachedex-report') }}]
recipe = slapos.cookbook:cron.d
cron-entries = ${cron:cron-entries}
name = generate-apachedex-report
# The goal is to be executed before logrotate log rotation.
# Here, logrotate-entry-base:frequency = daily, so we run at 23 o'clock every day.
frequency = 0 23 * * *
command = ${monitor-generate-apachedex-report-wrapper:wrapper-path}
[monitor-generate-apachedex-report-wrapper]
recipe = slapos.cookbook:wrapper
wrapper-path = ${directory:bin}/${:command}
command-line = "{{ parameter_dict['run-apachedex-location'] }}" "{{ parameter_dict['apachedex-location'] }}" "${directory:apachedex}" ${monitor-publish-parameters:monitor-base-url}/private/apachedex --apache-log-list "${apachedex-parameters:apache-log-list}" --configuration ${apachedex-parameters:configuration}
command = generate-apachedex-report
[monitor-apachedex-report-config]
recipe = slapos.recipe.template
output = ${directory:etc}/${:_buildout_section_name_}
inline =
{% for line in slapparameter_dict['apachedex-configuration'] %}
{# apachedex config files use shlex.split, so we need to quote the arguments. #}
{# BBB: in python 3 we can use shlex.quote instead. #}
{{ repr(line.encode('utf-8')) }}
{% endfor %}
[apachedex-parameters]
apache-log-list = ${rsyslogd-cfg-parameter-dict:access-log-file}
configuration = ${monitor-apachedex-report-config:output}
promise-threshold = {{ slapparameter_dict['apachedex-promise-threshold'] }}
[{{ section('monitor-promise-apachedex-result') }}]
<= monitor-promise-base
promise = check_command_execute
name = check-apachedex-result.py
config-command = "{{ parameter_dict['promise-check-apachedex-result'] }}" --apachedex_path "${directory:apachedex}" --status_file ${monitor-directory:private}/apachedex.report.json --threshold "${apachedex-parameters:promise-threshold}"
[{{ section('promise-check-computer-memory') }}]
<= monitor-promise-base
promise = check_command_execute
name = check-computer-memory.py
config-command = "{{ parameter_dict["check-computer-memory-binary"] }}" -db ${monitor-instance-parameter:collector-db} --threshold "{{ slapparameter_dict["computer-memory-percent-threshold"] }}" --unit percent
[monitor-instance-parameter]
monitor-httpd-ipv6 = {{ (ipv6_set | list)[0] }}
monitor-httpd-port = {{ next_port() }}
monitor-title = {{ slapparameter_dict['name'] }}
password = {{ slapparameter_dict['monitor-passwd'] }}
[buildout]
extends =
{{ template_monitor }}
parts +=
{{ part_list | join('\n ') }}
stack/erp5-zope2/instance-erp5.cfg.in 0 → 100644
View file @ 8eeb74fe
{% import "root_common" as root_common with context -%}
{% import "caucase" as caucase with context %}
{% set frontend_dict = slapparameter_dict.get('frontend', {}) -%}
{% set has_frontend = frontend_dict.get('software-url', '') != '' -%}
{% set site_id = slapparameter_dict.get('site-id', 'erp5') -%}
{% set inituser_login = slapparameter_dict.get('inituser-login', 'zope') -%}
{% set publish_dict = {'site-id': site_id, 'inituser-login': inituser_login} -%}
{% set has_posftix = slapparameter_dict.get('smtp', {}).get('postmaster') -%}
{% set jupyter_dict = slapparameter_dict.get('jupyter', {}) -%}
{% set has_jupyter = jupyter_dict.get('enable', jupyter_enable_default.lower() in ('true', 'yes')) -%}
{% set jupyter_zope_family = jupyter_dict.get('zope-family', '') -%}
{% set wcfs_dict = slapparameter_dict.get('wcfs', {}) -%}
{% set wcfs_enable = wcfs_dict.get('enable', wcfs_enable_default.lower() in ('true', 'yes')) -%}
{% set test_runner_enabled = slapparameter_dict.get('test-runner', {}).get('enabled', True) -%}
{% set test_runner_node_count = slapparameter_dict.get('test-runner', {}).get('node-count', 3) -%}
{% set test_runner_extra_database_count = slapparameter_dict.get('test-runner', {}).get('extra-database-count', 3) -%}
{% set test_runner_total_database_count = test_runner_node_count * (1 + test_runner_extra_database_count) -%}
{# Backward compatibility for mariadb.test-database-amount #}
{% set mariadb_test_database_amount = slapparameter_dict.get('mariadb', {}).get('test-database-amount') -%}
{% if mariadb_test_database_amount is not none -%}
{% set test_runner_total_database_count = mariadb_test_database_amount %}
{% set test_runner_enabled = mariadb_test_database_amount > 0 %}
{% endif -%}
{# Backward compatibility for cloudooo-url #}
{% if slapparameter_dict.get('cloudooo-url') -%}
{% do slapparameter_dict.setdefault('cloudooo-url-list', slapparameter_dict['cloudooo-url'].split(',')) %}
{% endif -%}
{% set test_runner_random_activity_priority = slapparameter_dict.get('test-runner', {}).get('random-activity-priority') -%}
{% set monitor_base_url_dict = {} -%}
{% set monitor_dict = slapparameter_dict.get('monitor', {}) %}
{% set use_ipv6 = slapparameter_dict.get('use-ipv6', False) -%}
{% set partition_thread_count_list = [] -%}
{% set zope_partition_dict = slapparameter_dict.get('zope-partition-dict', {'1': {}}) -%}
{% set zope_family_override_dict = slapparameter_dict.get('family-override', {}) -%}
{% for zope_parameter_dict in six.itervalues(zope_partition_dict) -%}
{# Apply some zope_parameter_dict default values, to avoid duplication. -#}
{% do zope_parameter_dict.setdefault('thread-amount', 4) -%}
{% do zope_parameter_dict.setdefault('instance-count', 1) -%}
{% do partition_thread_count_list.append(zope_parameter_dict['thread-amount'] * zope_parameter_dict['instance-count']) -%}
{% endfor -%}
[request-common]
<= request-common-base
config-use-ipv6 = {{ dumps(slapparameter_dict.get('use-ipv6', False)) }}
config-computer-memory-percent-threshold = {{ dumps(monitor_dict.get('computer-memory-percent-threshold', 80)) }}
{% set caucase_dict = slapparameter_dict.get('caucase', {}) -%}
{% set caucase_url = caucase_dict.get('url', '') -%}
{% macro request(name, software_type, config_key, config, ret={'url': True}, key_config={}) -%}
{% do config.update(slapparameter_dict.get(config_key, {})) -%}
{% set section = 'request-' ~ name -%}
[{{ section }}]
<= request-common
name = {{ name }}
software-type = {{ software_type }}
return = {{ ' '.join(ret) }}
{% for ret, publish in six.iteritems(ret) -%}
{% if publish -%}
{% do publish_dict.__setitem__(name ~ '-' ~ ret, '${' ~ section ~ ':connection-' ~ ret ~ '}') %}
{% endif -%}
{% if ret == "monitor-base-url" -%}
{% do monitor_base_url_dict.__setitem__(section, '${' ~ section ~ ':connection-' ~ ret ~ '}') -%}
{% endif -%}
{% endfor -%}
{{ root_common.sla(name) }}
{% for k, v in six.iteritems(config) -%}
config-{{ k }} = {{ dumps(v) }}
{% endfor -%}
{% for k, v in six.iteritems(key_config) -%}
config-{{ k }} = {{ '${' ~ v ~ '}' }}
{% endfor -%}
config-name = {{ name }}
{% endmacro -%}
[directory]
recipe = slapos.cookbook:mkdirectory
bin = ${buildout:directory}/bin
service-on-watch = ${buildout:directory}/etc/service
srv = ${buildout:directory}/srv
tmp = ${buildout:directory}/tmp
backup-caucased = ${:srv}/backup/caucased
{% if not caucase_url -%}
{% if use_ipv6 -%}
{% set caucase_host = '[' ~ (ipv6_set | list)[0] ~ ']' %}
{%- else -%}
{% set caucase_host = (ipv4_set | list)[0] %}
{%- endif %}
{% set caucase_port = caucase_dict.get('base-port', 8890) -%}
{% set caucase_netloc = caucase_host ~ ':' ~ caucase_port -%}
{% set caucase_url = 'http://' ~ caucase_netloc -%}
{{ caucase.caucased(
prefix='caucased',
buildout_bin_directory=bin_directory,
caucased_path='${directory:service-on-watch}/caucased',
backup_dir='${directory:backup-caucased}',
data_dir='${directory:srv}/caucased',
netloc=caucase_netloc,
tmp='${directory:tmp}',
service_auto_approve_count=caucase_dict.get('service-auto-approve-amount', 1),
user_auto_approve_count=caucase_dict.get('user-auto-approve-amount', 0),
key_len=caucase_dict.get('key-length', 2048),
)}}
{% do root_common.section('caucased') -%}
{% do root_common.section('caucased-promise') -%}
{% endif -%}
{% do publish_dict.__setitem__('caucase-http-url', caucase_url) -%}
{% set balancer_dict = slapparameter_dict.setdefault('balancer', {}) -%}
{% do balancer_dict.setdefault('ssl', {}).setdefault('caucase-url', caucase_url) -%}
{% do balancer_dict.setdefault('tcpv4-port', 2150) -%}
{% do balancer_dict.__setitem__('haproxy-server-check-path', balancer_dict.get('haproxy-server-check-path', '/') % {'site-id': site_id}) -%}
{% set routing_path_template_field_dict = {"site-id": site_id} -%}
{% macro expandRoutingPath(output, input) -%}
{% for outer_prefix, inner_prefix in input -%}
{% do output.append((outer_prefix, inner_prefix % routing_path_template_field_dict)) -%}
{% endfor -%}
{% endmacro -%}
{% set path_routing_list = [] -%}
{% do expandRoutingPath(
path_routing_list,
balancer_dict.get(
'path-routing-list',
(('/', '/'), ),
),
) -%}
{% do balancer_dict.__setitem__('path-routing-list', path_routing_list) -%}
{% set family_path_routing_dict = {} -%}
{% for name, family_path_routing_list in balancer_dict.get(
'family-path-routing-dict',
{},
).items() -%}
{% set path_routing_list = [] -%}
{% do expandRoutingPath(path_routing_list, family_path_routing_list) -%}
{% do family_path_routing_dict.__setitem__(name, path_routing_list) -%}
{% endfor -%}
{% do balancer_dict.__setitem__('family-path-routing-dict', family_path_routing_dict) -%}
{{ request('memcached-persistent', 'kumofs', 'kumofs', {'tcpv4-port': 2000}, {'url': True, 'monitor-base-url': False}, key_config={'monitor-passwd': 'monitor-htpasswd:passwd'}) }}
{{ request('memcached-volatile', 'kumofs', 'memcached', {'tcpv4-port': 2010, 'ram-storage-size': 64}, {'url': True, 'monitor-base-url': False}, key_config={'monitor-passwd': 'monitor-htpasswd:passwd'}) }}
{# Notes on max-connection-count: On a standard ERP5, each transaction
can have 4 connections to mariadb: activities, catalog, deferred and
transactionless. Count 5 to have some headroom. Multiply by the total
number of zope threads for all processes from all partitions to get the
expected number of connections. Add 50 for have some more zope-independent
headroom (automated probes, replication, ...).
-#}
{{ request('mariadb', 'mariadb', 'mariadb',
{
'tcpv4-port': 2099,
'max-slowqueries-threshold': monitor_dict.get('max-slowqueries-threshold', 1000),
'slowest-query-threshold': monitor_dict.get('slowest-query-threshold', ''),
'test-database-amount': test_runner_total_database_count,
'max-connection-count': sum(partition_thread_count_list) * 5 + 50,
},
{
'database-list': True,
'test-database-list': True,
'monitor-base-url': False,
},
key_config={'monitor-passwd': 'monitor-htpasswd:passwd'},
) }}
{% if has_posftix -%}
{{ request('smtp', 'postfix', 'smtp', {'tcpv4-port': 2025, 'smtpd-sasl-user': 'erp5@nowhere'}, key_config={'smtpd-sasl-password': 'publish-early:smtpd-sasl-password', 'monitor-passwd': 'monitor-htpasswd:passwd'}) }}
{%- else %}
[request-smtp]
# Placeholder smtp service URL
connection-url = smtp://127.0.0.2:0/
{%- endif %}
{# ZODB -#}
{% set zodb_dict = {} -%}
{% set storage_dict = {} -%}
{% set mountpoints = set() -%}
{% for zodb in slapparameter_dict.get('zodb') or ({'type': 'zeo', 'server': {}},) -%}
{% do mountpoints.add(zodb.setdefault('mount-point', '/')) -%}
{% set name = zodb.pop('name', 'root') -%}
{% do assert(name not in zodb_dict, name, zodb_dict) -%}
{% do zodb_dict.__setitem__(name, zodb) -%}
{% if 'server' in zodb -%}
{% do storage_dict.setdefault(zodb['type'], {}).__setitem__(name, zodb.pop('server')) -%}
{% endif -%}
{% endfor -%}
{% do assert(len(mountpoints) == len(zodb_dict)) -%}
{% set neo = [] -%}
{% for server_type, server_dict in six.iteritems(storage_dict) -%}
{% if server_type == 'neo' -%}
{% set ((name, server_dict),) = server_dict.items() -%}
{% do neo.append(server_dict.get('cluster')) -%}
{% do server_dict.update(cluster='${publish-early:neo-cluster}') -%}
{{ root_common.request_neo(server_dict, 'zodb-neo', 'neo-', monitor_base_url_dict) }}
{% set client_dict = zodb_dict[name].setdefault('storage-dict', {}) -%}
{% for k in 'ssl', '_ca', '_cert', '_key' -%}
{% do k in server_dict and client_dict.setdefault(k, server_dict[k]) -%}
{% endfor -%}
{% else -%}
{{ assert(server_type == 'zeo', server_type) -}}
{# BBB: for compatibility, keep 'zodb' as partition_reference for ZEO -#}
{{ request('zodb', 'zodb-' ~ server_type, 'zodb-' ~ server_type, {'tcpv4-port': 2100, 'zodb-dict': server_dict}, dict.fromkeys(('storage-dict', 'tidstorage-ip', 'tidstorage-port', 'monitor-base-url')), key_config={'monitor-passwd': 'monitor-htpasswd:passwd'}) }}
{% endif -%}
{% endfor -%}
[request-zodb-base]
config-zodb-dict = {{ dumps(zodb_dict) }}
{% for server_type, server_dict in six.iteritems(storage_dict) -%}
{% if server_type == 'neo' -%}
config-neo-cluster = ${publish-early:neo-cluster}
config-neo-name = {{ server_dict.keys()[0] }}
config-neo-masters = ${publish-early:neo-masters}
{% else -%}
config-zodb-zeo = ${request-zodb:connection-storage-dict}
config-tidstorage-ip = ${request-zodb:connection-tidstorage-ip}
config-tidstorage-port = ${request-zodb:connection-tidstorage-port}
{% endif -%}
{% endfor -%}
{% set zope_address_list_id_dict = {} -%}
{% if zope_partition_dict -%}
[request-zope-base]
<= request-common
request-zodb-base
return =
zope-address-list
hosts-dict
monitor-base-url
software-release-url
{%- if test_runner_enabled %}
test-runner-address-list
{% endif %}
{% set bt5_default_list = [
'erp5_full_text_mroonga_catalog',
'erp5_configurator_standard',
'erp5_configurator_maxma_demo',
'erp5_configurator_run_my_doc',
] -%}
{% if has_jupyter -%}
{% do bt5_default_list.append('erp5_data_notebook') -%}
{% endif -%}
config-bt5 = {{ dumps(slapparameter_dict.get('bt5', ' '.join(bt5_default_list))) }}
config-bt5-repository-url = {{ dumps(slapparameter_dict.get('bt5-repository-url', local_bt5_repository)) }}
config-cloudooo-url-list = {{ dumps(slapparameter_dict.get('cloudooo-url-list', default_cloudooo_url_list)) }}
config-caucase-url = {{ dumps(caucase_url) }}
config-deadlock-debugger-password = ${publish-early:deadlock-debugger-password}
config-developer-list = {{ dumps(slapparameter_dict.get('developer-list', [inituser_login])) }}
config-selenium-server-configuration-dict = {{ dumps(slapparameter_dict.get('selenium-server-configuration-dict', {})) }}
config-hosts-dict = {{ dumps(slapparameter_dict.get('hosts-dict', {})) }}
config-computer-hosts-dict = {{ dumps(slapparameter_dict.get('computer-hosts-dict', {})) }}
config-hostalias-dict = {{ dumps(slapparameter_dict.get('hostalias-dict', {})) }}
config-id-store-interval = {{ dumps(slapparameter_dict.get('id-store-interval')) }}
config-zope-longrequest-logger-error-threshold = {{ dumps(monitor_dict.get('zope-longrequest-logger-error-threshold', 20)) }}
config-zope-longrequest-logger-maximum-delay = {{ dumps(monitor_dict.get('zope-longrequest-logger-maximum-delay', 0)) }}
config-inituser-login = {{ dumps(inituser_login) }}
config-inituser-password = ${publish-early:inituser-password}
config-kumofs-url = ${request-memcached-persistent:connection-url}
config-memcached-url = ${request-memcached-volatile:connection-url}
config-monitor-passwd = ${monitor-htpasswd:passwd}
config-mysql-test-url-list = ${request-mariadb:connection-test-database-list}
config-mysql-url-list = ${request-mariadb:connection-database-list}
config-site-id = {{ dumps(site_id) }}
config-smtp-url = ${request-smtp:connection-url}
config-timezone = {{ dumps(slapparameter_dict.get('timezone', 'UTC')) }}
config-cloudooo-retry-count = {{ slapparameter_dict.get('cloudooo-retry-count', 2) }}
config-wendelin-core-zblk-fmt = {{ dumps(slapparameter_dict.get('wendelin-core-zblk-fmt', '')) }}
config-wsgi = {{ dumps(slapparameter_dict.get('wsgi', True)) }}
config-test-runner-enabled = {{ dumps(test_runner_enabled) }}
config-test-runner-node-count = {{ dumps(test_runner_node_count) }}
config-test-runner-random-activity-priority = {{ dumps(test_runner_random_activity_priority) }}
config-wcfs_enable = {{ dumps(wcfs_enable) }}
config-test-runner-configuration = {{ dumps(slapparameter_dict.get('test-runner', {})) }}
software-type = zope
{% set global_publisher_timeout = slapparameter_dict.get('publisher-timeout', 300) -%}
{% set global_activity_timeout = slapparameter_dict.get('activity-timeout') -%}
{% set zope_family_dict = {} -%}
{% set zope_family_name_list = [] -%}
{% set zope_backend_path_dict = {} -%}
{% set ssl_authentication_dict = {} -%}
{% set balancer_timeout_dict = {} -%}
{% set jupyter_zope_family_default = [] -%}
{% for custom_name, zope_parameter_dict in six.iteritems(zope_partition_dict) -%}
{% set partition_name = 'zope-' ~ custom_name -%}
{% set section_name = 'request-' ~ partition_name -%}
{% set promise_software_url_section_name = 'promise-software-url' ~ partition_name -%}
{% set zope_family = zope_parameter_dict.get('family', 'default') -%}
{% do zope_family_name_list.append(zope_family) %}
{% set backend_path = zope_parameter_dict.get('backend-path', '') % {'site-id': site_id} %}
{# # default jupyter zope family is first zope family. -#}
{# # use list.append() to update it, because in jinja2 set changes only local scope. -#}
{% if not jupyter_zope_family_default -%}
{% do jupyter_zope_family_default.append(zope_family) -%}
{% endif -%}
{% do zope_family_dict.setdefault(zope_family, []).append(section_name) -%}
{% do zope_backend_path_dict.__setitem__(zope_family, backend_path) -%}
{% do ssl_authentication_dict.__setitem__(zope_family, zope_parameter_dict.get('ssl-authentication', False)) -%}
{% set current_zope_family_override_dict = zope_family_override_dict.get(zope_family, {}) -%}
{% do balancer_timeout_dict.__setitem__(zope_family, current_zope_family_override_dict.get('publisher-timeout', global_publisher_timeout)) -%}
[{{ section_name }}]
<= request-zope-base
name = {{ partition_name }}
{% do monitor_base_url_dict.__setitem__(section_name, '${' ~ section_name ~ ':connection-monitor-base-url}') -%}
{{ root_common.sla(partition_name) }}
config-name = {{ dumps(custom_name) }}
config-instance-count = {{ dumps(zope_parameter_dict['instance-count']) }}
config-private-dev-shm = {{ zope_parameter_dict.get('private-dev-shm', '') }}
config-thread-amount = {{ dumps(zope_parameter_dict['thread-amount']) }}
config-timerserver-interval = {{ dumps(zope_parameter_dict.get('timerserver-interval', 1)) }}
config-longrequest-logger-interval = {{ dumps(zope_parameter_dict.get('longrequest-logger-interval', -1)) }}
config-longrequest-logger-timeout = {{ dumps(zope_parameter_dict.get('longrequest-logger-timeout', 1)) }}
config-large-file-threshold = {{ dumps(zope_parameter_dict.get('large-file-threshold', "10MB")) }}
config-port-base = {{ dumps(zope_parameter_dict.get('port-base', 2200)) }}
{# BBB: zope_parameter_dict used to contain 'webdav', so fallback to it -#}
config-webdav = {{ dumps(current_zope_family_override_dict.get('webdav', zope_parameter_dict.get('webdav', False))) }}
config-publisher-timeout = {{ dumps(current_zope_family_override_dict.get('publisher-timeout', global_publisher_timeout)) }}
config-activity-timeout = {{ dumps(current_zope_family_override_dict.get('activity-timeout', global_activity_timeout)) }}
{% if test_runner_enabled -%}
config-test-runner-apache-url-list = ${publish-early:{{ zope_family }}-test-runner-url-list}
[{{ promise_software_url_section_name }}]
# Promise to wait for zope partition to use the expected software URL,
# used on upgrades.
recipe = slapos.cookbook:check_parameter
value = {{ '${' ~ section_name ~ ':connection-software-release-url}' }}
expected-not-value =
expected-value = ${slap-connection:software-release-url}
path = ${directory:bin}/${:_buildout_section_name_}
{% do root_common.section(promise_software_url_section_name) -%}
{% endif -%}
{% endfor -%}
{# if not explicitly configured, connect jupyter to first zope family, which -#}
{# will be 'default' if zope families are not configured also -#}
{% if not jupyter_zope_family and jupyter_zope_family_default -%}
{% set jupyter_zope_family = jupyter_zope_family_default[0] -%}
{% endif -%}
{# We need to concatenate lists that we cannot read as lists, so this gets hairy. -#}
{% set zope_family_parameter_dict = {} -%}
{% for family_name, zope_section_id_list in zope_family_dict.items() -%}
{% for zope_section_id in zope_section_id_list -%}
{% set parameter_name = 'zope-family-entry-' ~ zope_section_id -%}
{% do zope_address_list_id_dict.__setitem__(zope_section_id, parameter_name) -%}
{% do zope_family_parameter_dict.setdefault(family_name, []).append(parameter_name) -%}
{% endfor -%}
{% if has_frontend -%}
{% set frontend_name = 'frontend-' ~ family_name -%}
{% do publish_dict.__setitem__('family-' ~ family_name, '${' ~ frontend_name ~ ':connection-site_url}' ) -%}
[{{ frontend_name }}]
<= request-frontend-base
name = {{ frontend_name }}
config-url = ${request-balancer:connection-{{ family_name }}-v6}
{% else -%}
{% do publish_dict.__setitem__('family-' ~ family_name, '${request-balancer:connection-' ~ family_name ~ '}' ) -%}
{% do publish_dict.__setitem__('family-' ~ family_name ~ '-v6', '${request-balancer:connection-' ~ family_name ~ '-v6}' ) -%}
{% endif -%}
{% endfor -%}
{% if has_jupyter -%}
{# request jupyter connected to balancer of proper zope family -#}
{{ request('jupyter', 'jupyter', 'jupyter', {}, key_config={'erp5-url': 'request-balancer:connection-' ~ jupyter_zope_family}) }}
{% if has_frontend -%}
[frontend-jupyter]
<= request-frontend-base
name = frontend-jupyter
config-url = ${request-jupyter:connection-url}
{# # override jupyter-url in publish_dict with frontend address -#}
{% do publish_dict.__setitem__('jupyter-url', '${frontend-jupyter:connection-site_url}') -%}
{% endif -%}
{%- endif %}
{% if wcfs_enable -%}
{# request WCFS connected to ZODB -#}
{% do root_common.section('request-wcfs') -%}
{{ request('wcfs', 'wcfs', 'wcfs', {}, {}) }}
[request-wcfs]
<= request-common
request-zodb-base
{%- endif %}
{% set balancer_ret_dict = {'monitor-base-url': False} -%}
{% for family in zope_family_dict -%}
{% do balancer_ret_dict.__setitem__(family, False) -%}
{% do balancer_ret_dict.__setitem__(family + '-v6', False) -%}
{% if test_runner_enabled -%}
{% do balancer_ret_dict.__setitem__(family + '-test-runner-url-list', False) -%}
{% endif -%}
{% endfor -%}
{% set balancer_key_config_dict = {
'monitor-passwd': 'monitor-htpasswd:passwd',
} -%}
{% for zope_section_id, name in zope_address_list_id_dict.items() -%}
{% do balancer_key_config_dict.__setitem__(
name,
zope_section_id + ':connection-zope-address-list',
) -%}
{% if test_runner_enabled -%}
{% do balancer_key_config_dict.__setitem__(
name + '-test-runner-address-list',
zope_section_id + ':connection-test-runner-address-list',
) -%}
{% endif -%}
{% endfor -%}
{{ request(
name='balancer',
software_type='balancer',
config_key='balancer',
config={
'zope-family-dict': zope_family_parameter_dict,
'backend-path-dict': zope_backend_path_dict,
'ssl-authentication-dict': ssl_authentication_dict,
'timeout-dict': balancer_timeout_dict,
'apachedex-promise-threshold': monitor_dict.get('apachedex-promise-threshold', 70),
'apachedex-configuration': monitor_dict.get(
'apachedex-configuration',
[
'--logformat', '%h %l %u %t "%r" %>s %O "%{Referer}i" "%{User-Agent}i" %{ms}T',
'--erp5-base', '+erp5', '.*/VirtualHostRoot/erp5(/|\\?|$)',
'--base', '+other', '/',
'--skip-user-agent', 'Zabbix',
'--error-detail',
'--js-embed',
'--quiet',
],
),
},
ret=balancer_ret_dict,
key_config=balancer_key_config_dict,
) }}
[request-frontend-base]
{% if has_frontend -%}
<= request-common
recipe = slapos.cookbook:request
software-url = {{ dumps(frontend_dict['software-url']) }}
software-type = {{ dumps(frontend_dict.get('software-type', 'RootSoftwareInstance')) }}
{{ root_common.sla('frontend', True) }}
shared = true
{% set config_dict = {
'type': 'zope',
} -%}
{% if frontend_dict.get('domain') -%}
{% do config_dict.__setitem__('custom_domain', frontend_dict['domain']) -%}
{% endif -%}
{% if frontend_dict.get('virtualhostroot-http-port') -%}
{% do config_dict.__setitem__('virtualhostroot-http-port', frontend_dict['virtualhostroot-http-port']) -%}
{% endif -%}
{% if frontend_dict.get('virtualhostroot-https-port') -%}
{% do config_dict.__setitem__('virtualhostroot-https-port', frontend_dict['virtualhostroot-https-port']) -%}
{% endif -%}
{% for name, value in config_dict.items() -%}
config-{{ name }} = {{ value }}
{% endfor -%}
return = site_url
{% endif -%}
{% endif -%}{# if zope_partition_dict -#}
[publish]
<= monitor-publish
recipe = slapos.cookbook:publish.serialised
-extends = publish-early
{% if zope_address_list_id_dict -%}
{#
Pick any published hosts-dict, they are expected to be identical - and there is
no way to check here.
-#}
hosts-dict = {{ '${' ~ next(iter(zope_address_list_id_dict)) ~ ':connection-hosts-dict}' }}
{% endif -%}
{% for name, value in publish_dict.items() -%}
{{ name }} = {{ value }}
{% endfor -%}
{% if test_runner_enabled -%}
{% for zope_family_name in zope_family_name_list -%}
{{ zope_family_name }}-test-runner-url-list = ${request-balancer:connection-{{ zope_family_name }}-test-runner-url-list}
{% endfor -%}
{% endif -%}
[publish-early]
recipe = slapos.cookbook:publish-early
-init =
inituser-password gen-password:passwd
deadlock-debugger-password gen-deadlock-debugger-password:passwd
{%- if has_posftix %}
smtpd-sasl-password gen-smtpd-sasl-password:passwd
{%- endif %}
{%- if test_runner_enabled %}
{%- for zope_family_name in zope_family_name_list %}
{{ zope_family_name }}-test-runner-url-list default-balancer-test-runner-url-list:{{ zope_family_name }}
{%- endfor %}
{%- endif %}
{%- if neo %}
neo-cluster gen-neo-cluster:name
neo-admins neo-cluster:admins
neo-masters neo-cluster:masters
{%- if neo[0] %}
neo-cluster = {{ dumps(neo[0]) }}
{%- endif %}
{%- endif %}
{%- set inituser_password = slapparameter_dict.get('inituser-password') %}
{%- if inituser_password %}
inituser-password = {{ dumps(inituser_password) }}
{%- endif %}
{%- set deadlock_debugger_password = slapparameter_dict.get('deadlock-debugger-password') -%}
{%- if deadlock_debugger_password %}
deadlock-debugger-password = {{ dumps(deadlock_debugger_password) }}
{% endif %}
{%- if test_runner_enabled %}
[default-balancer-test-runner-url-list]
recipe =
{%- for zope_family_name in zope_family_name_list %}
{{ zope_family_name }} = not-ready
{%- endfor %}
{%- endif %}
[gen-password]
recipe = slapos.cookbook:generate.password
storage-path =
[gen-deadlock-debugger-password]
<= gen-password
[gen-neo-cluster-base]
<= gen-password
[gen-neo-cluster]
name = neo-${gen-neo-cluster-base:passwd}
[gen-smtpd-sasl-password]
< = gen-password
{{ root_common.common_section() }}
[monitor-conf-parameters]
monitor-title = ERP5
password = ${monitor-htpasswd:passwd}
[monitor-base-url-dict]
{% for key, value in monitor_base_url_dict.items() -%}
{{ key }} = {{ value }}
{% endfor %}
stack/erp5-zope2/instance-kumofs.cfg.in 0 → 100644
View file @ 8eeb74fe
{% set use_ipv6 = slapparameter_dict.get('use-ipv6', False) -%}
[buildout]
extends =
{{ template_monitor }}
parts +=
publish
kumofs-instance
logrotate-entry-kumofs
resiliency-exclude-file
promise-kumofs-server
promise-kumofs-server-listen
promise-kumofs-gateway
promise-kumofs-manager
promise-check-computer-memory
[publish]
recipe = slapos.cookbook:publish.serialised
{% if use_ipv6 -%}
url = memcached://[${kumofs-instance:ip}]:${kumofs-instance:gateway-port}/
{% else -%}
url = memcached://${kumofs-instance:ip}:${kumofs-instance:gateway-port}/
{% endif -%}
monitor-base-url = ${monitor-publish-parameters:monitor-base-url}
[kumofs-instance]
recipe = slapos.cookbook:generic.kumofs
# Network options
{% if use_ipv6 -%}
ip = {{ (ipv6_set | list)[0] }}
address-family = inet6
{% else -%}
ip = {{ (ipv4_set | list)[0] }}
address-family = inet4
{% endif -%}
{% set tcpv4_port = slapparameter_dict['tcpv4-port'] -%}
manager-port = {{ tcpv4_port }}
server-port = {{ tcpv4_port + 1 }}
server-listen-port = {{ tcpv4_port + 2 }}
gateway-port = {{ tcpv4_port + 3 }}
# Paths: Data
{% set ram_storage_size = slapparameter_dict.get('ram-storage-size') -%}
{% if ram_storage_size -%}
data-path = *#capsiz={{ ram_storage_size }}m
{% else -%}
# (with 10M buckets and HDBTLARGE option)
data-path = ${directory:kumofs-data}/kumodb.tch#bnum=10485760#opts=l
{% endif -%}
# Paths: Running wrappers
gateway-wrapper = ${directory:services}/kumofs_gateway
manager-wrapper = ${directory:services}/kumofs_manager
server-wrapper = ${directory:services}/kumofs_server
# Paths: Logs
kumo-gateway-log = ${directory:log}/kumo-gateway.log
kumo-manager-log = ${directory:log}/kumo-manager.log
kumo-server-log = ${directory:log}/kumo-server.log
# Binary information
kumo-gateway-binary = {{ parameter_dict['kumo-location'] }}/bin/kumo-gateway
kumo-manager-binary = {{ parameter_dict['kumo-location'] }}/bin/kumo-manager
kumo-server-binary = {{ parameter_dict['kumo-location'] }}/bin/kumo-server
shell-path = {{ parameter_dict['dash-location'] }}/bin/dash
[logrotate-entry-kumofs]
< = logrotate-entry-base
name = kumofs
log = ${kumofs-instance:kumo-gateway-log} ${kumofs-instance:kumo-manager-log} ${kumofs-instance:kumo-server-log}
[directory]
recipe = slapos.cookbook:mkdirectory
log = ${buildout:directory}/var/log
services = ${buildout:directory}/etc/run
plugin = ${buildout:directory}/etc/plugin
srv = ${buildout:directory}/srv
kumofs-data = ${:srv}/kumofs
[resiliency-exclude-file]
# Generate rdiff exclude file in case of resiliency
recipe = slapos.recipe.template:jinja2
inline = {{ '{{ "**\\n" }}' }}
output = ${directory:srv}/exporter.exclude
# Deploy zope promises scripts
[promise-template]
<= monitor-promise-base
promise = check_socket_listening
config-host = ${kumofs-instance:ip}
config-port = ${kumofs-instance:server-listen-port}
[promise-kumofs-server]
<= promise-template
name = kumofs-server.py
config-port = ${kumofs-instance:server-port}
[promise-kumofs-server-listen]
<= promise-template
name = kumofs-server-listen.py
config-port = ${kumofs-instance:server-listen-port}
[promise-kumofs-gateway]
<= promise-template
name = kumofs-gateway.py
config-port = ${kumofs-instance:gateway-port}
[promise-kumofs-manager]
<= promise-template
name = kumofs-manager.py
config-port = ${kumofs-instance:manager-port}
[promise-check-computer-memory]
<= monitor-promise-base
promise = check_command_execute
name = check-computer-memory.py
config-command = "{{ parameter_dict["check-computer-memory-binary"] }}" -db ${monitor-instance-parameter:collector-db} --threshold "{{ slapparameter_dict["computer-memory-percent-threshold"] }}" --unit percent
[monitor-instance-parameter]
monitor-httpd-ipv6 = {{ (ipv6_set | list)[0] }}
monitor-httpd-port = {{ tcpv4_port + 4 }}
monitor-title = {{ slapparameter_dict['name'] }}
password = {{ slapparameter_dict['monitor-passwd'] }}
stack/erp5-zope2/instance-mariadb-resiliency-after-import-script.sh.in 0 → 100644
View file @ 8eeb74fe
#!{{ dash }}
# DO NOT RUN THIS SCRIPT ON PRODUCTION INSTANCE
# OR MYSQL DATA WILL BE ERASED.
# This script will import the dump of the mysql database to the real
# database. It is launched by the clone (importer) instance of webrunner
# in the end of the import script.
# Depending on the output, it will create a file containing
# the status of the restoration (success or failure)
set -e
mysql_executable='{{ mysql_executable }}'
mariadb_data_directory='{{ mariadb_data_directory }}'
mariadb_backup_directory='{{ mariadb_backup_directory }}'
pid_file='{{ pid_file }}'
binlog_path='{{ binlog_path }}'
server_executable='{{ server_executable }}'
# Make sure mariadb is not already running
if [ -e "$pid_file" ]; then
if ! pid=$(cat "$pid_file"); then
echo "Cannot read Mariadb pidfile, assuming running. Aborting."
exit 1
fi
if kill -0 "$pid"; then
echo "Mariadb is already running with pid $pid. Aborting."
exit 1
fi
fi
echo "Deleting existing database..."
find "$mariadb_data_directory" -mindepth 1 -delete
# $binlog_path can be empty if incremental_backup_retention_days <= -1
if [ -n "$binlog_path" ]; then
new_binlog_directory="$(dirname "$binlog_path")"
binlog_index_file="$new_binlog_directory/binlog.index"
if [ -e "$binlog_index_file" ]; then
echo "Adapting binlog database to new paths..."
old_binlog_directory="$(dirname $(head -n 1 $binlog_index_file))"
sed -e "s|$old_binlog_directory|$new_binlog_directory|g" $binlog_index_file > $binlog_index_file
fi
fi
echo "Starting mariadb..."
"$server_executable" --innodb-flush-method=nosync --skip-innodb-doublewrite --innodb-flush-log-at-trx-commit=0 --sync-frm=0 --slow-query-log=0 --skip-log-bin &
mysqld_pid=$!
trap "kill $mysqld_pid" EXIT TERM INT
sleep 30
# If mysql has stopped, abort
if ! [ -d /proc/$mysql_pid ]; then
echo "mysqld exited, aborting."
exit 1
fi
echo "Importing data..."
# Use latest dump XXX can contain funny characters
dump=$(ls -r "$mariadb_backup_directory" | head -1)
zcat "$mariadb_backup_directory/$dump" | $mysql_executable || {
RESTORE_EXIT_CODE=$?
echo 'Backup restoration failed.'
exit $RESTORE_EXIT_CODE
}
echo 'Backup restoration successfully completed.'
stack/erp5-zope2/instance-mariadb-start-clone-from-backup.sh.in 0 → 100644
View file @ 8eeb74fe
#!{{ dash }}
set -eu
if [ $# -ne 7 ]; then
echo "Bootstrap a mariadb instance from available backup data so it replicates from given master."
echo " $0 <BACKUP FILE> <MASTER HOST> <MASTER PORT> <MASTER USER> <MASTER SSL CA FILE> <MASTER SSL CERT FILE> <MASTER SSL KEY FILE>"
exit 1
fi
BACKUP=$1
MASTER_HOST=$2
MASTER_PORT=$3
MASTER_USER=$4
MASTER_SSL_CA=$5
MASTER_SSL_CERT=$6
MASTER_SSL_KEY=$7
CLIENT='{{ client }}'
DATA_DIRECTORY='{{ data_directory }}'
PID_FILE='{{ pid_file }}'
SERVER='{{ server }}'
UPDATE='{{ update }}'
SOCKET='{{ socket }}'
# Make sure mariadb is not already running
if [ -e "$PID_FILE" ]; then
PID=$(cat "$PID_FILE")
if [ $? -ne 0 ]; then
echo "Cannot read Mariadb pidfile, assuming running. Aborting."
exit 1
fi
if kill -0 "$PID"; then
echo "Mariadb is already running with pid $PID. Aborting."
exit 1
fi
fi
BACKUP_HEAD="$(zcat "$BACKUP" | head -n 100)"
SQL_CHANGE_MASTER=$(echo "$BACKUP_HEAD" | grep "^--\s*CHANGE MASTER TO " | sed "s/^--\s*//")
if [ -z "$SQL_CHANGE_MASTER" ]; then
echo "'CHANGE MASTER TO' statement not found in given backup file."
echo "Is replication enabled on future master ?"
exit 1
fi
SQL_SET_GTID="$(echo "$BACKUP_HEAD" | grep "^--\s*SET GLOBAL gtid_slave_pos=" | sed "s/^--\s*//")"
if [ -z "$SQL_SET_GTID" ]; then
echo "Info: GTID not found in backup, it will not be enabled."
MASTER_USE_GTID=0
else
echo "Info: GTID found in backup, it will be enabled."
MASTER_USE_GTID=1
fi
echo "EXISTING DATABASE CONTENT WILL BE DESTROYED"
echo "You have 5 seconds to interrupt this script..."
if sleep 5; then
echo "Expired, proceeding"
else
echo "Interrupted, aborting"
exit 1
fi
echo "Emptying data directory..."
find "$DATA_DIRECTORY" -mindepth 1 -delete
echo -n "Starting mariadb for backup restoration"
"$SERVER" --innodb-flush-method=nosync --skip-innodb-doublewrite --innodb-flush-log-at-trx-commit=0 --sync-frm=0 --slow-query-log=0 --skip-log-bin &
PID=$!
trap "kill $PID; wait; exit 1" EXIT
while true; do
if [ ! -e "/proc/$PID" ]; then
trap EXIT
echo "Service exited, check logs"
wait
exit 1
fi
test -e "$SOCKET" && break
echo -n .
sleep 0.5
done
"$UPDATE"
echo "Importing $BACKUP ..."
zcat "$BACKUP" | "$CLIENT"
echo "Configuring server as slave..."
if [ "$MASTER_USE_GTID" -eq 1 ]; then
"$CLIENT" -e "$SQL_SET_GTID"
MASTER_USE_GTID_SQL="slave_pos"
else
MASTER_USE_GTID_SQL="NO"
fi
"$CLIENT" -e "
CHANGE MASTER TO
MASTER_HOST='$MASTER_HOST',
MASTER_USER='$MASTER_USER',
MASTER_PORT=$MASTER_PORT,
MASTER_SSL=1,
MASTER_SSL_CA='$MASTER_SSL_CA',
MASTER_SSL_CERT='$MASTER_SSL_CERT',
MASTER_SSL_KEY='$MASTER_SSL_KEY',
MASTER_SSL_VERIFY_SERVER_CERT=1,
MASTER_USE_GTID=$MASTER_USE_GTID_SQL;
"
if [ "$MASTER_USE_GTID" -eq 0 ]; then
# No GTID, use binlog name & offset as provided by backup file.
# Example: CHANGE MASTER TO MASTER_LOG_FILE='binlog.003447', MASTER_LOG_POS=360;
# Notes:
# - Must happen after setting MASTER_HOST & MASTER_PORT.
# - Implicitly sets MASTER_USE_GTID=NO if it was set before.
"$CLIENT" -e "$SQL_CHANGE_MASTER"
fi
"$CLIENT" -e "START SLAVE;"
echo "Stopping mariadb..."
trap EXIT
kill $PID
wait
echo "Done. Start mariadb normally. You may use 'show slave status' SQL command to monitor progress."
stack/erp5-zope2/instance-mariadb.cfg.in 0 → 100644
View file @ 8eeb74fe
{% set part_list = [] -%}
{% macro section(name) %}{% do part_list.append(name) %}{{ name }}{% endmacro -%}
{% set use_ipv6 = slapparameter_dict.get('use-ipv6', False) -%}
{% set database_list = slapparameter_dict.get('database-list', [{'name': 'erp5', 'user': 'user', 'password': 'insecure', 'with-process-privilege': True}]) -%}
{% set test_database_list = [] %}
{% for database_count in range(slapparameter_dict.get('test-database-amount', 1)) -%}
{% do test_database_list.append({'name': 'erp5_test_' ~ database_count, 'user': 'testuser_' ~ database_count, 'password': 'testpassword' ~ database_count}) -%}
{% endfor -%}
{% set catalog_backup = slapparameter_dict.get('catalog-backup', {}) -%}
{% set backup_periodicity = slapparameter_dict.get('backup-periodicity', 'daily') -%}
{% set full_backup_retention_days = catalog_backup.get('full-retention-days', 7) -%}
{% set incremental_backup_retention_days = catalog_backup.get('incremental-retention-days', full_backup_retention_days) -%}
{% set port = slapparameter_dict['tcpv4-port'] %}
{% if use_ipv6 -%}
{% set ip = (ipv6_set | list)[0] -%}
{% else -%}
{% set ip = (ipv4_set | list)[0] -%}
{% endif -%}
{% set dash = parameter_dict['dash-location'] ~ '/bin/dash' %}
[{{ section('publish') }}]
recipe = slapos.cookbook:publish.serialised
-extends = publish-early
{% macro render_database_list(database_list) -%}
{% set publish_database_list = [] -%}
{% for database in database_list -%}
{% if database.get('user') -%}
{% do publish_database_list.append("mysql://" ~ database['user'] ~ ":" ~ database['password'] ~ "@" ~ ip ~ ":" ~ port ~ "/" ~ database['name']) -%}
{% else -%}
{% do publish_database_list.append("mysql://" ~ ip ~ ":" ~ port ~ "/" ~ database['name']) -%}
{% endif -%}
{% endfor -%}
{{ dumps(publish_database_list) }}
{% endmacro -%}
database-list = {{ render_database_list(database_list) }}
test-database-list = {{ render_database_list(test_database_list) }}
monitor-base-url = ${monitor-publish-parameters:monitor-base-url}
[publish-early]
recipe = slapos.cookbook:publish-early
-init =
server-id gen-server-id:value
{%- set server_id = slapparameter_dict.get('server-id') %}
{%- if server_id %}
server-id = {{ dumps(server_id) }}
{%- endif %}
[gen-server-id]
recipe = slapos.cookbook:random.integer
minimum = {{ dumps(1) }}
maximum = {{ dumps(2**32 - 1) }}
[simplefile]
recipe = slapos.recipe.template:jinja2
inline = {{ '{{ content }}' }}
{% macro simplefile(section_name, file_path, content, mode='') -%}
{% set content_section_name = section_name ~ '-content' -%}
[{{ content_section_name }}]
content = {{ dumps(content) }}
[{{ section(section_name) }}]
< = simplefile
output = {{ file_path }}
context = key content {{content_section_name}}:content
mode = {{ mode }}
{%- endmacro %}
{% set ssl_dict = {} -%}
{% macro sslfile(key, content, mode='644') -%}
{% set path = '${directory:mariadb-ssl}/' ~ key ~ '.pem' -%}
{% do ssl_dict.__setitem__(key, path) -%}
{{ simplefile('ssl-file-' ~ key, path, content, mode) }}
{%- endmacro %}
{% set ssl_parameter_dict = slapparameter_dict.get('ssl') -%}
{% if ssl_parameter_dict -%}
{% set base_directory = '${directory:mariadb-ssl}/' -%}
{# Note: The key content will be stored in .installed.cfg, and this template's
rendering, so the only point of mode is to avoid risking mariadb complaining
about laxist file mode. -#}
{{ sslfile('key', ssl_parameter_dict['key'], mode='600') }}
{{ sslfile('crt', ssl_parameter_dict['crt']) }}
{% if 'ca-crt' in ssl_parameter_dict -%}
{{ sslfile('ca-crt', ssl_parameter_dict['ca-crt']) }}
{% endif -%}
{% if 'crl' in ssl_parameter_dict -%}
{{ sslfile('crl', ssl_parameter_dict['crl']) }}
{% endif -%}
{%- endif %}
{% if full_backup_retention_days > -1 -%}
[{{ section('cron-entry-mariadb-backup') }}]
recipe = slapos.cookbook:cron.d
cron-entries = ${cron:cron-entries}
name = mariadb-backup
time = {{ dumps(backup_periodicity) }}
{# When binlogs are enabled:
# flush-logs: used so no manipulation on binlogs is needed to restore from
# full + binlogs. The first binlog after a dump starts from dump snapshot and
# can be fully restored.
# master-data: use value "2" as we are not in a replication case
#}
command = "${binary-wrap-mysqldump:wrapper-path}" --all-databases --flush-privileges --single-transaction --max-allowed-packet=128M {% if incremental_backup_retention_days > -1 %}--flush-logs --master-data=2 {% endif %}| {{ parameter_dict['gzip-location'] }}/bin/gzip > "${directory:mariadb-backup-full}/$({{ parameter_dict['coreutils-location'] }}/bin/date "+%Y%m%d%H%M%S").sql.gz"
{# KEEP GLOB PATTERN IN SYNC with generated filenames above
# YYYYmmddHHMMSS -#}
file-glob = ??????????????.sql.gz
{% if full_backup_retention_days > 0 -%}
[{{ section("cron-entry-mariadb-backup-expire") }}]
recipe = slapos.cookbook:cron.d
cron-entries = ${cron:cron-entries}
name = mariadb-backup-expire
time = {{ dumps(backup_periodicity) }}
command = {{ parameter_dict['findutils-location'] }}/bin/find "${directory:mariadb-backup-full}" -maxdepth 1 -name "${cron-entry-mariadb-backup:file-glob}" -daystart -mtime +{{ full_backup_retention_days }} -delete
{%- endif %}
{%- endif %}
[my-cnf-parameters]
ip = {{ ip }}
port = {{ port }}
socket = ${directory:run}/mariadb.sock
data-directory = ${directory:srv}/mariadb
tmp-directory = ${directory:tmp}
etc-directory = ${directory:etc}
plugin-directory = {{ dumps(parameter_dict['mroonga-mariadb-plugin-dir']) }}
groonga-plugins-path = {{ parameter_dict['groonga-plugins-path'] }}
pid-file = ${directory:run}/mariadb.pid
error-log = ${directory:log}/mariadb_error.log
slow-query-log = ${directory:log}/mariadb_slowquery.log
long-query-time = {{ dumps(slapparameter_dict.get('long-query-time', 1)) }}
max-connection-count = {{ dumps(slapparameter_dict.get('max-connection-count', 1000)) }}
innodb-buffer-pool-size = {{ dumps(slapparameter_dict.get('innodb-buffer-pool-size', 0)) }}
innodb-buffer-pool-instances = {{ dumps(slapparameter_dict.get('innodb-buffer-pool-instances', 0)) }}
innodb-log-file-size = {{ dumps(slapparameter_dict.get('innodb-log-file-size', 0)) }}
innodb-file-per-table = {{ dumps(slapparameter_dict.get('innodb-file-per-table', 0)) }}
innodb-log-buffer-size = {{ dumps(slapparameter_dict.get('innodb-log-buffer-size', 0)) }}
relaxed-writes = {{ dumps(slapparameter_dict.get('relaxed-writes', False)) }}
character-set-server = {{ dumps(slapparameter_dict.get('character-set-server', 'utf8mb4')) }}
collation-server = {{ dumps(slapparameter_dict.get('collation-server', 'utf8mb4_general_ci')) }}
{% if incremental_backup_retention_days > -1 -%}
binlog-path = ${directory:mariadb-backup-incremental}/binlog
# XXX: binlog rotation happens along with other log's rotation
binlog-expire-days = {{ dumps(incremental_backup_retention_days) }}
server-id = ${publish-early:server-id}
{% else %}
binlog-path =
{%- endif %}
{%- for key, value in ssl_dict.items() -%}
ssl-{{ key }} = {{ value }}
{% endfor %}
[my-cnf]
recipe = slapos.recipe.template:jinja2
output = ${directory:etc}/mariadb.cnf
url = {{ parameter_dict['template-my-cnf'] }}
context = section parameter_dict my-cnf-parameters
[init-script-parameters]
database-list = {{ dumps(database_list + test_database_list) }}
mroonga-mariadb-install-sql = {{ dumps(parameter_dict['mroonga-mariadb-install-sql']) }}
[init-script]
recipe = slapos.recipe.template:jinja2
# XXX: is there a better location ?
output = ${directory:etc}/mariadb_initial_setup.sql
url = {{ parameter_dict['template-mariadb-initial-setup'] }}
context = section parameter_dict init-script-parameters
[{{ section('update-mysql') }}]
recipe = slapos.cookbook:generic.mysql.wrap_update_mysql
output = ${directory:services}/mariadb_update
binary = ${binary-wrap-mysql_upgrade:wrapper-path}
mysql = ${binary-wrap-mysql:wrapper-path}
init-script = ${init-script:output}
mysql_tzinfo_to_sql = ${binary-wrap-mysql_tzinfo_to_sql:wrapper-path}
[{{ section('mysqld') }}]
recipe = slapos.recipe.template:jinja2
output = ${directory:services}/mariadb
url = {{ parameter_dict['template-mysqld-wrapper'] }}
context =
key defaults_file my-cnf:output
key datadir my-cnf-parameters:data-directory
key environ :environ
environ =
GRN_PLUGINS_PATH='${my-cnf-parameters:groonga-plugins-path}'
ODBCSYSINI='${my-cnf-parameters:etc-directory}'
LD_LIBRARY_PATH=$${LD_LIBRARY_PATH:+$LD_LIBRARY_PATH:}'{{ parameter_dict['unixodbc-location'] }}/lib'
{%- for variable in slapparameter_dict.get('environment-variables', ()) %}
{{ variable }}
{%- endfor %}
[{{ section('odbc-ini') }}]
recipe = slapos.recipe.template
output = ${directory:etc}/odbc.ini
inline = {{ dumps(slapparameter_dict.get('odbc-ini', ' ')) }}
[{{ section('logrotate-entry-mariadb') }}]
< = logrotate-entry-base
name = mariadb
log = ${my-cnf-parameters:error-log} ${my-cnf-parameters:slow-query-log}
post = "${binary-wrap-mysql:wrapper-path}" -B -e "FLUSH LOGS"
[{{ section('binary-link') }}]
recipe = slapos.cookbook:symbolic.link
target-directory = ${directory:bin}
link-binary = {{ dumps(parameter_dict['link-binary']) }}
[binary-wrap-base]
recipe = slapos.cookbook:wrapper
# Note: --defaults-file must be the first argument, otherwise wrapped binary
# will reject it.
command-line = "{{ parameter_dict['mariadb-location'] }}/bin/${:command}" --defaults-file="${my-cnf:output}"
wrapper-path = ${directory:bin}/${:command}
[binary-wrap-mysql]
< = binary-wrap-base
command = mysql
[binary-wrap-mysqldump]
< = binary-wrap-base
command = mysqldump
[binary-wrap-mysql_upgrade]
< = binary-wrap-base
command = mysql_upgrade
[binary-wrap-mysql_tzinfo_to_sql]
< = binary-wrap-base
command-line = "{{ parameter_dict['mariadb-location'] }}/bin/${:command}"
command = mysql_tzinfo_to_sql
[binary-wrap-pt-digest]
<= binary-wrap-base
command-line = "{{ parameter_dict['percona-tools-location'] }}/bin/${:command}"
command = pt-query-digest
[directory]
recipe = slapos.cookbook:mkdirectory
bin = ${buildout:directory}/bin
etc = ${buildout:directory}/etc
services = ${:etc}/run
plugin = ${:etc}/plugin
srv = ${buildout:directory}/srv
tmp = ${buildout:directory}/tmp
backup = ${:srv}/backup
mariadb-backup-full = ${:backup}/mariadb-full
mariadb-backup-incremental = ${:backup}/mariadb-incremental
mariadb-ssl = ${:etc}/mariadb-ssl
var = ${buildout:directory}/var
log = ${:var}/log
run = ${:var}/run
slowquery = ${monitor-directory:private}/slowquery_digest
[{{ section('resiliency-exclude-file') }}]
# Generate rdiff exclude file in case of resiliency
recipe = slapos.recipe.template:jinja2
inline = {{ '{{ "${my-cnf-parameters:data-directory}/**\\n${directory:mariadb-backup-incremental}/**\\n${directory:log}/**\\n${directory:tmp}/**\\n" }}' }}
output = ${directory:srv}/exporter.exclude
[{{ section("resiliency-identity-signature-script")}}]
# Generate identity script used by webrunner to check data integrity
recipe = slapos.cookbook:wrapper
command-line = {{ bin_directory }}/backup-identity-script-excluding-path --exclude-path "srv/backup/logrotate/**"
wrapper-path = ${directory:srv}/.backup_identity_script
mode = 770
[dash]
dash = {{ dumps(dash) }}
[{{ section('start-clone-from-backup') }}]
recipe = slapos.recipe.template:jinja2
url = {{ parameter_dict['mariadb-start-clone-from-backup'] }}
output = ${directory:bin}/start-clone-from-backup
context =
key dash dash:dash
key client binary-wrap-mysql:wrapper-path
key data_directory my-cnf-parameters:data-directory
key pid_file my-cnf-parameters:pid-file
key server mysqld:output
key update update-mysql:output
key socket my-cnf-parameters:socket
[{{ section('resiliency-after-import-script') }}]
# Generate after import script used by importer instance of webrunner
recipe = slapos.recipe.template:jinja2
url = {{ parameter_dict['mariadb-resiliency-after-import-script'] }}
output = ${directory:bin}/restore-from-backup
context =
key dash dash:dash
key mysql_executable binary-wrap-mysql:wrapper-path
key mariadb_data_directory my-cnf-parameters:data-directory
key mariadb_backup_directory directory:mariadb-backup-full
key pid_file my-cnf-parameters:pid-file
key binlog_path my-cnf-parameters:binlog-path
key server_executable mysqld:output
[{{ section('monitor-generate-mariadb-slow-query-report') }}]
recipe = slapos.cookbook:cron.d
cron-entries = ${cron:cron-entries}
name = generate-mariadb-slow-query-report
# The goal is to be executed before logrotate log rotation.
# Here, logrotate-entry-base:frequency = daily, so we run at 23 o'clock every day.
frequency = 0 23 * * *
command = ${monitor-generate-mariadb-slow-query-report-wrapper:output}
[monitor-generate-mariadb-slow-query-report-wrapper]
recipe = slapos.recipe.template:jinja2
url = {{ parameter_dict['mariadb-slow-query-report-script'] }}
output = ${directory:bin}/${:filename}
filename = generate-mariadb-slow-query-report
context =
raw slow_query_path ${directory:srv}/backup/logrotate/mariadb_slowquery.log
raw pt_query_exec ${binary-wrap-pt-digest:wrapper-path}
raw dash {{ parameter_dict['dash-location'] }}/bin/dash
raw xz {{ parameter_dict['xz-utils-location'] }}/bin/xz
key output_folder directory:slowquery
{%if slapparameter_dict.get('max-slowqueries-threshold') and slapparameter_dict.get('slowest-query-threshold') %}
[{{ section('monitor-promise-slowquery-result') }}]
<= monitor-promise-base
promise = check_command_execute
name = check-slow-query-pt-digest-result.py
config-command = "{{ parameter_dict['promise-check-slow-queries-digest-result'] }}" --ptdigest_path "${directory:slowquery}" --status_file ${monitor-directory:private}/mariadb_slow_query.report.json --max_queries_threshold "${:max_queries_threshold}" --slowest_query_threshold "${:slowest_queries_threshold}"
max_queries_threshold = {{ slapparameter_dict['max-slowqueries-threshold'] }}
slowest_queries_threshold = {{ slapparameter_dict['slowest-query-threshold'] }}
{%-endif%}
[{{ section('promise-check-computer-memory') }}]
<= monitor-promise-base
promise = check_command_execute
name = check-computer-memory.py
config-command = "{{ parameter_dict["check-computer-memory-binary"] }}" -db ${monitor-instance-parameter:collector-db} --threshold "{{ slapparameter_dict["computer-memory-percent-threshold"] }}" --unit percent
[{{ section('promise') }}]
<= monitor-promise-base
promise = check_command_execute
name = mariadb.py
config-command = "${binary-wrap-mysql:wrapper-path}" --execute ';' {% if database_list and database_list[0].get('user') %} --host="${my-cnf-parameters:ip}" --port="${my-cnf-parameters:port}" --user="{{ database_list[0]['user'] }}" --password="{{ database_list[0]['password'] }}" {% endif %}
[monitor-instance-parameter]
monitor-httpd-ipv6 = {{ (ipv6_set | list)[0] }}
monitor-httpd-port = {{ port + 1 }}
monitor-title = {{ slapparameter_dict['name'] }}
password = {{ slapparameter_dict['monitor-passwd'] }}
[buildout]
extends =
{{ template_monitor }}
parts +=
{{ part_list | join('\n ') }}
stack/erp5-zope2/instance-postfix.cfg.in 0 → 100644
View file @ 8eeb74fe
{% set part_list = [] -%}
{% macro section(name) %}{% do part_list.append(name) %}{{ name }}{% endmacro -%}
{% if slapparameter_dict['use-ipv6'] -%}
{% set ip = '[' ~ (ipv6_set | list)[0] ~ ']' -%}
{% else -%}
{% set ip = (ipv4_set | list)[0] -%}
{% endif -%}
{% set tcpv4_port = slapparameter_dict['tcpv4-port'] -%}
{% set relay = slapparameter_dict.get('relay', {}) -%}
{% set divert = slapparameter_dict.get('divert', []) -%}
{% set alias_dict = slapparameter_dict.get('alias-dict', {}) -%}
{% do alias_dict.setdefault('postmaster', [slapparameter_dict['postmaster']]) -%}
{% set smtpd_sasl_user = slapparameter_dict['smtpd-sasl-user'] -%}
{% set smtpd_sasl_password = slapparameter_dict['smtpd-sasl-password'] -%}
{% set milter_list = [] %}
[jinja2-template-base]
recipe = slapos.recipe.template:jinja2
[smtpd-password]
recipe = slapos.cookbook:generate.password
storage-path =
[{{ section('publish') }}]
recipe = slapos.cookbook:publish.serialised
url = {{ dumps('smtp://' ~ urllib.quote_plus(smtpd_sasl_user) ~ ':' ~ urllib.quote_plus(smtpd_sasl_password) ~ '@' ~ ip ~ ':' ~ tcpv4_port) }}
[directory]
recipe = slapos.cookbook:mkdirectory
etc = ${buildout:directory}/etc
plugin = ${:etc}/plugin
etc-postfix = ${:etc}/postfix
etc-cyrus = ${:etc}/cyrus
run = ${:etc}/run
bin = ${buildout:directory}/bin
usr = ${buildout:directory}/usr
srv = ${buildout:directory}/srv
var = ${buildout:directory}/var
var-log = ${:var}/log
var-lib = ${:var}/lib
var-lib-postfix = ${:var-lib}/postfix
var-spool = ${:var}/spool
var-spool-postfix = ${:var-spool}/postfix
# Not used at buildout level, presence needed by postfix.
var-spool-postfix-active = ${:var-spool-postfix}/active
var-spool-postfix-bounce = ${:var-spool-postfix}/bounce
var-spool-postfix-corrupt = ${:var-spool-postfix}/corrupt
var-spool-postfix-defer = ${:var-spool-postfix}/defer
var-spool-postfix-deferred = ${:var-spool-postfix}/deferred
var-spool-postfix-flush = ${:var-spool-postfix}/flush
var-spool-postfix-hold = ${:var-spool-postfix}/hold
var-spool-postfix-incoming = ${:var-spool-postfix}/incoming
var-spool-postfix-maildrop = ${:var-spool-postfix}/maildrop
var-spool-postfix-pid = ${:var-spool-postfix}/pid
var-spool-postfix-private = ${:var-spool-postfix}/private
var-spool-postfix-public = ${:var-spool-postfix}/public
var-spool-postfix-saved = ${:var-spool-postfix}/saved
var-spool-postfix-trace = ${:var-spool-postfix}/trace
# Used for ERP5 resiliency or (more probably)
# webrunner resiliency with erp5 inside.
[{{ section("resiliency-exclude-file") }}]
# Generate rdiff exclude file
recipe = slapos.recipe.template
inline = {{ '{{ "**\\n" }}' }}
output = ${directory:srv}/exporter.exclude
{% if divert -%}
{% set milter_port = tcpv4_port + 1 -%}
{% set socket = 'inet:' ~ ip ~ ':' ~ milter_port -%}
[{{ section('divert-milter') }}]
recipe = slapos.cookbook:wrapper
command-line =
'{{ parameter_dict['buildout-bin-directory'] }}/munnel'
--listen '{{ socket }}'
-- {{ ' '.join(divert) }}
wrapper-path = ${directory:run}/munnel
{% do milter_list.append(socket) -%}
[{{ section('munnel-promise') }}]
<= monitor-promise-base
promise = check_socket_listening
name = munnel.py
config-host = {{ ip }}
config-port = {{ milter_port }}
{% endif -%}
[configuration]
smtp = {{ dumps(tcpv4_port) }}
inet-interfaces = {{ dumps(ip) }}
alias-dict = {{ dumps(alias_dict) }}
relayhost = {{ dumps(relay.get('host')) }}
relay-sasl-credential = {{ dumps(relay.get('sasl-credential')) }}
cyrus-sasldb = ${directory:etc-cyrus}/postfix.gdbm
milter-list = {{ dumps(milter_list) }}
xz-utils-location = {{ dumps(parameter_dict['xz-utils-location']) }}
[userinfo]
recipe = slapos.cookbook:userinfo
[smtp-sasl-passwd]
< = jinja2-template-base
output = ${directory:etc-postfix}/sasl_passwd
{% if relay -%}
inline = {{ "{{ host }} {{ sasl_credential }}" }}
{%- else -%}
inline =
{%- endif %}
context =
key host configuration:relayhost
key sasl_credential configuration:relay-sasl-credential
[{{ section('cyrus-smtpd-conf') }}]
< = jinja2-template-base
output = ${directory:etc-cyrus}/smtpd.conf
inline =
pwcheck_method: auxprop
mech_list: PLAIN LOGIN
sasldb_path: {{ '{{ sasldb }}' }}
context =
key sasldb configuration:cyrus-sasldb
[{{ section('cyrus-smtpd-password') }}]
recipe = plone.recipe.command
stop-on-error = true
command =
rm -f '${configuration:cyrus-sasldb}' &&
echo '{{ smtpd_sasl_password }}' | '${wrapper-postfix-saslpasswd2:wrapper-path}' -pc '{{ smtpd_sasl_user }}'
update-command = ${:command}
[smtpd-ssl]
recipe = plone.recipe.command
stop-on-error = true
openssl = '{{ parameter_dict['openssl'] }}/bin/openssl'
cert = ${directory:etc-postfix}/smtpd.crt
key = ${directory:etc-postfix}/smtpd.pem
dh-512 = ${directory:etc-postfix}/dh512.pem
dh-2048 = ${directory:etc-postfix}/dh2048.pem
command =
${:openssl} dhparam -out '${:dh-512}' 512 &&
${:openssl} dhparam -out '${:dh-2048}' 2048 &&
${:update}
update =
${:openssl} req -newkey rsa -batch -new -x509 -days 3650 -nodes -keyout '${:key}' -out '${:cert}'
[{{ section('postfix-logrotate') }}]
recipe = slapos.cookbook:cron.d
cron-entries = ${cron:cron-entries}
name = postfix-logrotate
frequency = 0 0 * * *
command = ${directory:bin}/postfix logrotate
[postfix-main-cf-parameter]
postfix-location = {{ parameter_dict['postfix-location'] }}
[{{ section('postfix-main-cf') }}]
< = jinja2-template-base
output = ${directory:etc-postfix}/main.cf
url = {{ parameter_dict['template-postfix-main-cf'] }}
context =
key bin_directory directory:bin
key usr_directory directory:usr
key queue_directory directory:var-spool-postfix
key data_directory directory:var-lib-postfix
key spool_directory directory:var-spool
key mail_owner userinfo:pw-name
key setgid_group userinfo:gr-name
key inet_interfaces configuration:inet-interfaces
key relayhost configuration:relayhost
key sasl_passwd typed-paths:smtp-sasl-passwd
key aliases typed-paths:aliases
key milter_list configuration:milter-list
key cyrus_directory directory:etc-cyrus
key cert smtpd-ssl:cert
key key smtpd-ssl:key
key dh_512 smtpd-ssl:dh-512
key dh_2048 smtpd-ssl:dh-2048
key log_directory directory:var-log
key xz_utils_location configuration:xz-utils-location
key postfix_location postfix-main-cf-parameter:postfix-location
key etc_postfix directory:etc-postfix
[{{ section('postfix-master-cf') }}]
< = jinja2-template-base
output = ${directory:etc-postfix}/master.cf
url = {{ parameter_dict['template-postfix-master-cf'] }}
context = key smtp configuration:smtp
[aliases]
< = jinja2-template-base
url = {{ parameter_dict['template-postfix-aliases'] }}
output = ${directory:etc-postfix}/aliases
context =
key alias_dict configuration:alias-dict
[typed-paths]
# Postfix-friendly rendering of file paths, prefixed with database type.
aliases = hash:${aliases:output}
smtp-sasl-passwd = hash:${smtp-sasl-passwd:output}
[{{ section('postalias-db') }}]
recipe = plone.recipe.command
stop-on-error = true
command = '${wrapper-postalias:wrapper-path}' '${typed-paths:aliases}' '${typed-paths:smtp-sasl-passwd}'
update-command = ${:command}
[wrapper-postfix-saslpasswd2]
recipe = slapos.cookbook:wrapper
command-line = '{{ parameter_dict['cyrus-sasl-location'] }}/sbin/saslpasswd2' -f '${configuration:cyrus-sasldb}'
wrapper-path = ${directory:bin}/saslpasswd2
[base-wrapper]
recipe = slapos.cookbook:wrapper
environment =
MAIL_CONFIG=${directory:etc-postfix}
SASL_CONF_PATH=${directory:etc-cyrus}
[base-bin-wrapper]
< = base-wrapper
command-line = ${:path}/${:basename}
wrapper-path = ${directory:bin}/${:basename}
[base-bin-bin-wrapper]
< = base-bin-wrapper
path = {{ parameter_dict['postfix-location'] }}/usr/bin
[base-sbin-bin-wrapper]
< = base-bin-wrapper
path = {{ parameter_dict['postfix-location'] }}/usr/sbin
{% for extend, basename_list in (
(
'base-bin-bin-wrapper',
(
'mailq',
'newaliases',
),
),
(
'base-sbin-bin-wrapper',
(
'postalias',
'postcat',
'postconf',
'postdrop',
'postfix',
'postkick',
'postlock',
'postlog',
'postmap',
'postmulti',
'postqueue',
'postsuper',
'sendmail',
),
),
) %}
{% for basename in basename_list -%}
[{{ section('wrapper-' ~ basename) }}]
< = {{ extend }}
basename = {{ basename }}
{% endfor %}
{% endfor %}
[{{ section('postfix-symlinks-libexec') }}]
recipe = slapos.cookbook:symbolic.link
target-directory = ${directory:usr}
link-binary =
{{ parameter_dict['postfix-location'] }}/usr/libexec
[{{ section('service-postfix-master') }}]
< = base-wrapper
command-line = ${directory:usr}/libexec/postfix/master
wrapper-path = ${directory:run}/postfix-master
[{{ section('postfix-promise') }}]
<= monitor-promise-base
promise = check_socket_listening
name = postfix.py
config-host = {{ ip }}
config-port = {{ tcpv4_port }}
[{{ section('promise-check-computer-memory') }}]
<= monitor-promise-base
promise = check_command_execute
name = check-computer-memory.py
config-command = "{{ parameter_dict["check-computer-memory-binary"] }}" -db ${monitor-instance-parameter:collector-db} --threshold "{{ slapparameter_dict["computer-memory-percent-threshold"] }}" --unit percent
[monitor-instance-parameter]
monitor-httpd-ipv6 = {{ (ipv6_set | list)[0] }}
monitor-httpd-port = {{ tcpv4_port + 2 }}
monitor-title = {{ slapparameter_dict['name'] }}
password = {{ slapparameter_dict['monitor-passwd'] }}
[buildout]
extends =
{{ template_monitor }}
parts =
{{ part_list | join('\n ') }}
stack/erp5-zope2/instance-wcfs.cfg.in 0 → 100644
View file @ 8eeb74fe
{# instance that runs WCFS service associated with ZODB storage #}
{% from "instance_zodb_base" import zodb_dict with context %}
{# q(text) returns urllib.quote_plus(text) #}
{% macro q(text) %}{{ urllib_parse.quote_plus(text) }}{% endmacro %}
{# build zurl to connect to configured ZODB #}
{% if len(zodb_dict) != 1 -%}
{% do assert(False, ("WCFS supports only single ZODB storage", zodb_dict)) -%}
{% endif -%}
{% set db_name, zodb = zodb_dict.popitem() -%}
{% set z = zodb['storage-dict'] -%}
{% if zodb['type'] == 'zeo' -%}
{% set zurl = ('zeo://%s?storage=%s' % (z['server'], z['storage'])) -%}
{% elif zodb['type'] == 'neo' -%}
{# neo(s)://[credentials@]master1,master2,...,masterN/name?options #}
{# (see https://lab.nexedi.com/kirr/neo/blob/3e13fa06/go/neo/client.go#L417) #}
{# If 'ca' in storage-dict, ssl is true. #}
{# (see https://lab.nexedi.com/nexedi/slapos/blob/397726e1/stack/erp5/instance-zodb-base.cfg.in#L17-21) #}
{% if "ca" in z -%}
{# ca=ca.crt;cert=my.crt;key=my.key (see https://lab.nexedi.com/kirr/neo/blob/3e13fa06/go/neo/client.go#L428) #}
{% set zurl = 'neos://ca=%s;cert=%s;key=%s@' % (q(z.pop("ca")), q(z.pop("cert")), q(z.pop("key"))) -%}
{% else -%}
{% set zurl = 'neo://' -%}
{% endif -%}
{% set zurl = ('%s%s/%s' % (zurl, z.pop('master_nodes'), z.pop('name'))) -%}
{% set zurl = zurl + '?' + (z | dictsort | urlencode) -%}
{% else -%}
{% do assert(False, ("unsupported ZODB type", zodb)) -%}
{% endif -%}
[buildout]
extends = {{ template_monitor }}
parts +=
wcfs
wcfs-promise
publish
[directory]
recipe = slapos.cookbook:mkdirectory
etc = ${buildout:directory}/etc
log = ${:var}/log
run = ${:var}/run
services = ${:etc}/run
service-on-watch = ${:etc}/service
srv = ${buildout:directory}/srv
tmp = ${buildout:directory}/tmp
var = ${buildout:directory}/var
[wcfs]
recipe = slapos.cookbook:wrapper
command-line = {{ bin_directory }}/wcfs serve -log_dir=${directory:log} {{ zurl }}
wrapper-path = ${directory:service-on-watch}/wcfs
[wcfs-promise]
<= monitor-promise-base
promise = check_command_execute
name = ${:_buildout_section_name_}.py
config-command = {{ bin_directory }}/wcfs status {{ zurl }}
[publish]
recipe = slapos.cookbook:publish.serialised
serving-zurl = {{ zurl }}
stack/erp5-zope2/instance-zeo.cfg.in 0 → 100644
View file @ 8eeb74fe
{% set ports = itertools.count(slapparameter_dict['tcpv4-port']) -%}
{% set ipv4 = (ipv4_set | list)[0] -%}
{% set backup_periodicity = slapparameter_dict.get('backup-periodicity', 'daily') -%}
{% set part_list = [] -%}
{% macro section(name) %}{% do part_list.append(name) %}{{ name }}{% endmacro -%}
{% set storage_dict = {} -%}
{% set default_zodb_path = buildout_directory ~ '/srv/zodb' -%}
{% set default_backup_path = buildout_directory ~ '/srv/backup/zodb' -%}
{% set bin_directory = parameter_dict['buildout-bin-directory'] -%}
[zeo-base]
recipe = slapos.cookbook:zeo
log-path = ${directory:log}/${:base-name}.log
pid-path = ${directory:run}/${:base-name}.pid
conf-path = ${directory:etc}/${:base-name}.conf
wrapper-path = ${directory:services}/${:base-name}
binary-path = {{ bin_directory }}/runzeo
ip = {{ ipv4 }}
{% set known_tid_storage_identifier_dict = {} -%}
{% set zodb_dict = {} -%}
{% for name, zodb in six.iteritems(slapparameter_dict['zodb-dict']) -%}
{% do zodb_dict.setdefault(zodb.get('family', 'default').lower(), []).append((name, zodb)) -%}
{% endfor -%}
{% set tidstorage_port = slapparameter_dict['tcpv4-port'] + len(zodb_dict) -%}
{% for family, zodb in six.iteritems(zodb_dict) -%}
{% set storage_list = [] -%}
{% set current_port = next(ports) -%}
{% set known_tid_storage_identifier_host = (ipv4, current_port), -%}
{% for name, zodb in zodb -%}
{% do storage_dict.__setitem__(name, {'server': ipv4 ~ ':' ~ current_port, 'storage': name}) %}
{% set path = zodb.get('path', '%(zodb)s/%(name)s.fs') % {'zodb': default_zodb_path, 'name': name} -%}
{% do storage_list.append((name, path)) -%}
{% set backup_directory = zodb.get('backup', '%(backup)s/%(name)s') % {'backup': default_backup_path, 'name': name} -%}
{# BBB: No mount-point specified because they're meaningless for ZEO and
TIDStorage. Pass '' for compatibility, and not None
because this would disable TIDStorage bootstrapping. -#}
{% do known_tid_storage_identifier_dict.__setitem__(json_module.dumps(
(known_tid_storage_identifier_host, name)),
(path, backup_directory, '')) -%}
{% endfor -%}
{% set zeo_section_name = 'zeo-' ~ family %}
[{{ zeo_section_name }}]
< = zeo-base
base-name = zeo-{{ family }}
port = {{ current_port }}
storage = {{ dumps(storage_list) }}
[{{ section("logrotate-" ~ zeo_section_name) }}]
< = logrotate-entry-base
name = {{ "${" ~ zeo_section_name ~ ":base-name}" }}
log = {{ "${" ~ zeo_section_name ~ ":log-path}" }}
post = test ! -s {{ "${" ~ zeo_section_name ~":pid-path}" }} || {{ bin_directory }}/slapos-kill --pidfile {{ "${" ~ zeo_section_name ~ ":pid-path}" }} -s USR2
[{{ section(zeo_section_name ~ "-promise") }}]
<= monitor-promise-base
promise = check_socket_listening
name = zeo-{{ family }}.py
config-host = {{ "${" ~ zeo_section_name ~ ":ip}" }}
config-port = {{ "${" ~ zeo_section_name ~ ":port}" }}
{% endfor -%}
{% if backup_periodicity == 'never' -%}
{% set known_tid_storage_identifier_dict = () %}
{% set tidstorage_repozo_path = '' -%}
{% else -%}
[tidstorage]
recipe = slapos.cookbook:tidstorage
known-tid-storage-identifier-dict = {{ dumps(known_tid_storage_identifier_dict) }}
configuration-path = ${directory:etc}/tidstorage.py
ip = {{ ipv4 }}
port = {{ tidstorage_port }}
{% set tidstorage_repozo_path = slapparameter_dict.get('tidstorage-repozo-path', buildout_directory ~ '/srv/backup/tidstorage') -%}
timestamp-file-path = {{ tidstorage_repozo_path ~ '/repozo_tidstorage_timestamp.log' }}
{# BBB: recipe requires logfile-name for nothing because tidstorage runs in foreground mode -#}
logfile-name =
pidfile-name = ${directory:run}/tidstorage.pid
{# TODO: Add support for backup status file, so that the status file can be close to the ZODB (rather than close to the backup files). And do it efficiently, to not copy the whole status file every time. -#}
status-file = {{ tidstorage_repozo_path ~ '/tidstorage.tid' }}
tidstorage-repozo-binary = {{ bin_directory }}/tidstorage_repozo
tidstoraged-binary = {{ bin_directory }}/tidstoraged
repozo-binary = {{ bin_directory }}/repozo
repozo-wrapper = ${buildout:bin-directory}/tidstorage-repozo
{% if len(known_tid_storage_identifier_dict) > 1 -%}
tidstorage-wrapper = ${directory:services}/tidstoraged
[{{ section("promise-tidstorage") }}]
<= monitor-promise-base
promise = check_socket_listening
name = tidstorage.py
config-host = ${tidstorage:ip}
config-port = ${tidstorage:port}
{% endif -%}
[{{ section("cron-entry-tidstorage-backup") }}]
# TODO:
# - configurable full/incremental
# - configurable retention
recipe = slapos.cookbook:cron.d
cron-entries = ${cron:cron-entries}
name = tidstorage
time = {{ dumps(backup_periodicity) }}
command = ${tidstorage:repozo-wrapper}
# Used for ERP5 resiliency or (more probably)
# webrunner resiliency with erp5 inside.
[{{ section("resiliency-exclude-file") }}]
# Generate rdiff exclude file
recipe = slapos.recipe.template:jinja2
inline = {{ '{{ "${directory:zodb}/**\\n${directory:log}/**\\n" }}' }}
output = ${directory:srv}/exporter.exclude
[{{ section("resiliency-identity-signature-script")}}]
# Generate identity script used by webrunner to check data integrity
# It excludes repozo files as they already include a hash function
# used to check backups when rebuilding the datafs
recipe = slapos.cookbook:wrapper
command-line = {{ bin_directory }}/backup-identity-script-excluding-path --exclude-path "srv/backup/logrotate/**" --exclude-path "srv/backup/zodb/*/*fsz"
wrapper-path = ${directory:srv}/.backup_identity_script
mode = 770
[{{ section("resiliency-after-import-script") }}]
# Generate after import script used by importer instance of webrunner
recipe = collective.recipe.template
input = inline: #!/bin/sh
# DO NOT RUN THIS SCRIPT ON PRODUCTION INSTANCE
# OR ZODB DATA WILL BE ERASED.
# This script will restore the repozo backup to the real
# zodb location. It is launched by the clone (importer) instance of webrunner
# in the end of the import script.
# Depending on the output, it will create a file containing
# the status of the restoration (success or failure).
zodb_directory="${directory:zodb}"
zodb_backup_directory="{{ default_backup_path }}"
repozo="${tidstorage:repozo-binary}"
EXIT_CODE=0
{% for family, zodb in six.iteritems(zodb_dict) -%}
{% for name, zodb in zodb -%}
{% set zeo_section_name = 'zeo-' ~ family %}
storage_name="{{ name }}"
zodb_path="$storage_name.fs"
pid_file={{ "${" ~ zeo_section_name ~ ":pid-path}" }}
if [ -e "$pid_file" ]; then
pid=$(cat $pid_file) > /dev/null 2>&1
if kill -0 "$pid"; then
echo "Zeo is already running with pid $pid. Aborting."
exit 1
fi
fi
echo "Removing $zodb_path..."
echo "Restoring $storage_name into $zodb_path..."
$repozo --verify --repository="$zodb_backup_directory/$storage_name"
CURRENT_EXIT_CODE=$?
if [ ! "$CURRENT_EXIT_CODE"="0" ]; then
echo "$storage_name Backup verification failed. Backup data is inconsistent."
exit "$CURRENT_EXIT_CODE"
fi
$repozo --recover --output="$zodb_directory/$zodb_path" --repository="$zodb_backup_directory/$storage_name"
CURRENT_EXIT_CODE=$?
if [ ! "$CURRENT_EXIT_CODE"="0" ]; then
EXIT_CODE="$CURRENT_EXIT_CODE"
echo "$storage_name Backup restoration failed."
fi
{% endfor -%}
{% endfor -%}
exit $EXIT_CODE
output = ${directory:srv}/runner-import-restore
mode = 755
{% endif -%}
[{{ section('promise-check-computer-memory') }}]
<= monitor-promise-base
promise = check_command_execute
name = check-computer-memory.py
config-command = "{{ parameter_dict["check-computer-memory-binary"] }}" -db ${monitor-instance-parameter:collector-db} --threshold "{{ slapparameter_dict["computer-memory-percent-threshold"] }}" --unit percent
[publish]
recipe = slapos.cookbook:publish.serialised
storage-dict = {{ dumps(storage_dict) }}
{% if len(known_tid_storage_identifier_dict) > 1 -%}
tidstorage-ip = ${tidstorage:ip}
tidstorage-port = ${tidstorage:port}
{% else -%}
tidstorage-ip =
tidstorage-port =
{% endif -%}
monitor-base-url = ${monitor-publish-parameters:monitor-base-url}
[directory]
recipe = slapos.cookbook:mkdirectory
bin = ${buildout:directory}/bin
etc = ${buildout:directory}/etc
services = ${:etc}/run
plugin = ${:etc}/plugin
srv = ${buildout:directory}/srv
var = ${buildout:directory}/var
log = ${:var}/log
run = ${:var}/run
backup-zodb = {{ default_backup_path }}
zodb = {{ default_zodb_path }}
tidstorage = {{ tidstorage_repozo_path }}
[monitor-instance-parameter]
monitor-httpd-ipv6 = {{ (ipv6_set | list)[0] }}
monitor-httpd-port = {{ next(ports) }}
monitor-title = {{ slapparameter_dict['name'] }}
password = {{ slapparameter_dict['monitor-passwd'] }}
[buildout]
extends =
{{ template_monitor }}
parts +=
{{ part_list | join('\n ') }}
publish
stack/erp5-zope2/instance-zodb-base.cfg.in 0 → 100644
View file @ 8eeb74fe
{# base for instances that need to access ZODB storage #}
{# provides zodb_dict #}
{% set zodb_dict = slapparameter_dict['zodb-dict'] -%}
{% set zeo_dict = slapparameter_dict.get('zodb-zeo', {}) -%}
{% for name, zodb in six.iteritems(zodb_dict) -%}
{% set storage_dict = zodb.setdefault('storage-dict', {}) -%}
{% if zodb['type'] == 'zeo' -%}
{% do storage_dict.update(zeo_dict.get(name, ())) -%}
{% else -%}
{% if name == slapparameter_dict.get('neo-name') -%}
{% do storage_dict.update(master_nodes=slapparameter_dict['neo-masters'],
name=slapparameter_dict['neo-cluster']) -%}
{% endif -%}
{{ assert(storage_dict['master_nodes'], name) }}
{% if storage_dict.pop('ssl', 1) -%}
{% do storage_dict.update(ca='~/etc/ca.crt',
cert='~/etc/neo.crt',
key='~/etc/neo.key') -%}
{% endif -%}
{% endif -%}
{% endfor -%}
stack/erp5-zope2/instance-zope.cfg.in 0 → 100644
View file @ 8eeb74fe
{% from "instance_zodb_base" import zodb_dict with context %}
{% set wsgi = slapparameter_dict['wsgi'] -%}
{% set webdav = slapparameter_dict['webdav'] -%}
{% set use_ipv6 = slapparameter_dict.get('use-ipv6', False) -%}
{% set ports = itertools.count(slapparameter_dict['port-base']) -%}
{% set site_id = slapparameter_dict['site-id'] -%}
{% set instance_index_list = range(slapparameter_dict['instance-count']) -%}
{% set node_id_base = slapparameter_dict['name'] -%}
{% set selenium_server_configuration_dict = slapparameter_dict.get('selenium-server-configuration-dict', None) -%}
{% set node_id_index_format = '-%%0%ii' % (len(str(instance_index_list[-1])), ) -%}
{% set part_list = [] -%}
{% set publish_list = [] -%}
{% set test_runner_address_list = [] -%}
{% set test_runner_enabled = slapparameter_dict['test-runner-enabled'] -%}
{% set test_runner_node_count = slapparameter_dict['test-runner-node-count'] -%}
{% set test_runner_random_activity_priority = slapparameter_dict['test-runner-random-activity-priority'] -%}
{% set longrequest_logger_base_path = buildout_directory ~ '/var/log/longrequest_logger_' -%}
{% if webdav -%}
{% set timerserver_interval = 0 -%}
{% else -%}
{% set timerserver_interval = slapparameter_dict['timerserver-interval'] -%}
{%- endif %}
{% macro section(name) %}{% do part_list.append(name) %}{{ name }}{% endmacro -%}
{% set bin_directory = parameter_dict['buildout-bin-directory'] -%}
{#
XXX: This template only supports exactly one IPv4 and one IPv6 per
partition. No more (undefined result), no less (IndexError).
-#}
{% set ipv4 = (ipv4_set | list)[0] -%}
{% set publishable_hosts_dict = {} -%}
{% set port_dict = {} -%}
{% for alias, url in (
('erp5-memcached-volatile', slapparameter_dict['memcached-url']),
('erp5-memcached-persistent', slapparameter_dict['kumofs-url']),
('erp5-smtp', slapparameter_dict['smtp-url']),
) -%}
{% set parsed_url = urllib_parse.urlparse(url) -%}
{% do port_dict.__setitem__(alias, parsed_url.port) -%}
{% do publishable_hosts_dict.__setitem__(alias, parsed_url.hostname) -%}
{%- endfor %}
{% for i, url in enumerate(slapparameter_dict['mysql-url-list']) -%}
{% do publishable_hosts_dict.__setitem__(
'erp5-catalog-' ~ i,
urllib_parse.urlparse(url).hostname,
) -%}
{%- endfor %}
{% do publishable_hosts_dict.update(slapparameter_dict['hosts-dict']) -%}
{% set hosts_dict = publishable_hosts_dict.copy() -%}
{% do hosts_dict.update(slapparameter_dict['computer-hosts-dict'].get(computer, {})) -%}
[jinja2-template-base]
recipe = slapos.recipe.template:jinja2
[erp5-kernel]
recipe = slapos.cookbook:symbolic.link
link-binary = {{ parameter_dict['erp5-kernel-location'] }}/{{ parameter_dict['erp5-kernel-filename'] }}
target-directory = ${directory:erp5-kernel-dir}
[kernel-json-dir]
<= jinja2-template-base
template = {{ parameter_dict['kernel-json-location'] }}/{{ parameter_dict['kernel-json-filename'] }}
rendered = ${directory:erp5-kernel-dir}/kernel.json
context =
raw erp5_url {{ ipv4 }}
raw python_executable {{ parameter_dict['python-executable-for-kernel'] }}
raw kernel_dir ${erp5-kernel:target-directory}/{{ parameter_dict['erp5-kernel-filename'] }}
raw display_name ERP5
raw language_name python
[run-common]
<= userhosts-wrapper-base
environment-extra =
environment +=
TMP=${directory:tmp}
TMPDIR=${directory:tmp}
HOME=${buildout:directory}
PATH=${binary-link:target-directory}:{{ parameter_dict['coreutils'] }}/bin:{{ parameter_dict['perl_dbd_mariadb_path'] }}
TZ={{ slapparameter_dict['timezone'] }}
MATPLOTLIBRC={{ parameter_dict['matplotlibrc'] }}
INSTANCE_HOME=${:instance-home}
CAUCASE={{ slapparameter_dict['caucase-url'] }}
FONTCONFIG_FILE=${fontconfig-conf:output}
JUPYTER_PATH=${directory:jupyter-dir}
JUPYTER_CONFIG_DIR=${directory:jupyter-config-dir}
JUPYTER_RUNTIME_DIR=${directory:jupyter-runtime-dir}
{% if slapparameter_dict.get('wendelin-core-zblk-fmt') %}
WENDELIN_CORE_ZBLK_FMT={{ slapparameter_dict['wendelin-core-zblk-fmt'] }}
{% endif %}
WENDELIN_CORE_WCFS_AUTOSTART=no
{% if slapparameter_dict['wcfs_enable'] %}
WENDELIN_CORE_VIRTMEM=r:wcfs+w:uvmm
{% else %}
WENDELIN_CORE_VIRTMEM=rw:uvmm
{% endif %}
${:environment-extra}
[directory]
recipe = slapos.cookbook:mkdirectory
bin = ${buildout:directory}/bin
etc = ${buildout:directory}/etc
instance = ${:srv}/erp5shared
instance-constraint = ${:instance}/Constraint
instance-document = ${:instance}/Document
instance-etc = ${:instance}/etc
instance-etc-package-include = ${:instance}/etc/package-include
instance-extensions = ${:instance}/Extensions
instance-import = ${:instance}/import
instance-lib = ${:instance}/lib
instance-products = ${:instance}/Products
instance-propertysheet = ${:instance}/PropertySheet
instance-tests = ${:instance}/tests
log = ${:var}/log
run = ${:var}/run
services = ${:etc}/run
service-on-watch = ${:etc}/service
srv = ${buildout:directory}/srv
ca-dir = ${:srv}/ssl
tmp = ${buildout:directory}/tmp
var = ${buildout:directory}/var
plugin = ${:etc}/plugin
unit-test-path = ${:srv}/test-instance/unit_test
fontconfig-cache = ${buildout:directory}/.fontconfig
jupyter-dir = ${buildout:directory}/jupyter
jupyter-config-dir = ${:jupyter-dir}/etc
jupyter-runtime-dir = ${:jupyter-dir}/runtime
jupyter-kernel-dir = ${:jupyter-dir}/kernels
erp5-kernel-dir = ${:jupyter-kernel-dir}/ERP5
[fontconfig-conf]
recipe = slapos.recipe.template:jinja2
url = {{ parameter_dict['template-fonts-conf'] }}
output = ${directory:etc}/fonts.conf
context =
key cachedir directory:fontconfig-cache
key fonts :fonts
key includes :includes
fonts =
{% for font in parameter_dict['fonts'].splitlines() %}
{{ font }}
{% endfor %}
includes =
{% for include in parameter_dict['fontconfig-includes'].splitlines() %}
{{ include }}
{% endfor %}
# Used for ERP5 resiliency or (more probably)
# webrunner resiliency with erp5 inside.
[{{ section("resiliency-exclude-file") }}]
# Generate rdiff exclude file
recipe = slapos.recipe.template:jinja2
inline = {{ '{{ "${directory:log}/**\\n${directory:tmp}/**\\n" }}' }}
output = ${directory:srv}/exporter.exclude
[{{ section("resiliency-identity-signature-script")}}]
# Generate identity script used by webrunner to check data integrity
recipe = slapos.cookbook:wrapper
command-line = {{ bin_directory }}/backup-identity-script-excluding-path --exclude-path "srv/backup/logrotate/**"
wrapper-path = ${directory:srv}/.backup_identity_script
mode = 770
[binary-link]
recipe = slapos.cookbook:symbolic.link
target-directory = ${directory:bin}
link-binary = {{ dumps(parameter_dict['link-binary']) }}
{% if use_ipv6 -%}
{% set ipv6 = (ipv6_set | list)[0] -%}
[ipv6toipv4-base]
recipe = slapos.cookbook:ipv6toipv4
runner-path = ${directory:services}/${:base-name}
6tunnel-path = {{ parameter_dict['6tunnel'] }}/bin/6tunnel
shell-path = {{ parameter_dict['dash'] }}/bin/dash
ipv4 = {{ ipv4 }}
ipv6 = {{ ipv6 }}
{% endif -%}
[hosts-parameter]
# Used for both hosts and hostaliases sections.
host-dict = {{ dumps(hosts_dict) }}
hostalias-dict = {{ dumps(slapparameter_dict['hostalias-dict']) }}
# Note: there is a subtle difference between hosts and hostaliases files:
# - hosts files start with resolved, followed by alias(es) (only one alias per
# line in this case)
# - hostaliases start with alias, followed by resolved
# ...so it's not possible to merge these templates (not a big deal anyway).
[hostaliases]
< = jinja2-template-base
inline = {{ '
{% for alias, aliased in host_dict.items() -%}
{{ alias }} {{ aliased }}
{% endfor %}
' }}
output = ${directory:etc}/hostaliases
context = key host_dict hosts-parameter:hostalias-dict
[hosts]
< = jinja2-template-base
inline = {{ '
{% for alias, aliased in host_dict.items() -%}
{{ aliased }} {{ alias }}
{% endfor %}
' }}
output = ${directory:etc}/hosts
context = key host_dict hosts-parameter:host-dict
[userhosts-bin]
recipe = slapos.recipe.template:jinja2
inline =
#!{{ parameter_dict['dash'] }}/bin/dash
export HOSTS_FILE=${hosts:output}
export LD_PRELOAD={{ parameter_dict['userhosts'] }}:$LD_PRELOAD
exec "$@"
output = ${directory:bin}/userhosts
[userhosts-wrapper-base]
recipe = slapos.cookbook:wrapper
environment =
HOSTALIASES=${hostaliases:output}
command-line = '${userhosts-bin:output}' ${:wrapped-command-line}
{# Hack to deploy SSL certs via instance parameters -#}
{% for zodb in six.itervalues(zodb_dict) -%}
{% set storage_dict = zodb.setdefault('storage-dict', {}) -%}
{% if zodb['type'] == 'neo' and storage_dict.get('ssl', 1) -%}
{% for k, v in (('_ca', 'ca.crt'),
('_cert', 'neo.crt'),
('_key', 'neo.key')) -%}
{% if k in storage_dict -%}
[{{ section('neo-ssl-' + k[1:]) }}]
< = jinja2-template-base
output = ${directory:etc}/{{v}}
inline = {{'{{'}}pem}}
context = key pem :pem
pem = {{dumps(storage_dict.pop(k))}}
{% endif -%}
{% endfor -%}
{% endif -%}
{% endfor -%}
{# endhack -#}
[runzope-base]
<= run-common
instance-home = ${directory:instance}
{% if wsgi -%}
wrapped-command-line = '{{ bin_directory }}/runwsgi' {% if webdav %}-w{% endif %} {{ ipv4 }}:${:port} {% if timerserver_interval %}--timerserver-interval={{ timerserver_interval }}{% endif %} '${:configuration-file}'
{% else -%}
wrapped-command-line = '{{ bin_directory }}/runzope' -C '${:configuration-file}'
{%- endif %}
{%- set private_dev_shm = slapparameter_dict['private-dev-shm'] %}
{%- if private_dev_shm %}
private-tmpfs = {{ private_dev_shm }} /dev/shm
{%- endif %}
[{{ section('zcml') }}]
recipe = slapos.cookbook:copyfilelist
target-directory = ${directory:instance-etc}
file-list = {{ parameter_dict['site-zcml'] }}
[{{ section('zope-inituser') }}]
< = jinja2-template-base
output = ${directory:instance}/inituser
inline = {{ slapparameter_dict['inituser-login'] }}:{SHA}{{ base64.b64encode(hashlib.sha1(slapparameter_dict['inituser-password'].encode('utf-8')).digest()) }}
once = ${:output}_done
[zope-conf-parameter-base]
ip = {{ ipv4 }}
site-id = {{ site_id }}
{% if site_id -%}
mysql-url = {{ slapparameter_dict['mysql-url-list'][0] }}
inituser = {{ slapparameter_dict['inituser-login'] }}
{% set mysql = urllib_parse.urlsplit(slapparameter_dict['mysql-url-list'][0]) -%}
{% set mysql_db = mysql.path.split('/')[1] -%}
sql-connection-string = {{ '%s@erp5-catalog-0:%s %s %s' % (
mysql_db, mysql.port, mysql.username, mysql.password) }}
bt5 = {{ slapparameter_dict['bt5'] }}
bt5-repository-url = {{ slapparameter_dict['bt5-repository-url'] }}
id-store-interval = {{ dumps(slapparameter_dict['id-store-interval']) }}
home = ${buildout:directory}
cloudooo-url = {{ ",".join(slapparameter_dict['cloudooo-url-list']) }}
{% endif -%}
developer-list = {{ dumps(slapparameter_dict['developer-list']) }}
publisher-timeout = {{ dumps(slapparameter_dict['publisher-timeout']) }}
activity-timeout = {{ dumps(slapparameter_dict['activity-timeout']) }}
instance = ${directory:instance}
instance-products = ${directory:instance-products}
deadlock-path = /manage_debug_threads
deadlock-debugger-password = {{ dumps(slapparameter_dict['deadlock-debugger-password']) }}
{% if slapparameter_dict.get('tidstorage-ip') -%}
tidstorage-ip = {{ dumps(slapparameter_dict['tidstorage-ip']) }}
tidstorage-port = {{ dumps(slapparameter_dict['tidstorage-port']) }}
{% endif -%}
{% set thread_amount = slapparameter_dict['thread-amount'] -%}
{% set large_file_threshold = slapparameter_dict['large-file-threshold'] -%}
thread-amount = {{ thread_amount }}
webdav = {{ dumps(webdav) }}
wsgi = {{ dumps(wsgi) }}
timerserver-interval = {{ dumps(timerserver_interval) }}
[zope-conf-base]
< = jinja2-template-base
url = {{ parameter_dict['zope-conf-template'] }}
extensions =
jinja2.ext.do
jinja2.ext.loopcontrols
import-list =
rawfile root_common {{ root_common }}
{% macro zope(
index,
port,
longrequest_logger_timeout,
longrequest_logger_interval
) -%}
{% set name = 'zope-' ~ index -%}
{% set conf_name = name ~ '-conf' -%}
{% set conf_parameter_name = conf_name ~ '-param' -%}
{% set zope_tunnel_section_name = name ~ '-ipv6toipv4' -%}
{% set zope_tunnel_base_name = zope_tunnel_section_name -%}
[{{ conf_parameter_name }}]
< = zope-conf-parameter-base
pid-file = ${directory:run}/{{ name }}.pid
lock-file = ${directory:run}/{{ name }}.lock
port = {{ port }}
event-log = ${directory:log}/{{ name }}-event.log
z2-log = ${directory:log}/{{ name }}-Z2.log
node-id = {{ dumps(node_id_base ~ (node_id_index_format % index)) }}
{% set log_list = [] -%}
{% set import_set = set() -%}
{% for db_name, zodb in six.iteritems(zodb_dict) -%}
{% do zodb.setdefault('pool-size', thread_amount) -%}
{% if zodb['type'] == 'neo' -%}
{% do import_set.add('neo.client') -%}
{% set log = name ~ '-neo-' ~ db_name ~ '.log' -%}
{% do log_list.append('${directory:log}/' + log) -%}
{% do zodb['storage-dict'].update(logfile='~/var/log/'+log) -%}
{% endif -%}
{% endfor -%}
import-list = {{ dumps(list(import_set)) }}
zodb-dict = {{ dumps(zodb_dict) }}
large-file-threshold = {{ large_file_threshold }}
{% if longrequest_logger_interval > 0 -%}
longrequest-logger-file = {{ longrequest_logger_base_path ~ name ~ ".log" }}
longrequest-logger-timeout = {{ longrequest_logger_timeout }}
longrequest-logger-interval = {{ longrequest_logger_interval }}
{% else -%}
longrequest-logger-file =
{% endif -%}
[{{ conf_name }}]
< = zope-conf-base
output = ${directory:etc}/{{ name }}.conf
context =
section parameter_dict {{ conf_parameter_name }}
import os os
import re re
[{{ section(name) }}]
< = runzope-base
wrapper-path = ${directory:service-on-watch}/{{ name }}
configuration-file = {{ '${' ~ conf_name ~ ':output}' }}
{%- if wsgi %}
port = {{ port }}
{%- endif %}
hash-files =
${:configuration-file}
hash-existing-files =
${buildout:directory}/software_release/buildout.cfg
[{{ section("promise-" ~ name) }}]
{% if six.PY3 -%}
# Disable the promise in Python 3. ERP5 is not compatible with Python 3 yet, so
# the promise would always fail.
recipe =
{% else -%}
<= monitor-promise-base
promise = check_socket_listening
name = {{ name }}.py
config-host = {{ ipv4 }}
config-port = {{ port }}
{% endif -%}
{% if use_ipv6 -%}
[{{ zope_tunnel_section_name }}]
< = ipv6toipv4-base
base-name = {{ zope_tunnel_base_name }}
ipv6-port = {{ port }}
ipv4-port = {{ port }}
{% do publish_list.append(("[" ~ ipv6 ~ "]:" ~ port, thread_amount, webdav)) -%}
[{{ section("promise-tunnel-" ~ name) }}]
<= monitor-promise-base
promise = check_socket_listening
name = {{ zope_tunnel_base_name }}.py
config-host = {{ '${' ~ zope_tunnel_section_name ~ ':ipv6}' }}
config-port = {{ '${' ~ zope_tunnel_section_name ~ ':ipv6-port}' }}
{% else -%}
{% do publish_list.append((ipv4 ~ ":" ~ port, thread_amount, webdav)) -%}
{% endif -%}
{% if longrequest_logger_interval > 0 -%}
[{{ section('promise-check-' ~name ~ '-longrequest-error-log') }}]
<= monitor-promise-base
promise = check_error_on_zope_longrequest_log
name = {{'check-' ~ name ~ '-longrequest-error-log.py'}}
config-log-file = {{ '${' ~ conf_parameter_name ~ ':longrequest-logger-file}' }}
config-error-threshold = {{ slapparameter_dict["zope-longrequest-logger-error-threshold"] }}
config-maximum-delay = {{ slapparameter_dict["zope-longrequest-logger-maximum-delay"] }}
{% endif -%}
[{{ section('logrotate-entry-' ~ name) }}]
< = logrotate-entry-base
name = {{ name }}
log = {{ '${' ~ conf_parameter_name ~ ':event-log}' }} {{ '${' ~ conf_parameter_name ~ ':z2-log}' }} {{ '${' ~ conf_parameter_name ~ ':longrequest-logger-file}' }} {{ ' '.join(log_list) }}
post = test ! -s {{ '${' ~ conf_parameter_name ~ ':pid-file}' }} || {{ bin_directory }}/slapos-kill --pidfile {{ '${' ~ conf_parameter_name ~ ':pid-file}' }} -s USR2
{% endmacro -%}
{% for i in instance_index_list -%}
{{ zope(
i,
next(ports),
slapparameter_dict['longrequest-logger-timeout'],
slapparameter_dict['longrequest-logger-interval'],
) }}
{% endfor -%}
[{{ section("watch_activities") }}]
<= userhosts-wrapper-base
environment +=
MYSQL=${binary-link:target-directory}/mysql
wrapped-command-line = {{ parameter_dict['erp5-location'] }}/product/CMFActivity/bin/${:_buildout_section_name_} "-h erp5-catalog-0 -P {{ mysql.port }} -u {{ mysql.username }} -p{{ mysql.password}} {{ mysql_db }}" 5 3600
wrapper-path = ${buildout:bin-directory}/${:_buildout_section_name_}
[{{ section("wait_activities") }}]
<= watch_activities
{% if test_runner_enabled and test_runner_node_count -%}
{% for _ in range(test_runner_node_count) %}
{% do test_runner_address_list.append((ipv4, next(ports))) %}
{% endfor %}
{% if selenium_server_configuration_dict -%}
[test-zelenium-runner-parameter]
configuration = {{ dumps(selenium_server_configuration_dict) }}
user = {{ dumps(slapparameter_dict['inituser-login']) }}
password = {{ dumps(slapparameter_dict['inituser-password']) }}
bin-path = {{ bin_directory }}/{{ parameter_dict['egg-interpreter'] }}
[{{ section('test-zelenium-runner') }}]
<= jinja2-template-base
url = {{ parameter_dict['run-zelenium-template'] }}
output = ${directory:bin}/runTestSuite
context =
import json_module json
key configuration test-zelenium-runner-parameter:configuration
key user test-zelenium-runner-parameter:user
key password test-zelenium-runner-parameter:password
key bin_path test-zelenium-runner-parameter:bin-path
{% else -%}
[{{ section('run-unit-test-userhosts-wrapper') }}]
<= userhosts-wrapper-base
wrapped-command-line = ${runUnitTest:wrapper-path}
wrapper-path = ${buildout:bin-directory}/runUnitTest
[{{ section('run-test-suite-userhosts-wrapper') }}]
<= userhosts-wrapper-base
wrapped-command-line = ${runTestSuite:wrapper-path}
wrapper-path = ${buildout:bin-directory}/runTestSuite
{% set connection_string_list = [] -%}
{% for url in slapparameter_dict['mysql-test-url-list'] -%}
{% set parsed_url = urllib_parse.urlparse(url) -%}
{% do connection_string_list.append(
'%s@%s:%s %s %s' % (
parsed_url.path.lstrip('/'),
parsed_url.hostname,
parsed_url.port,
parsed_url.username,
parsed_url.password,
),
) -%}
{% endfor -%}
[run-test-common]
< = run-common
environment-extra =
REAL_INSTANCE_HOME=${:instance-home}
SQLBENCH_PATH={{ parameter_dict['sqlbench_path'] }}
TEST_CA_PATH=${directory:ca-dir}
ERP5_TEST_RUNNER_CONFIGURATION=${test-runner-configuration:output}
{%- if wsgi %}
erp5_wsgi=1
{%- endif %}
instance-home = ${directory:unit-test-path}
wrapper-path = ${directory:bin}/${:command-name}.real
command-line =
'{{ parameter_dict['bin-directory'] }}/${:command-name}'
${:command-line-extra}
--conversion_server_url={{ ",".join(slapparameter_dict['cloudooo-url-list']) }}
--conversion_server_retry_count={{ slapparameter_dict.get('cloudooo-retry-count', 2) }}
--volatile_memcached_server_hostname=erp5-memcached-volatile
--volatile_memcached_server_port={{ port_dict['erp5-memcached-volatile'] }}
--persistent_memcached_server_hostname=erp5-memcached-persistent
--persistent_memcached_server_port={{ port_dict['erp5-memcached-persistent'] }}
[test-runner-configuration]
recipe = slapos.recipe.template:jinja2
output = ${directory:etc}/${:_buildout_section_name_}.json
inline =
{{ json.dumps(slapparameter_dict['test-runner-configuration']) }}
[{{ section('runUnitTest') }}]
< = run-test-common
command-name = runUnitTest
command-line-extra =
--erp5_sql_connection_string '{{ connection_string_list[0] }}'
--extra_sql_connection_string_list '{{ ','.join(connection_string_list[1:]) }}'
--zserver {{ test_runner_address_list[0][0] ~ ':' ~ test_runner_address_list[0][1] }}
--zserver_frontend_url {{ slapparameter_dict['test-runner-apache-url-list'][0] }}
{% if test_runner_random_activity_priority is not none %}
--random_activity_priority={{ test_runner_random_activity_priority }}
{%- endif %}
[{{ section('runTestSuite') }}]
< = run-test-common
command-name = runTestSuite
command-line-extra =
--db_list '{{ ','.join(connection_string_list) }}'
environment-extra +=
{#- turn a list of (ip, port) in a list of 'ip:port' #}
{% set zserver_address_list = [] -%}
{% for ip, port in test_runner_address_list %}
{% do zserver_address_list.append(ip ~ ':' ~ port) %}
{% endfor -%}
zserver_address_list={{ ','.join(zserver_address_list) }}
zserver_frontend_url_list={{ ','.join(slapparameter_dict['test-runner-apache-url-list']) }}
[promise-test-runner-apache-url-executable]
# promise to wait for apache partition to have returned the parameter
recipe = slapos.cookbook:check_parameter
value = {{ slapparameter_dict['test-runner-apache-url-list'] }}
expected-not-value = not-ready
path = ${directory:bin}/${:_buildout_section_name_}
expected-value =
[{{ section("promise-test-runner-apache-url") }}]
<= monitor-promise-base
promise = check_command_execute
name = ${:_buildout_section_name_}.py
config-command = ${promise-test-runner-apache-url-executable:path}
{%- endif %}
{%- endif %}
[{{ section('promise-check-computer-memory') }}]
<= monitor-promise-base
promise = check_command_execute
name = check-computer-memory.py
config-command = "{{ parameter_dict["check-computer-memory-binary"] }}" -db ${monitor-instance-parameter:collector-db} --threshold "{{ slapparameter_dict["computer-memory-percent-threshold"] }}" --unit percent
[publish]
recipe = slapos.cookbook:publish.serialised
zope-address-list = {{ dumps(publish_list) }}
{#
Note: hosts_dist is generated at zope level rather than at erp5 (root partition)
level, as it is easier: we can access urls as python values trivially here.
This has the downside of making each zope partition publish the (hopefuly) same
dict toward erp5 partition, violating the DRY principle and making the intent
hard to guess.
-#}
hosts-dict = {{ dumps(publishable_hosts_dict) }}
monitor-base-url = ${monitor-publish-parameters:monitor-base-url}
test-runner-address-list = {{ dumps(test_runner_address_list) }}
software-release-url = ${slap-connection:software-release-url}
[monitor-instance-parameter]
monitor-httpd-ipv6 = {{ (ipv6_set | list)[0] }}
monitor-httpd-port = {{ next(ports) }}
monitor-title = {{ slapparameter_dict['name'] }}
password = {{ slapparameter_dict['monitor-passwd'] }}
[buildout]
extends =
{{ template_monitor }}
parts +=
{{ '\n '.join(part_list) }}
publish
erp5-kernel
kernel-json-dir
stack/erp5-zope2/instance.cfg.in 0 → 100644
View file @ 8eeb74fe
[buildout]
extends =
{{ instance_common_cfg }}
[default-dynamic-template-parameters]
bin-directory = {{ bin_directory }}
buildout-bin-directory = {{ buildout_bin_directory }}
check-computer-memory-binary = {{ bin_directory }}/check-computer-memory
[dynamic-template-postfix-parameters]
<= default-dynamic-template-parameters
cyrus-sasl-location = {{ cyrus_sasl_location }}
openssl = {{ openssl_location }}
postfix-location = {{ postfix_location }}
template-postfix-aliases = {{ template_postfix_aliases }}
template-postfix-main-cf = {{ template_postfix_main_cf }}
template-postfix-master-cf = {{ template_postfix_master_cf }}
xz-utils-location = {{ xz_utils_location }}
[dynamic-template-postfix]
< = jinja2-template-base
url = {{ template_postfix }}
filename = instance-postfix.cfg
extensions = jinja2.ext.do
extra-context =
section parameter_dict dynamic-template-postfix-parameters
import urllib urllib
[default-cloudooo-url-list]
recipe = slapos.recipe.build
default-url-list =
{{ default_cloudooo_url_list | indent }}
init =
# Expose the default cloudooo URLs, sorted using a pseudo-random seed,
# so that the same instance keep the same order
import six, random
seed = repr(sorted(six.iteritems(self.buildout['slap-connection'])))
default_cloudooo_url_list = [url.strip() for url in self.options['default-url-list'].splitlines()]
random.Random(seed).shuffle(default_cloudooo_url_list)
self.options['url-list'] = default_cloudooo_url_list
[dynamic-template-erp5-parameters]
default-cloudooo-url-list = ${default-cloudooo-url-list:url-list}
jupyter-enable-default = {{ jupyter_enable_default }}
wcfs-enable-default = {{ wcfs_enable_default }}
local-bt5-repository = {{ ' '.join(local_bt5_repository.split()) }}
[context]
root-common = {{ root_common }}
caucase-jinja2-library = {{ caucase_jinja2_library }}
template-zodb-base = {{ template_zodb_base }}
[dynamic-template-erp5]
<= jinja2-template-base
url = {{ template_erp5 }}
filename = instance-erp5.cfg
extra-context =
key default_cloudooo_url_list dynamic-template-erp5-parameters:default-cloudooo-url-list
key jupyter_enable_default dynamic-template-erp5-parameters:jupyter-enable-default
key wcfs_enable_default dynamic-template-erp5-parameters:wcfs-enable-default
key local_bt5_repository dynamic-template-erp5-parameters:local-bt5-repository
key openssl_location :openssl-location
import re re
import urllib_parse six.moves.urllib.parse
import-list =
file root_common context:root-common
file caucase context:caucase-jinja2-library
openssl-location = {{ openssl_location }}
[dynamic-template-balancer-parameters]
<= default-dynamic-template-parameters
openssl = {{ openssl_location }}
haproxy = {{ haproxy_location }}
rsyslogd = {{ rsyslogd_location }}
socat = {{ socat_location }}
apachedex-location = {{ bin_directory }}/apachedex
run-apachedex-location = {{ bin_directory }}/runApacheDex
promise-check-apachedex-result = {{ bin_directory }}/check-apachedex-result
template-haproxy-cfg = {{ template_haproxy_cfg }}
template-rsyslogd-cfg = {{ template_rsyslogd_cfg }}
# XXX: only used in software/slapos-master:
apache = {{ apache_location }}
template-apache-conf = {{ template_apache_conf }}
[dynamic-template-balancer]
<= jinja2-template-base
url = {{ template_balancer }}
filename = instance-balancer.cfg
extra-context =
section parameter_dict dynamic-template-balancer-parameters
import itertools itertools
import hashlib hashlib
import functools functools
import-list =
file caucase context:caucase-jinja2-library
[dynamic-template-zeo-parameters]
<= default-dynamic-template-parameters
[dynamic-template-zeo]
<= jinja2-template-base
url = {{ template_zeo }}
filename = instance-zeo.cfg
extra-context =
key buildout_directory buildout:directory
section parameter_dict dynamic-template-zeo-parameters
import json_module json
import itertools itertools
[dynamic-template-zope-parameters]
<= default-dynamic-template-parameters
zope-conf-template = {{ template_zope_conf }}
run-zelenium-template = {{ template_run_zelenium }}
6tunnel = {{ sixtunnel_location }}
coreutils = {{ coreutils_location }}
sqlbench_path = {{ mariadb_location }}/sql-bench
perl_dbd_mariadb_path = {{ perl_dbd_mariadb_path }}
dash = {{ dash_location }}
link-binary = {{ dumps(zope_link_binary) }}
fonts = {{ dumps(zope_fonts) }}
fontconfig-includes = {{ dumps(zope_fontconfig_includes) }}
template-fonts-conf = {{ dumps(template_fonts_conf) }}
userhosts = {{ userhosts_location }}/lib/libuserhosts.so
site-zcml = {{ site_zcml }}
extra-path-list = {{ dumps(extra_path_list) }}
matplotlibrc = {{ matplotlibrc_location }}
erp5-location = {{ erp5_location }}
egg-interpreter = {{egg_interpreter}}
erp5-kernel-location = {{ erp5_kernel_location }}
erp5-kernel-filename = {{ erp5_kernel_filename }}
kernel-json-location = {{ kernel_json_location }}
kernel-json-filename = {{ kernel_json_filename }}
python-executable-for-kernel = {{ python_executable_for_kernel }}
[dynamic-template-zope]
<= jinja2-template-base
url = {{ template_zope }}
filename = instance-zope.cfg
extra-context =
key buildout_directory buildout:directory
key root_common context:root-common
section parameter_dict dynamic-template-zope-parameters
import base64 base64
import urllib_parse six.moves.urllib.parse
import hashlib hashlib
import itertools itertools
import json json
import-list =
file instance_zodb_base context:template-zodb-base
[dynamic-template-kumofs-parameters]
<= default-dynamic-template-parameters
dash-location = {{ dash_location }}
dcron-location = {{ dcron_location }}
gzip-location = {{ gzip_location }}
kumo-location = {{ kumo_location }}
logrotate-location = {{ logrotate_location }}
[dynamic-template-kumofs]
<= jinja2-template-base
url = {{ template_kumofs }}
filename = instance-kumofs.cfg
extra-context =
section parameter_dict dynamic-template-kumofs-parameters
[dynamic-template-mariadb-parameters]
<= default-dynamic-template-parameters
bash = {{ bash_location }}
coreutils-location = {{ coreutils_location }}
dash-location = {{ dash_location }}
findutils-location = {{ findutils_location }}
gzip-location = {{ gzip_location }}
xz-utils-location = {{ xz_utils_location }}
mariadb-location = {{ mariadb_location }}
template-my-cnf = {{ template_my_cnf }}
template-mariadb-initial-setup = {{ template_mariadb_initial_setup }}
template-mysqld-wrapper = {{ template_mysqld_wrapper }}
link-binary = {{ dumps(mariadb_link_binary) }}
mariadb-resiliency-after-import-script = {{ mariadb_resiliency_after_import_script }}
mariadb-slow-query-report-script = {{ mariadb_slow_query_report_script }}
mariadb-start-clone-from-backup = {{ mariadb_start_clone_from_backup }}
promise-check-slow-queries-digest-result = {{ bin_directory }}/check-slow-queries-digest-result
percona-tools-location = {{ percona_toolkit_location }}
unixodbc-location = {{ unixodbc_location }}
mroonga-mariadb-install-sql = {{ mroonga_mariadb_install_sql }}
mroonga-mariadb-plugin-dir = {{ mroonga_mariadb_plugin_dir }}
groonga-plugins-path = {{ groonga_plugin_dir }}:{{ groonga_mysql_normalizer_plugin_dir }}
[dynamic-template-mariadb]
<= jinja2-template-base
url = {{ template_mariadb }}
filename = instance-mariadb.cfg
extra-context =
section parameter_dict dynamic-template-mariadb-parameters
# Keep a section for backward compatibility for removed types
# Once the section is removed, ghost instances will keep failing until
# garbage collection be implemented.
[dynamic-template-legacy]
recipe = collective.recipe.template
input = inline:[buildout]
eggs-directory = ${buildout:eggs-directory}
develop-eggs-directory = ${buildout:develop-eggs-directory}
offline = true
parts =
output = ${directory:directory/instance-legacy.cfg
mode = 644
# we need this value to be present in a section,
# for slapos.cookbook:switch-softwaretype to work
[dynamic-template-jupyter]
output = {{ template_jupyter_cfg }}
[dynamic-template-wcfs]
<= jinja2-template-base
url = {{ instance_wcfs_cfg_in }}
filename = instance_wcfs.cfg
extra-context =
section parameter_dict dynamic-template-zope-parameters
import urllib_parse six.moves.urllib.parse
import-list =
file instance_zodb_base context:template-zodb-base
[switch-softwaretype]
recipe = slapos.cookbook:switch-softwaretype
override = {{ dumps(override_switch_softwaretype |default) }}
# Public software types
default = dynamic-template-erp5:output
# BBB
RootSoftwareInstance = ${:default}
# Internal software types
kumofs = dynamic-template-kumofs:output
caucase = dynamic-template-caucase:output
mariadb = dynamic-template-mariadb:output
balancer = dynamic-template-balancer:output
postfix = dynamic-template-postfix:output
zodb-zeo = dynamic-template-zeo:output
zodb-neo = neo:output
zope = dynamic-template-zope:output
jupyter = dynamic-template-jupyter:output
wcfs = dynamic-template-wcfs:output
# Keep cloudooo backward compatibility
cloudooo = dynamic-template-legacy:output
caucase = dynamic-template-legacy:output
stack/erp5-zope2/mariadb_initial_setup.sql.in 0 → 100644
View file @ 8eeb74fe
USE mysql;
{% set mroonga = parameter_dict.get('mroonga', 'ha_mroonga.so') -%}
{% if mroonga %}
SOURCE {{ parameter_dict['mroonga-mariadb-install-sql'] }};
{% endif %}
DROP FUNCTION IF EXISTS sphinx_snippets;
{% macro database(name, user, password, with_process_privilege) -%}
CREATE DATABASE IF NOT EXISTS `{{ name }}`;
{% if user -%}
GRANT ALL PRIVILEGES ON `{{ name }}`.* TO `{{ user }}`@`%` IDENTIFIED BY '{{ password }}';
GRANT ALL PRIVILEGES ON `{{ name }}`.* TO `{{ user }}`@localhost IDENTIFIED BY '{{ password }}';
{% if with_process_privilege %}
GRANT PROCESS ON *.* TO `{{ user }}`@`%` IDENTIFIED BY '{{ password }}';
GRANT PROCESS ON *.* TO `{{ user }}`@localhost IDENTIFIED BY '{{ password }}';
{%- endif %}
{%- endif %}
{% endmacro -%}
{% for entry in parameter_dict['database-list'] -%}
{{ database(entry['name'], entry.get('user'), entry.get('password'), entry.get('with-process-privilege')) }}
{% endfor -%}
stack/erp5-zope2/my.cnf.in 0 → 100644
View file @ 8eeb74fe
{% set socket = parameter_dict['socket'] -%}
# ERP5 buildout my.cnf template based on my-huge.cnf shipped with mysql
# The MySQL server
[mysqld]
# ERP5 by default requires InnoDB storage. MySQL by default fallbacks to using
# different engine, like MyISAM. Such behaviour generates problems only, when
# tables requested as InnoDB are silently created with MyISAM engine.
#
# Loud fail is really required in such case.
sql_mode="NO_ENGINE_SUBSTITUTION"
skip_show_database
{% set ip = parameter_dict.get('ip') -%}
{% if ip -%}
bind_address = {{ ip }}
port = {{ parameter_dict['port'] }}
{% else -%}
skip_networking
{% endif -%}
socket = {{ socket }}
datadir = {{ parameter_dict['data-directory'] }}
tmpdir = {{ parameter_dict['tmp-directory'] }}
pid_file = {{ parameter_dict['pid-file'] }}
log_error = {{ parameter_dict['error-log'] }}
slow_query_log
slow_query_log_file = {{ parameter_dict['slow-query-log'] }}
long_query_time = {{ parameter_dict['long-query-time'] }}
max_allowed_packet = 128M
query_cache_size = 32M
innodb_file_per_table = {{ parameter_dict['innodb-file-per-table'] }}
default_time_zone = '+00:00'
plugin_load = ha_mroonga
plugin-dir = {{ parameter_dict['plugin-directory'] }}
max_connections = {{ parameter_dict['max-connection-count'] }}
{% set innodb_buffer_pool_size = parameter_dict['innodb-buffer-pool-size'] -%}
{% if innodb_buffer_pool_size %}innodb_buffer_pool_size = {{ innodb_buffer_pool_size }}{% endif %}
{% set innodb_buffer_pool_instances = parameter_dict['innodb-buffer-pool-instances'] -%}
{% if innodb_buffer_pool_instances %}innodb_buffer_pool_instances = {{ innodb_buffer_pool_instances }}{% endif %}
{% set innodb_log_file_size = parameter_dict['innodb-log-file-size'] -%}
{% if innodb_log_file_size %} innodb_log_file_size = {{ innodb_log_file_size }}{% endif %}
{% set innodb_log_buffer_size = parameter_dict['innodb-log-buffer-size'] -%}
{% if innodb_log_buffer_size %} innodb_log_buffer_size = {{ innodb_log_buffer_size }}{% endif %}
# very important to allow parallel indexing
# Note: this is compatible with binlog-based incremental backups, because ERP5
# doesn't use "insert ... select" (in any number of queries) pattern.
innodb_locks_unsafe_for_binlog = 1
{% set log_bin = parameter_dict['binlog-path'] -%}
{% if log_bin -%}
log_bin = {{ log_bin }}
{% set binlog_expire_days = parameter_dict['binlog-expire-days'] -%}
{% if binlog_expire_days > 0 %}expire_logs_days = {{ binlog_expire_days }}{% endif %}
server_id = {{ parameter_dict['server-id'] }}
# Note: increases writes on replica (it writes to relay log and to database as
# usual, but then also to binlog), but with the advantage of allowing such node
# to switch role later, or to allow another replica to catch up with it if the
# primary is not available anymore. This also increases the disk usage on the
# replica, as the binlogs are stored as usual.
log_slave_updates = 1
# Note: as of at least mariadb 10.3.27, the relay log name is not based on the
# hostname, but still there is a line in mariadb_error.log on every start
# warning about this missing option. So provide it.
relay-log = mariadb-relay-bin
{% endif %}
# Some dangerous settings you may want to uncomment temporarily
# if you only want performance or less disk access.
{% set x = '' if parameter_dict['relaxed-writes'] else '#' -%}
{{x}}innodb_flush_log_at_trx_commit = 0
{{x}}innodb_flush_method = nosync
{{x}}innodb_doublewrite = 0
{{x}}sync_frm = 0
character_set_server = {{ parameter_dict['character-set-server'] }}
collation_server = {{ parameter_dict['collation-server'] }}
skip_character_set_client_handshake
{% if 'ssl-key' in parameter_dict -%}
ssl_cert = {{ parameter_dict['ssl-crt'] }}
ssl_key = {{ parameter_dict['ssl-key'] }}
{% if 'ssl-ca-crt' in parameter_dict -%}
ssl_ca = {{ parameter_dict['ssl-ca-crt'] }}
{%- endif %}
{% if 'ssl-crl' in parameter_dict -%}
ssl_crl = {{ parameter_dict['ssl-crl'] }}
{%- endif %}
{% if 'ssl-cipher' in parameter_dict -%}
ssl_cipher = {{ parameter_dict['ssl-cipher'] }}
{%- endif %}
{%- endif %}
[client]
socket = {{ socket }}
user = root
[mysql]
no_auto_rehash
[mysqlhotcopy]
interactive_timeout
[mysqldump]
max_allowed_packet = 128M
stack/erp5-zope2/mysql-querydigest.sh.in 0 → 100644
View file @ 8eeb74fe
#!{{dash}}
# BEWARE: This file is operated by slapos node
set -e
SLOW_QUERY_PATH='{{slow_query_path}}'
OUTPUT_FOLDER='{{output_folder}}'
PT_QUERY_EXEC='{{pt_query_exec}}'
if [ ! -d "$OUTPUT_FOLDER" ]; then
echo "ERROR: output_folder don't exists"
exit 1
fi
dashed_today=$(date +%Y-%m-%d)
today=$(date -d "$dashed_today" +%Y%m%d)
SLOW_LOG="$SLOW_QUERY_PATH-$today"
OUTPUT_FILE="$OUTPUT_FOLDER/slowquery_digest.txt-$dashed_today"
if [ ! -f "$SLOW_LOG" ]; then
echo "ERROR: cannot read mysql slow query log file $SLOW_LOG. Exiting."
exit 1
fi
"$PT_QUERY_EXEC" "$SLOW_LOG" > "$OUTPUT_FILE"
{{ xz }} -9 "$OUTPUT_FILE"
stack/erp5-zope2/postfix_aliases.in 0 → 100644
View file @ 8eeb74fe
# See http://www.postfix.org/aliases.5.html for format
{% for name, alias_list in alias_dict.items() -%}
{{ name }}: {{ alias_list | join(', ') }}
{% endfor %}
stack/erp5-zope2/postfix_main.cf.in 0 → 100644
View file @ 8eeb74fe
# http://www.postfix.org/STANDARD_CONFIGURATION_README.html
# http://www.postfix.org/postconf.5.html
queue_directory = {{ queue_directory }}
command_directory = {{ bin_directory }}
daemon_directory = {{ usr_directory }}/libexec/postfix
data_directory = {{ data_directory }}
mail_owner = {{ mail_owner }}
alias_maps = {{ aliases }}
alias_database = {{ aliases }}
mail_spool_directory = {{ spool_directory }}
sendmail_path = {{ bin_directory }}/sendmail
newaliases_path = {{ bin_directory }}/newaliases
mailq_path = {{ bin_directory }}/mailq
setgid_group = {{ setgid_group }}
html_directory = no
manpage_directory = {{ postfix_location }}/usr/local/man
sample_directory = {{ postfix_location }}/etc/postfix
readme_directory = no
inet_interfaces = {{ inet_interfaces }}
smtp_bind_address = 0.0.0.0
smtp_bind_address6 = ::
compatibility_level = 3.6
smtputf8_enable = no
# Compared to default:
# - remove X-related variables, irrelevant for slapos, to be concise
# - add SASL_CONF_PATH to have per-partition cyrus-sasl configuration
import_environment =
MAIL_CONFIG MAIL_DEBUG MAIL_LOGTAG TZ LANG=C
SASL_CONF_PATH
# Mandatory sasl auth over TLS
# XXX: no man-in-the-middle protection
smtpd_tls_cert_file = {{ cert }}
smtpd_tls_key_file = {{ key }}
smtpd_tls_dh512_param_file = {{ dh_512 }}
{#
Note: 1024 vs. 2048 is not a typo, but what is actually recommended in
postfix documentation
-#}
smtpd_tls_dh1024_param_file = {{ dh_2048 }}
smtpd_tls_security_level = encrypt
smtpd_sasl_auth_enable = yes
# Reject as many bogus cases as soon as possible, so errors are visible to ERP5
# developper rather than relying on bounces.
smtpd_recipient_restrictions =
reject_non_fqdn_recipient
reject_unknown_recipient_domain
permit_sasl_authenticated
reject
# Do not allow mynetworks to send mails, only authenticated clients.
smtpd_relay_restrictions =
permit_sasl_authenticated
defer_unauth_destination
# We do not pass mail address in command lines, so accept those starting with
# a dash.
allow_min_user = yes
# Disable local delivery
local_transport = error
smtpd_milters ={{ '\n '.join(milter_list) }}
{% if relayhost -%}
relayhost = {{ relayhost }}
smtp_tls_security_level = encrypt
smtp_tls_session_cache_database = btree:{{ data_directory }}/smtp_scache
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = {{ sasl_passwd }}
smtp_sasl_tls_security_options = noanonymous
{%- endif %}
maillog_file = {{ log_directory }}/postfix.log
maillog_file_compressor = {{ xz_utils_location }}/bin/xz
maillog_file_prefixes = {{ log_directory }}
stack/erp5-zope2/postfix_master.cf.in 0 → 100644
View file @ 8eeb74fe
# http://www.postfix.org/master.5.html
# ==========================================================================
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (100)
# ==========================================================================
{{ smtp }} inet n - n - - smtpd
pickup unix n - n 60 1 pickup
cleanup unix n - n - 0 cleanup
qmgr unix n - n 300 1 qmgr
tlsmgr unix - - n 1000? 1 tlsmgr
rewrite unix - - n - - trivial-rewrite
bounce unix - - n - 0 bounce
defer unix - - n - 0 bounce
trace unix - - n - 0 bounce
verify unix - - n - 1 verify
flush unix n - n 1000? 0 flush
proxymap unix - - n - - proxymap
proxywrite unix - - n - 1 proxymap
smtp unix - - n - - smtp
relay unix - - n - - smtp
showq unix n - n - - showq
error unix - - n - - error
retry unix - - n - - error
discard unix - - n - - discard
local unix - n n - - local
virtual unix - n n - - virtual
lmtp unix - - n - - lmtp
anvil unix - - n - 1 anvil
scache unix - - n - 1 scache
postlog unix-dgram n - n - 1 postlogd
stack/erp5-zope2/rsyslogd.cfg.in 0 → 100644
View file @ 8eeb74fe
module(
load="imuxsock"
SysSock.Name="{{ parameter_dict['log-socket'] }}")
# Just simply output the raw line without any additional information, as
# haproxy emits enough information by itself
# Also cut out first empty space in msg, which is related to rsyslogd
# internal and end up cutting on 8k, as it's default of $MaxMessageSize
template(name="rawoutput" type="string" string="%msg:2:8192%\n")
$FileCreateMode 0644
$DirCreateMode 0700
$Umask 0022
$WorkDirectory {{ parameter_dict['spool-directory'] }}
local0.=info {{ parameter_dict['access-log-file'] }};rawoutput
local0.warning {{ parameter_dict['error-log-file'] }}
stack/erp5-zope2/run-zelenium-test.py.in 0 → 100644
View file @ 8eeb74fe
#!{{ bin_path }}
# BEWARE: This file is operated by slapos node
# BEWARE: It will be overwritten automatically
import argparse, os, sys, traceback
from erp5.util import taskdistribution
from lxml import etree
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import six
import time
from urllib import urlopen
import json
def main():
parser = argparse.ArgumentParser(description='Run a test suite.')
parser_configuration = {{ repr(configuration) }}
parser.add_argument('--test_suite', help='The test suite name')
parser.add_argument('--test_suite_title', help='The test suite title')
parser.add_argument('--test_node_title', help='The test node title')
parser.add_argument('--project_title', help='The project title')
parser.add_argument('--revision', help='The revision to test',
default='dummy_revision')
parser.add_argument('--node_quantity', help='ignored', type=int)
parser.add_argument('--master_url',
help='The Url of Master controling many suites')
args = parser.parse_args()
test_line_dict = {}
test_suite_title = args.test_suite_title or args.test_suite
test_suite = args.test_suite
revision = args.revision
if not parser_configuration['server-url'] or not parser_configuration['remote-access-url']:
sys.exit(-1)
if parser_configuration['run-only']:
url = "%s/erp5/portal_tests/%s/core/TestRunner.html" \
"?test=../test_suite_html" \
"&auto=on" \
"&resultsUrl=../getId" \
"&__ac_name=%s" \
"&__ac_password=%s" % (parser_configuration['remote-access-url'], parser_configuration['run-only'], {{ repr(user) }}, {{ repr(password) }})
else:
url = "%s/erp5/portal_tests/core/TestRunner.html" \
"?test=../test_suite_html" \
"&auto=on" \
"&resultsUrl=../getId" \
"&__ac_name=%s" \
"&__ac_password=%s" % (parser_configuration['remote-access-url'], {{ repr(user) }}, {{ repr(password) }})
# Wait until all activities are finished...
wait_url = "%s/erp5/ActivityTool_getSqlStatisticList" \
"?__ac_name=%s" \
"&__ac_password=%s" % (parser_configuration['remote-access-url'], {{ repr(user) }}, {{ repr(password) }})
while 1:
try:
response = urlopen(wait_url)
try:
if response.code == 500:
sys.exit(-1)
if response.code == 200:
static_dict = json.loads(response.read())
activity_list = []
for _, value in six.iteritems(static_dict):
activity_list += value['line_list']
if len(activity_list) == 0:
break
elif all(x['node'] == -2 for x in activity_list):
sys.exit(-1)
finally:
response.close()
except Exception:
traceback.print_exc()
time.sleep(600)
# Unsubscribe activity
# Launch Zuite_waitForActivities with activity suscribed may create conflict
activity_unsubscribe_url = "%s/erp5/portal_activities/unsubscribe" \
"?__ac_name=%s" \
"&__ac_password=%s" % (parser_configuration['remote-access-url'], {{ repr(user) }}, {{ repr(password) }})
print activity_unsubscribe_url
try:
response = urlopen(activity_unsubscribe_url)
try:
if response.code != 200:
sys.exit(-1)
finally:
response.close()
except Exception:
traceback.print_exc()
tool = taskdistribution.TaskDistributor(portal_url=args.master_url)
browser = webdriver.Remote(parser_configuration['server-url'] , parser_configuration['desired-capabilities'] )
try:
agent = browser.execute_script("return navigator.userAgent")
print url
print agent
start_time = time.time()
browser.get(url)
# Wait for Zelenium to be loaded
WebDriverWait(browser, 10).until(EC.presence_of_element_located((
By.XPATH, '//iframe[@id="testSuiteFrame"]'
)))
# XXX No idea how to wait for the iframe content to be loaded
time.sleep(5)
# Count number of test to be executed
test_count = browser.execute_script(
"return document.getElementById('testSuiteFrame').contentDocument.querySelector('tbody').children.length"
) - 1 # First child is the file name
# Wait for test to be executed
while 1:
try:
WebDriverWait(browser, parser_configuration['max-duration']).until(EC.presence_of_element_located((
By.XPATH, '//td[@id="testRuns" and contains(text(), "%i")]' % test_count
)))
break
except TimeoutException as error:
raise error
except Exception:
traceback.print_exc()
except:
test_line_dict['UnexpectedException'] = {
'test_count': 1,
'error_count': 0,
'failure_count': 1,
'skip_count': 0,
'duration': 1,
'command': url,
'stdout': agent,
'stderr': traceback.format_exc()
}
finally:
execution_duration = round(time.time() - start_time, 2)
if test_count:
test_execution_duration = execution_duration / test_count
else:
test_execution_duration = 0
html_parser = etree.HTMLParser(recover=True)
# body = etree.fromstring(browser.page_source.encode('UTF-8'), html_parser)
# test_count = int(body.xpath('//td[@id="testRuns"]')[0].text)
# failed_test_count = int(body.xpath('//td[@id="testFailures"]')[0].text)
# print 'Run %i, failed %i' % (test_count, failed_test_count)
# https://github.com/appium/appium/issues/5199
# browser.switch_to.frame(browser.find_element_by_id("testSuiteFrame"))
# iframe = etree.fromstring(browser.page_source.encode('UTF-8'), html_parser)
iframe = etree.fromstring(
browser.execute_script(
"return document.getElementById('testSuiteFrame').contentDocument.querySelector('html').innerHTML"
).encode('UTF-8'),
html_parser
)
# parse test result
tbody = iframe.xpath('.//body/table/tbody')[0]
for tr in tbody[1:]:
# First td is the main title
test_name = tr[0][0].text
if len(tr) == 1:
test_line_dict[test_name] = {
'test_count': 1,
'error_count': 0,
'failure_count': 1,
'skip_count': 0,
'duration': 0,
'command': url,
'stdout': agent,
'stderr': '',
'html_test_result': 'Not Executed due to selenium server related error'
}
else:
skip_count = error_count = 0
test_table = tr[1].xpath('.//table')[0]
status = tr.attrib.get('class')
if status is None or 'status_done' in status:
skip_count = 1
elif 'status_failed' in status:
error_count = 1
test_line_dict[test_name] = {
'test_count': 1,
'error_count': error_count,
'failure_count': 0,
'skip_count': skip_count,
'duration': test_execution_duration,
'command': url,
'stdout': agent,
'stderr': '',
'html_test_result': etree.tostring(test_table)
}
browser.quit()
try:
test_result = tool.createTestResult(revision = revision,
test_name_list = list(test_line_dict),
node_title = args.test_node_title,
test_title = test_suite_title,
project_title = args.project_title)
if test_result is None or not hasattr(args, 'master_url'):
return
# report test results
while 1:
test_result_line = test_result.start()
if not test_result_line:
print 'No test result anymore.'
break
print 'Submitting: "%s"' % test_result_line.name
# report status back to Nexedi ERP5
test_result_line.stop(**test_line_dict[test_result_line.name])
except:
# Catch any exception here, to warn user instead of being silent,
# by generating fake error result
traceback.print_exc()
result = dict(status_code=-1,
command=url,
stderr=traceback.format_exc(),
stdout='')
# XXX: inform test node master of error
raise EnvironmentError(result)
if __name__ == "__main__":
main()
stack/erp5-zope2/site.zcml 0 → 100644
View file @ 8eeb74fe
<configure
xmlns="http://namespaces.zope.org/zope"
xmlns:meta="http://namespaces.zope.org/meta"
xmlns:five="http://namespaces.zope.org/five">
<include package="Products.Five" />
<meta:redefinePermission from="zope2.Public" to="zope.Public" />
<!-- Load the meta -->
<include files="package-includes/*-meta.zcml" />
<five:loadProducts file="meta.zcml"/>
<!-- Load the configuration -->
<include files="package-includes/*-configure.zcml" />
<five:loadProducts />
<!-- Load the configuration overrides-->
<includeOverrides files="package-includes/*-overrides.zcml" />
<five:loadProductsOverrides />
<securityPolicy
component="AccessControl.security.SecurityPolicy" />
</configure>
stack/erp5-zope2/zope-versions.cfg 0 → 100644
View file @ 8eeb74fe
[versions]
Zope2 = 2.13.30
AccessControl = 2.13.16
Acquisition = 2.13.12
DateTime = 2.12.8
DocumentTemplate = 2.13.6
ExtensionClass = 2.13.2
Jinja2 = 2.8.1
MarkupSafe = 1.1.1
Missing = 2.13.1
MultiMapping = 2.13.0
Paste = 1.7.5.1
PasteDeploy = 1.3.4
PasteScript = 1.7.5
Persistence = 2.13.2
Products.BTreeFolder2 = 2.13.5
Products.ExternalMethod = 2.13.1
Products.MIMETools = 2.13.0
Products.MailHost = 2.13.4
Products.OFSP = 2.13.2
Products.PythonScripts = 2.13.2
Products.Sessions = 3.0
Products.StandardCacheManagers = 2.13.1
Products.TemporaryFolder = 3.0
Products.ZCTextIndex = 2.13.5
Products.ZCatalog = 2.13.30
Record = 2.13.0
RestrictedPython = 3.6.0
Sphinx = 1.0.8
ZConfig = 2.9.3
ZODB3 = 3.10.7
ZServer = 3.0
ZopeUndo = 2.12.0
appdirs = 1.4.3
configparser = 4.0.2
contextlib2 = 0.6.0.post1
distlib = 0.3.0
docutils = 0.12
filelock = 3.0.12
importlib-metadata = 1.3.0
importlib-resources = 1.0.2
initgroups = 2.13.0
mechanize = 0.2.5
more-itertools = 5.0.0
mr.developer = 1.34
packaging = 20.1
pathlib2 = 2.3.5
pluggy = 0.13.1
py = 1.8.1
pyparsing = 2.4.6
pytz = 2017.2
repoze.retry = 1.2
repoze.tm2 = 1.0
repoze.who = 2.0
scandir = 1.10.0
six = 1.14.0
tempstorage = 2.12.2
toml = 0.10.0
tox = 3.14.4
transaction = 1.1.1
typing = 3.7.4.1
virtualenv = 20.0.4
z3c.checkversions = 1.1
zExceptions = 2.13.0
zLOG = 2.11.2
zc.buildout = 2.3.1
zc.lockfile = 1.0.2
zc.recipe.egg = 2.0.5
zc.recipe.testrunner = 1.2.1
zdaemon = 2.0.7
zipp = 0.6.0
zope.annotation = 3.5.0
zope.broken = 3.6.0
zope.browser = 1.3
zope.browsermenu = 3.9.1
zope.browserpage = 3.12.2
zope.browserresource = 3.10.3
zope.component = 3.9.5
zope.configuration = 3.7.4
zope.container = 3.11.2
zope.contentprovider = 3.7.2
zope.contenttype = 3.5.5
zope.deferredimport = 3.5.3
zope.dottedname = 3.4.6
zope.event = 3.5.2
zope.exceptions = 3.6.2
zope.filerepresentation = 3.6.1
zope.i18n = 3.7.4
zope.i18nmessageid = 3.5.3
zope.interface = 3.6.8
zope.lifecycleevent = 3.6.2
zope.location = 3.9.1
zope.pagetemplate = 3.5.2
zope.processlifetime = 1.0
zope.proxy = 3.6.1
zope.ptresource = 3.9.0
zope.publisher = 3.12.6
zope.schema = 3.7.1
zope.security = 3.7.4
zope.sendmail = 3.7.5
zope.sequencesort = 3.4.0
zope.site = 3.9.2
zope.size = 3.4.1
zope.structuredtext = 3.5.1
zope.tal = 3.5.2
zope.tales = 3.5.3
zope.testbrowser = 3.11.1
zope.testing = 3.9.7
zope.traversing = 3.13.2
zope.viewlet = 3.7.2
stack/erp5-zope2/zope.conf.in 0 → 100644
View file @ 8eeb74fe
{% set slapparameter_dict = {} %}{# dummy -#}
{% import "root_common" as root_common with context -%}
{% set node_id = parameter_dict['node-id'] -%}
# Note: Environment is setup in running wrapper script, as zope.conf is read
# too late for some components.
%define INSTANCE {{ parameter_dict['instance'] }}
instancehome $INSTANCE
zserver-threads {{ parameter_dict['thread-amount'] }}
# When ownership checking is enabled, the roles a script runs as are the
# intersection between user's roles and script owner's roles. This means
# that revoking a code author's access to the system prevent all scripts
# owned by that user from being of much use.
# This is not how ERP5 approaches development: Managers write code,
# Managers must be trustable and trusted, and their past work should not be
# revoked when their account is terminated.
skip-ownership-checking true
lock-filename {{ parameter_dict['lock-file'] }}
pid-filename {{ parameter_dict['pid-file'] }}
default-zpublisher-encoding utf-8
rest-input-encoding utf-8
rest-output-encoding utf-8
# XXX: isn't this entry implicit ?
products {{ parameter_dict['instance-products'] }}
# Magic parameter to use the first entry of X-Forwarded-For as the source IP address.
# (see monkey patches in ERP5Type/patches/HTTPRequest.py and ERP5Type/patches/http_server.py)
# * Frontend HTTP server should drop incoming X-Forwarded-For.
# * Communication between frontend and backend should use SSL Client Authentication.
# * Backend proxy drops incoming X-Forwarded-For without valid SSL Client Authentification.
trusted-proxy 0.0.0.0
{% if not parameter_dict['wsgi'] -%}
{% if parameter_dict['webdav'] -%}
<webdav-source-server>
address {{ parameter_dict['ip'] }}:{{ parameter_dict['port'] }}
force-connection-close off
</webdav-source-server>
{% else %}
<http-server>
address {{ parameter_dict['ip'] }}:{{ parameter_dict['port'] }}
</http-server>
{% endif %}
{%- endif %}
<zoperunner>
program $INSTANCE/bin/runzope
</zoperunner>
<product-config DeadlockDebugger>
dump_url {{ parameter_dict['deadlock-path'] }}
secret {{ parameter_dict['deadlock-debugger-password'] }}
</product-config>
{% if 'longrequest-logger-interval' in parameter_dict -%}
<product-config LongRequestLogger>
logfile {{ parameter_dict['longrequest-logger-file'] }}
timeout {{ parameter_dict['longrequest-logger-timeout'] }}
interval {{ parameter_dict['longrequest-logger-interval'] }}
</product-config>
{% endif -%}
{% if 'large-file-threshold' in parameter_dict -%}
large-file-threshold {{ parameter_dict['large-file-threshold'] }}
{% endif -%}
{% if 'tidstorage-ip' in parameter_dict -%}
<product-config TIDStorage>
backend-ip {{ parameter_dict['tidstorage-ip'] }}
backend-port {{ parameter_dict['tidstorage-port'] }}
</product-config>
{% endif -%}
<product-config CMFActivity>
node-id {{ node_id }}
</product-config>
{% if not parameter_dict['wsgi'] -%}
{% set timerserver_interval = parameter_dict['timerserver-interval'] -%}
{% if timerserver_interval -%}
%import Products.TimerService.timerserver
<timer-server>
interval {{ timerserver_interval }}
</timer-server>
{% endif %}
{% endif -%}
{% set sql_connection_string = parameter_dict.get('sql-connection-string') -%}
{% if sql_connection_string -%}
{% set bt5_repository_url = [] -%}
{% set sr_path = os.path.realpath(parameter_dict['home'] + '/software_release') + '/' -%}
{% for url in parameter_dict['bt5-repository-url'].split() -%}
{% if url.startswith(sr_path) -%}
{% set url = '~/software_release/' + url[len(sr_path):] -%}
{% endif -%}
{% do bt5_repository_url.append(url) -%}
{% endfor -%}
<product-config initsite>
id {{ parameter_dict['site-id'] }}
owner {{ parameter_dict['inituser'] }}
erp5_sql_connection_string {{ sql_connection_string }}
cmf_activity_sql_connection_string {{ sql_connection_string }}
bt5_repository_url {{ ' '.join(bt5_repository_url) }}
bt5 {{ parameter_dict['bt5'] }}
{%- if parameter_dict['id-store-interval'] != None %}
id_store_interval {{ parameter_dict['id-store-interval'] }}
{%- endif %}
cloudooo_url {{ parameter_dict['cloudooo-url'] }}
</product-config>
{% endif -%}
<eventlog>
level info
<logfile>
dateformat
path {{ parameter_dict['event-log'] }}
</logfile>
</eventlog>
<logger access>
level WARN
<logfile>
dateformat
format %(message)s
path {{ parameter_dict['z2-log'] }}
</logfile>
</logger>
<zodb_db temporary>
<temporarystorage>
name temporary storage for sessioning
</temporarystorage>
mount-point /temp_folder
container-class Products.TemporaryFolder.TemporaryContainer
</zodb_db>
{% set developer_list = parameter_dict['developer-list'] -%}
{% set publisher_timeout = parameter_dict['publisher-timeout'] -%}
{% set activity_timeout = parameter_dict['activity-timeout'] -%}
{% if developer_list or publisher_timeout or activity_timeout -%}
%import Products.ERP5Type
<ERP5Type erp5>
{% if developer_list %}developers {{ developer_list | join(' ') }}{% endif %}
{% if publisher_timeout %}publisher-timeout {{ publisher_timeout }}{% endif %}
{% if activity_timeout %}activity-timeout {{ activity_timeout }}{% endif %}
</ERP5Type>
{% endif -%}
{% for m in parameter_dict['import-list'] -%}
%import {{ m }}
{% endfor -%}
{% set type_dict = {'neo': 'NEOStorage', 'zeo': 'zeoclient'} %}
{% for name, zodb_dict in six.iteritems(parameter_dict['zodb-dict']) %}
<zodb_db {{ name }}>
{%- set storage_type = type_dict[zodb_dict.pop('type')] %}
{%- set storage_dict = zodb_dict.pop('storage-dict') %}
{%- do root_common.apply_overrides(zodb_dict, node_id) %}
{%- for key, value in six.iteritems(zodb_dict) %}
{{ key }} {{ value }}
{%- endfor %}
<{{ storage_type }}>
{%- do root_common.apply_overrides(storage_dict, node_id) %}
{%- for key, value in six.iteritems(storage_dict) %}
{{ key }} {{ value }}
{%- endfor %}
</{{ storage_type }}>
</zodb_db>
{% endfor -%}
stack/erp5-zope2/ztk-versions.cfg 0 → 100644
View file @ 8eeb74fe
[versions]
# ZTK
zope.annotation = 3.5.0
zope.applicationcontrol = 3.5.5
zope.authentication = 3.7.1
zope.broken = 3.6.0
zope.browser = 1.3
zope.browsermenu = 3.9.1
zope.browserpage = 3.12.2
zope.browserresource = 3.10.3
zope.cachedescriptors = 3.5.1
zope.catalog = 3.8.2
zope.component = 3.9.5
zope.componentvocabulary = 1.0.1
zope.configuration = 3.7.4
zope.container = 3.11.2
zope.contentprovider = 3.7.2
zope.contenttype = 3.5.5
zope.copy = 3.5.0
zope.copypastemove = 3.7.0
zope.datetime = 3.4.1
zope.deferredimport = 3.5.3
zope.deprecation = 3.4.1
zope.dottedname = 3.4.6
zope.dublincore = 3.7.1
zope.error = 3.7.4
zope.event = 3.5.2
zope.exceptions = 3.6.2
zope.filerepresentation = 3.6.1
zope.formlib = 4.0.6
zope.hookable = 3.4.1
zope.i18n = 3.7.4
zope.i18nmessageid = 3.5.3
zope.index = 3.6.4
zope.interface = 3.6.7
zope.intid = 3.7.2
zope.keyreference = 3.6.4
zope.lifecycleevent = 3.6.2
zope.location = 3.9.1
zope.login = 1.0.0
zope.mimetype = 1.3.1
zope.minmax = 1.1.2
zope.pagetemplate = 3.5.2
zope.password = 3.6.1
zope.pluggableauth = 1.0.3
zope.principalannotation = 3.6.1
zope.principalregistry = 3.7.1
zope.processlifetime = 1.0
zope.proxy = 3.6.1
zope.ptresource = 3.9.0
zope.publisher = 3.12.6
zope.ramcache = 1.0
zope.schema = 3.7.1
zope.security = 3.7.4
zope.securitypolicy = 3.7.0
zope.sendmail = 3.7.5
zope.sequencesort = 3.4.0
zope.server = 3.6.3
zope.session = 3.9.5
zope.site = 3.9.2
zope.size = 3.4.1
zope.structuredtext = 3.5.1
zope.tal = 3.5.2
zope.tales = 3.5.3
zope.testing = 3.9.7
zope.traversing = 3.13.2
zope.viewlet = 3.7.2
# Deprecating
zope.documenttemplate = 3.4.3
# Dependencies
# Needed for the mechanize 0.1.x.
ClientForm = 0.2.10
distribute = 0.6.49
docutils = 0.7
Jinja2 = 2.5.5
# Newer versions of mechanize are not fully py24 compatible.
mechanize = 0.1.11
Paste = 1.7.5.1
PasteDeploy = 1.3.4
PasteScript = 1.7.5
py = 1.3.4
Pygments = 1.3.1
python-gettext = 1.0
pytz = 2013b
RestrictedPython = 3.6.0
setuptools = 12.2
Sphinx = 1.0.8
transaction = 1.1.1
unittest2 = 0.5.1
z3c.recipe.sphinxdoc = 0.0.8
zc.buildout = 2.3.1
zc.lockfile = 1.0.2
ZConfig = 2.8.0
zc.recipe.egg = 1.3.2
zc.recipe.testrunner = 1.2.1
zc.resourcelibrary = 1.3.4
zdaemon = 2.0.7
ZODB3 = 3.9.7
zope.mkzeoinstance = 3.9.6
# toolchain
argparse = 1.1
coverage = 3.5.3
lxml = 2.2.8
mr.developer = 1.25
tl.eggdeps = 0.4
nose = 1.1.2
z3c.checkversions = 0.4.2
z3c.recipe.compattest = 0.12.2
z3c.recipe.depgraph = 0.5
zope.kgs = 1.2.0
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7