software/end-to-end-testing: add end-to-end-testing software

Use runTestSuite interface to launch tests that request instances
on actual SlapOS cloud.

software/end-to-end-testing/README.md

+# End To End Testing
+
+This software release is used to run end-to-end test of SlapOS softwares on an actual SlapOS cloud such as Rapid.Space. Since it can supply softwares and request instances on an actual cloud, it needs a SlaPOS client certificate and the URLs of your tests.
+
+## Input parameters
+
+```
+{
+  "client.crt": <content of client.crt>,
+  "client.key": <content of client.key>,
+  "master-url": <url of SlapOS master>,
+  "tests": [
+  {
+    "url": "<url of test1 script>",
+    "md5sum": "MD5sum of test1 script"
+  },
+  {
+    "url": "<url of test2 python script>",
+    "md5sum": "MD5sum of test2 python script"
+  },
+  ...
+  ]
+}
+```
+Example:
+(`e2e-parameters.json`)
+```
+{
+  "client.crt": "Certificate:...-----END CERTIFICATE-----\n",
+
+  "master-url": "https://slap.vifib.com",
+  "tests": [
+    {
+      "url": "https://lab.nexedi.com/lu.xu/slapos/raw/feat/end-to-end-testing/software/end-to-end-testing/test_test.py",
+      "md5sum": "c074373dbb4154aa924ef5781dade7a0"
+    }
+  ]
+}
+```
+
+## Generate client certificate
+
+Follow [How To Set Up SlapOS Client](https://handbook.rapid.space/user/rapidspace-HowTo.Setup.SlapOS.Client) to prepare `slapos-client.cfg` if you don't have one.
+
+A convenience script `generate_parameters.py` is provided to compute these parameters in JSON format from an existing SlapOS client configuration:
+
+```
+python3 generate_parameters.py --cfg <absolute path to slapos-client.cfg> -o <output path>
+```
+
+
+## Adding tests
+
+There are 3 example tests available in end-to-end testing SR:
+- test_test.py
+  Simple successful test and failed test
+- test_kvm.py
+  Request a KVM instance with published SR and verify one of the connection parameters
+- test_health.py
+  Request a Monitor instance with published SR and log promises output
+
+All tests should be written in Python with a `.py` extension and should have names that start with `test_`.
+
+Once your test is prepared, you have the option to input a URL and its corresponding md5sum as parameters. This will enable the end-to-end testing instance to automatically detect the test.
+
+## Running tests
+
+### In Nexedi ERP5 automated test environment (testnodes)
+
+
+When performing tests on a software release (SR) using the ERR5 test suite, you can use "SlapOS.SoftwareReleases.IntegrationTest". Additionally, make sure to fill in the "Slapos Parameters" field with the content of the input parameter mentioned above.
+
+### In Theia locally
+
+#### Using virtualenv
+
+Using slapos.core for quick testing:
+```
+python3 -m venv testenv
+source testenv/bin/activate
+pip install -e path/to/my/slapos.core
+export SLAPOS_E2E_TEST_CLIENT_CFG=my_test_client_cfg
+export SLAPOS_E2E_TEST_LOG_FILE=my_test_log_file
+python -m unittest my_test_file_in_development
+# edit e2e.py in path/to/my/slapos.core or the instanciated one if improvements are needed
+```
+
+#### Using an instance of software/end-to-end-testing
+
+1. Setup and instantiate the runner
+```
+slapos supply ~/srv/project/slapos/software/end-to-end-testing/software.cfg slaprunner
+slapos request <e2e_instance_name> ~/srv/project/slapos/software/end-to-end-testing/software.cfg --parameters-file <e2e_parameter_json_file>
+```
+Your tests should be listed in <e2e_parameter_json_file> with URL and MD5sum
+
+2. Go to instance directory and run test
+```
+cd ~/srv/runner/instance/slappartX
+./bin/runTestSuite
+```
+Downloaded tests and the reusable `e2e.py` script can be found in the `~/srv/runner/instance/slappartX/var/tests/` directory.
+
+To quickly test, you have the option to modify the test script directly here(`~/srv/runner/instance/slappartX/var/tests/`). After making the necessary changes, you can relaunch the tests by running `./bin/runTestSuite`.
+
+## FAQ
+
+Q1. What is the difference between `slapos-sr-testing` and `end-to-end-testing`?
+
+- slapos-sr-testing requests SRs on a slapproxy in an SlapOSStandalone inside the slapos-sr-testing instance (same kind of thing as Theia or webrunner).
+
+- end-to-end-testing requests SRs on the actual master, in real compute nodes (COMP-XXX). To do this it needs a slapos certificate which is passed as instance parameter to end-to-end-testing instance in the test suite.
+
+So unlike slapos-sr-testing, end-to-end-testing does not contain the SRs it tests, it merely runs the python tests scripts (like mentioned `test_kvm.py`) and integrates with the ERP5 test suite. This also means we cannot access the files in the partition of the tested SRs, as those are on other computers. All we have access to is what a normal user requesting on panel would have access to.

software/end-to-end-testing/buildout.hash.cfg

+[instance.cfg]
+filename = instance.cfg.in
+md5sum = 962830010e0a257d52c22141db3d34cf

software/end-to-end-testing/generate_parameters.py

+import argparse
+import configparser
+import json
+
+
+def main():
+  parser = argparse.ArgumentParser()
+  parser.add_argument('--cfg', required=True)
+  parser.add_argument('-o', '--output', required=True)
+  args = parser.parse_args()
+
+  configp = configparser.ConfigParser()
+  configp.read(args.cfg)
+
+  with open(configp.get('slapconsole', 'cert_file')) as f:
+    crt = f.read()
+
+  with open(configp.get('slapconsole', 'key_file')) as f:
+    key = f.read()
+
+  url = configp.get('slapos', 'master_url')
+
+  with open(args.output, 'w') as f:
+    json.dump(
+      {
+        'client.crt': crt,
+        'client.key': key,
+        'master-url': url
+      }, f, indent=2)
+
+
+if __name__ == '__main__':
+  main()

software/end-to-end-testing/instance-input-schema.json

+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "type": "object",
+  "properties": {
+    "client.crt": {
+      "type": "string",
+      "pattern": "^Certificate:.+\\n-----END CERTIFICATE-----\\n$",
+      "default": "Certificate:\ndefault-client-crt\n-----END CERTIFICATE-----\n"
+    },
+    "client.key": {
+      "type": "string",
+      "pattern": "^-----BEGIN PRIVATE KEY-----.+\\n-----END PRIVATE KEY-----\\n$",
+      "default": "-----BEGIN PRIVATE KEY-----\ndefault-client-key\n-----END PRIVATE KEY-----\n"
+    },
+    "master-url": {
+      "type": "string",
+      "format": "uri",
+      "default": "https://slap.vifib.com"
+    },
+    "tests": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "properties": {
+          "url": {
+            "type": "string",
+            "format": "uri"
+          },
+          "md5sum": {
+            "type": "string",
+            "pattern": "^[a-f0-9]{32}$"
+          }
+        },
+        "additionalProperties": false
+      }
+    }
+  },
+  "additionalProperties": false
+}

software/end-to-end-testing/instance-output-schema.json

+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "description": "Values returned by End to End Testing instanciation",
+  "additionalProperties": false,
+  "properties": {},
+  "type": "object"
+}

software/end-to-end-testing/instance.cfg.in

+[buildout]
+eggs-directory = ${buildout:eggs-directory}
+develop-eggs-directory = ${buildout:develop-eggs-directory}
+
+extends =
+  ${nxdtest-instance.cfg:output}
+
+parts =
+  .nxdtest
+
+[slap-configuration]
+recipe = slapos.cookbook:slapconfiguration.serialised
+computer = $${slap-connection:computer-id}
+partition = $${slap-connection:partition-id}
+url = $${slap-connection:server-url}
+key = $${slap-connection:key-file}
+cert = $${slap-connection:cert-file}
+
+
+[directory]
+recipe = slapos.cookbook:mkdirectory
+home = $${buildout:directory}
+bin = $${buildout:directory}/bin
+etc = $${buildout:directory}/etc
+var = $${buildout:directory}/var
+cfg = $${buildout:directory}/.slapos
+nxdtestdir = $${:var}/nxdtest
+log = $${:var}/log
+
+
+[client.crt]
+recipe = slapos.recipe.template
+output = $${directory:cfg}/client.crt
+inline = $${slap-configuration:configuration.client.crt}
+
+
+[client.key]
+recipe = slapos.recipe.template
+output = $${directory:cfg}/client.key
+inline = $${slap-configuration:configuration.client.key}
+
+
+[slapos-client.cfg]
+recipe = slapos.recipe.template
+output = $${directory:cfg}/slapos-client.cfg
+inline =
+  [slapos]
+  master_url = $${slap-configuration:configuration.master-url}
+
+  [slapconsole]
+  cert_file = $${client.crt:output}
+  key_file = $${client.key:output}
+
+
+[env.sh]
+recipe = slapos.recipe.template:jinja2
+output = $${directory:cfg}/env.sh
+inline =
+  export HOME=$${directory:home}
+
+
+[runTestSuite]
+# extended from stack/nxdtest
+env.sh = $${env.sh:output}
+workdir = $${directory:nxdtestdir}
+
+
+[.nxdtest]
+recipe = slapos.recipe.template:jinja2
+output = $${runTestSuite:workdir}/.nxdtest
+python_for_test = ${python_for_test:executable}
+testdir = $${tests:location}
+log-output = $${directory:log}/e2e-testing.log
+context =
+  key python_for_test :python_for_test
+  key testdir :testdir
+  key slapos_cfg slapos-client.cfg:output
+  key log_file :log-output
+inline =
+  import os
+  directory = "{{ testdir }}"
+  slapos_cfg = {{ repr(slapos_cfg) }}
+  log_file = {{ repr(log_file) }}
+  dir_list = os.listdir(directory)
+  for filename in dir_list:
+    name, ext = os.path.splitext(filename)
+    if name.startswith('test') and ext == '.py':
+      TestCase(
+        name,
+        [{{ repr(python_for_test) }} , '-m', 'unittest',  '-v', name],
+        cwd=directory,
+        env={'SLAPOS_E2E_TEST_CLIENT_CFG': slapos_cfg,
+             'SLAPOS_E2E_TEST_LOG_FILE': log_file},
+        summaryf=UnitTest.summary,
+      )
+
+[tests]
+recipe = slapos.recipe.build
+tests = $${slap-configuration:configuration.tests}
+location = $${directory:var}/tests
+install =
+  import os
+  os.mkdir(location)
+  buildout_offline = self.buildout['buildout']['offline']
+  try:
+    # Allow to do self.download() which can only be used in "online" mode
+    self.buildout['buildout']['offline'] = 'false'
+    for i, test in enumerate(options['tests']):
+      tmp = self.download(test['url'], test['md5sum'])
+      path = os.path.join(location, 'test%s.py' % i)
+      os.rename(tmp, path)
+  finally:
+    # reset the parameter
+    self.buildout['buildout']['offline'] = buildout_offline

software/end-to-end-testing/software.cfg

+[buildout]
+extends =
+  ../../component/pygolang/buildout.cfg
+  ../../stack/slapos.cfg
+  ../../stack/nxdtest.cfg
+  buildout.hash.cfg
+
+parts =
+  instance.cfg
+  slapos-cookbook
+
+[instance.cfg]
+recipe = slapos.recipe.template
+output = ${buildout:directory}/instance.cfg
+url = ${:_profile_base_location_}/${:filename}
+
+[e2e.py]
+recipe = slapos.recipe.build:download
+output = ${buildout:directory}/${:filename}
+url = ${:_profile_base_location_}/${:filename}
+
+[python_for_test]
+<= python-interpreter
+interpreter = python_for_test
+executable = ${buildout:bin-directory}/${:interpreter}
+depends = ${lxml-python:egg}
+eggs =
+  ${pygolang:egg}
+  slapos.core
+  websocket-client
+  requests
+
+[versions]
+websocket-client = 1.4.2

software/end-to-end-testing/software.cfg.json

+{
+  "name": "End-To-End-Testing",
+  "description": "End-To-End Testing on SlapOS Cloud",
+  "serialisation": "json-in-xml",
+  "software-type": {
+    "default": {
+      "title": "default",
+      "software-type": "default",
+      "description": "default",
+      "request": "instance-input-schema.json",
+      "response": "instance-output-schema.json"
+    }
+  }
+}

software/end-to-end-testing/tests/test_health.py

+import slapos.testing.e2e as e2e
+import time
+
+
+class HealthTest(e2e.EndToEndTestCase):
+
+  @classmethod
+  def test_health_promise_feed(self):
+    instance_name = e2e.time.strftime('e2e-test-health-%Y-%B-%d-%H:%M:%S')
+    product = self.product.slapmonitor
+    parameter_dict = {}
+    self.request(
+      self.product.slapmonitor,
+      instance_name,
+      software_type='default',
+      filter_kw={"computer_guid": "COMP-4057"})
+    self.waitUntilGreen(instance_name)
+    self.connection_dict = self.getInstanceInfos(instance_name).connection_dict
+
+    resp, url = self.waitUntilMonitorURLReady(instance_name=instance_name)
+    self.getMonitorPromises(resp.content)

software/end-to-end-testing/tests/test_kvm.py

+import slapos.testing.e2e as e2e
+
+
+class KvmTest(e2e.EndToEndTestCase):
+
+  def test(self):
+
+    instance_name = e2e.time.strftime('e2e-test-kvm-%Y-%B-%d-%H:%M:%S')
+    # instance_name = 'e2e-kvm-test' # avoid timestamp to reuse instance
+    self.request(self.product.kvm, instance_name)
+    self.waitUntilGreen(instance_name)
+    connection_dict = self.request(self.product.kvm, instance_name)
+    self.assertIn('url', connection_dict)

software/end-to-end-testing/tests/test_test.py

+import unittest
+
+
+class Test(unittest.TestCase):
+
+  def test_fail(self):
+    self.assertEqual(0, 1)
+
+  def test_succeed(self):
+    self.assertEqual(0, 0)