See merge request nexedi/slapos!1486

Use runTestSuite interface to launch tests that request instances
on actual SlapOS cloud.

As described in https://github.com/msgpack/msgpack-python#major-breaking-changes-in-msgpack-10
raw is False by default for unpacker so we receive strings, not bytes

See merge request nexedi/slapos!1490

ZODB4 does not support python3

These must be installed through dependencies, but we explicitly need
scripts from supervisor because this is used by slapos-core tests

on python3 we use lzma directly

This software only use nodejs for a simple build step, it should be OK
to use any version.

Hello up there.

I've noticed that wendelin.core tests became failing to build and
automatically disabled some time ago. Not good. I've tried to fix build
failures. Please see individual patches for details.

/cc @levin.zimmermann
/reviewed-by @jerome
/reviewed-on nexedi/slapos!1492

After probably 02fad6a8 (component/{numpy,scipy}: Version up for Python3.)
pygolang test build started to fail in numpy compilation as shown in the
appendix. The failure is correct as pygolang never explicitly depended on
${numpy:egg}. We did not noticed before probably because in earlier versions
numpy did not required Cython to be present at install time.

-> Fix it by explicitly adding ${numpy:egg} to pygolang[all_test] dependencies.

Apendix. Log of test build failure

    networkcache: Trying to download pypi:numpy=1.22.0 from network cache...
    Getting distribution for 'numpy==1.22.0'.
    WARNING: The easy_install command is deprecated and will be removed in a future version.
    Running from numpy source directory.
    Processing numpy/random/_bounded_integers.pxd.in
    Processing numpy/random/_pcg64.pyx
    Traceback (most recent call last):
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/tools/cythonize.py", line 53, in process_pyx
        import Cython
    ModuleNotFoundError: No module named 'Cython'

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/tools/cythonize.py", line 234, in <module>
        main()
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/tools/cythonize.py", line 230, in main
        find_process_files(root_dir)
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/tools/cythonize.py", line 221, in find_process_files
        process(root_dir, fromfile, tofile, function, hash_db)
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/tools/cythonize.py", line 187, in process
        processor_function(fromfile, tofile)
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/tools/cythonize.py", line 60, in process_pyx
        raise OSError(msg) from e
    OSError: Cython needs to be installed in Python as a module
    Traceback (most recent call last):
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 154, in save_modules
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 195, in setup_context
      File "<string>", line 7, in setup_context
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 250, in run_setup
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 45, in _execfile
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/setup.py", line 450, in <module>
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/setup.py", line 432, in setup_package
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/setup.py", line 237, in generate_cython
    RuntimeError: Running cythonize failed!

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 10, in <module>
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/command/easy_install.py", line 2317, in main
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/__init__.py", line 162, in setup
      File "/srv/slapgrid/slappart49/srv/runner/shared/python3/a066243faa0ae8c256b7daa6207cc9a6/lib/python3.9/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/srv/slapgrid/slappart49/srv/runner/shared/python3/a066243faa0ae8c256b7daa6207cc9a6/lib/python3.9/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/srv/slapgrid/slappart49/srv/runner/shared/python3/a066243faa0ae8c256b7daa6207cc9a6/lib/python3.9/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/command/easy_install.py", line 424, in run
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/command/easy_install.py", line 666, in easy_install
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/command/easy_install.py", line 711, in install_item
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/command/easy_install.py", line 896, in install_eggs
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/command/easy_install.py", line 1164, in build_and_install
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/command/easy_install.py", line 1150, in run_setup
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 253, in run_setup
      File "/srv/slapgrid/slappart49/srv/runner/shared/python3/a066243faa0ae8c256b7daa6207cc9a6/lib/python3.9/contextlib.py", line 137, in __exit__
        self.gen.throw(typ, value, traceback)
      File "<string>", line 7, in setup_context
      File "/srv/slapgrid/slappart49/srv/runner/shared/python3/a066243faa0ae8c256b7daa6207cc9a6/lib/python3.9/contextlib.py", line 137, in __exit__
        self.gen.throw(typ, value, traceback)
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 195, in setup_context
      File "/srv/slapgrid/slappart49/srv/runner/shared/python3/a066243faa0ae8c256b7daa6207cc9a6/lib/python3.9/contextlib.py", line 137, in __exit__
        self.gen.throw(typ, value, traceback)
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 166, in save_modules
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 141, in resume
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/_vendor/six.py", line 685, in reraise
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 154, in save_modules
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 195, in setup_context
      File "<string>", line 7, in setup_context
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 250, in run_setup
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 45, in _execfile
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/setup.py", line 450, in <module>
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/setup.py", line 432, in setup_package
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/setup.py", line 237, in generate_cython
    RuntimeError: Running cythonize failed!
    Cythonizing sources
    An error occurred when trying to install numpy 1.22.0. Look above this message for any errors that were output by easy_install.
    While:
      Installing gpython.
      Base installation request: 'pygolang[all_test]'
        Requirement of pygolang[all_test]: wheel
        Requirement of pygolang[all_test]: setuptools_dso>=2.8
        Requirement of pygolang[all_test]: setuptools
        Requirement of pygolang[all_test]: pytest
        Requirement of pygolang[all_test]: numpy
        Requirement of pygolang[all_test]: ipython
        Requirement of pygolang[all_test]: cython<3
        Requirement of pygolang[all_test]: geventmp
        Requirement of pygolang[all_test]: setuptools_dso>=2.8
        Requirement of pygolang[all_test]: decorator
        Requirement of pygolang[all_test]: six
        Requirement of pygolang[all_test]: gevent
        Requirement of setuptools_dso>=2.8: setuptools
        Requirement of pytest: wcwidth
        Requirement of pytest: six>=1.10.0
        Requirement of pytest: py>=1.5.0
        Requirement of pytest: pluggy<1.0,>=0.12
        Requirement of pytest: packaging
        Requirement of pytest: more-itertools>=4.0.0
        Requirement of pytest: attrs>=17.4.0
        Requirement of pytest: atomicwrites>=1.0
      Getting distribution for 'numpy==1.22.0'.
    Error: Couldn't install: numpy 1.22.0

I already fixed ZEO4-wc2 installation once in 97832d95 (ZEO: Fix
ZEO4-wc2 installation). That worked but, as it turned out, not fully:
Even though correct ZEO4-wc2-repository is cloned, and correct ZEO4-wc2
develop-egg is installed, we don't tell buildout which egg-version of
ZEO to use. This leads to the situation when buildout picks up _latest_
ZEO egg, when another egg, which depends on ZEO, is installed.

For example wendelin.core/test-zodb4-wc2 was failing as

    Updating ZEO4-wc2-repository.
    Installing ZEO4-wc2.
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    Installing ZEO.
    Installing zodbtools.
    While:
      Installing zodbtools.
      Base installation request: 'zodbtools'
        Requirement of zodbtools==0.0.0.dev8: dateparser
        Requirement of zodbtools==0.0.0.dev8: six
        Requirement of zodbtools==0.0.0.dev8: pygolang>=0.0.0.dev6
        Requirement of zodbtools==0.0.0.dev8: zope.interface
        Requirement of zodbtools==0.0.0.dev8: zodburi
        Requirement of zodbtools==0.0.0.dev8: ZODB
        Requirement of dateparser: tzlocal
        Requirement of dateparser: regex!=2019.02.19
        Requirement of dateparser: pytz
        Requirement of dateparser: python-dateutil
        Requirement of pygolang>=0.0.0.dev6: Importing
        Requirement of pygolang>=0.0.0.dev6: decorator
        Requirement of pygolang>=0.0.0.dev6: six
        Requirement of pygolang>=0.0.0.dev6: gevent
        Requirement of zope.interface: setuptools
        Requirement of zodburi: ZEO
        Requirement of zodburi: ZConfig
        Requirement of zodburi: ZODB
        Requirement of ZODB: zodbpickle>=0.6.0
        Requirement of ZODB: zope.interface
        Requirement of ZODB: zc.lockfile
        Requirement of ZODB: six
        Requirement of ZODB: transaction>=1.5.0
        Requirement of ZODB: ZConfig
        Requirement of ZODB: BTrees>=4.2.0
        Requirement of ZODB: persistent>=4.2.0
        Requirement of tzlocal: pytz
        Requirement of python-dateutil: six>=1.5
        Requirement of gevent: greenlet>=0.4.17
        Requirement of gevent: setuptools
        Requirement of gevent: zope.interface
        Requirement of gevent: zope.event
        Requirement of ZEO: trollius
        Requirement of ZEO: futures
        Requirement of ZEO: zope.interface
        Requirement of ZEO: zdaemon
        Requirement of ZEO: ZConfig
        Requirement of ZEO: zc.lockfile
        Requirement of ZEO: persistent>=4.1.0
        Requirement of ZEO: transaction>=2.0.3
        Requirement of ZEO: six
        Requirement of ZEO: ZODB>=5.1.1
        Requirement of zodbpickle>=0.6.0: setuptools
        Requirement of zc.lockfile: setuptools
        Requirement of transaction>=1.5.0: zope.interface
        Requirement of BTrees>=4.2.0: zope.interface>=5.0.0
        Requirement of BTrees>=4.2.0: persistent>=4.1.0
        Requirement of persistent>=4.2.0: cffi
        Requirement of persistent>=4.2.0: zope.interface
        Requirement of zope.event: setuptools
      Getting distribution for 'trollius'.
    Error: Picked: trollius = 2.2.1

-> Fix that by specifying which egg-version ZEO4/ZEO4-wc2 should be using.

For the reference for ZODB we already do the same - specify egg-version for all
ZODB5, ZODB4-wc2 and ZODB4:

https://lab.nexedi.com/nexedi/slapos/blob/6765c349/component/ZODB/buildout.cfg#L52-72

ZEO[test] requires ZopeUndo, but we now never pin that anywhere in e.g. stack/slapos.cfg .
Previously it was working without explicit pinning because Zope2, contrary to Zope4, pins ZopeUndo to 2.12.0.

Build of test-zodb5.cfg was failing as

    Installing wendelin.core-python.
    While:
      Installing wendelin.core-python.
      Base installation request: 'wendelin.core[test]', 'pygolang[pyx.build]', 'neoppod[tests]', 'ZEO[test]'
        Requirement of ZEO[test]==5.4.0: zope.testrunner
        Requirement of ZEO[test]==5.4.0: zdaemon
        Requirement of ZEO[test]==5.4.0: msgpack<1
        Requirement of ZEO[test]==5.4.0: mock
        Requirement of ZEO[test]==5.4.0: transaction
        Requirement of ZEO[test]==5.4.0: manuel
        Requirement of ZEO[test]==5.4.0: zope.testing
        Requirement of ZEO[test]==5.4.0: ZopeUndo
        Requirement of ZEO[test]==5.4.0: ZODB>=5.5.1
        Requirement of ZEO[test]==5.4.0: ZConfig
        Requirement of ZEO[test]==5.4.0: trollius
        Requirement of ZEO[test]==5.4.0: futures
        Requirement of ZEO[test]==5.4.0: zope.interface
        Requirement of ZEO[test]==5.4.0: zdaemon
        Requirement of ZEO[test]==5.4.0: ZConfig
        Requirement of ZEO[test]==5.4.0: zc.lockfile
        Requirement of ZEO[test]==5.4.0: persistent>=4.1.0
        Requirement of ZEO[test]==5.4.0: transaction>=2.0.3
        Requirement of ZEO[test]==5.4.0: six
        Requirement of ZEO[test]==5.4.0: ZODB>=5.1.1
        Requirement of neoppod[tests]==1.12.0: neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-pymysql,storage-sqlite]
        Requirement of neoppod[tests]==1.12.0: psutil>=2
        Requirement of neoppod[tests]==1.12.0: zope.testing
        Requirement of neoppod[tests]==1.12.0: coverage
        Requirement of neoppod[tests]==1.12.0: python-dateutil
        Requirement of neoppod[tests]==1.12.0: msgpack>=0.5.6
        Requirement of pygolang[pyx.build]==0.1: wheel
        Requirement of pygolang[pyx.build]==0.1: setuptools_dso>=1.7
        Requirement of pygolang[pyx.build]==0.1: setuptools
        Requirement of pygolang[pyx.build]==0.1: cython
        Requirement of pygolang[pyx.build]==0.1: Importing
        Requirement of pygolang[pyx.build]==0.1: decorator
        Requirement of pygolang[pyx.build]==0.1: six
        Requirement of pygolang[pyx.build]==0.1: gevent
        Requirement of wendelin.core[test]: ZEO
        Requirement of wendelin.core[test]: neoppod
        Requirement of wendelin.core[test]: scipy
        Requirement of wendelin.core[test]: pytest
        Requirement of wendelin.core[test]: psutil
        Requirement of wendelin.core[test]: six
        Requirement of wendelin.core[test]: pygolang>=0.1
        Requirement of wendelin.core[test]: zodbtools>=0.0.0.dev8
        Requirement of wendelin.core[test]: ZODB>=4
        Requirement of wendelin.core[test]: numpy
        Requirement of zope.testrunner: zope.interface
        Requirement of zope.testrunner: zope.exceptions
        Requirement of zope.testrunner: six
        Requirement of zope.testrunner: setuptools
        Requirement of zdaemon: setuptools
        Requirement of zdaemon: ZConfig
        Requirement of mock: funcsigs>=1
        Requirement of mock: six
        Requirement of transaction: zope.interface
        Requirement of manuel: six
        Requirement of manuel: setuptools
        Requirement of zope.testing: setuptools
      Getting distribution for 'ZopeUndo'.
    Error: Picked: ZopeUndo = 6.0

-> pin ZopeUndo to 5.0 - the last version that supports both py2 and py3.

After d8409763 (NEO: default ZODB was changed to 5) wendelin.core tests
started to fail to build because files, that the tests extend from, were
renamed on neoppod side. Example failure:

    An internal error occurred due to a bug in either zc.buildout or in a
    recipe being used:
    Traceback (most recent call last):
      File "/opt/slapos/eggs/zc.buildout-2.7.1+slapos019-py3.7.egg/zc/buildout/buildout.py", line 2359, in main
        user_defaults, command, args)
      File "/opt/slapos/eggs/zc.buildout-2.7.1+slapos019-py3.7.egg/zc/buildout/buildout.py", line 312, in __init__
        data['buildout'], override))
      File "/opt/slapos/eggs/zc.buildout-2.7.1+slapos019-py3.7.egg/zc/buildout/buildout.py", line 1974, in _open
        seen, processing))
      File "/opt/slapos/eggs/zc.buildout-2.7.1+slapos019-py3.7.egg/zc/buildout/buildout.py", line 1974, in _open
        seen, processing))
      File "/opt/slapos/eggs/zc.buildout-2.7.1+slapos019-py3.7.egg/zc/buildout/buildout.py", line 1974, in _open
        seen, processing))
      File "/opt/slapos/eggs/zc.buildout-2.7.1+slapos019-py3.7.egg/zc/buildout/buildout.py", line 1940, in _open
        with open(downloaded_filename) as fp:
    FileNotFoundError: [Errno 2] No such file or directory: '/srv/slapgrid/slappart49/srv/project/slapos/software/neoppod/software-zodb5.cfg'

-> Fix it by adjusting wendelin.core tests correspondingly.

Similarly to 339490eb (components/zodbtools: reorganize test profiles)
make each test profile to explicitly indicate which configuration it
tests against.

Drop support for testing pristine ZODB4 - now only ZODB5 and ZODB4-wc2 remain.

We can drop test coverage for ZODB4 because slapos mainline switched to
ZODB5 to be used by default in 1c51c4cd (stack/erp5: version up all zope
stack (Zope 4.8.7), and because wendelin.core 2 works only with ZODB5
and ZODB4-wc2, but not with plain ZODB4.

In Wendelin we mostly always want to run our zopes with the soft limit
set to the hard limit.

/reviewed-by @jerome, @jm, @rafael, @vpelletier
/reviewed-on !1465

---

This patch was already added in !1451, but
removed (by force push), due to issues discussed in !1451 (comment 193296).

nexedi/erp5@39369169 added the functionality to ERP5 to
set its soft limit of allowed open file descriptors to the system wide hard limit.
This parameter is useful for Wendelin based instances where the 1024
limit is easily reached.

With this patch, this parameter can also be set via SlapOS, which
simplifies usage of the Wendelin SR.

/reviewed-by @jerome, @jm, @rafael, @vpelletier
/reviewed-on nexedi/slapos!1465

---

This patch was already added in nexedi/slapos!1451, but
removed (by force push), due to issues discussed in nexedi/slapos!1451 (comment 193296).

Comments need to be at the start of the line.

The SLA unique_by_network was not requested at all when any SLA was set
in the resilience. Now we put unique_by_network for all instances
without SLA.

this is needed for slapos-node package compilation

This is a fixup of 5f6a2eb8.

util-linux needs gcc 8+ in order to avoid error:

libuuid/src/gen_uuid.c:536:33: error: initializer element is not constant

See merge request !1489

See merge request !1457