In the test output, there are lots of tracebacks like this one:

    "./software/caddy-frontend/test/test.py", line 534, in do_GET
        for header in config.pop('X-Drop-Header', '').split():
    AttributeError: 'NoneType' object has no attribute 'split'

this does not seem to impact the test result, but it makes the result hard to
read.

Now that slapos-node-software service does not use --all we have to use another
approach for this test which re-installs an already installed software.

Use the slapos-node-software-all services which was made for similar use cases.

This was changed as part of b049c057 (upgrade to slapos.toolbox version
0.123, 2021-07-19), but this is not a slapos.toolbox promise plugin, it's
still an old style promise.

when running on test nodes some promise fail because testnode itself has
monitoring and the promise connects to testnode's own monitoring.

This reveals a real problem, but we need to fix testnode first to be able
to use this.

With gcc 6.3.0 we see build failures:

    error: no matching function for call to 'tesseract::STRING::STRING()'

According to https://github.com/tesseract-ocr/tesseract/issues/3210#issuecomment-774599019
minimal gcc version is version 7.

This probably started to break since d6dd75df (component/ghostscript:
version up 9.54.0, 2021-05-25) and 5f616dd7 (component/tesseract: version
up 4.1.1, 2021-05-20).

This gcc is needed for both tesseract and ghostscript, because ghostscript
build again its embedded copy of tesseract.

In b049c057 (upgrade to slapos.toolbox version 0.123, 2021-07-19), we
translated check-secure = 1 to config-http-code = 401, but check-secure = 1
also accepted responses with status code 200.

Thoses promises where checking servers without authentication enabled, they
reply with 200, so update the promises accordingly to expect 200.

This fixes a username/password confusion from c536eb7c (stack/monitor:
change frontend url promise to use username/password, 2021-07-26)

This change has no effect on stand-alone setups (which is what is
automatically deployed).
On setups configured to replicate from another database, it allows them to
takeover the role of the primary server, which allows rebuilding a
redundant setup. On such setup, the cost of this option is the extra I/O
load of binlogs being written to (in addition to the relay log and
database), and the corresponding disk space use.

Commit 7d1ea024 (version up: zc.buildout 2.7.1+slapos013) switched buildout to
not use egg wheels, even for wheels that are source-only without any binaries.
As the result some eggs stopped to install because they setup_require other
eggs and neither buildout handles setup_requires automatically, nor there is
explicit setup_require declaration in corresponding components.

An example of such egg is python-dateutil: it setup_requires setuptools-scm,
and without switching python-dateutil to zc.recipe.egg:custom with
setup_requires the build fails.

The way 7d1ea024 used to workaround this is to mark all such eggs with :whl
suffix marking them that "it is ok to use wheel for this particular egg". The
complete list of eggs for which such markings was done in 7d1ea024 is:

    ipykernel
    plantuml
    mock
    PasteScript
    backports.functools-lru-cache
    cliff
    stevedore
    CacheControl
    pytest-runner
    jsonschema
    lockfile
    python-dateutil

However this list is not really complete because after 7d1ea024 the build
started to fail for e.g. zodbtools test:

        Traceback (most recent call last):
          File "<string>", line 10, in <module>
          File "/srv/slapgrid/slappart16/srv/testnode/dfp/soft/c670f74c8de08964f63c4b9f7d48c3d1/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 2321, in main
          ...
          File "/srv/slapgrid/slappart16/srv/testnode/dfp/soft/c670f74c8de08964f63c4b9f7d48c3d1/eggs/setuptools-44.1.1-py2.7.egg/pkg_resources/__init__.py", line 786, in resolve
            raise DistributionNotFound(req, requirers)
        pkg_resources.DistributionNotFound: The 'setuptools-scm' distribution was not found and is required by the application
        An error occurred when trying to install pytest 4.6.11. Look above this message for any errors that were output by easy_install.
        While:
          Installing zodbtools-python.
          Base installation request: 'zodbtools[test]'
            Requirement of zodbtools[test]: mock
            Requirement of zodbtools[test]: pytz
            Requirement of zodbtools[test]: freezegun
            Requirement of zodbtools[test]: pytest
            Requirement of zodbtools[test]: dateparser
            Requirement of zodbtools[test]: six
            Requirement of zodbtools[test]: pygolang>=0.0.0.dev6
            Requirement of zodbtools[test]: zope.interface
            Requirement of zodbtools[test]: zodburi
            Requirement of zodbtools[test]: ZODB
            Requirement of mock: funcsigs>=1
            Requirement of mock: six
            Requirement of freezegun: python-dateutil>=2.7
          Getting distribution for 'pytest==4.6.11'.
        Error: Couldn't install: pytest 4.6.11

and similarly for pygolang and nxdtest tests.

I would be more comfortable if the fix would be to teach buildout to either
handle setup_requires automatically, or to automatically install wheels if the
wheel is source-only. However, given the way - that 7d1ea024 started to use
to handle such situations - let's fix this via explicitly marking pytest, and
other related eggs that setup_require something and are source-only, as :whl.

NOTE we did not notice the problem earlier because on testnodes, after the
switch to new buildout, pytest egg was already installed and incremental
rebuild did not need to reinstall pytest egg. So most of pygolang and zodbtools
test results remained in "ok" state, and only zodbtools-kirr, that was recently
reconfigured or changed testnode, indicated this issue. That's why the problem
remained unnoticed for a long time.

/reviewed-by @jerome
/reviewed-on nexedi/slapos!1026

See merge request nexedi/slapos!1022

This introduces some breaking changes to the check_url_available
promise:

- http_code has been renamed to http-code
- check-secure = 1 has been deprecated; http-code = 401 should be used
  instead

See merge request nexedi/slapos!1011

also apply the change in software/theia/test/dummy/software.cfg for consistency

  This is a partial port, just to have more tests that
  weren't afected by the patches for keep apache as entry-point
  of the backend.

See merge request nexedi/slapos!1009

This reverts commit cf99b892

See merge request !1020

In 6e696faa "v↑ tempstorage 2.12.2 -> 5.2(ZODB5) 3-nxd(ZODB4-wc2)" we upgraded
tempstorage for ZODB5 and ZODB4-wc2, but were careful and left tempstorage,
that was being used with ZODB4, intact. However @tomo suggests that we should
also upgrade tempstorage for ZODB4 because of the loadBefore fixes and not to
further grow divergence in between ZODB4 and ZODB4-wc2:

    I understand that keeping the same version of tempstorage for ZODB 4 is to
    be on the safe side but I'd rather change it to tempstorage4-wc2 anyway. I mean
    we have a bugfix but we are afraid to apply it. Also, The less difference
    between ZODB 4 and ZODB 4-wc2 the better (everything will be more tested this way).
    ( nexedi/slapos!1009 (comment 137985) )

-> So let's do that.

This was introduced recently, in 0aecb41a (software: Use local tmp for
caucase, 2021-05-18). This is harmless, but uses a weird-looking directory

With the default umask of running slapos node, the default umask create files
that are only readable by user, not by group:

    slapuser10@COMP-1234:~$ ls -al /opt/slapgrid/bf0ccebdbd76d491bcbe16b7998529b2/parts/tesseract/share/tessdata/
    total 125296
    drwxr-x--- 4 slapsoft slapsoft 4096 Jul 13 05:57 .
    drwxr-x--- 3 slapsoft slapsoft 4096 Jul 13 05:57 ..
    -rw------- 1 slapsoft slapsoft 44366093 Jul 13 05:57 chi_sim.traineddata
    drwxr-x--- 2 slapsoft slapsoft 4096 Jul 13 05:57 configs
    -rw------- 1 slapsoft slapsoft 23466654 Jul 13 05:57 eng.traineddata
    -rw------- 1 slapsoft slapsoft 14213351 Jul 13 05:57 fra.traineddata
    -rw------- 1 slapsoft slapsoft 35659159 Jul 13 05:57 jpn.traineddata
    -rw------- 1 slapsoft slapsoft 10562874 Jul 13 05:57 osd.traineddata
    -rw-r--r-- 1 slapsoft slapsoft 572 Jul 13 05:57 pdf.ttf
    drwxr-x--- 2 slapsoft slapsoft 4096 Jul 13 05:57 tessconfigs

but when using multiple users, partitions users are members of slapsoft group,
so these files needs to be group readable.

See merge request nexedi/slapos!1015

See merge request nexedi/slapos!1012

listening-ip parameter is now mandatory

tempstorage 2.12.2 is from 2012 and misses loadBefore fixes which are
important when ERP5 is run with ZODB5 or ZODB4-wc2:

https://github.com/zopefoundation/tempstorage/commit/d187a86d
https://github.com/zopefoundation/tempstorage/pull/16

When the first loadBefore fix is not there, the problem manifests itself as
Zope not starting correctly with the following error in its log:

    2021-07-02 08:37:25,579 ERROR Zope.ZODBMountPoint Failed to mount database. <class 'ZODB.POSException.ReadConflictError'> (^@^@^@^@^@^@^@^@)
    Traceback (most recent call last):
     File ".../eggs/Zope2-2.13.30-py2.7.egg/Products/ZODBMountPoint/MountedObject.py", line 248, in _getOrOpenObject
       root = conn.root()
     File ".../parts/ZODB4-wc2/src/ZODB/Connection.py", line 405, in root
       return RootConvenience(self.get(z64))
     File ".../parts/ZODB4-wc2/src/ZODB/Connection.py", line 276, in get
       p, serial = self._load(oid)
     File ".../parts/ZODB4-wc2/src/ZODB/Connection.py", line 246, in _load
       return self._loadFrom(oid, self._storage)
     File ".../parts/ZODB4-wc2/src/ZODB/Connection.py", line 255, in _loadFrom
       raise ReadConflictError(oid)
    ReadConflictError: ^@^@^@^@^@^@^@^@

When the second loadBefore fix is not there, the problem manifests
itself as data corruption and sporadic POSKeyErrors
(https://github.com/zopefoundation/tempstorage/issues/8).

-> Fix both problem by using recent tempstorage with ZODB5 and ZODB4:

- ZODB5: just use tempstorage 5.2 egg
- ZODB4-wc2: use tempstorage 3 + loadBefore backports:
  https://lab.nexedi.com/nexedi/tempstorage/commits/3-nxd
- ZODB4: stays at 2.12.2 for now for this change to be as risk-free as
  possible. However, due to the fixes and e.g. https://github.com/zopefoundation/tempstorage/issues/8#issuecomment-873894941
  it makes sense to switch ZODB4 to tempstorage 3-nxd as well.

/cc @tomo, @romain, @rafael

https://github.com/restic/rest-server

Now that slapos node instance no longer use --all ( nexedi/slapos.core!278 ),
some software release test revealed some problems with softwares,
either missing promises or maybe something that needs to be handled in slapos.core.

Using --all was wrong and we want to stop doing this now. Instead of fixing these software now, we keep the old behavior
of using --all only for the softwares with missing promises.


See merge request nexedi/slapos!969