In addition to linux-image we have to explicitly install/upgrade kernel
headers as well because once they are installed via nxd-fuse-dkms ->
dkms -> linux-headers-amd64, they won't be upgraded, and if
linux-image-amd64 gets upgraded, but headers are not, nxd-fuse-dkms will
be skipped to recompile for upgraded kernel.

nexedi/slapos.package!133 (comment 121068)

See merge request nexedi/slapos.package!133

Debian 9 and Debian10 ship linux-4.9.x and linux-4.19.x correspondingly.
However kernel packages on those distributions include ABINAME as
package version, for example linux-image-4.9.0-13-amd64
means linux 4.9.x with ABINAME=13.

The ABINAME is there because Linux sometimes breaks ABI compatibility in
small places and Debian is very strict on not throwing ABI changes onto
users unless requested.  However even with that protection Debian
"strongly recommends" to explicitly install just linux-image-amd64 -
without ABINAME - which is just a dependency package that depends on the
latest kernel in particular distribution series:

https://wiki.debian.org/DebianKernelABIChanges

The reason for this recommendation is that upstream Linux rarely breaks
any ABI and even if there is a breakage it is so small and in obscure
places that in practice it does not affect people.

Today, for Debian9, we are explicitly requesting to install Linux 4.9
with ABINAME=13. That stops on 4.9.228 while current Linux 4.9 is
4.9.240 bumped to ABI 14. Before ABI=13, we were requesting to install
ABI=11 which stopped on 4.9.189 and was v↑'ed in 1f249bf7 (playbook:
debian9: kernel v↑  (4.9.189-3 -> 4.9.228-1). In other words by
explicitly specifying linux ABINAME we prevent to keep on updating Linux
kernel to latest _stable_ updates provided by upstream Linux and the
distribution.

Another inconvenience of installing Linux with explicit ABINAME is
interaction with nxd-fuse.ko: this module comes with nxd-fuse-dkms
package, which uses dkms to build itself, and dkms recommends
linux-headers-amd64 for modules that it manages to be able to build.

However we recently saw an issue when linux-headers-amd64 was installed
as latest and depending on linux-headers-14-amd64 (NOTE ABINAME=14),
while the kernel on that server was only linux-image-4.9.0-13-amd64
(NOTE ABINAME=13) as requested by our playbook. As the result nxd-fuse
was skipped to compile, failed to be loaded and FUSE became non-working
on that machine:

    nexedi/slapos.package!132 (comment 120438)

    root@rapidspace-testnode-005:~# apt install nxd-fuse-dkms
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following NEW packages will be installed:
      nxd-fuse-dkms
    0 upgraded, 1 newly installed, 0 to remove and 105 not upgraded.
    Need to get 0 B/58.8 kB of archives.
    After this operation, 295 kB of additional disk space will be used.
    Selecting previously unselected package nxd-fuse-dkms.
    (Reading database ... 80295 files and directories currently installed.)
    Preparing to unpack .../nxd-fuse-dkms_4.9.nxd3+debian2_all.deb ...
    Unpacking nxd-fuse-dkms (4.9.nxd3+debian2) ...
    Setting up nxd-fuse-dkms (4.9.nxd3+debian2) ...
    Loading new nxd-fuse-4.9.nxd3+debian2 DKMS files...
    Building for 4.9.0-13-amd64
    Module build for kernel 4.9.0-13-amd64 was skipped since the
    kernel headers for this kernel does not seem to be installed.

-> Fix it by requiring only linux-image-amd64 without specifying ABINAME and
relying on upstream Linux and distro to provide stable updates for the kernel.

See merge request nexedi/slapos.package!134

See merge request nexedi/slapos.package!132

See merge request nexedi/slapos.package!131

See merge request nexedi/slapos.package!130

Wendelin.core 2 tests need to be able to mlock 64M to run.
Increase corresponding limit for all subprocessed spawned by cron, which
covers slapos -> supervisord -> ...

See added comments in slapos_limit.conf for why this has to be done in
/etc/security/limits.d/slapos.conf instead of
/etc/systemd/system/cron.service.d/override.conf

I testd this to work on a KVM of mine with manually putting
/etc/security/limits.d/slapos.conf in place. Before the patch:

    https://erp5.nexedi.net/test_result_module/20201009-341BCC3C/17
    (lots of "OSError: [Errno 1] Operation not permitted" for mlock requests)

After the patch:

    https://erp5.nexedi.net/test_result_module/20201009-6B97A93B/17
    (test.wcfs/fs passes correctly)

/cc @rafael, @tomo, @luke, @Nicolas

nxd-fuse is fuse package from those distribution with backported
git.kernel.org/linus/ad2ba64dd489 for wendelin.core 2 to be able to
fully and explicitly control OS pagecache for files that wcfs serves.
This patch is very essential for wendelin.core 2.

The sources for nxd-fuse package are located in fs/fuse/ here:

https://lab.nexedi.com/kirr/linux/commits/nxd/fuse/debian/buster	(Debian 10)
https://lab.nexedi.com/kirr/linux/commits/nxd/fuse/debian/stretch	(Debian 9)
https://lab.nexedi.com/kirr/linux/commits/nxd/fuse/ubuntu/xenial	(Ubuntu 16.04)

I tested the package by myself for a long time on my Debian10-based
laptop via go-fuse tests and while working on wendelin.core. Ubuntu
package was also tested for a long time on my KVM via manually running
the same tests and, recently, via running wendelin.core 2 tests on
testnode located on the same KVM. I tested Debian 9 module only briefly
on my Debian 9 KVM, but given that the diff in between nxd-fuse and
std/ fs/fuse/ for that kernel is essentially only
git.kernel.org/linus/ad2ba64dd489, and that the debianisation was
already tested well via Debian10, and that go-fuse and wendelin.core 2
tests pass, it should be ok.

The packages are built on OBS here:

https://build.opensuse.org/package/show/home:VIFIBnexedi/nxd-fuse
https://build.opensuse.org/package/view_file/home:VIFIBnexedi/nxd-fuse/README

I've used vifib-kernel role as the place where to add corresponding
play. Nxd-fuse is tightly related to kernel and requires reboot after
installation, which, if I understood correctly, needs infrastructure
internal to that role to work correctly.

/cc @rafael, @tomo, @vpelletier

There are many fixes in between those versions, including security ones:

https://salsa.debian.org/kernel-team/linux/-/raw/stretch/debian/changelog

those 2 modules can make a system crash at startup. It is true for
Supermicro motherboards and winterfell motherboards.

See https://www.thomas-krenn.com/en/wiki/Resolve_mei:_Init_hw_failure_or_mei:_initialization_failed

This reverts commit 280c9763.

In commit 433f3d22, we disable the test
runners so the instances went back to previous organisation and so the
ports.

See merge request nexedi/slapos.package!128

The 1.0.163 version has instantiation fixed.

It's not needed in this context.

See merge request nexedi/slapos.package!127

See merge request nexedi/slapos.package!126

Also add important note why the SR shall be kept as is.

See merge request nexedi/slapos.package!125

python-requests package doesn't exist anymore on Ubuntu 20.04 so we
convert the upload script to support both python2 and python3 and we use
python3 in Debian >= 10 and Ubuntu >= 20.04.

This let us clean up the imt-vm-bootstrap playbook for Debian 10 (remove
manual installation of python-requests package).

See also nexedi/slapos!720