Similarly to f365a440 (software/ors-amarisoft: Merge gnb.jinja2.cfg into
enb.jinja2.cfg) merge LTE and NR configuration files for UEsim into single
config as the preparation for MultiRU - there UEsim will be also handling TDD,
FDD, LTE and NR cells and UEs all at the same time.

In this patch for now we only move code without changing it and wrap parts with
`if do_lte` and `if do_nr` correspondingly.  The end result of rendered
ue-lte.cfg and ue-nr.cfg stays practically the same as shown in the appendix.

/cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus

--------

Appendix. Diff for rendered ue-*.cfg before and after this patch.

```
$ ./pythonwitheggs slapos-render-config.py && git diff --no-index -w config/old/ config/out/
```

```diff
diff --git a/config/old/ue-lte.cfg b/config/out/ue-lte.cfg
index 7f1530daf..836a95e79 100644
--- a/config/old/ue-lte.cfg
+++ b/config/out/ue-lte.cfg
@@ -1,5 +1,6 @@

 #define N_RB_DL             50
+
 {
   log_options: "all.level=error,all.max_size=0,nas.level=debug,nas.max_size=1,rrc.level=debug,rrc.max_size=1,phy.level=info,file.rotate=1G,file.path=/dev/null",
   log_filename: "log/ue.log",
@@ -14,7 +15,8 @@ rf_driver: {
 },
   tx_gain: 60,
   rx_gain: 40,
-  cell_groups: [{
+  cell_groups: [
+  {
     multi_ue: true,
     cells: [
       {
@@ -27,7 +29,8 @@ rf_driver: {
     ],
     pdcch_decode_opt: false,
     pdcch_decode_opt_threshold: 0.1,
-  }],
+  },
+  ],
   ue_list: [
     {
     sim_algo: "milenage",
@@ -38,8 +41,8 @@ rf_driver: {
     impi: "",
     imsi: "001010123456789",
     K: "00112233445566778899aabbccddeeff",
-    rue_addr: "host1",
     ue_category: 12,
+    rue_addr: "host1",
     tun_setup_script: "ue-ifup",
     apn: "internet",
     }
diff --git a/config/old/ue-nr.cfg b/config/out/ue-nr.cfg
index 41758ab6f..631e1ab1e 100644
--- a/config/old/ue-nr.cfg
+++ b/config/out/ue-nr.cfg
@@ -1,3 +1,5 @@
+
+
 {
   log_options: "all.level=error,all.max_size=0,nas.level=debug,nas.max_size=1,rrc.level=debug,rrc.max_size=1,phy.level=info,file.rotate=1G,file.path=/dev/null",
   log_filename: "log/ue.log",
@@ -12,7 +14,8 @@ rf_driver: {
 },
   tx_gain: 60,
   rx_gain: 40,
-    cell_groups: [{
+  cell_groups: [
+  {
     group_type: "nr",
     multi_ue: false,
     cells: [{
@@ -26,7 +29,8 @@ rf_driver: {
       n_antenna_ul: 2,
       }
     ],
-    }],
+  },
+  ],
   ue_list: [
     {
     sim_algo: "milenage",
@@ -37,11 +41,11 @@ rf_driver: {
     impi: "",
     imsi: "001010123456789",
     K: "00112233445566778899aabbccddeeff",
-      rue_addr: "host2",
     as_release: 15,
     ue_category: "nr",
+    rue_addr: "host2",
     tun_setup_script: "ue-ifup",
     apn: "internet",
     }
-    ]
+  ],
 }
```

This is useful during development while working on ue-*.jinja2.cfg

We are going to generate more and more type of files via slapos-render-config:
currently we generate only enb.cfg and gnb.cfg, but soon it will be also
ue-*.cfg, drb_*.cfg, asn etc.

So it will be handy to easily separate generated files from their source, to
make snapshots of generated stuff for comparison, and to also e.g. to be able
to copy generated configs to a callbox/simbox for testing.

-> Move the place where generated files are emitted from config/ to config/out/

To run tapsplit we use plone.recipe.command with both command and
update-command set to `tapsplit ...`. But tapsplit, when run, fully
recreates and reinitializes subtap interfaces, which leads to
interfering with running enb because subtap interfaces, that enb
started to use, are removed. This is not desirable behaviour.

What we need:

1) create subtap interfaces only once and keep them stable
2) until configuration changes which should lead to
   * subtaps recreated, and
   * enb restarted

Carefully reading plone.recipe.command documentation shows:

  command
    Command to run when the buildout part is installed.

  update-command
    Command to run when the buildout part is updated. This happens when
    buildout is run BUT THE CONFIGURATION FOR THIS BUILDOUT PART HAS NOT
    CHANGED.

  (emphasis mine)

So the fix looks to be to make update-command noop - this fulfills
requirement "1". For "2" - I've verified that when configuration
changes, e.g. number of RU changes, buildout reinstalls [vtap] section
from scratch, and it also should restart enb, because generated enb.cfg
changes.

So the fix is fully correct because it satifies all needed requirements.

Amends 49ce8ef5 (software/ors-amarisoft: Provide dedicated TAP interface for each Radio Unit)
/cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus

While not faulty in itself, f365a440 caused ors-amarisoft testsuite to start failing with

    ERROR: test_gnb_conf (testTDD.TestGNBParameters1)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File ".../slapos-repository/software/ors-amarisoft/test/testTDD.py", line 298, in test_gnb_conf
        test_gnb_conf1(self)
      File ".../slapos-repository/software/ors-amarisoft/test/testTDD.py", line 184, in test_gnb_conf1
        self.assertEqual(conf['tx_gain'], gnb_param_dict1['tx_gain'])
    KeyError: 'tx_gain'

( https://erp5js.nexedi.net/#/test_result_module/20240119-1C2BDD94/45 )

This happens because previously gnb.jinja2.cfg emitted tx_gain/rx_gain and everything else for SDR case when `ru != "m2ru"`:

    https://lab.nexedi.com/nexedi/slapos/blob/d078060c/software/ors-amarisoft/config/gnb.jinja2.cfg#L22-55

while enb.jinja2.cfg had this only under `if bbu == "ors"`:

    https://lab.nexedi.com/nexedi/slapos/blob/d078060c/software/ors-amarisoft/config/enb.jinja2.cfg#L44-114

so since bbu != "ors" when tests are run via `testTDD.py` and `testFDD.py`
(and `testTDD-ORS.py` is not  ran at all as I explained in nexedi/slapos.toolbox!129)
this resulted in gnb.cfg not to have tx_gain/rx_gain for bbu != ors case and
corresponding test failure.

There is no real problem for ORS since for `bbu==ors` tx_gain/rx_gain are emitted.

-> Fix testsuite to pass again similarly to how it was done in b32b4a8e
   (software/ors-amarisoft: general improvement for RU (logs/promises/input
   parameters) for now).

In the future ORS case will be verified by dedicated test_ors.py which will
have checks for tx_gain and rx_gain unconditionally applied, while support for
all other cases will be handled by generic test.py which will have different
logic for how rf_driver section is generated and verified.

https://lab.nexedi.com/kirr/slapos/blob/b21d2636/software/ors-amarisoft/test/test_ors.py
https://lab.nexedi.com/kirr/slapos/blob/b21d2636/software/ors-amarisoft/test/test.py

See merge request nexedi/slapos!1500

Directly expose all passlib.hash supported hashes, using a `passwd-`
prefix. For example, to access `sha256_crypt`, use `passwd-sha256-crypt`
option name.

  [secret]
  recipe = slapos.cookbook:generate.password

  [config-file]
  hashed-password = ${secret:passwd-sha256-crypt}

This changes the format of storage-path, it used to be the password in
plain text, it is now a mapping also containing hashed passwords, to
have the same hashed passwords for each buildout run.

This needs collaboration from publish_early recipe, because .pop(k) does
raised a KeyError with the dict.__missing__  approach.

In MultiRU there will be only one ENB which supports all TDD, FDD, LTE and NR
cells and different types of radio units - all at the same time. This patch is
preparatory step for that: it merges gnb configuration template into enb
configuration template, so that enb.jinja2.cfg now serves both enb and gnb
instances.

In this patch for now we only move code from gnb.jinja2.cfg without changing
it(*) and wrap parts with `if do_lte` and `if do_nr` correspondingly.

The end result of rendered enb.cfg and gnb.cfg stays the same modulo space
changes and added innoccent `#define TDD` as Appendix shows.

(*) the only exception is set up of gtp_addr which has practically the same
code, was wrapped with `if mme_list` in enb and `if amf_list` in gnb, and is now
wrapped with `if mme_list or amf_list`.

/cc @lu.xu, @tomo, @xavier_thompson, @Daetalus
/reviewed-by @jhuge
/reviewed-on !1512

--------

Appendix. Diff for rendered enb.cfg and gnb.cfg before and after this patch.

```
$ ./pythonwitheggs slapos-render-config.py && (git diff -w --no-index config/enb.cfg.old config/enb.cfg ; git diff -w --no-index config/gnb.cfg.old config/gnb.cfg)
```

```diff
diff --git a/config/enb.cfg.old b/config/enb.cfg
index fdf3ab24d..cb46697ea 100644
--- a/config/enb.cfg.old
+++ b/config/enb.cfg
@@ -1,11 +1,15 @@

+
 #define TDD                 1

 #define N_RB_DL             50
+
 #define N_ANTENNA_DL        2
+
 #define N_ANTENNA_UL        2

+
 {

   log_options: "all.level=error,all.max_size=0,nas.level=debug,nas.max_size=1,s1ap.level=debug,s1ap.max_size=1,x2ap.level=debug,x2ap.max_size=1,rrc.level=debug,rrc.max_size=1,phy.level=info,file.rota>
@@ -24,7 +28,6 @@
   rx_gain: 43,

   com_addr: "127.0.1.2:9001",
-
   mme_list: [

     {
@@ -33,9 +36,9 @@

   ],

+
   gtp_addr: "127.0.1.1",

-
   enb_id: 0x1A2D0,

   cell_list: [{
@@ -49,7 +52,6 @@
       ],
     }
   ],
-
   cell_default: {
     plmn_list: [
       "00101",
@@ -226,4 +228,6 @@
   meas_gap_config: "gp0",
   ho_from_meas: true,
   },
+
+
 }
\ No newline at end of file
diff --git a/config/gnb.cfg.old b/config/gnb.cfg
index e3d671e09..4e47a2094 100644
--- a/config/gnb.cfg.old
+++ b/config/gnb.cfg
@@ -1,15 +1,21 @@

+
+#define TDD                 1
+
+
 #define N_ANTENNA_DL        2

 #define N_ANTENNA_UL        2

+
 {

   log_options: "all.level=error,all.max_size=0,nas.level=debug,nas.max_size=1,ngap.level=debug,ngap.max_size=1,xnap.level=debug,xnap.max_size=1,rrc.level=debug,rrc.max_size=1,phy.level=info,file.rota>

   log_filename: "log/gnb.log",

+
   rf_driver: {
       name: "sdr",
       args: "dev0=/dev/sdr0",
@@ -30,15 +36,17 @@
   ],

-
-
   gtp_addr: "127.0.1.1",

   gnb_id_bits: 28,
   gnb_id: 0x12345,
   en_dc_support: true,
-  cell_list: [],
+
+  cell_list: [
+  ],
+
+
   nr_cell_list: [
   {
     rf_port: 0,
```

It stopped to be used after 49ce8ef5 (software/ors-amarisoft: Provide dedicated
TAP interface for each Radio Unit).

/cc @lu.xu, @tomo, @xavier_thompson, @Daetalus
/reviewed-by @jhuge
/reviewed-on nexedi/slapos!1512

Because

1) those services are needed and used only by ru/ promises like check_cpri_lock
   and check_rx_saturated.
2) in general we will need to initialize and setup radio units not only in eNB -
   for example UEsim will use the same code library to initialize radio units.
   Thus the proper place to keep everything required for RU to be operational have to
   be located inside ru/ and activated by that radio-units library.

Push corresponding code from instance-enb to ru/ and do only minor adjustments
to instance-gnb trying not to break it, since gnb does not currently use rulib,
and because in the future gnb will be replaced by enb which will be serving
both lte and nr cells in the same service.

/cc @lu.xu, @tomo, @xavier_thompson, @Daetalus
/reviewed-by @jhuge
/reviewed-on nexedi/slapos!1511

Because:

- ssh server is needed for and used by ru/lopcomm/ only
- in general we will need to initialize and setup radio units not only in eNB -
  for example UEsim will use the same code library to initialize radio units.
  Thus the proper place to keep everything required for RU to be operational have to
  be located inside ru/ and activated by that radio-units library.

/cc @lu.xu, @tomo, @xavier_thompson, @Daetalus
/reviewed-by @jhuge
/reviewed-on nexedi/slapos!1510

Dnsmasq insists on dhcp-range's prefixlen to be at most 64, which
triggers the following error if original slaptap is wider than that:

    dnsmasq: prefix length must be at least 64 at line 5 of /srv/slapgrid/slappart6/etc/dnsmasq.cfg

-> Fix it by capping provided range to /64

/cc @tomo, @xavier_thompson, @Daetalus
/reported-by @lu.xu
/reviewed-by @jhuge
/reviewed-on !1509

our new nexedi.org currently no longer serve static pages with links,
using absolute links should allow us to have test passing until this is
fixed.

also expose MB_UNAGGREGATED_QUERY_ROW_LIMIT and MB_AGGREGATED_QUERY_ROW_LIMIT

These partition references should be kept short, they are a mechanism
to use a short path for unix sockets, because unix socket paths can not
exceed 108 characters.

When running the test in theia, this was causing errors:

   # [ALERT]    (100453) : config : parsing [/srv/slapgrid/slappart15/srv/runner/instance/slappart7/tmp/inst/with-max-rlimit-nofile6/etc/haproxy.cfg:37] : log : socket path '/srv/slapgrid/slappart15/srv/runner/instance/slappart7/tmp/inst/with-max-rlimit-nofile6/var/run/log.sock' too long (max 97

Currently, due to ensure_ascii=True default of json.dumps, we are
insisting on our JSON schemas to be ascii-only and all other characters
to be represented by \uxxxx escapes. So far this was not problematic as
all our schemas contains only ASCII characters, but upcoming
ors-amarisoft changes want to use e.g. "→" symbol:

    https://lab.nexedi.com/kirr/slapos/blob/b51f5523/software/ors-amarisoft/software.cfg.json#L15

which currently results in failure of json-schema test:

    FAIL: test_ors-amarisoft_software_cfg_json_format (slapos.test.test_json_schema.TestJSONSchemaValidation)
    ...

    First differing element 14:
    '      "title": "\\u2192  eNB/gNB | Radio Unit",'
    '      "title": "→  eNB/gNB | Radio Unit",'

And in general, in 2023 I think there is no reason to insist on our schemas to
be ASCII-only: say if one wants to describe something about "α" parameter. It
would be good to use that α character directly and seeing it in the editor,
instead of using escapes all the time.

As indicated by below stackoverflow answer "JSON spec requires UTF-8 support by
decoders": https://stackoverflow.com/a/594881/9456786 , and indeed checking
JSON specification also confirms that by default JSON decoders shall use UTF-8:

    https://datatracker.ietf.org/doc/html/rfc7159#section-8.1

This way, I think, we can switch to UTF-8 safely.

/reviewed-by @jerome, @lu.xu
/reviewed-on nexedi/slapos!1498

See merge request nexedi/slapos!1499

/reviewed-by @kirr and @jerome
/reviewed-on nexedi/slapos!1494

wendelin.core 2.0.alpha3.post9 supports golang 1.21: with this new
version we can set golang 1.21 as the new default.

/reviewed-by @kirr and @jerome
/reviewed-on nexedi/slapos!1494

Go1.21 already has the fifth minor revision (released 2023-12-05) and
should therefore already be sufficiently stable. Furthermore we need it
to fix a bug in a NEO/go dependency [1].

Please find all details in the official release note:

    https://go.dev/doc/go1.21

It was released on 2023-08-08 [2]. Due to the go promise of compatibility
most software should still compile without any problems.

In golang < 1.21 we needed to patch golang to fix
https://github.com/golang/go/issues/42525. In golang 1.21 we still need
to apply a fix, but can't apply the old patch because the code changed.
In golang > 1.21 this problem is already fixed with https://go-review.googlesource.com/c/go/+/520055.

Because of https://github.com/golang/go/commit/092671423cd95eaa6df93eb29442fef41504d097
'TestUnshareMountNameSpace' fails on golang 1.21 [3]. In golang 1.20 this test
was skipped [4]. To fix this failure the additional patch
'skip-unshare-mount-test.patch' has been added.

---

[1] nexedi/wendelin.core!22 (comment 195769)]
[2] https://go.dev/doc/devel/release
[3] --- FAIL: TestUnshareMountNameSpace (0.18s)
    exec_linux_test.go:243: unshare failed: exit status 2
        unshare: mount /tmp/TestUnshareMountNameSpace2210137852/001 failed: 0x1
[4] === RUN TestUnshareMountNameSpace exec_linux_test.go:333: kernel prohibits unshare in unprivileged process, unless using user namespace — SKIP: TestUnshareMountNameSpace (0.00s)

/reviewed-by @kirr and @jerome
/reviewed-on nexedi/slapos!1494

This version is compatible with gcc 12

using https://github.com/ilevkivskyi/com2ann

drop versions not running on debian 12 and add new versions

This was disabled to "keep compatibility with current font selection
problems", but I don't think we need to care about compatibility for
this. This was anyway causing a problem that system fontconfig will be
used if it's available, so it's better to define things explictly to
prevent falling back to system configuration.

This caused cloudooo to select different fonts, because LibreOffice-bin
comes with some Noto fonts and fontconfig now prefers Noto font