See merge request nexedi/slapos!1503

Split command which is used to split pgdump files before push to gitlab backup repository crash during
backup process. This problem occurs with version starting from 9.2, older versions doesn't have this problem.
The version 9.1 is used for gitlab-backup so.

For more details, see: nexedi/slapos!1503 (diffs, comment 197515)

See merge request nexedi/slapos!1506

While not faulty in itself, f365a440 caused ors-amarisoft testsuite to start failing with

    ERROR: test_gnb_conf (testTDD.TestGNBParameters1)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File ".../slapos-repository/software/ors-amarisoft/test/testTDD.py", line 298, in test_gnb_conf
        test_gnb_conf1(self)
      File ".../slapos-repository/software/ors-amarisoft/test/testTDD.py", line 184, in test_gnb_conf1
        self.assertEqual(conf['tx_gain'], gnb_param_dict1['tx_gain'])
    KeyError: 'tx_gain'

( https://erp5js.nexedi.net/#/test_result_module/20240119-1C2BDD94/45 )

This happens because previously gnb.jinja2.cfg emitted tx_gain/rx_gain and everything else for SDR case when `ru != "m2ru"`:

    https://lab.nexedi.com/nexedi/slapos/blob/d078060c/software/ors-amarisoft/config/gnb.jinja2.cfg#L22-55

while enb.jinja2.cfg had this only under `if bbu == "ors"`:

    https://lab.nexedi.com/nexedi/slapos/blob/d078060c/software/ors-amarisoft/config/enb.jinja2.cfg#L44-114

so since bbu != "ors" when tests are run via `testTDD.py` and `testFDD.py`
(and `testTDD-ORS.py` is not  ran at all as I explained in nexedi/slapos.toolbox!129)
this resulted in gnb.cfg not to have tx_gain/rx_gain for bbu != ors case and
corresponding test failure.

There is no real problem for ORS since for `bbu==ors` tx_gain/rx_gain are emitted.

-> Fix testsuite to pass again similarly to how it was done in b32b4a8e
   (software/ors-amarisoft: general improvement for RU (logs/promises/input
   parameters) for now).

In the future ORS case will be verified by dedicated test_ors.py which will
have checks for tx_gain and rx_gain unconditionally applied, while support for
all other cases will be handled by generic test.py which will have different
logic for how rf_driver section is generated and verified.

https://lab.nexedi.com/kirr/slapos/blob/b21d2636/software/ors-amarisoft/test/test_ors.py
https://lab.nexedi.com/kirr/slapos/blob/b21d2636/software/ors-amarisoft/test/test.py

This makes urljoin works as expected and generally makes sense because
this is a collection.

See merge request nexedi/slapos!1500

Directly expose all passlib.hash supported hashes, using a `passwd-`
prefix. For example, to access `sha256_crypt`, use `passwd-sha256-crypt`
option name.

  [secret]
  recipe = slapos.cookbook:generate.password

  [config-file]
  hashed-password = ${secret:passwd-sha256-crypt}

This changes the format of storage-path, it used to be the password in
plain text, it is now a mapping also containing hashed passwords, to
have the same hashed passwords for each buildout run.

This needs collaboration from publish_early recipe, because .pop(k) does
raised a KeyError with the dict.__missing__  approach.

In MultiRU there will be only one ENB which supports all TDD, FDD, LTE and NR
cells and different types of radio units - all at the same time. This patch is
preparatory step for that: it merges gnb configuration template into enb
configuration template, so that enb.jinja2.cfg now serves both enb and gnb
instances.

In this patch for now we only move code from gnb.jinja2.cfg without changing
it(*) and wrap parts with `if do_lte` and `if do_nr` correspondingly.

The end result of rendered enb.cfg and gnb.cfg stays the same modulo space
changes and added innoccent `#define TDD` as Appendix shows.

(*) the only exception is set up of gtp_addr which has practically the same
code, was wrapped with `if mme_list` in enb and `if amf_list` in gnb, and is now
wrapped with `if mme_list or amf_list`.

/cc @lu.xu, @tomo, @xavier_thompson, @Daetalus
/reviewed-by @jhuge
/reviewed-on !1512

--------

Appendix. Diff for rendered enb.cfg and gnb.cfg before and after this patch.

```
$ ./pythonwitheggs slapos-render-config.py && (git diff -w --no-index config/enb.cfg.old config/enb.cfg ; git diff -w --no-index config/gnb.cfg.old config/gnb.cfg)
```

```diff
diff --git a/config/enb.cfg.old b/config/enb.cfg
index fdf3ab24d..cb46697ea 100644
--- a/config/enb.cfg.old
+++ b/config/enb.cfg
@@ -1,11 +1,15 @@

+
 #define TDD                 1

 #define N_RB_DL             50
+
 #define N_ANTENNA_DL        2
+
 #define N_ANTENNA_UL        2

+
 {

   log_options: "all.level=error,all.max_size=0,nas.level=debug,nas.max_size=1,s1ap.level=debug,s1ap.max_size=1,x2ap.level=debug,x2ap.max_size=1,rrc.level=debug,rrc.max_size=1,phy.level=info,file.rota>
@@ -24,7 +28,6 @@
   rx_gain: 43,

   com_addr: "127.0.1.2:9001",
-
   mme_list: [

     {
@@ -33,9 +36,9 @@

   ],

+
   gtp_addr: "127.0.1.1",

-
   enb_id: 0x1A2D0,

   cell_list: [{
@@ -49,7 +52,6 @@
       ],
     }
   ],
-
   cell_default: {
     plmn_list: [
       "00101",
@@ -226,4 +228,6 @@
   meas_gap_config: "gp0",
   ho_from_meas: true,
   },
+
+
 }
\ No newline at end of file
diff --git a/config/gnb.cfg.old b/config/gnb.cfg
index e3d671e09..4e47a2094 100644
--- a/config/gnb.cfg.old
+++ b/config/gnb.cfg
@@ -1,15 +1,21 @@

+
+#define TDD                 1
+
+
 #define N_ANTENNA_DL        2

 #define N_ANTENNA_UL        2

+
 {

   log_options: "all.level=error,all.max_size=0,nas.level=debug,nas.max_size=1,ngap.level=debug,ngap.max_size=1,xnap.level=debug,xnap.max_size=1,rrc.level=debug,rrc.max_size=1,phy.level=info,file.rota>

   log_filename: "log/gnb.log",

+
   rf_driver: {
       name: "sdr",
       args: "dev0=/dev/sdr0",
@@ -30,15 +36,17 @@
   ],

-
-
   gtp_addr: "127.0.1.1",

   gnb_id_bits: 28,
   gnb_id: 0x12345,
   en_dc_support: true,
-  cell_list: [],
+
+  cell_list: [
+  ],
+
+
   nr_cell_list: [
   {
     rf_port: 0,
```

It stopped to be used after 49ce8ef5 (software/ors-amarisoft: Provide dedicated
TAP interface for each Radio Unit).

/cc @lu.xu, @tomo, @xavier_thompson, @Daetalus
/reviewed-by @jhuge
/reviewed-on nexedi/slapos!1512

Because

1) those services are needed and used only by ru/ promises like check_cpri_lock
   and check_rx_saturated.
2) in general we will need to initialize and setup radio units not only in eNB -
   for example UEsim will use the same code library to initialize radio units.
   Thus the proper place to keep everything required for RU to be operational have to
   be located inside ru/ and activated by that radio-units library.

Push corresponding code from instance-enb to ru/ and do only minor adjustments
to instance-gnb trying not to break it, since gnb does not currently use rulib,
and because in the future gnb will be replaced by enb which will be serving
both lte and nr cells in the same service.

/cc @lu.xu, @tomo, @xavier_thompson, @Daetalus
/reviewed-by @jhuge
/reviewed-on nexedi/slapos!1511

Because:

- ssh server is needed for and used by ru/lopcomm/ only
- in general we will need to initialize and setup radio units not only in eNB -
  for example UEsim will use the same code library to initialize radio units.
  Thus the proper place to keep everything required for RU to be operational have to
  be located inside ru/ and activated by that radio-units library.

/cc @lu.xu, @tomo, @xavier_thompson, @Daetalus
/reviewed-by @jhuge
/reviewed-on nexedi/slapos!1510

Dnsmasq insists on dhcp-range's prefixlen to be at most 64, which
triggers the following error if original slaptap is wider than that:

    dnsmasq: prefix length must be at least 64 at line 5 of /srv/slapgrid/slappart6/etc/dnsmasq.cfg

-> Fix it by capping provided range to /64

/cc @tomo, @xavier_thompson, @Daetalus
/reported-by @lu.xu
/reviewed-by @jhuge
/reviewed-on nexedi/slapos!1509

our new nexedi.org currently no longer serve static pages with links,
using absolute links should allow us to have test passing until this is
fixed.

also expose MB_UNAGGREGATED_QUERY_ROW_LIMIT and MB_AGGREGATED_QUERY_ROW_LIMIT

These partition references should be kept short, they are a mechanism
to use a short path for unix sockets, because unix socket paths can not
exceed 108 characters.

When running the test in theia, this was causing errors:

   # [ALERT]    (100453) : config : parsing [/srv/slapgrid/slappart15/srv/runner/instance/slappart7/tmp/inst/with-max-rlimit-nofile6/etc/haproxy.cfg:37] : log : socket path '/srv/slapgrid/slappart15/srv/runner/instance/slappart7/tmp/inst/with-max-rlimit-nofile6/var/run/log.sock' too long (max 97

Currently, due to ensure_ascii=True default of json.dumps, we are
insisting on our JSON schemas to be ascii-only and all other characters
to be represented by \uxxxx escapes. So far this was not problematic as
all our schemas contains only ASCII characters, but upcoming
ors-amarisoft changes want to use e.g. "→" symbol:

    https://lab.nexedi.com/kirr/slapos/blob/b51f5523/software/ors-amarisoft/software.cfg.json#L15

which currently results in failure of json-schema test:

    FAIL: test_ors-amarisoft_software_cfg_json_format (slapos.test.test_json_schema.TestJSONSchemaValidation)
    ...

    First differing element 14:
    '      "title": "\\u2192  eNB/gNB | Radio Unit",'
    '      "title": "→  eNB/gNB | Radio Unit",'

And in general, in 2023 I think there is no reason to insist on our schemas to
be ASCII-only: say if one wants to describe something about "α" parameter. It
would be good to use that α character directly and seeing it in the editor,
instead of using escapes all the time.

As indicated by below stackoverflow answer "JSON spec requires UTF-8 support by
decoders": https://stackoverflow.com/a/594881/9456786 , and indeed checking
JSON specification also confirms that by default JSON decoders shall use UTF-8:

    https://datatracker.ietf.org/doc/html/rfc7159#section-8.1

This way, I think, we can switch to UTF-8 safely.

/reviewed-by @jerome, @lu.xu
/reviewed-on nexedi/slapos!1498