1) Fix amari.kpi to handle stats messages in enb.xlog that come with multiple cells.
   Previously such messages were leading to the following errors on KPI calculator place (e.g. Wendelin):

        xlte.amari.kpi.LogError: t1731059787.321: stats describes 2 cells;  but only single-cell configurations are supported

2) Fix amari.drb to generate x.drb_stats messages when an UE is associated to multiple cells due to e.g. Carrier Aggregation.
   Previously CA was leading to

        raise RuntimeError(("ue #%s belongs to %d cells;  "+
            "but only single-cell configurations are supported") % (ue_id, len(ju(['cells']))))

   error on eNB side.

+ minor fixes and enhancements done along the way.

Please see individual patches for details.

An example enb.xlog for eNB with 2 cells and an UE Carrier-Aggregated to both cells is here:

https://lab.nexedi.com/kirr/misc/-/blob/6a04cf3/lte/20241111-2cell.xlog

And here is how it looks when visualized via kpidemo.py :

https://lab.nexedi.com/kirr/misc/-/blob/6a04cf3/lte/20241111-2cell.png

Kirill

/cc @lu.xu
/reviewed-by @paul.graydon
/reviewed-on !7

./kpi.py:658:9: local variable 'τ' is assigned to but never used

My mistake from 2824f50d (kpi: Calc: Add support for E-UTRAN IP Throughput KPI)

Add explanatory comments and fix typos noticed while working on
drb/multicell topic.

It helps debugging and testing to be able to see UE transmission states
in human-readable form.

Since 2a016d48 (Draft support for E-UTRAN IP Throughput KPI) there was a
hardcoded limitation that x.drb_stats generation works with 1-cell
configurations only. However we do use multicell eNB configurations and
on such configurations `xamari xlog x.drb_stats` was failing on eNB
side with

    raise RuntimeError(("ue #%s belongs to %d cells;  "+
        "but only single-cell configurations are supported") % (ue_id, len(ju(['cells']))))

because an UE might be associated with multiple cells of one eNB
due to e.g. Carrier Aggregation.

Now, after we did preparatory amari.drb refactoring and taught BitSync
to be multicell-aware, we can finally remove the limitation and correct
the Sampler to handle UEs associated with multiple cells at the same
time.

The handling is mostly straightforward: for every frame the Sampler needs
to estimate time of active transmission and amount of transmitted bytes.
The transmitted amount is Σcell(tx_bytes) and we estimate transmission
time on different cells C₁ and C₂ as

    tx_time ∈ [max(t₁,t₂), min(t₁+t₂, δt/tti)]

In other words when transmission to/from UE on C₁ and C₂ is going in
parallel in one frame, the time it takes to transmit current part of UEs
data

    1) cannot be less than time spent on each cell
       (equals to the maximum if transmission on Ci was fully performed
        in the same subframes where transmission on Cj was active)

    2) cannot be more than the sum of time spent transmitting on each cell
       (equals to the sum if transmission on C₁ and C₂ was performed in
        distinct subframes of the frame), and

    3) cannot be more than the whole frame
       (t₁+t₂ could become more than that if there is some overlap in
        subframes where C₁ and C₂ transmissions were done)

The patch implements this logic when updating data flows of an UE.

Amari.drb now works for multicell configurations.

There are one-cell unit tests for Sampler already added in 2a016d48 and
now I manually verified that `xamari xlog x.drb_stats` works ok with
multicell eNB configurations.

Unittests for Sampler in multicell environment are left as TODO.

To be able to handle multicell configurations rework _BitSync to handle
UEs that might be associated with multiple cells at the same time, e.g.
in the case of Carrier Aggregation.

For this BitSync interface is reworked correspondingly, then it uses
tx_bytes total->percell splitter, we added in the previous patch, then
it uses BitSync1 helpers, which each work on the stream of one
particular cell, and then the result is combined back into multicell _Utx.

The internals of new BitSync1 are very similar to original old BitSync implementation.
The new BitSync, that wraps many BitSync1s, is new.

The Sampler still accepts UEs with one cell only.
In the next patch we will update the Sampler to handle multicell
configurations as well.

To be able to compute E-UTRAN IP Throughput amari.drb uses Sampler which
detects and extracts separate transmission samples from 100Hz log of
ue_get[stats]. The Sampler, in turn uses help from _BitSync for correct
operations because Amarisoft LTEENB updates counters for dl_total_bytes
and dl_tx at different times, and _BitSync synchronizes streams of those
updates in time. _BitSync itself works by correlating amount of
transmitted data (tx_bytes) and transport blocks (#tx) and shifting some
amount of #tx to previous frame based on the correlation. See d102ffaa
(amari.drb: Start of the package) for details.

This works ok for configurations with 1 cell, but does not work out of the box
for multicell configurations because #tx is per-cell value, while e.g.
dl_total_bytes is reported by LTEENB only as a value aggregated over all cells.
That's why original implementation in d102ffaa had an assert that the
number of cells an UE is associated with is 1.

-> Implement custom filter that splits overall tx_bytes into per-cell
   parts via heuristic based on reported per-cell bitrates to workaround
   that: the more cell bitrate is the more is the part of tx_bytes that
   gets associated to this cell.

In basic implementation the heuristic would be to divide tx_bytes as

	tx_bytes(cell) = tx_bytes·β/Σcells(β)		; β is bitrate of a cell

but given that for every frame _BitSync works by computing things based
on neighbour frame as well we do it as

	tx_bytes(cell) = tx_bytes·(β₁+β₂)/Σcells(β₁+β₂)

This should make the heuristic a bit more stable.

This patch comes with tx_bytes splitter filter itself only. In the next
patch we will use _CTXBytesSplitter to implement multicell-awareness for
_BitSync.

We will need to know current cell DL/UL bitrate for multicell BitSync in
the next patches.

Continue preparatory steps to support multicell configurations and for
that split the class that tracks UE transmission state into global part
(_Utx) and per-cell parts (_UCtx).

Everywhere else in the code we still assert that the number of cells an UE
attached to is 1, so no support for multicell yet - only preparatory
non-functional changes currently.

Similarly to previous patch this is preparatory non-functional change to
support multicell configurations.

Previously the number of transmitted transport blocks was passed around
as separate argument but with multiple cells each cell will have its own
information about how many TB were transmitted/received and we will need
to maintain those .tx and .retx in per-cell data structure.

Start preparing to that by moving .tx and .retx to be tracked in UE
transmission state instead of being passed around separately.

No support for multicell yet - only preparatory non-functional changes currently.

Since 2a016d48 (Draft support for E-UTRAN IP Throughput KPI) there was a
hardcoded limitation that x.drb_stats generation works with 1-cell
configurations only. However we do use multicell eNB configurations and
on such configurations `xamari xlog x.drb_stats` was failing on eNB
side with

    raise RuntimeError(("ue #%s belongs to %d cells;  "+
        "but only single-cell configurations are supported") % (ue_id, len(ju(['cells']))))

Start fixing that.

As the first preparatory step to support multiple cells with x.drb_stats
reorganize amari/drb.py code a bit: _QCI_Flow works at whole QCI
transmission level which can aggregate and cover multiple cells, but
xl_use_avg and rank are per cell things and they can be different for
different cells.

-> Start preparing to handle them in per-cell way by adjusting the code
   to take those values into account at higher level in computation stack
   where we still have cell context.

No support for multicell yet - only preparatory non-functional changes
currently.

When computing Measurements amari.kpi derives period from stats messages
- for example 60s if `xamari xlog` was running with stats/60s. But
inside that period there could be multiple x.drb_stats messages - for
example if xlog was running with `stats/60s x.drb_stats/10s`.
However the code was handling only the last such x.drb_stats instead of
accumulating counters from there during the period.

As the result, if. e.g. during the period there were 90B/9s and 2B/0.2s
x.drb_stats events, only the latter was accounted giving wrong data for
transmitted amount and transmission time.

-> Fix it.

Starting from 71087f67 (amari.kpi: New package with driver for Amarisoft
LTE stack to retrieve KPI-related measurements from logs) a limitation
was hardcoded that only 1-cell configurations are supported. However we
do use multicell eNB configurations and computing E-RAB Accessibility
KPI was failing on them as e.g.:

    xlte.amari.kpi.LogError: t1731059787.321: stats describes 2 cells;  but only single-cell configurations are supported

-> Remove that limitation and handle stats covering multiple cells.

Add new --password option and wire it to go to amari.connect(password=...)

Support for password-based authentication in amari.connect was just
added in the previous patch.

We need to extend filtering of logged fields on service attach a bit
since now the first service message can be both 'ready' and
'authenticate', and besides e.g. 'message' we don't want to log what was
the 'challenge'.

/reported-and-tested-by @lu.xu
/reviewed-on !6

Sometimes Amarisoft services are setup to require custom
challenge/response authentication upon connecting to their WebSocket.
In 61ad9032 (amari: Add functionality to interoperate with an Amarisoft
LTE service via WebSocket) I've put TODO for that but now we start to
need it.

-> Implement corresponding support for password-based authentication.

In the next patch we will teach XLog to use it.

See https://tech-academy.amarisoft.com/lteenb.doc#Startup for details of
the authentication handshake protocol.

/reported-and-tested-by @lu.xu
/reviewed-on !6

Contrary to earfcn, where band can be automatically deduced from earfcn
number because 4G bands never overlap, most functions in nrarfcn accept
as input parameters both nr_arfcn and band, because 5G bands can and do
overlap. As the result it is possible to invoke e.g. dl2ul with
dl_nr_arfcn being outside of downlink spectrum of specified band.

However in b8065120 I've made a thinko and handled such situation with
simple assert which does not lead to useful error feedback from a user
perspective, for example:

    In [2]: xnrarfcn.dl2ul(10000, 1)
    ---------------------------------------------------------------------------
    AssertionError                            Traceback (most recent call last)
    Cell In[2], line 1
    ----> 1 n.dl2ul(10000, 1)

    File ~/src/wendelin/xlte/nrarfcn.py:85, in dl2ul(dl_nr_arfcn, band)
         83 if dl_lo == 'N/A':
         84     raise AssertionError('band%r does not have downlink spectrum' % band)
    ---> 85 assert dl_lo <= dl_nr_arfcn <= dl_hi
         86 ul_lo, ul_hi = nr.get_nrarfcn_range(band, 'ul')
         87 if ul_lo == 'N/A':

    AssertionError:

The issue here is that asserts can be used to only verify internal
invariants, and that reported error does not provide details about which
nrarfcn and band were used in the query.

-> Fix this by providing details in the error reported to incorrect
module usage, and by consistently raising ValueError for "invalid
parameters" cases.

The reported error for above example now becomes

    ValueError: band1: NR-ARFCN=10000 is outside of downlink spectrum

Do a package for converting DL/UL NR-ARFCN in between each other and to
convert DL NR-ARFCN to SSB NR-ARFCN. The API mimics xlte.earfcn added in 6cb9d37f.

xlte.nrarfcn complements pypi.org/project/nrarfcn, which we use here under the hood.

See package documentation for API details.

Do a package which provides calculations like EARFCN -> frequency,
EARFCN -> band info, and to convert DL/UL EARFCN in between each other.

I was hoping to find something ready on the net, but could find only
pypi.org/project/nrarfcn for 5G, while for LTE everything I found
was of lesser quality and capability.

-> So do it myself.

See package documentation for API details.

Before this patch we were supporting only boolean option flags - with, for
example, stats[rf] meaning stats query with {"rf": True} arguments. Now we add
support for arbitrary types, so that it is possible to specify e.g. integer or
string query options, as well as some boolean flag set to false.

This should be good for generality.

For backward compatibility the old way to implicitly specify "on" flags is
continued to be supported.

If the parsed period was '60s' we were printing it back as '60.0s' on str.
Fix it by using %g insted of %s.

Adjust plain writer to append to log file instead of truncating it on
the open. The rotating writers are already ok as they use "a" mode by default.

- add Calc.cum to aggregate Measurements.

- add ΣMeasurement type to represent result of this. It is very similar
  to Measurement, but every field comes accompanied with information
  about how much time there was no data for that field. In other words
  it is not all or nothing for NA in the result. For example a field
  might be present 90% of the time and NA only 10% of the time. We want to
  preserver knowledge about that 90% of valid values in the result. And we
  also want to know how much time there was no data.

- amend kpidemo.py and kpidemo.ipynb to demonstrate this.

XLTE should be ready to be tried to be used for real now.

That's python versions we care about. Add explicit tests to cover that.

Xavier reports that str|None is supported only by python ≥ 3.10, while
we still should care to support at least 3.9 - e.g. SlapOS uses it by
default as well as Debian 11.

Let's not delve deep into typing game. If we cannot express things
easily we can omit the type completely or express it in comments.

/reported-by @xavier_thompson at kirr/xlte!5 (comment 182930)

Rotate output enb.xlog ourselves at sync points so that nothing is lost
in the output (hello `logrotate copytruncate`) and so that we can emit
pre- and post- logrotate syncs.

Reuse logging's RotatingFileHandler and TimedRotatingFileHandler to
implement actual rotation, but carefully wrap them in our writer
classes so that we emit exactly the output we prepared explicitly
without any headers prepended by logging, and that we explicitly control
when rotation happens.

/proposed-for-review-at !5

Nxdtest[1] is tox-like tool to run tests under Nexedi testing
infrastructure.

[1] https://lab.nexedi.com/nexedi/nxdtest

py3 worked without it ok, but py2 was failing to import xlte.demo
without demo/__init__.py present.

For py3 it does not matter, but on py2 %s prettifies result a bit:

    kirr@deca:~$ python
    Python 2.7.18 (default, Jul 14 2021, 08:11:37)
    >>> str(0.1-0.01)
    '0.09'
    >>> repr(0.1-0.01)
    '0.09000000000000001'

    kirr@deca:~$ python3
    Python 3.9.2 (default, Feb 28 2021, 17:03:44)
    >>> str(0.1-0.01)
    '0.09000000000000001'
    >>> repr(0.1-0.01)
    '0.09000000000000001'

Should make the diff in between master and py2 a bit smaller.

This will make diff in between master and py2 backport a bit smaller.

Rework XLog protocol to come with periodic sync events that come from time to
time so that xlog stream becomes self-synchronizing. Sync events should be
useful for Wendelin to start reading xlog stream from any point, and to verify
that the stream is ok by matching its content vs messages schedule coming in
the syncs.

Teach xlog.Reader to read streams in reverse order from end to start. This
should be useful to look at tail of a log without reading it in full from the
start.

Teach xlog.Reader to reconstruct messages timestamps for xlog streams produced
with Amarisoft releases < 2022-12-01. There messages do not have .utc field
added in https://support.amarisoft.com/issues/21934 and come with only .time
field that represent internal eNB time using clock originating at eNB startup.
We combine message.time and δ(utc, enb.time) from sync to build message.timestamp .

See individual patches for details and
!3 for preliminary discussion.

/reviewed-by @xavier_thompson
/reviewed-on !4

We currently emit information about local time in events, and
information about on-service time in messages. Events don't have
information about on-service time and messages don't carry information
about local time. That is mostly ok, since primary xlog setup is to run
on the same machine, where eNB runs because on-service .utc correlates
with .time in events.

However for eNB < 2022-12-01 on-service time includes only .time field
without .utc field with .time representing "time passed since when eNB
was started". This way for enb.xlog streams generated on older systems
it is not possible for xlog.Reader to know the absolute timestamps of
read messages.

To fix this we amend "attach" and "sync" events to carry both local and
on-service times. This way xlog.Reader, after seeing e.g. "sync" with
.time and only .srv_time without .srv_utc, should be able to
correlate local and on-service clocks and to approximate srv_utc as

	srv_utc' = srv_time' + (time - srv_time)

where time and srv_time correspond to last synchronization, and
srv_time' is what xlog.Reader retrieves for a further-read message in
question.

See !3 for related discussion.

This functionality is useful to look at tail of a log without reading it
in full from the start.

.pos verification should be there from xlog.Reader start in 0633d26f
(amari.xlog += Reader)

This way xlog.Reader can be sure that if it looked around in such a
window and did not find a sync, then something is not good with the
stream and it does not need to go beyond that limit looking around.

This is a change of the protocol. But it is early days and existing logs
- that we use in the demo, are all below 1000 lines limit, so they will
continue to be loaded ok.

No direct test for actual Loss Of Sync detection - this functionality is
draft for now and should be improved later. However for no-LOS cases
xlog.Reader is already covered with tests.

Let's use "sync(reason=start)" instead of dedicated "start" event for
uniformity. Periodic syncs are now "sync(reason=periodic)" and after
logrotation support there will be also "pre-logrotate" and
"post-logrotate" reasons. Emit "sync(reason=stop)" at xlog shutdown for
uniformity and to make it more clear from looking at just enb.xlog about
what is xlog state at the end.

Stop requiring "start" to be present in the header - we will soon rework
xlog reader to look around for nearby sync automatically so that reading
could be started from any position in the stream.

We emit config_get after every attach from the beginning of xlog in
e0cc8a38 (amari.xlog: Initial draft). The reasoning here is that it is
useful by default to know configuration of a service.

In the previous patch we added sync events so that xlog stream becomes
self-synchronizing. To continue that line it is now useful to have that
config_get emitted after every such synchronization point instead of
only after attaching to the service. That's what hereby patch does.

As a bonus the code is reworked in a way that config_get setup is not
hardcoded anymore and config_get periodicity now can be controlled by
users via explicitly specifying config_get in the logspec.

So that xlog stream becomes self-synchronized and could be used even if
we start reading it from some intermediate point instead of only from
the beginning.

We will need this in general - to be able to start reading long log not
only from its beginning, and also in particular for Wendelin systems
where logs are uploaded by Fluentd in chunks and some chunks could be
potentially lost.

Sync events are emitted always unconditionally with default sync
interval being 10x the longest specified period. We also provide users a
way to control sync periodicity via explicitly specifying
"meta.sync/period" query in the logspec.

See !3 (comment 175796) and
further for related discussion.

This is change of xlog protocol. But it is early days and the only
direct consumer of xlog is amari.kpi which we adjust accordingly. So it
should be ok.

We will soon need to construct logspecs not only by way of parsing.

In 79d10eb9 that patch wired ctx through xlog callchains and added
corresponding handling of cancellation. But I overlooked one place where
plain sleep was used.

-> Fix it.