1. 12 Oct, 2023 7 commits
    • Kirill Smelkov's avatar
      software/ors-amarisoft: Switch *.json.log to emit `data` field via JSON · f32db64f
      Kirill Smelkov authored
      ORS software-release emits many *.json.log files with information about current
      state of eNB and radio units. Those files are used by code in slapos.toolbox to
      implement corresponding promises. The files, it seems, try to match JSON Lines
      format(*) with every line containing a JSON object describing timestamp, log
      level and log payload of corresponding log entry.
      
      So far all is good. One peculiarity, however, is that, contrary to everything
      else, the content of the `data` field in the logs is not emitted via
      JSON-encoding. Initially, starting from 4968d55c (ors-amarisoft: get eNB stats
      and add promises), that `data` payload was being emitted via %s py formatting,
      which is similar to what JSON-encoding would produce, but is not it exactly.
      For example if log payload is {'abc': 123} then %s and JSON would quote `abc`
      differently:
      
          {'abc': 123}  # %s
          {"abc": 123}  # JSON
      
      That would not be a big deal if what %s produces would be still valid JSON
      accepted by JSON-decoder, but unfortunately it is not so: JSONDecodeError is
      raised when {'abc': 123} is tried to be decoded.
      
      The code in slapos.toolbox, so far, is handling this situation by feeding to
      JSON-decoder original input with ' characters replaced by "
      
          json.loads(line.decode().replace("'", '"'))     (+)
      
      which more or less works in basic cases, but breaks when log payload contains '
      character itself. For example for {"a'bc": 123} replacing gives {"a"bc": 123},
      which, being invalid JSON, raises
      
          JSONDecodeError: Expecting ':' delimiter: line 1 column 5 (char 4)
      
      when fed to json.loads() .
      
      I've actually hit this problem for real with amarisoft-rf-info.json.log where
      the data part contains output of `rf_info` looking e.g. as
      
          TX underflows=0 RX overflows=0
          TRX SDR driver 2023-09-07, API v15/18
          PCIe CPRI /dev/sdr0@0:
            Hardware ID: 0x4b12
            DNA: [0x0048248a334a7054]
            Serial: ''
            FPGA revision: 2023-06-23  10:05:24
            FPGA vccint: 0.98 V
            FPGA vccaux: 1.76 V
            FPGA vccbram: 0.98 V
            FPGA temperature: 71.9 °C
            Clock tune: 0.0 ppm
            NUMA: 0
            CPRI_option: '5' (x8) lock=no
            DMA0: TX fifo: 66.67us  Usage=16/32768 (0%)
            DMA0: RX fifo: 66.67us  Usage=16/32768 (0%)
            DMA0 Underflows: 0
            DMA0 Overflows: 0
          PCIe CPRI /dev/sdr0@1:
            Hardware ID: 0x4b12
            DNA: [0x0048248a334a7054]
            Serial: ''
            FPGA revision: 2023-06-23  10:05:24
            FPGA vccint: 0.98 V
            FPGA vccaux: 1.77 V
            FPGA vccbram: 0.98 V
            FPGA temperature: 71.7 °C
            Clock tune: 0.0 ppm
            NUMA: 0
            CPRI_option: '5' (x8) lock=HW+SW rx/tx=46.606us
              Port #0: T14=46.606us
            DMA0: TX fifo: 66.67us  Usage=16/32768 (0%)
            DMA0: RX fifo: 66.67us  Usage=16/32768 (0%)
            DMA0 Underflows: 0
            DMA0 Overflows: 0
      
      which lead to breakage when trying to load that log due to ' symbols.
      
      The fix is simple: change the code, that emits *.json.log to emit data payload
      via json.dumps instead of %s. We can do that without breaking anything because,
      contrary to enb.xlog, those *.json.log are not uploaded to any separate system
      and currently are only used to implement promises without preserving log files
      in any storage for any non-small amount of time. In other words, currently
      those *.json.log are temporary files whose format can be adjusted without
      strongly caring about backward compatibility.
      
      b32b4a8e (software/ors-amarisoft: general improvement for RU (logs/promises/input parameters))
      already changed %s to json.dumps for amarisoft-stats.json.log , but left all other logs at %s.
      
      -> Fix this for everything by replacing %s to json.dumps in all generated logs.
      -> Corresponding slapos.toolbox adjustments are in nexedi/slapos.toolbox!120.
      
      /reviewed-on nexedi/slapos!1447
      /reviewed-by @jhuge, @jerome, @tomo
      /cc @lu.xu, @xavier_thompson, @Daetalus
      
      (*) https://jsonlines.org/
      (+) see e.g. https://lab.nexedi.com/nexedi/slapos.toolbox/blob/453dce5f/slapos/promise/plugin/util.py#L50
      f32db64f
    • Titouan Soulard's avatar
    • Titouan Soulard's avatar
      update dead links · 188f6518
      Titouan Soulard authored
      188f6518
    • Titouan Soulard's avatar
      remove unused components · a11779b6
      Titouan Soulard authored
      a11779b6
    • Jérome Perrin's avatar
      776411e0
    • Jérome Perrin's avatar
      software/osie-coupler/test: adjust test · 9718ecf0
      Jérome Perrin authored
      after 6e8b97ee (Publish IPv6 URL to SlapOs master., 2023-10-02) the
      parameters are different
      9718ecf0
    • Jérome Perrin's avatar
      software/seleniumserver: update snapshots to fix tests · c60ef89b
      Jérome Perrin authored
      after recent updates, probably 3c514224 (component/freetype: version
      up 2.13.2, including security fixes., 2023-10-10), the rendering is
      different.
      c60ef89b
  2. 11 Oct, 2023 2 commits
  3. 10 Oct, 2023 6 commits
  4. 09 Oct, 2023 3 commits
  5. 06 Oct, 2023 4 commits
  6. 05 Oct, 2023 4 commits
  7. 04 Oct, 2023 4 commits
    • Jérome Perrin's avatar
      component/git: version up 2.42.0 · e5892c7a
      Jérome Perrin authored
      e5892c7a
    • Jérome Perrin's avatar
    • Łukasz Nowak's avatar
      rapid-cdn: Handle correctly wildcard domains · a039c8cf
      Łukasz Nowak authored
      While generating haproxy configuration (including it's CRT list) the specific
      order of entries is used, so that wildcard domains end up last. Thanks to this
      they work as a catch-all and allow specific domain to take precedence. Care
      is taken to support *.example.example.com and *.example.com situation - so
      tree like possibility of wildcards.
      
      Anonymous in-place ACL are used per each domain, instead of per-shared
      instance grouping in order to avoid situation like *.example.com and
      example.com having single ACL, thus resulting with catch-all kicking in too
      fast.
      
      For the precision in the haproxy configuration and simplifcation of the regular
      expressions the -m reg is used, so that host_only can be applied, which also
      lowercases the hostname.
      
      Notes:
       * test00cluster_request_instance_parameter_dict changed due to sorting slaves
         in test's requestSlaves
       * the test infrastructure has been improved to assure repetition of the
         situation
       * tests in TestSlaveHostHaproxyClash are asserting that correct domain AND
         that specific certificate have been used while serving given frontend
         configuration
      a039c8cf
    • Joanne Hugé's avatar
  8. 02 Oct, 2023 5 commits
    • Jérome Perrin's avatar
      component/ca-certificates: version up 20230311 · 5f05b413
      Jérome Perrin authored
      A similar patches as ca-certificates-sbin-dir was
      applied upstream as 4f0d3ec7aa4ebc91793245ed66c0e24d7150782b , the rest
      of our patch was to use mkdir -p instead of mkdir, we keep this part in
      ca-certificates-mkdir-p.patch
      
      This introduces a new patch to not depend on cryptography, which is used
      only to print a warning on the console when an expired certificate is
      used.
      5f05b413
    • Jérome Perrin's avatar
      software/slapos-sr-testing: disallow usage of system python3 · 24f5a96c
      Jérome Perrin authored
      We had a mechanism to catch usage of system python2, but using not
      for system python3, which cause the same kind of problems.
      24f5a96c
    • Jérome Perrin's avatar
      component/glib: put slapos python3 in PATH · 698f69d5
      Jérome Perrin authored
      On old debian (9) where python3 is python3.5 this component fails to
      build with an error like:
      
          Configuring gdbus-example-objectmanager-visibility.h with command
          Running command: /opt/slapgrid/shared/glib/60e920f1feec2451d51bb344cfcad9ab/.build/glib-2.76.3/tools/gen-visibility-macros.py 2.0 visibility-macros GDBUS_OBJECT_MANAGER_EXAMPLE /opt/slapgrid/shared/glib/60e920f1feec2451d51bb344cfcad9ab/.build/glib-2.76.3/builddir/gio/tests/gdbus-object-manager-example/gdbus-example-objectmanager-visibility.h
          --- stdout ---
          --- stderr ---
            File "/opt/slapgrid/shared/glib/60e920f1feec2451d51bb344cfcad9ab/.build/glib-2.76.3/tools/gen-visibility-macros.py", line 37
              """
              ^
          SyntaxError: invalid syntax
      
      This is because this gen-visibility-macros.py script is executable with
      a shebang:
      
          #!/bin/env python3
      
      for python 3 softwares, this is slapos python, because the python
      section from component/defaults.cfg injects the slapos' python in PATH,
      but for python 2 software, slapos' python 2 is injected in path and
      `python3` resolves to system python, which in that case fails because
      f-strings are SyntaxError but more generally this showed a dependency
      to system python, but we can not rely on system python here.
      698f69d5
    • Ivan Tyagov's avatar
      Publish IPv6 URL to SlapOs master. · 6e8b97ee
      Ivan Tyagov authored
      See merge request !1438
      6e8b97ee
    • Ivan Tyagov's avatar
      Publish IPv6 URL to SlapOs master. · 5eac88ee
      Ivan Tyagov authored
      5eac88ee
  9. 29 Sep, 2023 3 commits
  10. 28 Sep, 2023 2 commits
    • Thomas Gambier's avatar
    • Jérome Perrin's avatar
      stack/erp5: serve balancer requests when client certificate is not verified · d58bbbba
      Jérome Perrin authored
      We configure haproxy with "verify optional", which makes haproxy request
      a client certificate, but accept the case where client does not present
      a certificate, but as described in [1], if client present a certificate
      and this certificate can not be verified, handshake is aborted. This is
      not what we want, we want to treat the case of a non verified
      certificate same as the case of the absence of certificate.
      
      This configures haproxy accordingly, using "crt-ignore-err all" to allow
      handshake anyway.
      
      Once this was fixed, there was a remaining problem with
      client_cert_verified acl, haproxy acl are OR, but this rule was supposed
      to be a AND (client present a certificate AND it is verified), this was
      rewritten to use inline condition which are AND.
      
      [1]: https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.1-verify
      
      Also adjust test_x_forwarded_for_stripped_when_no_certificate to assert
      that there is no X-Forwarded-For header at all when no client
      certificate.
      d58bbbba