1. 09 Dec, 2021 3 commits
    • Kirill Smelkov's avatar
      golang += patches to fix tests under user namespaces · c19097ef
      Kirill Smelkov authored
      If we enter user namespace via regular unshare without help from SUID
      newuidmap/newgidmap, all supplementary groups are mapped to -1. As the result
      when Go test tries to chown to a supplementary group, it gets EINVAL:
      
      https://github.com/golang/go/issues/42525
      
      -> work it around with patch to skip this chown tests.
      
      A more proper, longer-term fix would be to fix Linux kernel to allow writes to
      /proc/self/gid_map to setup mapping not only to original gid, but to all
      original supplementary groups as well here:
      
      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/kernel/user_namespace.c?id=v5.16-rc4-0-g0fcfb00b28c0#n1143
      
      this fix, even if accepted by upstream, would be long to be waited for to
      propagate to distribution kernels that we currently use. So we go with this
      workaround for now.
      
      --------
      
      Another patch is to fix the following TestSCMCredentials failure:
      
          === RUN   TestSCMCredentials
              creds_test.go:81: WriteMsgUnix failed with invalid argument, want EPERM
          --- FAIL: TestSCMCredentials (0.00s)
      
      There the code tries to send uid0/gid0 credentials from non-zero uid and
      expects EPERM reject from kernel. However under `unshare -Umc` uid0/gid0 are
      not mapped to anywhere and so implicitly map to -1 and are rejected with EINVAL
      by the kernel.
      c19097ef
    • Kirill Smelkov's avatar
      component/nxdtest: Prepare for nexedi/nxdtest!13 (II) · 7b92083b
      Kirill Smelkov authored
      4) Hook in python-prctl, as it becomes nxdtest dependency: see
         nexedi/nxdtest!13 (79d13eff)
      7b92083b
    • Kirill Smelkov's avatar
      component/nxdtest: Prepare for nexedi/nxdtest!13 · 64a6e702
      Kirill Smelkov authored
      Prepare for upcoming nxdtest changes to run each testcase with its own
      /tmp and /dev/shm:
      
      1) put unshare from SlapOS component into $PATH, so that our version is
         used even if OS provides /bin/unshare. As @jerome explains we need
         features that were added relatively recently and are missing in unshare
         on Debian 10: nexedi/nxdtest!13 (comment 146752)
      
         It is anyway better to "isolate" from OS by using our own component
         instead of system-provided one.
      
         Correspondingly adjust util-linux to enable unshare in its build.
      
      2) similarly to "1" adjust util-linux to enable mount so that our version
         is used instead of /bin/mount. For example on Debian 9, even if we
         successfully enter user/mount namespace with `unshare -Umc`
      
             /bin/mount -t tmpfs none /tmp
      
         complains that
      
             mount: only root can use "--types" option
      
         -> Fix it the same way as with unshare by forcing usage of
         SlapOS-provided mount.
      
      3) rework how nxdtest script is generated and split it into .nxdtest.pyexe and
         nxdtest itself. .nxdtest.pyexe is python interpreter via which nxdtest is run.
         This interpreter has all eggs required by nxdtest in sys.path, so that
         nxdtest could spawn its trun.py via sys.executable. If we don't care to have
         properly setup sys.executable, trun.py will fail when importing any module that
         nxdtest.py could already successfully import.
      
         Initially I tried to workaround this issue via adjusting $PYTHONPATH <-
         sys.path in main nxdtest script, but @jerome points out that, $PYTHONPATH,
         if set, also affects processes that trun.py spawns, which is not good:
      
         nexedi/slapos!1095 (comment 146799)
      
         -> so fix this via running nxdtest via environment where sys.executable is
         properly setup python interpreter with path for all eggs that nxdtest has
         access to.
      
         Because we already have half-way workarounds for similar problem in several
         places, and because running a script with correctly setup sys.executable is
         generally better, I would say it should be a good idea to rework
         zc.recipe.egg:scripts to generate all scripts to work this way, but I do not
         want to fight about it.
      
         So let's leave this scheme nxdtest-specific for now.
      
      /cc @tomo
      /helped-by @jerome
      64a6e702
  2. 07 Dec, 2021 1 commit
  3. 06 Dec, 2021 4 commits
  4. 03 Dec, 2021 11 commits
  5. 02 Dec, 2021 12 commits
  6. 01 Dec, 2021 5 commits
  7. 30 Nov, 2021 4 commits
    • Julien Muchembled's avatar
    • Julien Muchembled's avatar
    • Julien Muchembled's avatar
      mariadb: build RocksDB without fallocate · dc16dc04
      Julien Muchembled authored
      https://github.com/facebook/rocksdb/commit/7291a3f813e563efbd6870465b1063a115480373
      is not enough.
      
      Anway, given how SST files are written, fallocate
      is unlikely to be useful, even less on SSD.
      
      RocksDB seems to have options to disable at runtime (see env/io_posix.cc)
      but I see nothing for that in MariaDB configuration.
      
      Steps to reproduce:
      
      With a Btrfs partition (no specific mkfs option and for example 2GB size)
      and following MariaDB configuration:
      
        [mysqld]
        datadir = /tmp/datadir
        disable-log-bin
        plugin-load = ha_rocksdb
        rocksdb_default_cf_options = compression=kNoCompression;bottommost_compression=kZSTD
      
      execute the following queries with `mysql -uroot`:
      
      > USE test;
      > CREATE TABLE t(x MEDIUMBLOB) ENGINE=RocksDB;
      > INSERT INTO t VALUES(repeat('.',16777215));
      > INSERT INTO t SELECT x FROM t UNION ALL SELECT x FROM t;
      > INSERT INTO t SELECT x FROM t UNION ALL SELECT x FROM t;
      > INSERT INTO t SELECT x FROM t UNION ALL SELECT x FROM t;
      
      Disk usage can be checked with:
      
      > SHOW ENGINE RocksDB STATUS \G
      
      Level    Files   Size     ...
      ------------------------- ...
        L0      1/0   144.00 MB ...
        L2      1/0   288.00 MB ...
       Sum      2/0   432.00 MB ...
      
      and this can be compacted as follows:
      
      > SET GLOBAL rocksdb_compact_cf='default';
      
      > SHOW ENGINE RocksDB STATUS \G
      
      Level    Files   Size     ...
      ------------------------- ...
        L0      0/0    0.00 KB  ...
        L1      0/0    0.00 KB  ...
        L2      1/0   16.28 KB  ...
       Sum      1/0   16.28 KB  ...
      
      Now using `compsize` (on Debian, it's provided by btrfs-compsize package):
      
      # with -DWITH_FALLOCATE=NO
      
      $ compsize /tmp/datadir/#rocksdb/000029.sst
      Type       Perc     Disk Usage   Uncompressed Referenced
      TOTAL      100%       20K          20K          20K
      none       100%       20K          20K          20K
      
      # without -DWITH_FALLOCATE=NO
      
      $ compsize /tmp/datadir/#rocksdb/000029.sst
      Type       Perc     Disk Usage   Uncompressed Referenced
      TOTAL      100%       35M          35M          20K
      none       100%       35M          35M          20K
      dc16dc04
    • Julien Muchembled's avatar