BOM(CloudOOO) was failing as

    RuntimeError: Unsupported url: 'https://lab.nexedi.com/bk/onlyoffice_core/repository/archive.tar.bz2?ref=8a40eb47bd80a40ecde14c223525b21852d2fc9f'

See previous patch for how it was crashing.

Needed to improve version detection heuristic because with only trimming
-src suffix the version was detected as "2" and name as "icu4c-58".

This emits useful information instead of further crash if an url could
not be parsef successfully as e.g. in below.

Before the patch:

    Traceback (most recent call last):
      File "/srv/slapgrid/slappart5/srv/project/nxd-bom/nxdbom/__init__.py", line 434, in <module>
        main()
      File "/srv/slapgrid/slappart5/srv/project/nxd-bom/nxdbom/__init__.py", line 423, in main
        bom = bom_software(arg)
      File "/srv/slapgrid/slappart5/srv/project/nxd-bom/nxdbom/__init__.py", line 108, in bom_software
        addbom(geturl(part), '')    # XXX detect kind?
      File "/srv/slapgrid/slappart5/srv/project/nxd-bom/nxdbom/__init__.py", line 56, in addbom
        ver = removeprefix(ver, name+'-')   # wendelin.core-2.0.alpha3-0-g6315384  -> 2.0.alpha3-0-g6315384
      File "/srv/slapgrid/slappart5/srv/project/nxd-bom/nxdbom/__init__.py", line 384, in removeprefix
        if s.startswith(prefix):
    AttributeError: 'NoneType' object has no attribute 'startswith'

After the patch:

    Traceback (most recent call last):
      File "/srv/slapgrid/slappart5/srv/project/nxd-bom/nxdbom/__init__.py", line 435, in <module>
        main()
      File "/srv/slapgrid/slappart5/srv/project/nxd-bom/nxdbom/__init__.py", line 424, in main
        bom = bom_software(arg)
      File "/srv/slapgrid/slappart5/srv/project/nxd-bom/nxdbom/__init__.py", line 109, in bom_software
        addbom(geturl(part), '')    # XXX detect kind?
      File "/srv/slapgrid/slappart5/srv/project/nxd-bom/nxdbom/__init__.py", line 56, in addbom
        assert ver is not None, urlpath
    AssertionError: https://github.com/unicode-org/icu/releases/download/release-58-2/icu4c-58_2-src.tgz

Fix it similarly to how readline is handled in ERP5 SR.

Without the fix cups version was not detected.

Occurs in Cloudooo.

Withthout this patch LibreOffice version was not detected and instead
was put into name.

Noticed while doing BOM(Cloudooo).

This covers Nexedi versions of e.g. zc.buildout and zc.recipe.egg.
Without the fix those eggs were reported as e.g.

    zc.buildout                  2.7.1+slapos019 https://pypi.org/project/zc.buildout/2.7.1+slapos019/

which was not correct.

Occurs in SlapOS-node.

namever restructuring caugh a thinko/bug in a2107a92 (Handle
%(__buildout_space_n__)s & co in .installed.cfg) - there test eggs were
wrongly emitted into BOM with .egg suffix in version.

Occured during Caucase.

- handle hash in url of slapos.recipe.build:download
  A hash there is instruction to verify checksum of the tarball, not
  part of the real URL.

- fix namever to detect that '-Country' is not version suffix.

Before this patch BOM entry for GeoLite2-Country was emittted as:

    GeoLite2                     Country.tar.gz#dc6224c648350d90f344a0c5c3ca5474 http://geolite.maxmind.com/download/geoip/database/GeoLite2-Country.tar.gz#dc6224c648350d90f344a0c5c3ca5474

Noticed while computing BOM(Re6stnet).

We need special care because configparser ignores nonexisting arguments
by default.

For example an egg might be refered to as 'cython' and placed on the
filesystem as 'Cython' (titled). And for cython-zstd it is placed on the
filesytem as cython_zstd ('-' -> '_'). Try to account for that.

Without the fix nxdbom breaks as

	ValueError: egg cython-zstd not found

ERP5 SR uses both of them, and without dedicated handling nxdbom breaks as

    AssertionError: (PkgInfo(name='readline', version='8.1', kind='', url='http://ftp.gnu.org/gnu/readline/readline-8.1.tar.gz'), PkgInfo(name='readline', version='5.2', kind='', url='http://ftp.gnu.org/gnu/readline/readline-5.2.tar.gz'))

readline is the only item that ERP5 SR uses with multiple versions.

Simply ignore it from the BOM point of view.

For example SOAPpy-0.12.0nxd001 is available only there, and

	https://pypi.org/project/SOAPpy/0.12.0nxd001/

is 404.

Even if slapos.buildout indicates that an egg was locally patched via
+slapospatchedXXX suffix, the URL of the egg should not have it.

For example here is how it occurs in .installed.cfg for ERP5 SR:

    [jupyter]
    eggs = jupyter
            jupyter_client
            jupyter_console
            ...
            jupyterlab
            jupyterlab-launcher%(__buildout_space_n__)s

and without the fix it breaks as

    configparser.InterpolationMissingOptionError: Bad value substitution: option 'eggs' in section 'zzz' contains an interpolation key '__buildout_space_n__' which is not a valid option name. Raw value: '\naaa\nbbb%(__buildout_space_n__)s'

This happens for real with msgpack egg / C-library.

-> Fix handling of such situation via using (name, kind) instead of only
name as BOM key. Without the fix added test breaks as

    def addbom(urlpath, kind, version=None):
        name, ver = namever(urlpath)
        if version is not None:
            assert ver is None
            ver = version
        ver = removeprefix(ver, name+'-')   # wendelin.core-2.0.alpha3-0-g6315384  -> 2.0.alpha3-0-g6315384
        if '//' in urlpath:
            url = urlpath
        else:
            if kind == 'egg':
                # XXX not strictly correct -> better retrieve the actual URL, but buildout does not save it in installed.cfg
                url = 'https://pypi.org/project/%s/%s/' % (name, ver)
            else:
                raise NotImplementedError('TODO url for kind %r  (urlpath: %r)' % (kind, urlpath))

        info = PkgInfo(name, ver, kind, url)
        if name in bom:
>           assert bom[name] == info,  (bom[name], info)
E           AssertionError: (PkgInfo(name='msgpack', version='0.5.4', kind='', url='http://downloads.sourceforge.net/project/msgpack/msgpack/cpp/msgpack-0.5.4.tar.gz'), PkgInfo(name='msgpack', version='0.6.2', kind='egg', url='https://pypi.org/project/msgpack/0.6.2/'))

-> in such a case the recipe uses section name as the default name for installed egg.

Notes:

- egg can come with options, e.g. 'neoppod[admin, ctl, master]'
- eggs are separated by \n, not space
- we ignore *.egg-link because those softwares are reported due to their
  original downloading sections

Without this fix SR files fetched via slapos.recipe.build:download and
slapos.recipe.build:download-unpacked produce leat to error e.g. like

urlpath = '/SR-BASE/stack/erp5/instance-mariadb-resiliency-after-import-script.sh.in', kind = '', version = None

    def addbom(urlpath, kind, version=None):
        name, ver = namever(urlpath)
        if version is not None:
            assert ver is None
            ver = version
        ver = removeprefix(ver, name+'-')   # wendelin.core-2.0.alpha3-0-g6315384  -> 2.0.alpha3-0-g6315384
        if '//' in urlpath:
            url = urlpath
        else:
            if kind == 'egg':
                # XXX not strictly correct -> better retrieve the actual URL, but buildout does not save it in installed.cfg
                url = 'https://pypi.org/project/%s/%s/' % (name, ver)
            else:
>               raise NotImplementedError('TODO url for kind %r  (urlpath: %r)' % (kind, urlpath))
E               NotImplementedError: TODO url for kind ''  (urlpath: '/SR-BASE/stack/erp5/instance-mariadb-resiliency-after-import-script.sh.in')

nxdbom/__init__.py:64: NotImplementedError

e.g.

	https://raw.githubusercontent.com/zuphilip/ocropy-models/master/en-default.pyrnn.gz

is fetched via slapos.recipe.build:download in ERP5 SR.

Hit issues with those urls on ERP5 SR because they were not recognized previously.

Hit complaints about those files being non-configuration on ERP5 SR.

Without the fix configparser complains about bad interpolation:

    configparser.InterpolationSyntaxError: '%' must be followed by '%' or '(', found: '%20Latest%20Stable/2.0.10/zabbix-2.0.10.tar.gz'

Ignore them instead of complaining.

We will be adding more changes to improve nxdbom. Going without
testsuite is not practical because there will be no easy way to catch
regressions.

We will need to add tests and, maybe in the future, helper packages.

nxd-bom file refers to it as

	See COPYING file for full licensing terms.
	See https://www.nexedi.com/licensing for rationale and options.

Jérome writes at https://www.erp5.com/group_section/forum/nxd-bom-XopAhODA1A/view?list_start=5&reset=1#2059332074 :

    I have a few suggestions for this tool:

    - we could also list if a component had patches applied ( for example in
      https://lab.nexedi.com/nexedi/nxd-bom/blob/9273cfb92e1163e786397b7665c62d61830fbb9b/example/ors-bom.txt#L36
      slapos does not really install http://www.python.org/ftp/python/2.7.18/Python-2.7.18.tar.xz it also
      applies a series of patches
      ( https://lab.nexedi.com/nexedi/slapos/blob/035f3382a4d768ea9c702842f3c3c2a1c20d6a21/component/python-2.7/buildout.cfg#L40-45 ).

    - output a machine readable format such as bom.json

-> Make corresponding TODO notes.

Program nxd-bom is intended to help generating bill of material for a
software-release or slapos node.

Generation of the BOM for a software-release is already draftly
implemented. Generation of the BOM for slapos node is thought-out, but
not currently implemented.

In provided example/ors-bom.txt it could be seen how both parts should
look like. And nxd-bom documents in corresponding TODO section how
BOM(slapos-node) should be computed.

I originally used this draft script to build ors-bom.txt as requested by
a customer, and I no longer actively work on it. However I suggest we
incrementally improve it as needed and maybe also integrate it to be run
automatically in _all_ our builds, so that whenever an SR is built, the
BOM.txt is also built automatically for it.

An utility to help preparing bill of materials for Nexedi software.