(new commit message)

This adds a generic mechanism in the Download API to rewrite the URL
to be downloaded, possibly with extra headers. Substitued groups from
the matching regular expression can be optionally quoted.

.netrc:

macdef buildout:HOSTNAME
  RULE_1_REGEX
    RULE_1_NEW_URL HEADER1=VALUE1 HEADER2=VALUE2 ...
  RULE_2_REGEX
    RULE_2_NEW_URL ...
  ...

macdef buildout:OTHER_HOSTNAME
  ...

A rewriting rule is defined by a pair of lines with optional indentation
(only there for readability).

The first line of a rule is a regex that matches fully against
the path?query part of the URL.

If the second line is empty, the request isn't changed.
Else parts are parsed using shell-like syntax:
- the first one must produce the full URL to download
- the next ones are optional headers to send
- each part is subject to regex substitution using the
  Python Format Specification Mini-Language:
  captured grouped from the first line are numbered starting from 1,
  {0} is the base URL (scheme://location) and
  the optional `quote` attribute returns the urlencoded value

A use case is to work around
  https://gitlab.com/gitlab-org/gitlab/-/issues/19189
for example with the following .netrc:

    macdef buildout:lab.nexedi.com
      /(.+)/-/raw/([^/]+)/(.+)
        {0}/api/v4/projects/{1.quote}/repository/files/{3.quote}/raw?ref={2} PRIVATE-TOKEN=<ACCESS_TOKEN>

See merge request !36
and merge request !34

- Fixup "[feat] zc.recipe.egg: Support on the fly patches."

This change is to rewrite URL of file to download through buildout.download.Download using macdef definition in netrc file.
The rewrite is based on regular expression, captures are substituted after optionally urlencoding them..

This is how macdef definition is used to rewrite URL:

.netrc:

machine HOSTNAME
login foo
password bar

macdef buildout:HOSTNAME
  REGEX_STRING
    TEMPLATE_1 HEADER1=VALUE1 HEADER2=VALUE2 ...
  ANOTHER_REGEX
    TEMPLATE_2 HEADER1=VALUE1 HEADER2=VALUE2 ...
  ...

macdef ...

HEADER1=VALUE1 string are optional, they are used to set the header. Similar to
the command `curl --header "HEADER1: VALUE1" ...`. Headers can
be repeated as it's needed.

REGEX_STRING is used to match the path and query (if present) of the url we are trying to download.
for example: the regex `/(.*)/-/raw/([\w\-]+)/(.*)` for url
'https://lab.nexedi.com/namespace/project/-/raw/master/README.md'

                /(.*)/-/raw/([\w\-]+)/(.*)
      {0}         {1}          {2}     {3}

TEMPLATE is the new full url with scheme and authority (netloc). All captures are used to format
the template and the headers. It's possible to encode string while formatting, for now only quote
method is supported. Captured groups from REGEX_STRING start at {1}; {0} is the base url.

Example of template and headers:

{0}/api/v4/projects/{1.quote}/repository/files/{3}/raw?ref={2} Authentication={2}

With the regex and template above, the url 'https://lab.nexedi.com/namespace/project/-/raw/master/README.md'
is rewritten to: https://lab.nexedi.com/api/v4/projects/namespace%2Fproject/repository/files/README.md/raw?ref=master

{0} is the base URL (https://lab.nexedi.com)
{1} match 'namespace/project'
{2} match 'master' and
{3} match 'README.md'

`namespace/project` is changed to `namespace%2Fproject` since it's encoded encoded with `quote`.

Some uses cases:

machine lab.nexedi.com
login ignored
password <ACCESS_TOKEN>

macdef buildout:lab.nexedi.com
  /(.*)/-/raw/([\w\-]+)/(.*)
    {0}/api/v4/projects/{1.quote}/repository/files/{3.quote}/raw?ref={2} PRIVATE-TOKEN=<ACCESS_TOKEN>

or

macdef buildout:lab.nexedi.com
  /(.*)/raw/([\w\-]+)/(.*)
    {0}/api/v4/projects/{1.quote}/repository/files/{3.quote}/raw?ref={2} Authorization="Bearer <OAUTH-TOKEN>"
  /(.*)/-/raw/([\w\-]+)/(.*)
    {0}/api/v4/projects/{1.quote}/repository/files/{3.quote}/raw?ref={2} PRIVATE-TOKEN=<ACCESS_TOKEN>

See merge request !37

- Fixup "[feat] Use pip install --editable in easy_install.develop"

See merge request !35

- Fixup "[feat] Use pip install --editable in easy_install.develop"
- Fixup "[wkrd] Use pip install --editable --user"

See merge request !33

- Fixup "Use pip install --editable in easy_install.develop"

Fix output renormalisation when the Python version string contains more
than one digit in the minor version number, i.e for Python >= 3.10.

Let zc.recipe.egg:develop reinstall if .egg-info or .dist-info is
missing from the folder referenced by .egg-link.

Store a part's installed paths in `self[part].installed_files` before
calling `recipe.update()`, to allow the part's recipe to read them if
needed by looking up `options.installed_files`.

Delete this attribute just after the call to ensure this data remains
private to the part.

This will allow zc.recipe.egg:develop to remember in `update()` which
.egg-link it installed in `install()`; inferring this would otherwise
require essentially reinstalling the egg from the setup path, as that
is all the recipe gets as input.

Storing this under `__buildout_installed__` in the part's options was
considered, but some recipes take all the recorded key/value pairs as
input, so it would be a breaking change.

Enable zc.buildout.easy_install.build, which builds a project manually
from an unpacked sdist archive and is used by zc.recipe.egg:custom, to
build a project which only has a pyproject.toml but no setup.py.

Mark .dist-info eggs installed by buildout as develop to distinguish
them from .dist-info eggs from the system and decide how to sign the
egg accordingly (with directory hash or with version).

Adapt tests to ignore warning generated by setuptools not finding
zc.recipe.egg on first index lookup, because it's indexed as
zc-recipe-egg, because since PEP 625 the sdist filename is
zc_recipe_egg-xyz.tar.gz.

Since PEP 625, sdist filenames replace . and - in the project name by
_, such as e.g. zc.buildout==3.0.1 yielding zc_buildout-3.0.1.tar.gz.

However, when looking up available dists in a package index, when an
sdist with such a normalized filename is found, setuptools interprets
the filename by replacing _ with - and uses that as the dist's name;
e.g. zc_buildout-3.0.1.tar.gz yields zc-buildout. This causes lookups
for the name with . (e.g.: zc.buildout) to fail.

Workaround by also looking up the name with . replaced by _, and then
fixing the names of found dists back to expected name.

Prior to pip 21.1, pip install --editable --target fails because it
results in wrong parameters being passed to setup.py develop by pip.

Prior to setuptools 45.2.0, both pip install --editable --target and
pip install --editable --prefix fail because the temporary install
directory used internally by pip is not added to PYTHONPATH prior
to pip calling setup.py develop. In later version setuptools emits a
warning instead of an error.

Temporarily override PYTHONUSERBASE to point to the target directory,
so as to emulate --prefix=<dir> with PYTHONUSERBASE=<dir> and --user.

This is needed for Python2 because pip 21.1 and setuptools 45.2.0 are
both Python3 only.

Instead of running python setup.py develop directly. This will allow
using zc.buildout.easy_install.develop on recent projects that have
only a pyproject.toml. It also fixes develop leaving build artifacts
in the source directory that caused later runs to do nothing, e.g.
preventing develop-eggs to be rebuilt when a build dependency passed
in setup-eggs option of zc.recipe.egg:develop changed.

A verbosity parameter to tune verbosity of pip is added, with adjusted
values for the case of buildout:develop and of zc.recipe.egg:develop,
so as to remain close to the previous behavior with regards to logs.

Technical details:

For packages using PEP-660-style editable installs, supported by more
recent versions of pip, pip will not delegate to `setup.py develop` -
enabling editable installs for pure pyproject.toml projects - and will
instead generate a .dist-info metadata folder but not a .egg-link.

Since buildout currently requires a .egg-link, as it does not support
PEP 660's mechanism that relies on having a sites-packages directory,
we need to create this .egg-link after the fact. The tricky part is
finding out where the .egg-link should point: the path containing the
pyproject.toml, or a subdirectory?

For this we make use of *.dist-info/top_level.txt to first determine
the list of top-level packages, and then importlib to extract info
from the PEP-660-style install.

If top_level.txt does not exist, is empty, or otherwise does not list
any package that resolves to an import path, fallback to the path of
the folder that contains the pyproject.toml as the .egg-link target.
If it lists multiple packages that resolve to different import paths,
arbitrarily use the first one and emit a warning.

Support namespace packages where `spec.submodule_search_locations` is
a `_NamespacePath` object instead of a simple `list` and also support
cases where the layout of the source project does not follow the same
structure as the package tree - meaning some custom magic might be
involved in making editable imports work as intended.

Trigger uninstall + install of eggs installed with zc.recipe.egg:custom
or :develop when pinned versions of setup-eggs have changed. To achieve
this the versions of setup-eggs are included in the section: this makes
them part of its signature so that when they change, buildout will call
`uninstall` and `install` for this section instead of just `update`.

Unlike other zc.recipe.egg entry points, :custom stores the path of the
installed egg; thus `uninstall` will remove it fully, leaving `install`
to reinstall it cleanly from scrach.

In the case of :develop, `uninstall` matters little as only the path of
the installed `.egg-link` is stored. Instead `install` must be fixed to
actually rebuild the egg in-place in the source directory and `develop`
should do nothing.

The main issue lies in `zc.buildout.easy_install.develop`: depending on
the build process, it may leave build artifacts in the source directory
that cause future runs to do nothing.

See merge request !31

Note: all changes with +slapos001 are tests-related.

- Fixup "[test] Add tests for build dependencies"
- Fixup "[feat] zc.recipe.egg: Improve on the fly patches."
- Fixup "[fix] Fix working set sorting".
- Fixup "[feat] Add dependencies in __buildout_signature__"

Pinning zc.buildout = >.1 is now invalid, so use >0.1 instead.

Adapt filter in tests for pip emitting Python2.7 deprecation warnings
to pip >= 20.2.2 and < 21.0 - these versions seem to accidentally add
a redundant bit of message to the warning.

The testing framework, some individual tests and 'buildout:develop'
call python setup.py <command> manually, which is deprecated by
setuptools. Depending on the setuptools version, suppress the
warning for now to avoid polluting the logs.

INFO logs from setuptools which were previously not emitted because
setuptool's ad-hoc legacy logger defaulted to WARNING and above may
now be emitted because setuptools now logs to the root logger, thus
the global root logger's level configuration applied.

This caused undesired 'root: <some setuptools info>' messages to be
emitted and caused many tests to fail due to unexpected outputs.

See merge request !30

When specifying an alternate URL as fallback, the main URL is always
used for both downloading & uploading from/to networkcache.

If slapos.libnetworkcache is importable, install it in bootstrap and
in buildout upgrade - the places where bin/buildout is (re)generated
- as though it were a dependency of zc.buildout.

This is a hack to propagate libnetworkcache as a soft dependency.

Like for URL that contain credentials, we still skip auth challenge
because it's faster and:
- we only support one auth scheme (basic)
- netrc provides no way to specify realms, which seem anyway to be
  less and less used (https://stackoverflow.com/q/69303610 reports
  that recent browsers don't display them anymore)

See merge request !25

This retries with the alternate URL in case of HTTPError with the main
one.

Used by slapos.recipe.build:download* and slapos.recipe.cmmi recipes.

- Py3: stop using legacy API of urllib.request and
       fix download of http(s) URLs containing user:passwd@
- Py2: avoid OOM when downloading huge files

This is implemented as a method in case we want to make it configurable
via [buildout].

An optimization is to avoid temporary file when possible: a rename
(or hard link) is not always possible (different mount points).

Another one is to not check md5sum twice when using cache file.

Fall-back mode is ignored if an MD5 checksum is given.

In case of checksum mismatch for a cached path, remove it and
download again, mainly to cover the following cases:
- the url content changes and the user updates the checksum
- buildout killed while downloading directly to cache
  (see above optimization)
- shutil.copyfile is interrupted

We want two identical sections at different URL to be able to produce
same signature. This feature is useful for slapos.recipe.cmmi's
shared parts.

This commit may fixup "Support ${:_profile_base_location_}." but is
purposefuly kept separate because it concerns parts signatures which
are an orthogonal feature.