While invoke setup.py certain eggs (like scikit-learn) launch
cetain custom builds (for cython) using subprocess and sys.executable.

This commit aims to preserve the sys.path over the runs, even if an
egg is using subprocess with the same python to build a component of
the egg.

This commit is part of fixup the "Add setup-eggs option in zc.recipe.egg:develop."

Now specified patches are automatically applied on required eggs as well.

Currently only zc.recipe.egg:custom supports setting environment
variables, and zc.recipe.egg:develop does not.

My motivation for allowing setting environment in :develop is
wendelin.core

    https://lab.nexedi.cn/nexedi/slapos/blob/b5faab3b/component/wendelin.core/buildout.cfg

There we have [wendelin.core] part which installs released egg from
pypi, and [wendelin.core-dev] part which installs wendelin.core from
its latest git version via zc.recipe.egg:develop .

The problem is, wendelin.core for setup.py to work, needs git available,
and with slapos we usually don't have git available on base system, so
we build it by our own and do something like

    [wendelin.core-dev]
    recipe = zc.recipe.egg:develop
    environment = wendelin.core-dev-env

    [wendelin.core-dev-env]
    # wendelin.core-dev needs git to build
    PATH = ${git:location}/bin:%(PATH)s

and the problem is environment does not currently work for
zc.recipe.egg:develop, and thus git is not found -> build fails.

~~~~

In order to support environment in :develop, we just move environment
setting/restoring bits from Custom to Base, and provide Base.install() which
uses this bits. Custom & Develop .install() becomes ._install() which gets
hooked into Base.install() .

I've tested the patch only manually, because currently automated tests are
broken in a lot of places for slapos.buildout and zc.recipe.egg .

/cc @kazuhiko, @Tyagov

- Support on the fly patches in zc.recipe.egg by ``EGGNAME-patches``,
  ``EGGNAME-patch-options``, ``EGGNAME-patch-binary`` (or
  ``patch-binary``) and ``EGGNAME-patch-revision`` options.

- Support on the fly patches in zc.recipe.egg:custom by ``patches``,
  ``patch-options``, ``patch-binary`` and ``patch-revision`` options.
  (options ``EGGNAME-*`` are also supported as well).

This is a rewrite of commit 6e4b8efe
("Support network cache in Download.download()").

Like for URL that contain credentials, we still skip auth challenge
because it's faster and:
- we only support one auth scheme (basic)
- netrc provides no way to specify realms, which seem anyway to be
  less and less used (https://stackoverflow.com/q/69303610 reports
  that recent browsers don't display them anymore)

See merge request nexedi/slapos.buildout!25

- Py3: stop using legacy API of urllib.request and
       fix download of http(s) URLs containing user:passwd@
- Py2: avoid OOM when downloading huge files

This is implemented as a method in case we want to make it configurable
via [buildout].

To be dropped once all buildout in the wild are able to upgrade to a
version that supports new names in expression of conditional sections
(see previous commit).

Adding new names for expression is currently not possible because
buildout aborts before it tries to upgrade (in-place or bootstrap).

For example, existing values were not enough to distinguish
'arm-linux-gnueabi' from 'arm-linux-gnueabihf'.

The 'multiarch' value is the output of

  $CC -dumpmachine

where CC defaults to 'gcc'.

See also https://wiki.debian.org/Multiarch/Tuples

(changed option or missing path)

Sometimes, most parts are reinstalled for a reason that the user
didn't think about and it can take time to understand why.
Explaining for all parts would be too verbose and useless because
many are reinstalled just because their dependencies changed.

This alternate URL is used in case of HTTPError with the main one.
In any case, the main URL is used for both downloading & uploading
from/to networkcache.

This will be exposed by slapos.recipe.build:download* and
slapos.recipe.cmmi recipes.

An optimization is to avoid temporary file when possible: a rename
(or hard link) is not always possible (different mount points).

Another one is to not check md5sum twice when using cache file.

Fall-back mode is ignored if an MD5 checksum is given.

In case of checksum mismatch for a cached path, remove it and
download again, mainly to cover the following cases:
- the url content changes and the user updates the checksum
- buildout killed while downloading directly to cache
  (see above optimization)
- shutil.copyfile is interrupted

Egg is installed from wheel when the requested version ends with :whl

Reuse more code from slapos.core to support case of parent directory
not writable.

When recipes mutate the options, we should not allow invalid syntax
otherwise it gets written in .installed.cfg and generate invalid
configparser syntax.

When recipe mutate options, options values are written to
.installed.cfg without escaping buildout substitution syntax,
so if a recipe sets an option value that could be interpreted
as a buildout substitution, it is written as is in .installed.cfg.

This can be a problem if options read from _read_installed_part_options
are accessed, like it's the case with slapos patched buildout which
saves installed options after an error with part installation or after
each part installation when running in verbose mode.

python ZipFile module does not support updating an entry in place, instead make
a new zip file and copy all entries.

This is useful when using OS Python & eggs.

Useless for SlapOS.

To set a default requirement for not-yet processed parts

The version of Python should not affect the behaviour of a recipe.
Anyway, it was already ignored for DEVELOP_DIST eggs.

This makes the slapos mechanism to share parts more efficient. And with
the upcoming changes in buildout & slapos.recipe.cmmi, there would be
no way for the slapos.reboostrap extension to prevent everything from
being rebuilt when reboostrapping to a different version of Python,
if bootstrap parts are shared.

The monkey-patch by slapos.reboostrap now becomes useless.

_install_and_load is slow, using cache saves time when there are
many sections.

Introduce a workaround for the shebang line length limitation
in zc.buildout.easy_install.

Useful when recipes generate non-string values to be reused by other
recipes.