Reuse more code from slapos.core to support case of parent directory
not writable.

When recipes mutate the options, we should not allow invalid syntax
otherwise it gets written in .installed.cfg and generate invalid
configparser syntax.

When recipe mutate options, options values are written to
.installed.cfg without escaping buildout substitution syntax,
so if a recipe sets an option value that could be interpreted
as a buildout substitution, it is written as is in .installed.cfg.

This can be a problem if options read from _read_installed_part_options
are accessed, like it's the case with slapos patched buildout which
saves installed options after an error with part installation or after
each part installation when running in verbose mode.

python ZipFile module does not support updating an entry in place, instead make
a new zip file and copy all entries.

Dists in python library (e.g. site-packages) using .dist-info format
are marked as DEVELOP_DIST by setuptools. Use their version to sign
these, and not their location like actual develop dists.

To set a default requirement for not-yet processed parts

The version of Python should not affect the behaviour of a recipe.
Anyway, it was already ignored for DEVELOP_DIST eggs.

This makes the slapos mechanism to share parts more efficient. And with
the upcoming changes in buildout & slapos.recipe.cmmi, there would be
no way for the slapos.reboostrap extension to prevent everything from
being rebuilt when reboostrapping to a different version of Python,
if bootstrap parts are shared.

The monkey-patch by slapos.reboostrap now becomes useless.

_install_and_load is slow, using cache saves time when there are
many sections.

Introduce a workaround for the shebang line length limitation
in zc.buildout.easy_install.

Useful when recipes generate non-string values to be reused by other
recipes.

Add referred parts' hash strings in __buildout_signature__, that
invokes rebuild of a part when one of its (recursive) dependencies
are modified.

Also remove duplicates and sort entries in __buildout_signature__.

Also, updating a part does not put it anymore at the end of the list of
installed parts, that was making .installed.cfg too big.

In SlapOS, bootstrap is called each time software release is invoked and
we do not want to delete develop-eggs directory.

This reverts commit 55d76b34.

https://bugs.launchpad.net/zc.buildout/+bug/144228

Even though such configuration is wrong...

This test asserts buildout's behavior with regards to download options,
and this was changed by the new algorithm for extends. Tests for the
new behavior have not been written yet.

The new algorithm avoids fetching the same extended file more than once
and correctly handles overriding values and += and -=:

The new algorithm starts as if there was a buildout file containing

```
[buildout]
extends =
  user/defaults.cfg # if it exists
  buildout.cfg # if it exists
  command_line_extends.cfg # if passed on the command line
```

The files are then fetched in depth-first-search postorder and fetching
child nodes in the order given by the extends directive, ignoring files
that have already been fetched.

The buildout dicts are then collected in order, and this linearisation
is then merged at the end, overriding the first configs collected with
the later ones. The first dict in the linearisation is not from a file,
but the dict of buildout's (hardcoded) defaults. This is equivalent to
acting as though every file that does not extend anything extends these
defaults.

The first time a file must be downloaded from a url, the linearisation
is merged with the configs already collected, and the resulting options
are then used to determine the download options for this download, and
every subsequent download.

This is a break with buildout's current logic for download options.

By analogy with classes in Python, we are computing a linearisation of
the class hierarchy to determine the method resolution order (MRO).
This algorithm is not the same as Python's MRO since Python 2.3 (C3).

It could be good to switch to a C3 linearisation like Python.

This avoids unecessary copies. This is a preparatory step to
reimplementing the extends algorithm. It may be that this breaks
the extends algorithm as it is currently implemented.

This avoids unecessary deepcopies. This is a preparatory step to
reimplementing the extends algorithm. It may be that this breaks
the extends algorithm as it is currently implemented.

This is useful when using OS Python & eggs.

Useless for SlapOS.

Also show `pip install <url-for-tar.gz-of-master-branch-on-gitlab>`.

This mode is similar to 'legacy' mode as it uses only the paths of
the currently running distributions for zc.buildout and dependencies,
but unlike 'legacy' mode it respects the order in which these appear
in sys.path, avoiding unexpected results.

This is now set to the default because it is closer to 'legacy' mode
and because it has a nice property: running in succession

  `buildout buildout:extra-paths= bootstrap` (1)
  `bin/buildout` (2)

will result in the (1) installing zc.buildout and its dependencies
in /eggs in isolation from the environment, and (2) using only the
paths of these in /eggs, i.e. continuing to operate in isolation,
even without setting extra-paths explictly.

Before this change, (2) would have still used  the whole `sys.path`
unless extra-paths was set otherwise.

With this change, we can remove 'extra-paths=legacy' in the tests
that previously required it.

Now that extra-paths is sys.path by default instead of the legacy
value of the specific paths of zc.buildout and its dependencies,
some tests fail because some eggs in sys.path that were previously
not in extra-paths were instead found on sys.path by package index's
search_path and subsequently installed in ./eggs, whereas they are
now directly seen as already installed at their original location.

This option determines what paths zc.buildout will scan for already
installed distributions. This defaults to sys.path and can be set to
an empty value to enable isolation. The special value 'legacy' yields
the previous behavior of scanning specifically the paths of the current
zc.buildout distribution and its dependencies.

In bootstrap we potentially copy eggs from the working set to ./eggs.
We then reconstruct the same working set using the moved locations.

This commits ensures we keep a correct working set order throughout
and that we avoid activating unintended dists.

If a dist in the computed working set is at a location shared with
other dists - such as site-packages - then when generating scripts
these other packages may overshadow the next items on the sys.path
and result in importing a different version than the one installed
and intended by buildout.

To avert this, a sort of the working set was introduced at various
points just before generating a script.

However, that sort put the paths referenced from an `.egg-link` in
./develop-eggs first. This is truly problematic because dists from
site-packages which are not eggs - e.g. dists installed with pip -
can become referenced as `.egg-link` during buildout bootstrap and
the sort then causes site-packages to be one of the first items in
sys.path.

In particular when running buildout bootstrap from a venv in which
zc.buildout was installed by pip, if any one of zc.buildout or its
dependencies from the venv meets the version requirements, then it
can cause the generated bin/buildout to import the dists only from
the venv's site-packages, even when some do not meet requirements.

To fix this, the sort now puts the dists from `./eggs` first as we
know their locations contain only a single dist, and then puts the
dists from ./develop-eggs which have locations inside the buildout
directory before the others.

The previous sort was also activating all the dists from the paths
of the already activated dists.

Note that this also means that any working set must be manipulated
with care in general to avoid activating unintended dists from the
locations of the already activated dists.

Replace `pkg_resources.Environment(ws.entries).best_match(req, ws)`
with `ws.find(req)`.

The first already starts by calling `ws.find(req)` to attempt to find
an already activated dist in the working set, but if none is found it
then proceeds to scan through the entries of the environment - i.e.
the locations of the directly-requested distributions, `ws.entries` -
to activate a dist at these locations if one matches.

This is problematic when directly-requested distributions are found in
a location shared by multiple dists, such as site-packages: this gives
that location precedence over the normal order of locations scanned by
easy_install, and can result in undesired versions being chosen over
versions available in ./eggs or in ./develop-eggs. The random aspect
of this is also problematic, as the order of paths considered will
depend on the order of the directly-requested distributions and where
they are found.

If zc.buildout or its dependencies have pinned versions that do not
match the currently running versions, they are now installed in the
local eggs directory from scratch according to the pinned versions.

In offline mode this merely ensures that versions that satisfy the
requirements are already available. This is the case when the eggs
are already installed, or when the running versions are a match to
the pinned versions or the absence of a pinned version.

If after this matching versions of zc.buildout and its dependencies
are not located in the local eggs or develop-eggs directories, they
are copied there as was already the case before.

When generating an environment dict for subprocess calls to pip,
os.environ was accidentally modified despite efforts to copy it
and modify only the copy, as copy.copy(os.environ) is not enough.

[ci skip]