zc.buildout 3.0.1+slapos001

Rebase, adapt and rework our patches on zc.buildout 3.0.1

This brings in pip-powered support for pyproject.toml specifications which are taking over legacy setup.py.

This is the central piece of a multi-MR work:

• slapos.rebootstrap!4 (merged) - Adapt slapos.rebootstrap to zc.buildout 3.0.1+slapos001
• slapos.rebootstrap!5 (merged) - Adapt rebootstrap to setuptools >= 52.0.0 without easy_install
• slapos!1550 (merged) - Adapt SlapOS SRs to use zc.buildout 3.0.1+slapos001
• !29 (merged) - Add a new patch over zc.buildout 2.7.1+slapos19 fixing buildout bootstrap to support installing zc.buildout 3.0.1 and 3.0.1+. This is needed to migrate to zc.buildout 3.0.1
• slapos.recipe.build!20 (merged) - Fix __init__.py files inside namespace slapos.recipe
• slapos.recipe.cmmi!17 (merged), rubygemsrecipe!10 (merged) - Adapt imports to moved path in slapos.recipe.build

This MR is to review our patches over zc.buildout 3.0.1, and not the changes brought by zc.buildout 3.0.1 compared to 2.7.1.

Notable details

Please read the corresponding commits for in-depth descriptions.

• Respect pinned versions in bootstrap: This is an improvement of a patch we already had to achieve the same thing. The previous patch hardcoded zc.buildout's dependencies: zc.buildout and setuptools. But zc.buildout 3.0.1 also has pip and wheel as dependencies. The new solution avoids hardcoding the dependencies.

• Reimplement the extends algorithm: A new implementation that linearises the graph of extended files to properly apply += and -= in the order of the linearisation. This is a brand new, more comprehensive implementation compared to our previous patch over the 2.7.1 implementation.
  Note: This changes the behavior of extends compared to zc.buildout 3.0.1. We prefer the new behavior.
  Future work: investigate using the same linearisation algorithm as Python's MRO, C3 linearisation - currently we use the linearisation algorithm Python used before C3.

• Enable partial support for += / -= with <=: This is a patch we already had before, so the behavior is the same. A solution for full support is discussed in the commit message; this solution would be facilitated by the new linearisation approach to handling extends.

• Use pip wheel + wheel.install_as_egg to produce genuine eggs every time. This also removes the need for .buildout-egg.

• Prevent buildout from implicitly installing setup_requires eggs defined in a setup.py using a specially crafted .pydistutils.cfg in a temporary directory that is set to $HOME for each pip wheel call.

• Add buildout:extra-paths option: This option determines what paths zc.buildout will scan for already installed distributions, in addition to ./eggs and ./develop-eggs. Special values sys.path, zc.buildout and legacy exist, corresponding respectively to the whole sys.path, the paths of zc.buildout and its dependencies in the same order as they appear in sys.path (the default value), and the legacy behavior (the paths of zc.buildout and its dependencies but in a weird order). Setting buildout:extra-paths= (empty value) enables some isolation from the packages available in the environment.

• Various fixes related to the order in which paths are inserted in sys.path in generated scripts: notably, when some distributions are found in site-packages, they should be inserted after distributions installed in ./eggs, as otherwise another version of the same package in site-packages might supersede the intended package in ./eggs.

• [DROPPED] Prevent buildout bootstrap from linking site-packages and .dist-info packages in ./develop-eggs. This caused too many tests to fail, so it was dropped for now. It may also have hindered legitimate use cases (although not for SlapOS). This means that care must be taken when running buildout bootstrap, as it may create .egg-link files in ./develop-eggs that introduce site-packages in the list of scanned paths until the .egg-link are deleted manually.

Future work

As mentioned:

• Investigate using C3 linearisation for extends

• Implement full support for += / -= with <=

Also:

• Adapt disabled extends tests: Some tests were disabled instead of being adapted after reimplementing extends.

• (Longstanding bug) Fix ${:_profile_base_location_} implementation: currently if ${:_profile_base_location_} is used inside the same section in multiple separate files in the extends graph, all will be substituted with the location of the file that occurs last in the extends linearisation order (i.e. the "most overriding").
  The particular case where ${:_profile_base_location_} for a given section occurs only in one file thus works as intended. This bug existed before the new extends linearisation algorithm - it was just less straightforward which location would be used.

• Support .dist-info distributions as installed by pip: pip install <somepackage results in package (the actual package) and package.dist-info (package metadata) being created next to each other in some directory, usually site-packages, usually with many other projects in the same directory. Currently setuptools and buildout do not distinguish between packages installed by pip (e.g. inside site-packages in a virtual environment) and develop distributions (such as from a git clone). Both are seen as DEVELOP_DIST - although a distinction is that setuptools uses the pkg_resource.DistInfoDistribution subclass for .dist-info distributions. This causes buildout bootstrap to link site-packages in ./develop-eggs, leading to unexpected consequences as this introduces site-packages into the set of paths checked for already-installed eggs. We propose a full solution: (1) Correctly distinguish .dist-info from DEVELOP_DIST. (2) bundle <somepackage>.dist-info and into a containing folder .dist (.dist being chosen by analogy with .egg and .egg-info). (3) This bundle could then be placed inside ./eggs and zc.buildout.easy_install adapted to properly handle .dist folders. This is partly what was already done previously to support installing packages with pip install, except the bundle was called somepackage.egg, causing issues (as it was not actually an .egg). That was then superseded by using pip wheel + setuptools.wheel.Wheel.install_as_egg. The goal here would be to let buildout bootstrap copy/bundle .dist-info packages from the environment in ./eggs, just like it does with .egg packages.