System Python and zc.buildout 1.5 ********************************* The 1.5 line of zc.buildout introduced a number of changes. Problems ======== As usual, please send questions and comments to the `distutils SIG mailing list <mailto://distutils-sig@python.org>`_. Report bugs using the `zc.buildout Launchpad Bug Tracker <https://launchpad.net/zc.buildout/+bugs>`_. If problems are keeping you from your work, here's an easy way to revert to the old code temporarily: switch to a custom "emergency" bootstrap script, available from http://svn.zope.org/repos/main/zc.buildout/branches/1.4/bootstrap/bootstrap.py . This customized script will select zc.buildout 1.4.4 by default. zc.buildout 1.4.4 will not upgrade itself unless you explicitly specify a new version. It will also prefer older versions of zc.recipe.egg and some other common recipes. If you have trouble with other recipes, consider using a standard buildout "versions" section to specify older versions of these, as described in the Buildout documentation (http://pypi.python.org/pypi/zc.buildout#repeatable-buildouts-controlling-eggs-used). Working with a System Python ============================ While there are a number of new features available in zc.buildout 1.5, the biggest is that Buildout itself supports usage with a system Python. This can work if you follow a couple of simple rules. 1. Use the new bootstrap.py (available from svn://svn.zope.org/repos/main/zc.buildout/trunk/bootstrap/bootstrap.py). 2. Use buildout recipes that have been upgraded to work with zc.buildout 1.5 and higher. Specifically, they should use ``zc.buildout.easy_install.sitepackage_safe_scripts`` to generate their scripts, if any, rather than ``zc.buildout.easy_install.scripts``. See the `Recipes That Support a System Python`_ section below for more details on recipes that are available as of this writing, and `Updating Recipes to Support a System Python`_ for instructions on how to update a recipe. Note that you should generally only need to update recipes that generate scripts. It's important to note that recipes not upgraded for zc.buildout 1.5.0 should continue to work--just not with a system Python. Using a system Python is inherently fragile. Using a clean, freshly-installed Python without customization in site-packages is more robust and repeatable. See some of the regression tests added to the 1.5.0 line for the kinds of issues that you can encounter with a system Python, and see http://pypi.python.org/pypi/z3c.recipe.scripts#including-site-packages-and-sitecustomize for more discussion. However, using a system Python can be very convenient, and the zc.buildout code for this feature has been tested by many users already. Moreover, it has automated tests to exercise the problems that have been encountered and fixed. Recipes That Support a System Python ==================================== zc.recipe.egg continues to generate old-style scripts that are not safe for use with a system Python. This was done for backwards compatibility, because it is integral to so many buildouts and used as a dependency of so many other recipes. If you want to generate new-style scripts that do support system Python usage, use z3c.recipe.scripts instead (http://pypi.python.org/pypi/z3c.recipe.scripts). z3c.recipe.scripts has the same script and interpreter generation options as zc.recipe.egg, plus a few more for the new features mentioned above. In the simplest case, you should be able to simply change ``recipe = zc.recipe.egg`` to ``recipe = z3c.recipe.scripts`` in the pertinent sections of your buildout configuration and your generated scripts will work with a system Python. Other updated recipes include zc.recipe.testrunner 1.4.0 and z3c.recipe.tag 0.4.0. Others should be updated soon: see their change documents for details, or see `Updating Recipes to Support a System Python`_ for instructions on how to update recipes yourself. Templates for creating Python scripts with the z3c.recipe.filetemplate recipe can be easily changed to support a system Python. - If you don't care about supporting relative paths, simply using a generated interpreter with the eggs you want should be sufficient, as it was before. For instance, if the interpreter is named "py", use ``#!${buildout:bin-directory/py}`` or ``#!/usr/bin/env ${buildout:bin-directory/py}``). - If you do care about relative paths, (``relative-paths = true`` in your buildout configuration), then z3c.recipe.scripts does require a bit more changes, as is usual for the relative path support in that package. First, use z3c.recipe.scripts to generate a script or interpreter with the dependencies you want. This will create a directory in ``parts`` that has a site.py and sitecustomize.py. Then, begin your script as in the snippet below. The example assumes that the z3c.recipe.scripts generated were from a Buildout configuration section labeled "scripts": adjust accordingly. :: #!${buildout:executable} -S ${python-relative-path-setup} import sys sys.path.insert(0, ${scripts:parts-directory|path-repr}) import site Updating Recipes to Support a System Python =========================================== You should generally only need to update recipes that generate scripts. These recipes need to change from using ``zc.buildout.easy_install.scripts`` to be using ``zc.buildout.easy_install.sitepackage_safe_scripts``. The signatures of the two functions are different. Please compare:: def scripts( reqs, working_set, executable, dest, scripts=None, extra_paths=(), arguments='', interpreter=None, initialization='', relative_paths=False, ): def sitepackage_safe_scripts( dest, working_set, executable, site_py_dest, reqs=(), scripts=None, interpreter=None, extra_paths=(), initialization='', include_site_packages=False, exec_sitecustomize=False, relative_paths=False, script_arguments='', script_initialization='', ): In most cases, the arguments are merely reordered. The ``reqs`` argument is no longer required in order to make it easier to generate an interpreter alone. The ``arguments`` argument was renamed to ``script_arguments`` to clarify that it did not affect interpreter generation. The only new required argument is ``site_py_dest``. It must be the path to a directory in which the customized site.py and sitecustomize.py files will be written. A typical generation in a recipe will look like this. (In the recipe's __init__ method...) :: self.options = options b_options = buildout['buildout'] options['parts-directory'] = os.path.join( b_options['parts-directory'], self.name) (In the recipe's install method...) :: options = self.options generated = [] if not os.path.exists(options['parts-directory']): os.mkdir(options['parts-directory']) generated.append(options['parts-directory']) Then ``options['parts-directory']`` can be used for the ``site_py_dest`` value. If you want to support the other arguments (``include_site_packages``, ``exec_sitecustomize``, ``script_initialization``, as well as the ``allowed-eggs-from-site-packages`` option), you might want to look at some of the code in svn://svn.zope.org/repos/main/zc.buildout/trunk/z3c.recipe.scripts\_/src/z3c/recipe/scripts/scripts.py . You might even be able to adopt some of it by subclassing or delegating. The Scripts class in that file is the closest to what you might be used to from zc.recipe.egg. Important note for recipe authors: the code in recipes is *always run without access to the site-packages*. This is irrespective of the ``include-site-packages`` option discussed elsewhere, which controls the software being built, but not the environment in which Buildout itself runs. virtualenv ========== Using virtualenv (http://pypi.python.org/pypi/virtualenv) with the --no-site-packages option already provided a simple way of using a system Python. This is intended to continue to work, and some automated tests exist to demonstrate this. However, it is only supported to the degree that people have found it to work in the past. The existing Buildout tests for virtualenv are only for problems encountered previously. They are very far from comprehensive. Using Buildout with a system python has at least three advantages over using Buildout in conjunction with virtualenv. They may or may not be pertinent to your desired usage. - Unlike ``virtualenv --no-site-packages``, Buildout's support allows you to choose to let packages from your system Python be available to your software (see ``include-site-packages`` in http://pypi.python.org/pypi/z3c.recipe.scripts). You can even specify which eggs installed in your system Python can be allowed to fulfill some of your packages' dependencies (see ``allowed-eggs-from-site-packages`` in http://pypi.python.org/pypi/z3c.recipe.scripts). At the expense of some repeatability and platform dependency, this flexibility means that, for instance, you can rely on difficult-to-build eggs like lxml coming from your system Python. - Buildout's implementation has a full set of automated tests. - An integral Buildout implementation means fewer steps and fewer dependencies to work with a system Python.