Commit 97b6bc7f authored by Kai Lautaportti's avatar Kai Lautaportti

Pass the augmented environment to the hook scripts.

The call signature for the hook scripts was changed by adding a third
parameter which is a dictionary containing the environment variables copied
from ``os.environ`` and augmented with the environment overrides from the
part configuration.

Existing hook scripts that accept only two arguments continue to work but
reading ``os.environ`` directly will not contain the overridden values.
parent 8308537a
Change History Change History
************** **************
1.4.1 (XXXX-XX-XX) 1.5.0 (XXXX-XX-XX)
================== ==================
- Refactored the environment variable handling logic. Python versions prior - Refactored the environment variable handling logic. Python versions prior
...@@ -11,15 +11,19 @@ Change History ...@@ -11,15 +11,19 @@ Change History
Instead of modifying ``os.environ`` directly we use the ``subprocess`` Instead of modifying ``os.environ`` directly we use the ``subprocess``
module to run the commands in child processes which are given an explicit module to run the commands in child processes which are given an explicit
environment which is a copy of the current ``os.environ`` augmented with environment which is a copy of the current ``os.environ`` augmented with
the per-part overrides. the per-part overrides. As a result, ``os.environ`` is no longer modified
by this recipe.
The `Python hook scripts`_ are passed the augmented environment dictionary
as a third parameter.
.. warning:: Existing hook scripts accepting only two parameters
continue to work but they do not have access to the modified
environment variables. To fix this they should be refactored
to accept the third parameter.
See https://github.com/hexagonit/hexagonit.recipe.cmmi/issues/issue/1/#issue/1/comment/605362 See https://github.com/hexagonit/hexagonit.recipe.cmmi/issues/issue/1/#issue/1/comment/605362
for details. for details.
.. warning:: Due to this change the hook scripts no longer have the
augmented environment. They can still access the buildout
configuration to read the overrides but need to do this
manually.
1.4.0 (2010-08-27) 1.4.0 (2010-08-27)
================== ==================
...@@ -34,7 +38,7 @@ Change History ...@@ -34,7 +38,7 @@ Change History
- the ``configure-command`` is not used to specify a custom configure command and - the ``configure-command`` is not used to specify a custom configure command and
- ``--prefix`` is not given explicitly in the ``configure-options`` option. - ``--prefix`` is not given explicitly in the ``configure-options`` option.
[dokai] [dokai]
- Removed the ``is_build_dir()`` heuristic. - Removed the ``is_build_dir()`` heuristic.
...@@ -45,7 +49,7 @@ Change History ...@@ -45,7 +49,7 @@ Change History
an error message if they were missing. However, the recipe is useful for an error message if they were missing. However, the recipe is useful for
building many different kinds of software packages and checking for building many different kinds of software packages and checking for
particular files limited its use severely. particular files limited its use severely.
Now the recipe omits any checks for particular files in the downloaded Now the recipe omits any checks for particular files in the downloaded
package. It is recommended that you use the ``md5sum`` option in your part package. It is recommended that you use the ``md5sum`` option in your part
configuration to assert that you are downloading the package you expect configuration to assert that you are downloading the package you expect
......
...@@ -69,17 +69,32 @@ Supported options ...@@ -69,17 +69,32 @@ Supported options
List of patch files to the applied to the extracted source. Each List of patch files to the applied to the extracted source. Each
file should be given on a separate line. file should be given on a separate line.
.. _Python hook scripts:
``pre-configure-hook`` ``pre-configure-hook``
Custom python script that will be executed before running the Custom python script that will be executed before running the
``configure`` script. The format of the options is:: ``configure`` script. The format of the options is::
/path/to/the/module.py:name_of_callable /path/to/the/module.py:name_of_callable
where the first part is a filesystem path to the python module and where the first part is a filesystem path to the python module and the
the second part is the name of the callable in the module that second part is the name of the callable in the module that will be called.
will be called. The callable will be passed two parameters: the The callable will be passed three parameters in the following order:
``options`` dictionary from the recipe and the global ``buildout``
dictionary. The callable is not expected to return anything. 1. The ``options`` dictionary from the recipe.
2. The global ``buildout`` dictionary.
3. A dictionary containing the current ``os.environ`` augmented with
the part specific overrides.
The callable is not expected to return anything.
.. note:: The ``os.environ`` is not modified so if the hook script is
interested in the environment variable overrides defined for the
part it needs to read them from the dictionary that is passed in
as the third parameter instead of accessing ``os.environ``
directly.
``pre-make-hook`` ``pre-make-hook``
Custom python script that will be executed before running Custom python script that will be executed before running
...@@ -102,18 +117,25 @@ Supported options ...@@ -102,18 +117,25 @@ Supported options
``environment-section`` ``environment-section``
Name of a section that provides environment variables that will be used to Name of a section that provides environment variables that will be used to
update ``os.environ`` before executing the recipe. augment the variables read from ``os.environ`` before executing the
recipe.
This recipe does not modify ``os.environ`` directly. External commands
run as part of the recipe (e.g. make, configure, etc.) get an augmented
environment when they are forked. Python hook scripts are passed the
augmented as a parameter.
The values of the environment variables may contain references to other The values of the environment variables may contain references to other
existing environment variables (including themselves) in the form of existing environment variables (including themselves) in the form of
Python string interpolation variables using the dictionary notation. These Python string interpolation variables using the dictionary notation. These
references will be expanded before ``os.environ`` is updated. This can be references will be expanded using values from ``os.environ``. This can be
used, for example, to append to the ``PATH`` variable, e.g.:: used, for example, to append to the ``PATH`` variable, e.g.::
[component] [component]
recipe = hexagonit.recipe.cmmi recipe = hexagonit.recipe.cmmi
environment-section = environment environment-section =
environment
[environment] [environment]
PATH = %(PATH)s:${buildout:directory}/bin PATH = %(PATH)s:${buildout:directory}/bin
...@@ -122,7 +144,7 @@ Supported options ...@@ -122,7 +144,7 @@ Supported options
A sequence of ``KEY=VALUE`` pairs separated by newlines that define A sequence of ``KEY=VALUE`` pairs separated by newlines that define
additional environment variables used to update ``os.environ`` before additional environment variables used to update ``os.environ`` before
executing the recipe. executing the recipe.
The semantics of this option are the same as ``environment-section``. If The semantics of this option are the same as ``environment-section``. If
both ``environment-section`` and ``environment`` are provided the values from both ``environment-section`` and ``environment`` are provided the values from
the former will be overridden by the latter allowing per-part customization. the former will be overridden by the latter allowing per-part customization.
...@@ -203,7 +225,7 @@ a custom location within the buildout:: ...@@ -203,7 +225,7 @@ a custom location within the buildout::
... configure-command = perl -I${buildout:perl_lib}/lib/perl5 Makefile.PL INSTALL_BASE=${buildout:perl_lib} ... configure-command = perl -I${buildout:perl_lib}/lib/perl5 Makefile.PL INSTALL_BASE=${buildout:perl_lib}
... url = file://%s/Foo-Bar-0.0.0.tar.gz ... url = file://%s/Foo-Bar-0.0.0.tar.gz
... """ % src) ... """ % src)
>>> print system(buildout) >>> print system(buildout)
Uninstalling package. Uninstalling package.
Installing foobar. Installing foobar.
...@@ -254,15 +276,15 @@ Makefile and using explicit ``make`` options to control the build process. ...@@ -254,15 +276,15 @@ Makefile and using explicit ``make`` options to control the build process.
Installing checkouts Installing checkouts
==================== ====================
Sometimes instead of downloading and building an existing tarball we Sometimes instead of downloading and building an existing tarball we need to
need to work with code that is already available on the filesystem, work with code that is already available on the filesystem, for example an SVN
for example an SVN checkout. checkout.
Instead of providing the ``url`` option we will provide a ``path`` Instead of providing the ``url`` option we will provide a ``path`` option to
option to the directory containing the source code. the directory containing the source code.
Let's demonstrate this by first unpacking our test package to the Let's demonstrate this by first unpacking our test package to the filesystem
filesystem and building that. and building that.
>>> checkout_dir = tmpdir('checkout') >>> checkout_dir = tmpdir('checkout')
>>> import setuptools.archive_util >>> import setuptools.archive_util
...@@ -290,21 +312,21 @@ filesystem and building that. ...@@ -290,21 +312,21 @@ filesystem and building that.
building package building package
installing package installing package
Since using the ``path`` implies that the source code has been Since using the ``path`` implies that the source code has been acquired
acquired outside of the control of the recipe also the responsibility outside of the control of the recipe also the responsibility of managing it is
of managing it is outside of the recipe. outside of the recipe.
Depending on the software you may need to manually run ``make clean`` Depending on the software you may need to manually run ``make clean`` etc.
etc. between buildout runs if you make changes to the code. Also, the between buildout runs if you make changes to the code. Also, the
``keep-compile-dir`` has no effect when ``path`` is used. ``keep-compile-dir`` has no effect when ``path`` is used.
Advanced configuration Advanced configuration
====================== ======================
The above options are enough to build most packages. However, in some The above options are enough to build most packages. However, in some cases it
cases it is not enough and we need to control the build process is not enough and we need to control the build process more. Let's try again
more. Let's try again with a new buildout and provide more options. with a new buildout and provide more options.
>>> write('buildout.cfg', >>> write('buildout.cfg',
... """ ... """
...@@ -335,10 +357,9 @@ more. Let's try again with a new buildout and provide more options. ...@@ -335,10 +357,9 @@ more. Let's try again with a new buildout and provide more options.
... patches/Makefile.dist.patch ... patches/Makefile.dist.patch
... """ % dict(src=src)) ... """ % dict(src=src))
This configuration uses custom configure options, an environment This configuration uses custom configure options, an environment section,
section, per-part customization to the environment, custom prefix, per-part customization to the environment, custom prefix, multiple make
multiple make targets and also patches the source code before the targets and also patches the source code before the scripts are run.
scripts are run.
>>> print system(buildout) >>> print system(buildout)
Uninstalling package. Uninstalling package.
...@@ -358,14 +379,14 @@ scripts are run. ...@@ -358,14 +379,14 @@ scripts are run.
Customizing the build process Customizing the build process
============================= =============================
Sometimes even the above is not enough and you need to be able to Sometimes even the above is not enough and you need to be able to control the
control the process in even more detail. One such use case would be to process in even more detail. One such use case would be to perform dynamic
perform dynamic substitutions on the source code (possible based on substitutions on the source code (possible based on information from the
information from the buildout) which cannot be done with static buildout) which cannot be done with static patches or to simply run arbitrary
patches or to simply run arbitrary commands. commands.
The recipe allows you to write custom python scripts that hook into The recipe allows you to write custom python scripts that hook into the build
the build process. You can define a script to be run: process. You can define a script to be run:
- before the configure script is executed (pre-configure-hook) - before the configure script is executed (pre-configure-hook)
- before the make process is executed (pre-make-hook) - before the make process is executed (pre-make-hook)
...@@ -375,13 +396,22 @@ Each option needs to contain the following information ...@@ -375,13 +396,22 @@ Each option needs to contain the following information
/full/path/to/the/python/module.py:name_of_callable /full/path/to/the/python/module.py:name_of_callable
where the callable object (here name_of_callable) is expected to take where the callable object (here name_of_callable) is expected to take three
two parameters, the ``options`` dictionary from the recipe and the parameters:
global ``buildout`` dictionary.
1. The ``options`` dictionary from the recipe.
2. The global ``buildout`` dictionary.
3. A dictionary containing the current ``os.environ`` augmented with
the part specific overrides.
Let's create a simple python script to demonstrate the These parameters should provide the callable all the necessary information to
functionality. You can naturally have separate scripts for each hook perform any part specific customization to the build process.
or simply use just one or two hooks. Here we use just a single module.
Let's create a simple python script to demonstrate the functionality. You can
naturally have separate modules for each hook or simply use just one or two
hooks. Here we use just a single module.
>>> hooks = tmpdir('hooks') >>> hooks = tmpdir('hooks')
>>> write(hooks, 'customhandlers.py', >>> write(hooks, 'customhandlers.py',
...@@ -389,15 +419,15 @@ or simply use just one or two hooks. Here we use just a single module. ...@@ -389,15 +419,15 @@ or simply use just one or two hooks. Here we use just a single module.
... import logging ... import logging
... log = logging.getLogger('hook') ... log = logging.getLogger('hook')
... ...
... def preconfigure(options, buildout): ... def preconfigure(options, buildout, environment):
... log.info('This is pre-configure-hook!') ... log.info('This is pre-configure-hook!')
... ...
... def premake(options, buildout): ... def premake(options, buildout, environment):
... log.info('This is pre-make-hook!') ... log.info('This is pre-make-hook!')
... ...
... def postmake(options, buildout): ... def postmake(options, buildout, environment):
... log.info('This is post-make-hook!') ... log.info('This is post-make-hook!')
... ...
... """) ... """)
and a new buildout to try it out and a new buildout to try it out
...@@ -431,8 +461,8 @@ and a new buildout to try it out ...@@ -431,8 +461,8 @@ and a new buildout to try it out
hook: This is post-make-hook! hook: This is post-make-hook!
For even more specific needs you can write your own recipe that uses For even more specific needs you can write your own recipe that uses
``hexagonit.recipe.cmmi`` and set the ``keep-compile-dir`` option to ``hexagonit.recipe.cmmi`` and set the ``keep-compile-dir`` option to ``true``.
``true``. You can then continue from where this recipe finished by You can then continue from where this recipe finished by reading the location
reading the location of the compile directory from of the compile directory from ``options['compile-directory']`` from your own
``options['compile-directory']`` from your own recipe. recipe.
...@@ -73,8 +73,15 @@ class Recipe(object): ...@@ -73,8 +73,15 @@ class Recipe(object):
filename, callable = script.split(':') filename, callable = script.split(':')
filename = os.path.abspath(filename) filename = os.path.abspath(filename)
module = imp.load_source('script', filename) module = imp.load_source('script', filename)
# Run the script with all options script = getattr(module, callable.strip())
getattr(module, callable.strip())(self.options, self.buildout)
try:
script(self.options, self.buildout, self.augmented_environment())
except TypeError:
# BBB: Support hook scripts that do not take the environment as
# the third parameter
script(self.options, self.buildout)
def run(self, cmd): def run(self, cmd):
"""Run the given ``cmd`` in a child process.""" """Run the given ``cmd`` in a child process."""
......
...@@ -29,6 +29,9 @@ class NonInformativeTests(unittest.TestCase): ...@@ -29,6 +29,9 @@ class NonInformativeTests(unittest.TestCase):
def tearDown(self): def tearDown(self):
shutil.rmtree(self.dir) shutil.rmtree(self.dir)
for var in os.environ.keys():
if var.startswith('HRC_'):
del os.environ[var]
def write_file(self, filename, contents, mode=stat.S_IREAD|stat.S_IWUSR): def write_file(self, filename, contents, mode=stat.S_IREAD|stat.S_IWUSR):
path = os.path.join(self.dir, filename) path = os.path.join(self.dir, filename)
...@@ -151,6 +154,48 @@ class NonInformativeTests(unittest.TestCase): ...@@ -151,6 +154,48 @@ class NonInformativeTests(unittest.TestCase):
'url' : 'file://%s/testdata/package-0.0.0.tar.gz' % os.path.dirname(__file__)}) 'url' : 'file://%s/testdata/package-0.0.0.tar.gz' % os.path.dirname(__file__)})
self.assertRaises(zc.buildout.UserError, lambda:recipe.run('this-command-does-not-exist')) self.assertRaises(zc.buildout.UserError, lambda:recipe.run('this-command-does-not-exist'))
def test_call_script__bbb_for_callable_with_two_parameters(self):
recipe = self.make_recipe({}, 'test', {
'url' : 'file://%s/testdata/package-0.0.0.tar.gz' % os.path.dirname(__file__),
})
# The hook script does not return anything so we (ab)use exceptions
# as a mechanism for asserting the function behaviour.
filename = os.path.join(self.dir, 'hooks.py')
script = open(filename, 'w')
script.write('def my_hook(options, buildout): raise ValueError("I got called")\n')
script.close()
try:
recipe.call_script('%s:my_hook' % filename)
self.fail("The hook script was not called.")
except ValueError, e:
self.assertEquals(str(e), 'I got called')
def test_call_script__augmented_environment_as_third_parameter(self):
os.environ['HRC_SENTINEL'] = 'sentinel'
os.environ['HRC_TESTVAR'] = 'foo'
recipe = self.make_recipe({}, 'test', {
'url' : 'file://%s/testdata/package-0.0.0.tar.gz' % os.path.dirname(__file__),
'environment' : 'HRC_TESTVAR=bar'
})
# The hook script does not return anything so we (ab)use exceptions
# as a mechanism for asserting the function behaviour.
filename = os.path.join(self.dir, 'hooks.py')
script = open(filename, 'w')
script.write('def my_hook(options, buildout, env): raise ValueError("%(HRC_SENTINEL)s %(HRC_TESTVAR)s" % env)\n')
script.close()
try:
recipe.call_script('%s:my_hook' % filename)
self.fail("The hook script was not called.")
except ValueError, e:
self.assertEquals(str(e), 'sentinel bar')
def test_suite(): def test_suite():
suite = unittest.TestSuite(( suite = unittest.TestSuite((
doctest.DocFileSuite( doctest.DocFileSuite(
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment