Skip to content

S
slapos.buildout
Commit 2ba9d0b2 authored by amos's avatar amos
Browse files
Options

Merged amos-dependency-links branch to trunk (r80906:81181). 

Now there's a buildout option (use-dependency-links) that allows you to 
disable setuptool dependency_links metadata.


git-svn-id: http://svn.zope.org/repos/main/zc.buildout/trunk@81182 62d5b8a3-27da-0310-9561-8e5933582275
parent e1df3174
Expand all Show whitespace changes
Inline Side-by-side
Showing with 441 additions and 68 deletions
CHANGES.txt
View file @ 2ba9d0b2
......@@ -14,6 +14,22 @@ Change History
1.0.0b31 (2007-???)
=====================
Feature Changes
---------------
- Added a configuration option that allows buildouts to ignore
dependency_links metadata specified in setup. By default
dependency_links in setup are used in addition to buildout specified
find-links. This can make it hard to control where eggs come
from. Here's how to tell buildout to ignore URLs in
dependency_links::
[buildout]
use-dependency-links = false
By default use-dependency-links is true, which matches the behavior
of previous versions of buildout.
Bugs Fixed
----------
......
src/zc/buildout/buildout.py
View file @ 2ba9d0b2
......@@ -163,6 +163,12 @@ class Buildout(UserDict.DictMixin):
prefer_final)
zc.buildout.easy_install.prefer_final(prefer_final=='true')
use_dependency_links = options.get('use-dependency-links', 'true')
if use_dependency_links not in ('true', 'false'):
self._error('Invalid value for use-dependency-links option: %s',
use_dependency_links)
zc.buildout.easy_install.use_dependency_links(
use_dependency_links == 'true')
download_cache = options.get('download-cache')
if download_cache:
......@@ -642,7 +648,7 @@ class Buildout(UserDict.DictMixin):
self['buildout']['eggs-directory'],
links = self['buildout'].get('find-links', '').split(),
index = self['buildout'].get('index'),
path = [self['buildout']['develop-eggs-directory']],
path = [self['buildout']['develop-eggs-directory']]
)
upgraded = []
......
src/zc/buildout/buildout.txt
View file @ 2ba9d0b2
......@@ -154,7 +154,7 @@ Currently, recipes must define 3 methods [#future_recipe_methods]_:
The constructor is responsible for updating a parts options to reflect
data read from other sections. The buildout system keeps track of
whether a part specification has changed. A part specification has
changed if it's options, after ajusting for data read from other
changed if it's options, after adjusting for data read from other
sections, has changed, or if the recipe has changed. Only the options
for the part are considered. If data are read from other sections,
then that information has to be reflected in the parts options. In
......@@ -174,7 +174,7 @@ has changed, then the part is uninstalled and reinstalled. The
buildout only looks at the part's options, so any data used to
configure the part needs to be reflected in the part's options. It is
the job of a recipe constructor to make sure that the options include
all relevent data.
all relevant data.
Of course, parts are also uninstalled if they are no-longer used.
......@@ -368,7 +368,7 @@ then raising a (or an instance of a subclass of a)
zc.buildout.UserError exception. Raising an error other than a
UserError still displays the error, but labels it as a bug in the
buildout software or recipe. In the sample above, of someone gives a
non-existant directory to create the directory in:
non-existent directory to create the directory in:
>>> write(sample_buildout, 'buildout.cfg',
......@@ -464,7 +464,7 @@ leave previously created paths in place:
OSError:
[Errno 17] File exists: '/sample-buildout/bin'
We meant to create a directiry bins, but typed bin. Now foo was
We meant to create a directory bins, but typed bin. Now foo was
left behind.
>>> os.path.exists('foo')
......@@ -582,7 +582,7 @@ It's critical that recipes clean up partial effects when errors
occur. Because recipes most commonly create files and directories,
buildout provides a helper API for removing created files when an
error occurs. Option objects have a created method that can be called
to record files as they are created. If the install or update methof
to record files as they are created. If the install or update method
returns with an error, then any registered paths are removed
automatically. The method returns the files registered and can be
used to return the files created. Let's use this API to simplify the
......@@ -633,7 +633,7 @@ returns the registered paths. We did this for illustrative purposes.
It would be simpler just to return the paths as before.
If we rerun the buildout, again, we'll get the error and no
directiories will be created:
directories will be created:
>>> print system(buildout),
Develop: '/sample-buildout/recipes'
......@@ -685,7 +685,7 @@ extensions are:
- option names are case sensitive
- option values can ue a substitution syntax, described below, to
- option values can use a substitution syntax, described below, to
refer to option values in specific sections.
The ConfigParser syntax is very flexible. Section names can contain
......@@ -821,7 +821,7 @@ restriction might be relaxed in future releases.
Automatic part selection and ordering
-------------------------------------
When a section with a recipe is refered to, either through variable
When a section with a recipe is referred to, either through variable
substitution or by an initializing recipe, the section is treated as a
part and added to the part list before the referencing part. For
example, we can leave data-dir out of the parts list:
......@@ -1076,7 +1076,7 @@ we'll set up a web server with some configuration files.
recipe recipes:debug
Here we specified a URL for the file we extended. The file we
downloaded, itself refered to a file on the server using a relative
downloaded, itself referred to a file on the server using a relative
URL reference. Relative references are interpreted relative to the
base URL when they appear in configuration files loaded via URL.
......@@ -1282,7 +1282,7 @@ When the buildout is run the service will be installed
<BLANKLINE>
The service has been installed. If the buildout is run again with no
changes, the serivce shouldn't be changed.
changes, the service shouldn't be changed.
>>> print system(buildout)
Develop: '/sample-buildout/recipes'
......@@ -1424,7 +1424,7 @@ is run before the directory is deleted.
recipe recipes:debug
<BLANKLINE>
Now we will return the registeration to normal for the benefit of the
Now we will return the registration to normal for the benefit of the
rest of the examples.
>>> write(sample_buildout, 'recipes', 'setup.py',
......@@ -2004,7 +2004,7 @@ Note that a basic setup.cfg was created for us.
- setuptools-0.6-py2.3.egg
- zc.buildout-1.0-py2.3.egg
(We list both the eggs and develop-eggs diectories because the
(We list both the eggs and develop-eggs directories because the
buildout or setuptools egg could be installed in the develop-eggs
directory if the original buildout had develop eggs for either
buildout or setuptools.)
......@@ -2060,35 +2060,49 @@ running the buildout when not connected to the internet. It also
makes buildouts run much faster. This option is typically set using
the buildout -o option.
Prefering Final Releases
------------------------
Preferring Final Releases
-------------------------
Currently, when searching for new releases, the newest available
release is used. This isn't usually ideal, as you may get development
releaes or alpha releases not ready to be widely used. You can
request that final releases be prefered using the prefer final option
in the buildout section::
release is used. This isn't usually ideal, as you may get a
development release or alpha releases not ready to be widely used.
You can request that final releases be preferred using the prefer
final option in the buildout section::
[buildout]
...
prefer-final = true
When the prefer-final option is set to true, then when searching for
new releases, final releases are prefered. If there are final
new releases, final releases are preferred. If there are final
releases that satisfy distribution requirements, then those releases
are used even if newer non-final releases are available. The buildout
prefer-final option can be used to override this behavior.
In buildout version 2, final releases will be prefered by default.
In buildout version 2, final releases will be preferred by default.
You will then need to use a false value for prefer-final to get the
newset releases.
newest releases.
Dependency links
----------------
By default buildout will obey the setuptools dependency_links metadata
when it looks for dependencies. This behavior can be controlled with
the use-dependency-links buildout option::
[buildout]
...
use-dependency-links = false
The option defaults to true. If you set it to false, then dependency
links are only looked for in the locations specified by find-links.
Controlling the installation database
-------------------------------------
The buildout installed uption is used to specify the file used to save
The buildout installed option is used to specify the file used to save
information on installed parts. This option is initialized to
".installed.cfg", but it can be overridded in the configuration file
".installed.cfg", but it can be overridden in the configuration file
or on the command line:
>>> write('buildout.cfg',
......@@ -2119,7 +2133,7 @@ or on the command line:
d recipes
The installation database can be disabled by supplying an empty
buildout installed opttion:
buildout installed option:
>>> os.remove('inst.cfg')
>>> print system(buildout+' buildout:installed='),
......@@ -2212,7 +2226,7 @@ egg to be built:
>>> print system(os.path.join(sample_bootstrapped, 'bin', 'buildout')),
Develop: '/sample-bootstrapped/demo'
Now we can add the extensions option. We were a bit tricly and ran
Now we can add the extensions option. We were a bit tricky and ran
the buildout once with the demo develop egg defined but without the
extension option. This is because extensions are loaded before the
buildout creates develop eggs. We needed to use a separate buildout
......@@ -2235,7 +2249,7 @@ We see that our extension is loaded and executed:
.. [#future_recipe_methods] In the future, additional mathods may be
.. [#future_recipe_methods] In the future, additional methods may be
added. Older recipes with fewer methods will still be
supported.
......
src/zc/buildout/dependencylinks.txt 0 → 100644
View file @ 2ba9d0b2
Dependency links
----------------
By default buildout will obey the setuptools dependency_links metadata
when it looks for dependencies. This behavior can be controlled with
the use-dependency-links buildout option.
[buildout]
...
use-dependency-links = false
The option defaults to true. If you set it to false, then dependency
links are only looked for in the locations specified by find-links.
Let's see this feature in action. To begin, let's create a new egg
repository. This repository uses the same sample eggs as the normal
testing repository.
>>> link_server2 = start_server(sample_eggs)
Turn on logging on this server so that we can see when eggs are pulled
from it.
>>> get(link_server2 + 'enable_server_logging')
GET 200 /enable_server_logging
''
Let's create a develop egg in our buildout that specifies
dependency_links which point to the new server.
>>> mkdir(sample_buildout, 'depdemo')
>>> write(sample_buildout, 'depdemo', 'dependencydemo.py',
... 'import eggrecipedemoneeded')
>>> write(sample_buildout, 'depdemo', 'setup.py',
... '''from setuptools import setup; setup(
... name='depdemo', py_modules=['dependencydemo'],
... install_requires = 'demoneeded',
... dependency_links = ['%s'],
... zip_safe=True, version='1')
... ''' % link_server2)
Now let's configure the buildout to use the develop egg.
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
... develop = depdemo
... parts = eggs
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = depdemo
... ''')
Now we can run the buildout.
>>> print system(buildout)
GET 200 /
GET 200 /demoneeded-1.2c1.zip
Develop: '/sample-buildout/depdemo'
Installing eggs.
Getting distribution for 'demoneeded'.
Got demoneeded 1.2c1.
<BLANKLINE>
Notice that the egg was retrieved from the logging server.
Now let's change the egg so that it doesn't specify dependency links.
>>> write(sample_buildout, 'depdemo', 'setup.py',
... '''from setuptools import setup; setup(
... name='depdemo', py_modules=['dependencydemo'],
... install_requires = 'demoneeded',
... zip_safe=True, version='1')
... ''')
Now we'll remove the existing dependency egg, and rerunning the
buildout to see where the egg comes from this time.
>>> from glob import glob
>>> from os.path import join
>>> def remove_demoneeded_egg():
... for egg in glob(join(sample_buildout, 'eggs', 'demoneeded*.egg')):
... remove(sample_buildout, 'eggs', egg)
>>> remove_demoneeded_egg()
>>> print system(buildout)
Develop: '/sample-buildout/depdemo'
Updating eggs.
Couldn't find index page for 'demoneeded' (maybe misspelled?)
Getting distribution for 'demoneeded'.
While:
Updating eggs.
Getting distribution for 'demoneeded'.
Error: Couldn't find a distribution for 'demoneeded'.
<BLANKLINE>
Now it can't find the dependency since neither the buildout
configuration nor setup specifies where to look.
Let's change things so that the buildout configuration specifies where
to look for eggs.
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
... develop = depdemo
... parts = eggs
... find-links = %s
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = depdemo
... ''' % link_server)
>>> print system(buildout)
Develop: '/sample-buildout/depdemo'
Installing eggs.
Getting distribution for 'demoneeded'.
Got demoneeded 1.2c1.
<BLANKLINE>
This time the dependency egg was found on the server without logging
configured.
Now let's change things once again so that both buildout and setup
specify different places to look for the dependency egg.
>>> write(sample_buildout, 'depdemo', 'setup.py',
... '''from setuptools import setup; setup(
... name='depdemo', py_modules=['dependencydemo'],
... install_requires = 'demoneeded',
... dependency_links = ['%s'],
... zip_safe=True, version='1')
... ''' % link_server2)
>>> remove_demoneeded_egg()
>>> print system(buildout) #doctest: +ELLIPSIS
GET 200 /...
Develop: '/sample-buildout/depdemo'
Updating eggs.
Getting distribution for 'demoneeded'.
Got demoneeded 1.2c1.
<BLANKLINE>
So when both setuptools and buildout specify places to search for
eggs, the dependency_links takes precedence over find-links.
There is a buildout option that you can specify to change this
behavior. It is the use-dependency-links option. This option defaults
to true. When you specify false for this option, buildout will ignore
dependency_links and only look for eggs using find-links.
Here is an example of using this option to disable dependency_links.
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
... develop = depdemo
... parts = eggs
... find-links = %s
... use-dependency-links = false
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = depdemo
... ''' % link_server)
>>> remove_demoneeded_egg()
>>> print system(buildout)
Develop: '/sample-buildout/depdemo'
Updating eggs.
Getting distribution for 'demoneeded'.
Got demoneeded 1.2c1.
<BLANKLINE>
Notice that this time the egg isn't downloaded from the logging server.
If we set the option to true, things return to the way they were
before. The dependency's are looked for first in the logging server.
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
... develop = depdemo
... parts = eggs
... find-links = %s
... use-dependency-links = true
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = depdemo
... ''' % link_server)
>>> remove_demoneeded_egg()
>>> print system(buildout) #doctest: +ELLIPSIS
GET 200 /...
Develop: '/sample-buildout/depdemo'
Updating eggs.
Getting distribution for 'demoneeded'.
Got demoneeded 1.2c1.
<BLANKLINE>
src/zc/buildout/easy_install.py
View file @ 2ba9d0b2
......@@ -115,6 +115,7 @@ class Installer:
_download_cache = None
_install_from_cache = False
_prefer_final = True
_use_dependency_links = True
def __init__(self,
dest=None,
......@@ -125,6 +126,7 @@ class Installer:
path=None,
newest=True,
versions=None,
use_dependency_links=None,
):
self._dest = dest
......@@ -135,8 +137,8 @@ class Installer:
links = ()
index = 'file://' + self._download_cache
if use_dependency_links is not None:
self._use_dependency_links = use_dependency_links
self._links = links = list(_fix_file_links(links))
if self._download_cache and (self._download_cache not in links):
links.insert(0, self._download_cache)
......@@ -503,10 +505,10 @@ class Installer:
else:
dists = [dist]
# XXX Need test for this
for dist in dists:
if (dist.has_metadata('dependency_links.txt')
and not self._install_from_cache
and self._use_dependency_links
):
for link in dist.get_metadata_lines('dependency_links.txt'):
link = link.strip()
......@@ -709,12 +711,19 @@ def prefer_final(setting=None):
Installer._prefer_final = bool(setting)
return old
def use_dependency_links(setting=None):
old = Installer._use_dependency_links
if setting is not None:
Installer._use_dependency_links = bool(setting)
return old
def install(specs, dest,
links=(), index=None,
executable=sys.executable, always_unzip=False,
path=None, working_set=None, newest=True, versions=None):
path=None, working_set=None, newest=True, versions=None,
use_dependency_links=None):
installer = Installer(dest, links, index, executable, always_unzip, path,
newest, versions)
newest, versions, use_dependency_links)
return installer.install(specs, working_set)
......
src/zc/buildout/easy_install.txt
View file @ 2ba9d0b2
This diff is collapsed.
src/zc/buildout/testing.txt
View file @ 2ba9d0b2
......@@ -97,7 +97,7 @@ number of names to the test namespace:
and:
>>> get(server_url+'enable_server_logging')
>>> get(server_url+'disable_server_logging')
This can be useful to see how buildout is interacting with a
server.
......
src/zc/buildout/tests.py
View file @ 2ba9d0b2
......@@ -2087,19 +2087,27 @@ need to make it to the download cache.
"""
def create_egg(name, version, dest):
def create_egg(name, version, dest, install_requires=None,
dependency_links=None):
d = tempfile.mkdtemp()
if dest=='available':
extras = dict(x=['x'])
else:
extras = {}
if dependency_links:
links = 'dependency_links = %s, ' % dependency_links
else:
links = ''
if install_requires:
requires = 'install_requires = %s, ' % install_requires
else:
requires = ''
try:
open(os.path.join(d, 'setup.py'), 'w').write(
'from setuptools import setup\n'
'setup(name=%r, version=%r, extras_require=%r, zip_safe=True,\n'
' py_modules=["setup"]\n)'
% (name, str(version), extras)
' %s %s py_modules=["setup"]\n)'
% (name, str(version), extras, requires, links)
)
zc.buildout.testing.bdist_egg(d, sys.executable, os.path.abspath(dest))
finally:
......@@ -2314,9 +2322,7 @@ We get an error if we specify anything but true or false:
"""
# XXX Tests needed:
# Link added from package meta data
......@@ -2330,12 +2336,12 @@ def create_sample_eggs(test, executable=sys.executable):
write(tmp, 'README.txt', '')
for i in (0, 1, 2):
write(tmp, 'eggrecipedemobeeded.py', 'y=%s\n' % i)
write(tmp, 'eggrecipedemoneeded.py', 'y=%s\n' % i)
c1 = i==2 and 'c1' or ''
write(
tmp, 'setup.py',
"from setuptools import setup\n"
"setup(name='demoneeded', py_modules=['eggrecipedemobeeded'],"
"setup(name='demoneeded', py_modules=['eggrecipedemoneeded'],"
" zip_safe=True, version='1.%s%s', author='bob', url='bob', "
"author_email='bob')\n"
% (i, c1)
......@@ -2346,18 +2352,18 @@ def create_sample_eggs(test, executable=sys.executable):
tmp, 'setup.py',
"from setuptools import setup\n"
"setup(name='other', zip_safe=False, version='1.0', "
"py_modules=['eggrecipedemobeeded'])\n"
"py_modules=['eggrecipedemoneeded'])\n"
)
zc.buildout.testing.bdist_egg(tmp, executable, dest)
os.remove(os.path.join(tmp, 'eggrecipedemobeeded.py'))
os.remove(os.path.join(tmp, 'eggrecipedemoneeded.py'))
for i in (1, 2, 3, 4):
write(
tmp, 'eggrecipedemo.py',
'import eggrecipedemobeeded\n'
'import eggrecipedemoneeded\n'
'x=%s\n'
'def main(): print x, eggrecipedemobeeded.y\n'
'def main(): print x, eggrecipedemoneeded.y\n'
% i)
c1 = i==4 and 'c1' or ''
write(
......@@ -2564,7 +2570,7 @@ def test_suite():
),
doctest.DocFileSuite(
'easy_install.txt', 'downloadcache.txt',
'easy_install.txt', 'downloadcache.txt', 'dependencylinks.txt',
setUp=easy_install_SetUp,
tearDown=zc.buildout.testing.buildoutTearDown,
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7