Merged amos-dependency-links branch to trunk (r80906:81181).

Now there's a buildout option (use-dependency-links) that allows you to 
disable setuptool dependency_links metadata.


git-svn-id: http://svn.zope.org/repos/main/zc.buildout/trunk@81182 62d5b8a3-27da-0310-9561-8e5933582275

CHANGES.txt

@@ -14,6 +14,22 @@ Change History
 1.0.0b31 (2007-???)
 =====================
 
+Feature Changes
+---------------
+
+- Added a configuration option that allows buildouts to ignore
+  dependency_links metadata specified in setup. By default
+  dependency_links in setup are used in addition to buildout specified
+  find-links. This can make it hard to control where eggs come
+  from. Here's how to tell buildout to ignore URLs in
+  dependency_links::
+
+    [buildout]
+    use-dependency-links = false
+
+  By default use-dependency-links is true, which matches the behavior
+  of previous versions of buildout.
+
 Bugs Fixed
 ----------
 

src/zc/buildout/buildout.py

@@ -163,6 +163,12 @@ class Buildout(UserDict.DictMixin):
                         prefer_final)
         zc.buildout.easy_install.prefer_final(prefer_final=='true')
         
+        use_dependency_links = options.get('use-dependency-links', 'true')
+        if use_dependency_links not in ('true', 'false'):
+            self._error('Invalid value for use-dependency-links option: %s',
+                        use_dependency_links)
+        zc.buildout.easy_install.use_dependency_links(
+            use_dependency_links == 'true')
 
         download_cache = options.get('download-cache')
         if download_cache:
@@ -642,7 +648,7 @@ class Buildout(UserDict.DictMixin):
             self['buildout']['eggs-directory'],
             links = self['buildout'].get('find-links', '').split(),
             index = self['buildout'].get('index'),
-            path = [self['buildout']['develop-eggs-directory']],
+            path = [self['buildout']['develop-eggs-directory']]
             )
 
         upgraded = []

src/zc/buildout/buildout.txt

@@ -154,7 +154,7 @@ Currently, recipes must define 3 methods [#future_recipe_methods]_:
 The constructor is responsible for updating a parts options to reflect
 data read from other sections.  The buildout system keeps track of
 whether a part specification has changed.  A part specification has
-changed if it's options, after ajusting for data read from other
+changed if it's options, after adjusting for data read from other
 sections, has changed, or if the recipe has changed.  Only the options
 for the part are considered.  If data are read from other sections,
 then that information has to be reflected in the parts options.  In
@@ -174,7 +174,7 @@ has changed, then the part is uninstalled and reinstalled.  The
 buildout only looks at the part's options, so any data used to
 configure the part needs to be reflected in the part's options.  It is
 the job of a recipe constructor to make sure that the options include
-all relevent data.
+all relevant data.
 
 Of course, parts are also uninstalled if they are no-longer used.
 
@@ -368,7 +368,7 @@ then raising a (or an instance of a subclass of a)
 zc.buildout.UserError exception.  Raising an error other than a
 UserError still displays the error, but labels it as a bug in the
 buildout software or recipe. In the sample above, of someone gives a
-non-existant directory to create the directory in:
+non-existent directory to create the directory in:
 
 
     >>> write(sample_buildout, 'buildout.cfg',
@@ -464,7 +464,7 @@ leave previously created paths in place:
     OSError:
     [Errno 17] File exists: '/sample-buildout/bin'
 
-We meant to create a directiry bins, but typed bin.  Now foo was
+We meant to create a directory bins, but typed bin.  Now foo was
 left behind.
 
     >>> os.path.exists('foo')
@@ -582,7 +582,7 @@ It's critical that recipes clean up partial effects when errors
 occur.  Because recipes most commonly create files and directories,
 buildout provides a helper API for removing created files when an
 error occurs.  Option objects have a created method that can be called
-to record files as they are created.  If the install or update methof
+to record files as they are created.  If the install or update method
 returns with an error, then any registered paths are removed
 automatically.  The method returns the files registered and can be
 used to return the files created.  Let's use this API to simplify the
@@ -633,7 +633,7 @@ returns the registered paths.  We did this for illustrative purposes.
 It would be simpler just to return the paths as before.
 
 If we rerun the buildout, again, we'll get the error and no
-directiories will be created:
+directories will be created:
 
     >>> print system(buildout),
     Develop: '/sample-buildout/recipes'
@@ -685,7 +685,7 @@ extensions are:
 
 - option names are case sensitive
 
-- option values can ue a substitution syntax, described below, to
+- option values can use a substitution syntax, described below, to
   refer to option values in specific sections.
 
 The ConfigParser syntax is very flexible.  Section names can contain
@@ -821,7 +821,7 @@ restriction might be relaxed in future releases.
 Automatic part selection and ordering
 -------------------------------------
 
-When a section with a recipe is refered to, either through variable
+When a section with a recipe is referred to, either through variable
 substitution or by an initializing recipe, the section is treated as a
 part and added to the part list before the referencing part.  For
 example, we can leave data-dir out of the parts list:
@@ -1076,7 +1076,7 @@ we'll set up a web server with some configuration files.
     recipe recipes:debug
 
 Here we specified a URL for the file we extended.  The file we
-downloaded, itself refered to a file on the server using a relative
+downloaded, itself referred to a file on the server using a relative
 URL reference.  Relative references are interpreted relative to the
 base URL when they appear in configuration files loaded via URL.
 
@@ -1282,7 +1282,7 @@ When the buildout is run the service will be installed
     <BLANKLINE>
 
 The service has been installed. If the buildout is run again with no
-changes, the serivce shouldn't be changed.
+changes, the service shouldn't be changed.
 
     >>> print system(buildout)
     Develop: '/sample-buildout/recipes'
@@ -1424,7 +1424,7 @@ is run before the directory is deleted.
     recipe recipes:debug
     <BLANKLINE>
 
-Now we will return the registeration to normal for the benefit of the
+Now we will return the registration to normal for the benefit of the
 rest of the examples.
 
     >>> write(sample_buildout, 'recipes', 'setup.py',
@@ -2004,7 +2004,7 @@ Note that a basic setup.cfg was created for us.
     -  setuptools-0.6-py2.3.egg
     -  zc.buildout-1.0-py2.3.egg
 
-(We list both the eggs and develop-eggs diectories because the
+(We list both the eggs and develop-eggs directories because the
 buildout or setuptools egg could be installed in the develop-eggs
 directory if the original buildout had develop eggs for either
 buildout or setuptools.)
@@ -2060,35 +2060,49 @@ running the buildout when not connected to the internet.  It also
 makes buildouts run much faster. This option is typically set using
 the buildout -o option.
 
-Prefering Final Releases
-------------------------
+Preferring Final Releases
+-------------------------
 
 Currently, when searching for new releases, the newest available
-release is used.  This isn't usually ideal, as you may get development
-releaes or alpha releases not ready to be widely used.  You can
-request that final releases be prefered using the prefer final option
-in the buildout section::
+release is used.  This isn't usually ideal, as you may get a
+development release or alpha releases not ready to be widely used.
+You can request that final releases be preferred using the prefer
+final option in the buildout section::
 
   [buildout]
   ...
   prefer-final = true
 
 When the prefer-final option is set to true, then when searching for
-new releases, final releases are prefered.  If there are final
+new releases, final releases are preferred.  If there are final
 releases that satisfy distribution requirements, then those releases
 are used even if newer non-final releases are available.  The buildout
 prefer-final option can be used to override this behavior.
 
-In buildout version 2, final releases will be prefered by default.
+In buildout version 2, final releases will be preferred by default.
 You will then need to use a false value for prefer-final to get the
-newset releases.
+newest releases.
+
+Dependency links
+----------------
+
+By default buildout will obey the setuptools dependency_links metadata
+when it looks for dependencies. This behavior can be controlled with
+the use-dependency-links buildout option::
+
+  [buildout]
+  ...
+  use-dependency-links = false
+
+The option defaults to true. If you set it to false, then dependency
+links are only looked for in the locations specified by find-links.
 
 Controlling the installation database
 -------------------------------------
 
-The buildout installed uption is used to specify the file used to save
+The buildout installed option is used to specify the file used to save
 information on installed parts.  This option is initialized to
-".installed.cfg", but it can be overridded in the configuration file
+".installed.cfg", but it can be overridden in the configuration file
 or on the command line:
 
     >>> write('buildout.cfg', 
@@ -2119,7 +2133,7 @@ or on the command line:
     d  recipes
 
 The installation database can be disabled by supplying an empty
-buildout installed opttion:
+buildout installed option:
 
     >>> os.remove('inst.cfg')
     >>> print system(buildout+' buildout:installed='),
@@ -2212,7 +2226,7 @@ egg to be built:
     >>> print system(os.path.join(sample_bootstrapped, 'bin', 'buildout')),
     Develop: '/sample-bootstrapped/demo'
 
-Now we can add the extensions option.  We were a bit tricly and ran
+Now we can add the extensions option.  We were a bit tricky and ran
 the buildout once with the demo develop egg defined but without the
 extension option.  This is because extensions are loaded before the
 buildout creates develop eggs. We needed to use a separate buildout
@@ -2235,7 +2249,7 @@ We see that our extension is loaded and executed:
 
 
 
-.. [#future_recipe_methods] In the future, additional mathods may be
+.. [#future_recipe_methods] In the future, additional methods may be
        added. Older recipes with fewer methods will still be
        supported.
 

src/zc/buildout/dependencylinks.txt

+Dependency links
+----------------
+
+By default buildout will obey the setuptools dependency_links metadata
+when it looks for dependencies. This behavior can be controlled with
+the use-dependency-links buildout option.
+
+  [buildout]
+  ...
+  use-dependency-links = false
+
+The option defaults to true. If you set it to false, then dependency
+links are only looked for in the locations specified by find-links.
+
+Let's see this feature in action.  To begin, let's create a new egg
+repository. This repository uses the same sample eggs as the normal
+testing repository.
+
+    >>> link_server2 = start_server(sample_eggs)
+
+Turn on logging on this server so that we can see when eggs are pulled
+from it.
+    
+    >>> get(link_server2 + 'enable_server_logging')
+    GET 200 /enable_server_logging
+    ''
+
+Let's create a develop egg in our buildout that specifies
+dependency_links which point to the new server.
+
+    >>> mkdir(sample_buildout, 'depdemo')
+    >>> write(sample_buildout, 'depdemo', 'dependencydemo.py',
+    ...       'import eggrecipedemoneeded')
+    >>> write(sample_buildout, 'depdemo', 'setup.py',
+    ... '''from setuptools import setup; setup(
+    ...     name='depdemo', py_modules=['dependencydemo'],
+    ...     install_requires = 'demoneeded',
+    ...     dependency_links = ['%s'],
+    ...     zip_safe=True, version='1')
+    ... ''' % link_server2)
+
+Now let's configure the buildout to use the develop egg.
+
+    >>> write(sample_buildout, 'buildout.cfg',
+    ... '''
+    ... [buildout]
+    ... develop = depdemo
+    ... parts = eggs
+    ...
+    ... [eggs]
+    ... recipe = zc.recipe.egg:eggs
+    ... eggs = depdemo
+    ... ''')
+
+Now we can run the buildout.
+
+    >>> print system(buildout)
+    GET 200 /
+    GET 200 /demoneeded-1.2c1.zip
+    Develop: '/sample-buildout/depdemo'
+    Installing eggs.
+    Getting distribution for 'demoneeded'.
+    Got demoneeded 1.2c1.
+    <BLANKLINE>
+
+Notice that the egg was retrieved from the logging server.
+
+Now let's change the egg so that it doesn't specify dependency links.
+
+    >>> write(sample_buildout, 'depdemo', 'setup.py',
+    ... '''from setuptools import setup; setup(
+    ...     name='depdemo', py_modules=['dependencydemo'],
+    ...     install_requires = 'demoneeded',
+    ...     zip_safe=True, version='1')
+    ... ''')
+
+Now we'll remove the existing dependency egg, and rerunning the
+buildout to see where the egg comes from this time.
+
+    >>> from glob import glob
+    >>> from os.path import join
+    >>> def remove_demoneeded_egg():
+    ...     for egg in glob(join(sample_buildout, 'eggs', 'demoneeded*.egg')):
+    ...         remove(sample_buildout, 'eggs', egg)
+    >>> remove_demoneeded_egg()
+    >>> print system(buildout)
+    Develop: '/sample-buildout/depdemo'
+    Updating eggs.
+    Couldn't find index page for 'demoneeded' (maybe misspelled?)
+    Getting distribution for 'demoneeded'.
+    While:
+      Updating eggs.
+      Getting distribution for 'demoneeded'.
+    Error: Couldn't find a distribution for 'demoneeded'.
+    <BLANKLINE>
+
+Now it can't find the dependency since neither the buildout
+configuration nor setup specifies where to look.
+
+Let's change things so that the buildout configuration specifies where
+to look for eggs.
+
+    >>> write(sample_buildout, 'buildout.cfg',
+    ... '''
+    ... [buildout]
+    ... develop = depdemo
+    ... parts = eggs
+    ... find-links = %s
+    ...
+    ... [eggs]
+    ... recipe = zc.recipe.egg:eggs
+    ... eggs = depdemo
+    ... ''' % link_server)
+
+    >>> print system(buildout)
+    Develop: '/sample-buildout/depdemo'
+    Installing eggs.
+    Getting distribution for 'demoneeded'.
+    Got demoneeded 1.2c1.
+    <BLANKLINE>
+
+This time the dependency egg was found on the server without logging
+configured.
+
+Now let's change things once again so that both buildout and setup
+specify different places to look for the dependency egg.
+
+    >>> write(sample_buildout, 'depdemo', 'setup.py',
+    ... '''from setuptools import setup; setup(
+    ...     name='depdemo', py_modules=['dependencydemo'],
+    ...     install_requires = 'demoneeded',
+    ...     dependency_links = ['%s'],
+    ...     zip_safe=True, version='1')
+    ... '''  % link_server2)
+
+    >>> remove_demoneeded_egg()
+    >>> print system(buildout) #doctest: +ELLIPSIS
+    GET 200 /...
+    Develop: '/sample-buildout/depdemo'
+    Updating eggs.
+    Getting distribution for 'demoneeded'.
+    Got demoneeded 1.2c1.
+    <BLANKLINE>
+
+So when both setuptools and buildout specify places to search for
+eggs, the dependency_links takes precedence over find-links.
+
+There is a buildout option that you can specify to change this
+behavior. It is the use-dependency-links option. This option defaults
+to true. When you specify false for this option, buildout will ignore
+dependency_links and only look for eggs using find-links.
+
+Here is an example of using this option to disable dependency_links.
+
+    >>> write(sample_buildout, 'buildout.cfg',
+    ... '''
+    ... [buildout]
+    ... develop = depdemo
+    ... parts = eggs
+    ... find-links = %s
+    ... use-dependency-links = false
+    ...
+    ... [eggs]
+    ... recipe = zc.recipe.egg:eggs
+    ... eggs = depdemo
+    ... ''' % link_server)
+
+    >>> remove_demoneeded_egg()
+    >>> print system(buildout)
+    Develop: '/sample-buildout/depdemo'
+    Updating eggs.
+    Getting distribution for 'demoneeded'.
+    Got demoneeded 1.2c1.
+    <BLANKLINE>
+
+Notice that this time the egg isn't downloaded from the logging server.
+
+If we set the option to true, things return to the way they were
+before. The dependency's are looked for first in the logging server.
+
+    >>> write(sample_buildout, 'buildout.cfg',
+    ... '''
+    ... [buildout]
+    ... develop = depdemo
+    ... parts = eggs
+    ... find-links = %s
+    ... use-dependency-links = true
+    ...
+    ... [eggs]
+    ... recipe = zc.recipe.egg:eggs
+    ... eggs = depdemo
+    ... ''' % link_server)
+    >>> remove_demoneeded_egg()
+    >>> print system(buildout) #doctest: +ELLIPSIS
+    GET 200 /...
+    Develop: '/sample-buildout/depdemo'
+    Updating eggs.
+    Getting distribution for 'demoneeded'.
+    Got demoneeded 1.2c1.
+    <BLANKLINE>

src/zc/buildout/easy_install.py

@@ -115,7 +115,8 @@ class Installer:
     _download_cache = None
     _install_from_cache = False
     _prefer_final = True
-
+    _use_dependency_links = True
+    
     def __init__(self,
                  dest=None,
                  links=(),
@@ -125,6 +126,7 @@ class Installer:
                  path=None,
                  newest=True,
                  versions=None,
+                 use_dependency_links=None,
                  ):
         self._dest = dest
 
@@ -135,8 +137,8 @@ class Installer:
             links = ()
             index = 'file://' + self._download_cache
 
-        
-        
+        if use_dependency_links is not None:
+            self._use_dependency_links = use_dependency_links
         self._links = links = list(_fix_file_links(links))
         if self._download_cache and (self._download_cache not in links):
             links.insert(0, self._download_cache)
@@ -503,10 +505,10 @@ class Installer:
         else:
             dists = [dist]
 
-        # XXX Need test for this
         for dist in dists:
             if (dist.has_metadata('dependency_links.txt')
                 and not self._install_from_cache
+                and self._use_dependency_links
                 ):
                 for link in dist.get_metadata_lines('dependency_links.txt'):
                     link = link.strip()
@@ -709,12 +711,19 @@ def prefer_final(setting=None):
         Installer._prefer_final = bool(setting)
     return old
 
+def use_dependency_links(setting=None):
+    old = Installer._use_dependency_links
+    if setting is not None:
+        Installer._use_dependency_links = bool(setting)
+    return old
+
 def install(specs, dest,
             links=(), index=None,
             executable=sys.executable, always_unzip=False,
-            path=None, working_set=None, newest=True, versions=None):
+            path=None, working_set=None, newest=True, versions=None,
+            use_dependency_links=None):
     installer = Installer(dest, links, index, executable, always_unzip, path,
-                          newest, versions)
+                          newest, versions, use_dependency_links)
     return installer.install(specs, working_set)
 
 

src/zc/buildout/testing.txt

@@ -97,7 +97,7 @@ number of names to the test namespace:
 
     and:
 
-       >>> get(server_url+'enable_server_logging')
+       >>> get(server_url+'disable_server_logging')
 
     This can be useful to see how buildout is interacting with a
     server.

src/zc/buildout/tests.py

@@ -2087,19 +2087,27 @@ need to make it to the download cache.
     
     """
 
-def create_egg(name, version, dest):
+def create_egg(name, version, dest, install_requires=None,
+               dependency_links=None):
     d = tempfile.mkdtemp()
     if dest=='available':
         extras = dict(x=['x'])
     else:
         extras = {}
-        
+    if dependency_links:
+        links = 'dependency_links = %s, ' % dependency_links
+    else:
+        links = ''
+    if install_requires:
+        requires = 'install_requires = %s, ' % install_requires
+    else:
+        requires = ''
     try:
         open(os.path.join(d, 'setup.py'), 'w').write(
             'from setuptools import setup\n'
             'setup(name=%r, version=%r, extras_require=%r, zip_safe=True,\n'
-            '      py_modules=["setup"]\n)'
-            % (name, str(version), extras)
+            '      %s %s py_modules=["setup"]\n)'
+            % (name, str(version), extras, requires, links)
             )
         zc.buildout.testing.bdist_egg(d, sys.executable, os.path.abspath(dest))
     finally:
@@ -2314,9 +2322,7 @@ We get an error if we specify anything but true or false:
     """
 
 
-# XXX Tests needed:
 
-# Link added from package meta data
 
 
 
@@ -2330,12 +2336,12 @@ def create_sample_eggs(test, executable=sys.executable):
         write(tmp, 'README.txt', '')
 
         for i in (0, 1, 2):
-            write(tmp, 'eggrecipedemobeeded.py', 'y=%s\n' % i)
+            write(tmp, 'eggrecipedemoneeded.py', 'y=%s\n' % i)
             c1 = i==2 and 'c1' or ''
             write(
                 tmp, 'setup.py',
                 "from setuptools import setup\n"
-                "setup(name='demoneeded', py_modules=['eggrecipedemobeeded'],"
+                "setup(name='demoneeded', py_modules=['eggrecipedemoneeded'],"
                 " zip_safe=True, version='1.%s%s', author='bob', url='bob', "
                 "author_email='bob')\n"
                 % (i, c1)
@@ -2346,18 +2352,18 @@ def create_sample_eggs(test, executable=sys.executable):
             tmp, 'setup.py',
             "from setuptools import setup\n"
             "setup(name='other', zip_safe=False, version='1.0', "
-            "py_modules=['eggrecipedemobeeded'])\n"
+            "py_modules=['eggrecipedemoneeded'])\n"
             )
         zc.buildout.testing.bdist_egg(tmp, executable, dest)
 
-        os.remove(os.path.join(tmp, 'eggrecipedemobeeded.py'))
+        os.remove(os.path.join(tmp, 'eggrecipedemoneeded.py'))
 
         for i in (1, 2, 3, 4):
             write(
                 tmp, 'eggrecipedemo.py',
-                'import eggrecipedemobeeded\n'
+                'import eggrecipedemoneeded\n'
                 'x=%s\n'
-                'def main(): print x, eggrecipedemobeeded.y\n'
+                'def main(): print x, eggrecipedemoneeded.y\n'
                 % i)
             c1 = i==4 and 'c1' or ''
             write(
@@ -2564,7 +2570,7 @@ def test_suite():
             ),
         
         doctest.DocFileSuite(
-            'easy_install.txt', 'downloadcache.txt',
+            'easy_install.txt', 'downloadcache.txt', 'dependencylinks.txt',
             setUp=easy_install_SetUp,
             tearDown=zc.buildout.testing.buildoutTearDown,