Commit 9affb95b authored by Jim Fulton's avatar Jim Fulton

Checking in initial work. Still need more tests and features.

parent deefd960
Zope Buildout
=============
The Zope Buildout project provides support for creating applications,
especially Pyton applications. It provides tools for assembling
applications from multiple parts, Python or otherwise. An application
may actually contain multiple programs, processes, and configuration
settings.
The word "buildout" refers to a description of a set of parts and the
software to create ans assemble them. It is often used informally to
refer to an installed system based on a buildout definition. For
example, if we are creating an application named "Foo", then "the Foo
buildout" is the collection of configuration and application-specific
software that allows an instance of the application to be created. We
may refer to such an instance of the application informally as "a Foo
buildout".
I expect that, for many Zope packages, we'll arrange the package
projects in subversion as buildouts. To work on the package, someone
will check the project out of Subversion and build it. Building it
will assemble all of packages and progras needed to work on it. For
example, a buildout for a project to provide a new security policy
will include the source of the policy and specifications to build the
application for working on it, including:
- a test runner
- a web server for running the user interface
- supporting packages
A buildout will typically contain a copy of bootstrap.py. When
someone checks out the project, they'll run bootstrap.py, which will
- create support directories, like bin, eggs, and work, as needed,
- download and install the zc.buildout and setuptools eggs,
- run bin/build (created by installing zc.buildout) to build the
application.
Buildouts are defined using configuration files. These files are
based on the Python ConfigParser module with some variable-definition
and substitution extensions.
The detailed documentation for the various parts of bukdout can be
found in the following files:
bootstrap.txt
Describes how to use the bootstrapping script
buildout.txt
Describes how to define and run buildouts. It also describes how
to write recipes.
recipes.txt
Documents the few built-in recipes.
##############################################################################
#
# Copyright (c) 2005 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""Bootstrap a buildout
$Id$
"""
import os, sys, urllib2
for d in 'eggs', 'bin':
if not os.path.exists(d):
os.mkdir(d)
ez = {}
exec urllib2.urlopen('http://peak.telecommunity.com/dist/ez_setup.py'
).read() in ez
ez['use_setuptools'](to_dir='eggs', download_delay=0)
import setuptools.command.easy_install
import pkg_resources
import setuptools.package_index
import distutils.dist
os.spawnle(os.P_WAIT, sys.executable, sys.executable, 'setup.py',
'-q', 'develop', '-m', '-x', '-d', 'eggs',
{'PYTHONPATH': os.path.dirname(pkg_resources.__file__)},
)
## easy = setuptools.command.easy_install.easy_install(
## distutils.dist.Distribution(),
## multi_version=True,
## exclude_scripts=True,
## sitepy_installed=True,
## install_dir='eggs',
## outputs=[],
## quiet=True,
## zip_ok=True,
## args=['zc.buildout'],
## )
## easy.finalize_options()
## easy.easy_install('zc.buildout')
env = pkg_resources.Environment(['eggs'])
ws = pkg_resources.WorkingSet()
sys.path[0:0] = [
d.location
for d in ws.resolve([pkg_resources.Requirement.parse('zc.buildout')], env)
]
import zc.buildout.egglinker
zc.buildout.egglinker.scripts(['zc.buildout'], 'bin', ['eggs'])
sys.exit(os.spawnl(os.P_WAIT, 'bin/buildout', 'bin/buildout'))
[buildout]
develop = eggrecipe testrunnerrecipe
parts = test
[test]
recipe = zc.recipe.testrunner
distributions = zc.buildout zc.recipe.egg
Buildout recipe for installing Python distutils distributions as eggs
from setuptools import setup, find_packages
setup(
name = "zc.recipe.egg",
version = "0.1",
packages = find_packages('src'),
include_package_data = True,
package_dir = {'':'src'},
namespace_packages = ['zc', 'zc.recipe'],
install_requires = ['zc.buildout'],
tests_require = ['zope.testing'],
test_suite = 'zc.recipe.eggs.tests.test_suite',
author = "Jim Fulton",
author_email = "jim@zope.com",
description = "Recipe for installing Python package distributions as eggs",
license = "ZPL 2.1",
keywords = "development build",
entry_points = {'zc.buildout': ['default = zc.recipe.egg:Egg']},
)
__import__('pkg_resources').declare_namespace(__name__)
__import__('pkg_resources').declare_namespace(__name__)
Installation of distributions as eggs
=====================================
The zc.recipe.egg ewcipe can be used to install various types if
distutils distributions as eggs. It takes a number of options:
distribution
The distribution specifies the distribution requirement.
This is a requirement as defined by setuptools.
find_links
A list of URLs, files, or directories to search for distributions.
To illustrate this, we've created a directory with some sample eggs:
>>> ls(sample_eggs)
- demo-0.1-py2.3.egg
- demo-0.2-py2.3.egg
- demo-0.3-py2.3.egg
- demoneeded-1.0-py2.3.egg
We have a sample buildout. Let's update it's configuration file to
install the demo package.
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = demo
...
... [demo]
... recipe = zc.recipe.egg
... distribution = demo <0.3
... find_links = %s
... """ % sample_eggs)
In this example, we limited ourself to revisions before 0.3. We also
specified where to find distributions using the find_links option.
Let's run the buildout:
>>> import os
>>> os.chdir(sample_buildout)
>>> runscript = os.path.join(sample_buildout, 'bin', 'buildout')
>>> print system(runscript),
Now, if we look at the buildout eggs directory:
>>> ls(sample_buildout, 'eggs')
- demo-0.2-py2.3.egg
- demoneeded-1.0-py2.3.egg
- zc.recipe.egg.egg-link
We see that we got an egg for demo that met the requirement, as well
as the egg for demoneeded, wich demo requires. (We also see an egg
link for the recipe. This egg link was actually created as part of
the sample buildout setup. Normally, when using the recipe, you'll get
a regular egg installation.)
The demo egg also defined a script and we see that the script was
installed as well:
>>> ls(sample_buildout, 'bin')
- buildout
- demo
- py_demo
Here, in addition to the buildout script, we see the demo script,
demo, and we see a script, py_demo, for giving us a Python prompt with
the path for demo and any eggs it depends on included in sys.path.
This is useful for testing.
If we run the demo script, it prints out some minimal data:
>>> print system(os.path.join(sample_buildout, 'bin', 'demo')),
2 1
The value it prints out happens to be some values defined in the
modules installed.
We can also run the py_demo script. Here we'll just print out
the bits if the path added to reflect the eggs:
>>> print system(os.path.join(sample_buildout, 'bin', 'py_demo'),
... """for p in sys.path[:3]:
... print p
... """).replace('>>> ', '').replace('... ', ''),
... # doctest: +ELLIPSIS
<BLANKLINE>
/usr/local/python/2.3.5/lib/python/setuptools-0.6b2-py2.3.egg
/tmp/tmpcy8MvGbuildout-tests/eggs/demo-0.2-py2.3.egg
/tmp/tmpcy8MvGbuildout-tests/eggs/demoneeded-1.0-py2.3.egg
<BLANKLINE>
from zc.recipe.egg.egg import Egg
##############################################################################
#
# Copyright (c) 2006 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""Install packages as eggs
$Id$
"""
import zc.buildout.egglinker
import zc.buildout.easy_install
class Egg:
def __init__(self, buildout, name, options):
self.buildout = buildout
self.name = name
self.options = options
def install(self):
distribution = self.options.get('distribution', self.name)
links = self.options.get(
'find_links',
self.buildout['buildout'].get('find_links'),
)
if links:
links = links.split()
else:
links = ()
buildout = self.buildout
zc.buildout.easy_install.install(
distribution,
buildout.eggs,
[buildout.buildout_path(link) for link in links],
always_copy = True,
)
zc.buildout.egglinker.scripts(
[distribution], buildout.bin, [buildout.eggs],
)
##############################################################################
#
# Copyright (c) 2006 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
import os, re, shutil, sys, tempfile
import pkg_resources
import zc.buildout.testing
import unittest
from zope.testing import doctest, renormalizing
def runsetup(d):
here = os.getcwd()
try:
os.chdir(d)
os.spawnle(
os.P_WAIT, sys.executable, sys.executable,
'setup.py', '-q', 'bdist_egg',
{'PYTHONPATH': os.path.dirname(pkg_resources.__file__)},
)
shutil.rmtree('build')
finally:
os.chdir(here)
def dirname(d, level=1):
if level == 0:
return d
return dirname(os.path.dirname(d), level-1)
def setUp(test):
zc.buildout.testing.buildoutSetUp(test)
open(os.path.join(test.globs['sample_buildout'],
'eggs', 'zc.recipe.egg.egg-link'),
'w').write(dirname(__file__, 4))
sample = tempfile.mkdtemp('eggtest')
test.globs['_sample_eggs_container'] = sample
test.globs['sample_eggs'] = os.path.join(sample, 'dist')
zc.buildout.testing.write(sample, 'README.txt', '')
zc.buildout.testing.write(sample, 'eggrecipedemobeeded.py', 'y=1\n')
zc.buildout.testing.write(
sample, 'setup.py',
"from setuptools import setup\n"
"setup(name='demoneeded', py_modules=['eggrecipedemobeeded'],"
" zip_safe=True, version='1.0')\n"
)
runsetup(sample)
os.remove(os.path.join(sample, 'eggrecipedemobeeded.py'))
for i in (1, 2, 3):
zc.buildout.testing.write(
sample, 'eggrecipedemo.py',
'import eggrecipedemobeeded\n'
'x=%s\n'
'def main(): print x, eggrecipedemobeeded.y\n'
% i)
zc.buildout.testing.write(
sample, 'setup.py',
"from setuptools import setup\n"
"setup(name='demo', py_modules=['eggrecipedemo'],"
" install_requires = 'demoneeded',"
" entry_points={'console_scripts': ['demo = eggrecipedemo:main']},"
" zip_safe=True, version='0.%s')\n" % i
)
runsetup(sample)
def tearDown(test):
shutil.rmtree(test.globs['_sample_eggs_container'])
zc.buildout.testing.buildoutTearDown(test)
def test_suite():
return unittest.TestSuite((
#doctest.DocTestSuite(),
doctest.DocFileSuite(
'README.txt',
setUp=setUp, tearDown=tearDown,
checker=renormalizing.RENormalizing([
(re.compile('\S+[/%(sep)s]'
'(\\w+-)[^ \t\n%(sep)s/]+.egg'
% dict(sep=os.path.sep)
),
'\\1-VVV-egg')
])
),
))
if __name__ == '__main__':
unittest.main(defaultTest='test_suite')
from setuptools import setup, find_packages
setup(
name = "zc.buildout",
version = "0.1",
packages = ['zc.buildout'],
package_dir = {'':'src'},
namespace_packages = ['zc'],
include_package_data = True,
tests_require = ['zope.testing'],
test_suite = 'zc.buildout.tests.test_suite',
author = "Jim Fulton",
author_email = "jim@zope.com",
description = "System for managing development buildouts",
license = "ZPL 2.1",
keywords = "development build",
install_requires = 'setuptools',
entry_points = {'console_scripts': ['buildout = zc.buildout.build:main']},
)
try:
__import__('pkg_resources').declare_namespace(__name__)
except:
# bootstrapping
pass
This diff is collapsed.
This diff is collapsed.
##############################################################################
#
# Copyright (c) 2005 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""Python easy_install API
This module provides a high-level Python API for installing packages.
It doesn't install scripts. It uses setuptools and requires it to be
installed.
$Id$
"""
# XXX needs doctest
import setuptools.command.easy_install
import pkg_resources
import setuptools.package_index
import distutils.dist
import distutils.log
def install(spec, dest, links=(), **kw):
index = setuptools.package_index.PackageIndex()
index.add_find_links(links)
easy = setuptools.command.easy_install.easy_install(
distutils.dist.Distribution(),
multi_version=True,
exclude_scripts=True,
sitepy_installed=True,
install_dir=dest,
outputs=[],
verbose = 0,
args = [spec],
find_links = links,
**kw
)
easy.finalize_options()
old_warn = distutils.log.warn
distutils.log.warn = lambda *a, **k: None
easy.easy_install(spec, deps=True)
distutils.log.warn = old_warn
##############################################################################
#
# Copyright (c) 2005 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""Egg linker -- Link eggs together to build applications
Egg linker is a script that generates startup scripts for eggs that
include an egg's working script in the generated script.
The egg linker module also exports helper functions of varous kinds to
assist in custom script generation.
$Id$
"""
# XXX needs doctest
import os
import sys
import pkg_resources
def distributions(reqs, eggss):
env = pkg_resources.Environment(eggss)
ws = pkg_resources.WorkingSet()
reqs = [pkg_resources.Requirement.parse(r) for r in reqs]
reqs.append(pkg_resources.Requirement.parse('setuptools'))
return ws.resolve(reqs, env=env)
def path(reqs, eggss):
dists = distributions(reqs, eggss)
return [dist.location for dist in dists]
def location(spec, eggss):
env = pkg_resources.Environment(eggss)
req = pkg_resources.Requirement.parse(spec)
dist = env.best_match(req, pkg_resources.WorkingSet())
return dist.location
def scripts(reqs, dest, eggss):
dists = distributions(reqs, eggss)
reqs = [pkg_resources.Requirement.parse(r) for r in reqs]
projects = [r.project_name for r in reqs]
path = "',\n '".join([dist.location for dist in dists])
for dist in dists:
if dist.project_name in projects:
for name in pkg_resources.get_entry_map(dist, 'console_scripts'):
_script(dist, name, path, os.path.join(dest, name))
_pyscript(path, os.path.join(dest, 'py_'+dist.project_name))
def _script(dist, name, path, dest):
open(dest, 'w').write(script_template % dict(
python = sys.executable,
path = path,
project = dist.project_name,
name = name,
))
try:
os.chmod(dest, 0755)
except (AttributeError, os.error):
pass
script_template = '''\
#!%(python)s
import sys
sys.path[0:0] = [
'%(path)s'
]
from pkg_resources import load_entry_point
sys.exit(
load_entry_point('%(project)s', 'console_scripts', '%(name)s')()
)
'''
def _pyscript(path, dest):
open(dest, 'w').write(py_script_template % dict(
python = sys.executable,
path = path,
))
try:
os.chmod(dest,0755)
except (AttributeError, os.error):
pass
py_script_template = '''\
#!%(python)s -i
import sys
sys.path[0:0] = [
'%(path)s'
]
'''
def main():
import pdb; pdb.set_trace()
##############################################################################
#
# Copyright (c) 2004 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""XXX short summary goes here.
$Id$
"""
import os, re, shutil, sys, tempfile, unittest
from zope.testing import doctest, renormalizing
def cat(dir, *names):
path = os.path.join(dir, *names)
print open(path).read(),
def ls(dir, *subs):
if subs:
dir = os.path.join(dir, *subs)
names = os.listdir(dir)
names.sort()
for name in names:
if os.path.isdir(os.path.join(dir, name)):
print 'd ',
else:
print '- ',
print name
def mkdir(dir, *subs):
if subs:
dir = os.path.join(dir, *subs)
os.mkdir(dir)
def write(dir, *args):
open(os.path.join(dir, *(args[:-1])), 'w').write(args[-1])
def system(command, input=''):
i, o = os.popen4(command)
if input:
i.write(input)
i.close()
return o.read()
def dirname(path, n=1):
if n <= 0:
return path
return dirname(os.path.dirname(path), n-1)
def buildoutSetUp(test):
sample = tempfile.mkdtemp('buildout-tests')
for name in ('bin', 'eggs', 'parts'):
os.mkdir(os.path.join(sample, name))
# make sure we can import zc.buildout and setuptools
import zc.buildout, setuptools
# Generate buildout script
dest = os.path.join(sample, 'bin', 'buildout')
open(dest, 'w').write(
script_template % dict(python=sys.executable, path=sys.path)
)
try:
os.chmod(dest, 0755)
except (AttributeError, os.error):
pass
open(os.path.join(sample, 'buildout.cfg'), 'w').write(
"[buildout]\nparts =\n"
)
open(os.path.join(sample, '.installed.cfg'), 'w').write(
"[buildout]\nparts =\n"
)
test.globs.update(dict(
__here = os.getcwd(),
sample_buildout = sample,
ls = ls,
cat = cat,
mkdir = mkdir,
write = write,
system = system,
__original_wd__ = os.getcwd(),
))
def buildoutTearDown(test):
shutil.rmtree(test.globs['sample_buildout'])
os.chdir(test.globs['__original_wd__'])
script_template = '''\
#!%(python)s
import sys
sys.path[0:0] = %(path)r
from pkg_resources import load_entry_point
sys.exit(load_entry_point('zc.buildout', 'console_scripts', 'buildout')())
'''
##############################################################################
#
# Copyright (c) 2004 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""XXX short summary goes here.
$Id$
"""
import unittest
from zope.testing import doctest
from zc.buildout.testing import buildoutSetUp, buildoutTearDown
def test_suite():
return unittest.TestSuite((
#doctest.DocTestSuite(),
doctest.DocFileSuite(
'buildout.txt',
setUp=buildoutSetUp, tearDown=buildoutTearDown,
),
))
if __name__ == '__main__':
unittest.main(defaultTest='test_suite')
Recipe for generating a custom test runner.
from setuptools import setup, find_packages
setup(
name = "zc.recipe.testrunner",
version = "0.1",
packages = find_packages('src'),
include_package_data = True,
package_dir = {'':'src'},
namespace_packages = ['zc', 'zc.recipe'],
install_requires = ['zc.buildout', 'zope.testing'],
dependency_links = ['http://download.zope.org/distribution/'],
test_suite = 'zc.recipe.testrunner.tests.test_suite',
author = "Jim Fulton",
author_email = "jim@zope.com",
description = "ZC Buildout recipe for creating test runners",
license = "ZPL 2.1",
keywords = "development build",
entry_points = {'zc.buildout':
['default = zc.recipe.testrunner:TestRunner']},
)
__import__('pkg_resources').declare_namespace(__name__)
__import__('pkg_resources').declare_namespace(__name__)
##############################################################################
#
# Copyright (c) 2006 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""A few built-in recipes
$Id$
"""
# XXX need tests
import os, sys
class TestRunner:
def __init__(self, buildout, name, options):
self.buildout = buildout
self.name = name
self.options = options
def install(self):
distributions = self.options['distributions'].split()
path = self.buildout.distributions_path(distributions+['zope.testing'])
locations = [self.buildout.distribution_location(distribution)
for distribution in distributions]
script = self.options.get('script', self.name)
script = self.buildout.buildout_path('bin', script)
open(script, 'w').write(tests_template % dict(
PYTHON=sys.executable,
PATH="',\n '".join(path),
TESTPATH="',\n '--test-path', '".join(locations),
))
try:
os.chmod(script, 0755)
except (AttributeError, os.error):
pass
tests_template = """#!%(PYTHON)s
import sys
sys.path[0:0] = [
'%(PATH)s',
]
from zope.testing import testrunner
defaults = [
'--test-path', '%(TESTPATH)s',
]
sys.exit(testrunner.run(defaults))
"""
- Missing tests. See XXXs
- Buildout command-line options:
--config -C specify a config files
--option -O specify options
- Python discovery support
(Perhaps this is best handled by DEFAULT section.
- Common recipes
- configure-make-make-install
- download, checkout
- Should ot be possible to provide multiple recipies?
Or should recipies be combined through inheritence (or
composition)?
- Python
- Need to better understand the way upgrading works in setuptools.
- Multiple setupfiles,
o extends, optionally-extends, extended-by, optionally-exteded-by
o instance.ini
o ~/buildout/default.ini
- Offline mode
- Local download cache
- Logging
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment