In case download from shacache fails, the upgrade should safely stop and be able to run again

  Include Debian 7 and 8
  Include ansible playbook to automate the build
  Include Ubuntu 15.04 and Ubuntu 14.04
  Include configuration for CentOS 6.7 and 7.2

  - Use crontab instead rc.local for start up
  - Include log on the run
  - Make script retry in case of failure (ie.: Network is not ready yet)
  - Pass PATH variable for slapos format, on cron it cannot find few binaries

 - ensure that gpg key for opensuse repo is present
 - pre-fill cache with specific node to faster bootstrap in re6stnet-upgrader
 - re6stnet-upgrader doesn't fail if previous version of package was also a re6st-node
 - repositories keys have to be installed before updating package cache
 - re6stnet-upgrader ensures that service is enabled
 - only update once package cache in re6stnet-upgrader

Makefile was so horrible and unreliable.
I tried https://waf.io/ which is nice but not suitable for this
(see the example 'wscript' at the end).

Functional improvements:
- better detection of what needs to be rebuilt or not
- reproducible tarballs, except for the re6stnet egg
  (and the main tarball if the egg is rebuilt)
- fewer temporary files

And support for OSC is back.

_______________________________________________________________________________

import os, shutil, subprocess, urllib
from waflib import Node, Utils

PREFIX = "opt/re6st"
BOOTSTRAP_URL = "http://downloads.buildout.org/1/bootstrap.py"
repo_dict = dict(
    re6stnet="http://git.erp5.org/repos/re6stnet.git",
    slapos="http://git.erp5.org/repos/slapos.git",
)

def configure(ctx):
    for name, url in repo_dict.iteritems():
        if ctx.path.find_node(name) is None:
            ctx.exec_command(("git", "clone", url), stdout=None, stderr=None)

def cfg(task):
    o, = task.outputs
    bld = task.generator.bld
    o.write(task.inputs[0].read() % dict(
        SLAPOS=bld.srcnode.search_node("slapos").abspath(),
        ROOT="${buildout:directory}/" + bld.bldnode.path_from(o.parent),
        TARGET="/"+PREFIX))

def bootstrap(task):
    b = task.outputs[0]
    d = b.parent.parent
    bootstrap = urllib.urlopen(BOOTSTRAP_URL).read()
    for cache in "download-cache", "extends-cache":
        cache = d.make_node(cache).abspath()
        if os.path.exists(cache):
            shutil.rmtree(cache)
        os.mkdir(cache)
    cwd = d.abspath()
    task.exec_command(("python2", "-S"), input=bootstrap, cwd=cwd)
    task.exec_command((b.abspath(), "buildout:parts=python"), cwd=cwd)

def sdist(task):
    r, p = task.inputs
    d = p.find_node("../../download-cache/dist")
    for x in d.ant_glob("re6stnet-*", quiet=True):
        x.delete()
    task.exec_command((p.abspath(), "setup.py", "sdist", "-d", d.abspath()),
                      cwd=r.abspath())

def build(bld):
    b = bld.bldnode.make_node(PREFIX)
    buildout_cfg = b.make_node("buildout.cfg")
    bld(source="buildout.cfg.in", target=buildout_cfg, rule=cfg)
    tg = bld(source=(buildout_cfg, "slapos"), rule=bootstrap,
             target=map(b.make_node, ("bin/buildout", "bin/python")))
    buildout, python = tg.target
    r = bld.path.find_node("re6stnet")
    tg = bld(source=(r, python), rule=sdist, update_outputs=True,
             target=r.make_node("re6stnet.egg-info/PKG-INFO"))
    bld(name="buildout", source=(buildout_cfg, tg.target),
        target=b.make_node(".installed.cfg"),
        rule=lambda task: task.exec_command((buildout.abspath(),),
            stdout=None, stderr=None, cwd=b.abspath()))

 ###

def h_file(fname, h_file=Utils.h_file):
    if os.path.isdir(fname):
        m = Utils.md5(fname)
        n = len(fname) + 1
        for dirpath, dirs, files in os.walk(fname):
            if dirpath.endswith(("/.git", ".egg-info")):
                del dirs[:]
                continue
            dirs.sort()
            files.sort()
            m.update(dirpath[n:] + '\0')
            for fname in files:
                m.update("%s\0%s\0" % (h_file(os.path.join(dirpath, fname)),
                                       fname))
        return m.digest()
    return h_file(fname)
Utils.h_file = h_file

def find_resource(self, lst):
    if isinstance(lst, str):
        lst = [x for x in Node.split_path(lst) if x and x != '.']
    return self.get_bld().search_node(lst) \
        or self.get_src().find_node(lst)
Node.Node.find_resource = find_resource

- With a Makefile instead of a shell script to make build files,
  it's easier to only redo the necessary parts.
- Do not bootstrap buildout in advance. This caused build failures on old
  dists when this was done with Python 2.7 (parts/buildout/site.py imported
  a module that does not exist on Python 2.6). Now everything can be prepared
  on a recent dist.
- Minimize list of build-deps. We don't need setuptools anymore.
  We also don't rely on system chrpath anymore because SLE_12 does not have it.
- Refactoring with upstream packaging.
- Debian: fix upgrade path from 're6stnet' package.
- Package re6st manpages and logrotate configuration.
- systemd integration on RPM-based dists.
- Fix shebang of OpenVPN/NetworkManager hooks.
- Do never ship *.py[co] files in packages. Remove them on uninstallation
  (not done yet for Arch).
- chrpath: Suse dists use RUNPATH instead of PATH
- Stop stripping ELF files.
- Greatly reduce size of packages by removing eggs/parts that were only useful
  during the build, like Perl.

TODO: integrate with osc working copy ?

For the record, here is an example to strip ELF files:

strip:
	cd $(DESTDIR)/$(TARGET) \
	&& s() { chmod u+w $$x && strip -v -R .comment -R .note "$$@" $$x; } \
	&& x=`find -type f -executable ! -name '*.so*' |xargs -r file | \
		sed -n 's/:.*ELF.*\(executable\|shared\).*//p'` && s \
	&& x=`find -type f -name '*.so*' |xargs -r file | \
		sed -n 's/:.*ELF.*shared.*//p'` && s --strip-unneeded