Commit 7f279836 authored by Bryton Lacquement's avatar Bryton Lacquement 🚪 Committed by Julien Muchembled

slapgrid-sr: do not rebootstrap unnecessarily

Currently, bootstrapBuildout is called unconditionally, and since
Software Releases use slapos.reboostrap, we end up with the
following behaviour in the best case:

    Processing software releases...
    Installing software release /srv/slapgrid/slappart9/srv/testnode/aai/software.cfg...
    Generated script '/srv/slapgrid/slappart9/srv/testnode/aai/soft/08f8010370c519147fe23fed3170be9e/bin/buildout'.
    slapos.rebootstrap: Make sure that the section 'python2.7' won't be reinstalled after rebootstrap.
    Unused options for buildout: 'allowed-eggs-from-site-packages' 'exec-sitecustomize' 'include-site-packages' 'unzip'.
    Updating xz-utils.
    Updating patch.
    ...
    Updating gcc.
    Updating python2.7.
    Stripping binaries ...
    Done.
    slapos.rebootstrap:
    ************ REBOOTSTRAP: IMPORTANT NOTICE ************
    bin/buildout is being reinstalled right now, as new python:
      /srv/slapgrid/slappart9/srv/testnode/aai/soft/08f8010370c519147fe23fed3170be9e/parts/python2.7/bin/python2.7
    is available, and buildout is using another python:
      /opt/slapgrid/b9f9e0967ab8491759399f306d65239a/parts/python2.7/bin/python2.7
    Buildout will be restarted automatically to have this change applied.
    ************ REBOOTSTRAP: IMPORTANT NOTICE ************

    Unused options for buildout: 'allowed-eggs-from-site-packages' 'exec-sitecustomize' 'include-site-packages' 'unzip'.
    Updating xz-utils.
    Updating patch.
    ...
    Updating gcc.
    Updating python2.7.
    Updating autoconf.
    ...

Which means that bin/buildout always runs twice: updating parts is usually fast
but loading extends and checking that binaries are stripped can take a while.

The idea of this commit is minimize the amount of work when bin/buildout
already exists, in particular if it is already changed by slapos.reboostrap
to use the built Python. We also hope this will avoid complete rebuilds when
building a different version of Python:

    Installing software release /srv/slapgrid/slappart1/srv/testnode/bsu/software.cfg...
    Generated script '/srv/slapgrid/slappart1/srv/testnode/bsu/soft/90adf14823b5e2bc7dd89ccfbd9388df/bin/buildout'.
    slapos.rebootstrap: Make sure that the section 'python3.5' won't be reinstalled after rebootstrap.
    Uninstalling python3.5.
    Uninstalling file.
    Uninstalling file-msooxml.
    Uninstalling gettext.
    Uninstalling lunzip.
    Uninstalling libxml2.
    Uninstalling sqlite3.
    Uninstalling openssl.
    Uninstalling ca-certificates.
    Uninstalling perl.
    Uninstalling gdbm.
    Uninstalling bzip2.
    Uninstalling libffi.
    Uninstalling libexpat.
    Uninstalling readline.
    Uninstalling ncurses.
    Uninstalling zlib.
    Uninstalling patch.
    Uninstalling xz-utils.
    Unused options for buildout: 'allowed-eggs-from-site-packages' 'exec-sitecustomize' 'include-site-packages' 'unzip'.
    Installing xz-utils.
    ...
    Stripping binaries ...
    Done.
    slapos.rebootstrap:
    ************ REBOOTSTRAP: IMPORTANT NOTICE ************
    bin/buildout is being reinstalled right now, as new python:
      /srv/slapgrid/slappart1/srv/testnode/bsu/soft/90adf14823b5e2bc7dd89ccfbd9388df/parts/python3.5/bin/python3.5
    is available, and buildout is using another python:
      /opt/slapgrid/b9f9e0967ab8491759399f306d65239a/parts/python2.7/bin/python2.7
    Buildout will be restarted automatically to have this change applied.
    ************ REBOOTSTRAP: IMPORTANT NOTICE ************

    Uninstalling template.
    ...
    Uninstalling m4.
    Unused options for buildout: 'allowed-eggs-from-site-packages' 'exec-sitecustomize' 'include-site-packages' 'unzip'.
    Updating xz-utils.
    ...
    Updating python3.5.
    Installing m4.
    ...

About the implementation, we could merely add a `if not os.path.exists(...):`.
I fear cases where a previous run may have left bin/buildout in a non-working
state, so I suggest the use of a marker to force bootstrap if a previous run
failed whereas the existing bin/buildout was reused.

/reviewed-on !54
parent 75c745c4
......@@ -28,6 +28,7 @@
#
##############################################################################
import errno
import os
import pkg_resources
import pwd
......@@ -254,6 +255,7 @@ class Software(object):
extends_cache = tempfile.mkdtemp()
self._set_ownership(extends_cache)
f = None
try:
buildout_cfg = os.path.join(self.software_path, 'buildout.cfg')
if not os.path.exists(buildout_cfg):
......@@ -262,18 +264,32 @@ class Software(object):
additional_parameters = list(self._additional_buildout_parameters(extends_cache))
additional_parameters.extend(['-c', buildout_cfg])
utils.bootstrapBuildout(path=self.software_path,
buildout=self.buildout,
logger=self.logger,
additional_buildout_parameter_list=additional_parameters)
buildout_binary = os.path.join(self.software_path, 'bin', 'buildout')
buildout_marker = buildout_binary + "-bootstrap-skipped"
try:
os.remove(buildout_marker)
except OSError as e:
if e.errno != errno.ENOENT:
raise
if os.path.exists(buildout_binary):
f = open(buildout_marker, "w")
if f is None:
utils.bootstrapBuildout(path=self.software_path,
buildout=self.buildout,
logger=self.logger,
additional_buildout_parameter_list=additional_parameters)
utils.launchBuildout(path=self.software_path,
buildout_binary=os.path.join(self.software_path, 'bin', 'buildout'),
buildout_binary=buildout_binary,
logger=self.logger,
additional_buildout_parameter_list=additional_parameters,
debug=self.buildout_debug)
if f is not None:
os.remove(buildout_marker)
finally:
shutil.rmtree(extends_cache)
if f is not None:
f.close()
def _create_buildout_profile(self, buildout_cfg, url):
with open(buildout_cfg, 'wb') as fout:
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment