Skip to content

S
slapos.core
Commit 5f0924ff authored by Cédric de Saint Martin's avatar Cédric de Saint Martin
Browse files
Options

Merge branch 'cliff'

parents 92980bc2 f469ae7d
Hide whitespace changes
Inline Side-by-side
Showing with 3344 additions and 2054 deletions
documentation/slapos.usage.rst deleted 100644 → 0
View file @ 92980bc2
=========================
SlapOS command line usage
=========================
Note:
-----
* Default configuration file of "Node" commands (slapos node, slapos supervisor) if not explicitly defined as first argument is:
/etc/opt/slapos/slapos.cfg or value of SLAPOS_CONFIGURATION environment variable if exists.
* Default configuration file of "Client" commands (slapos request, slapos supply, ...) if not explicitly defined as first argument is:
~/.slapos/slapos.cfg or value of SLAPOS_CLIENT_CONFIGURATION environment variable if exists.
* Default log file for Node commands is /var/log/[slapos-node-software.log | slapos-node-instance.log | slapos-node-report.log]. This one requires working log in slapgrid, currently log/console is a total mess.
* Default pid file for Node commands is: /var/run/[slapos-node-software.pid | slapos-node-instance.pid | slapos-node-report.pid].
* Default SlapOS Master is http://www.vifib.net. It can be changed by altering configuration files.
General commands
----------------
slapos
~~~~~~
Display help/usage.
SlapOS Client commands
----------------------
Those commands are used by clients (as human beings or programs) to manage their own instances.
slapos request
~~~~~~~~~~~~~~
Usage:
slapos request [slapos_configuration] <reference> <software_alias | software-url> [--node id=<computer guid> region=<region> network-type=<network> | location/to/node.json] [--configuration foo=value1 bar=value2 | location/to/configuration.json ] [--type type] [--slave]
Request an instance and get status and parameters of instance.
Examples:
* Request a wordpress instance named "mybeautifulinstance" on Node named "COMP-12345":
slapos request mybeautifulinstance wordpress --node id=COMP-12345
* Request a kvm instance named "mykvm" on Node named "COMP-12345", specifying nbd-host and nbd-ip parameters:
slapos request mykvm kvm --node id=COMP-12345 --configuration nbd-host=debian.nbd.vifib.org nbd-port=1024
XXX Change in slaplib: allow to fetch instance params without changing anything. i.e we should do "slapos request myalreadyrequestedinstance" to fetch connection parameters without erasing previously defined instance parameters.
slapos search
~~~~~~~~~~~~~
Note: Not yet implemented.
Usage:
slapos search <search parameters ex. computer region, instance reference, source_section, etc.>
Returns visible instances matching search parameters.
slapos supply
~~~~~~~~~~~~~
Usage:
slapos supply <software | software_group> <computer_guid | commputer_group>
Ask installation of a software on a specific node or group of nodes. Nodes will then be ready to accept instances of specified software.
Examples:
* Ask installation of wordpress Software Release on COMP-12345:
slapos supply wordpress COMP-12345
slapos remove
~~~~~~~~~~~~~
Usage:
slapos remove <software | software_group> <computer_guid | commputer_group>
Ask Removal of a software on a specific node or group of nodes. Existing instances won't work anymore.
XXX "slapos autounsupply a.k.a slapos cleanup"
Examples:
* Ask installation of wordpress Software Release on COMP-12345:
slapos supply wordpress COMP-12345
slapos autosupply
~~~~~~~~~~~~~~~~~
Note: Not yet implemented.
Usage:
slapos autosupply <software | software_group> <computer_guid | computer_group>
Like "slapos suppply", but on-demand. Software will be (re)installed only when at least one instance of this software is requested. When no instance of this software is deployed on the node, it will be uninstalled.
slapos console
~~~~~~~~~~~~~~
Enter in a python console with slap library imported. See "Slapconsole" section to have detailed documentation.
slapos <stop|start|destroy>
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Note: Not yet implemented.
Usage:
slapos <stop|start|destroy> <instance reference>
Ask start/stop/destruction of selected instance.
Example:
* Ask to stop "mywordpressinstance":
slapos stop mywordpressinstance
SlapOS Node commands
--------------------
This kind of commands are used to control the current SlapOS Node. Those commands are only useful for administrators of Nodes.
# XXX: add an environment variable for configuration file.
slapos node
~~~~~~~~~~~
Display status of Node and if not started, launch supervisor daemon.
Temporary note: equivalent of old slapgrid-supervisord + slapgrid-supervisorctl.
slapos node register
~~~~~~~~~~~~~~~~~~~~
Usage:
******
::
slapos node register <DESIRED NODE NAME> [--login LOGIN [--password PASSWORD]] [--interface-name INTERFACE] [--master-url URL <--master-url-web URL>] [--partition-number NUMBER] [--ipv4-local-network NETWORK] [--ipv6-interface INTERFACE] [--create-tap] [--dry-run]
If login is not provided, asks for user's vifib account then password.
Node will register itself, if not already done, to the SlapOS Master defined in configuration file, and will generate SlapOS configuration file.
XXX-Cedric should be like this: If desired node name is already taken, will raise an error.
XXX-Cedric: --master-url-web url will disappear in REST API. Currently, "register" uses SlapOS master web URL to register computer, so it needs the web URL (like http://www.vifib.net)
If Node is already registered (slapos.cfg and certificate already present), issues a warning, backups original configuration and creates new one.
XXX-Cedric should check for IPv6 in selected interface
Parameters:
***********
--login LOGIN Your SlapOS Master login. If not provided, asks it interactively.
--password PASSWORD Your SlapOS Master password. If not provided, asks it interactively. NOTE: giving password as parameter should be avoided for security reasons.
--interface-name INTERFACE Use interface as primary interface. IP of Partitions will be added to it. Defaults to "eth0".
--master-url URL URL of SlapOS Master REST API. defaults to "https://slap.vifib.com".
--master-url-web URL URL of SlapOS Master web access. defaults to "https://www.vifib.com".
--partition-number NUMBER Number of partitions that will have your SlapOS Node. defaults to "10".
--ipv4-local-network NETWORK Subnetwork used to assign local IPv4 addresses. It should be a not used network in order to avoid conflicts. defaults to 10.0.0.0/16.
-t, --create-tap Will trigger creation of one virtual "tap" interface per Partition and attach it to primary interface. Requires primary interface to be a bridge. defaults to false. Needed to host virtual machines.
-n, --dry-run Don't touch to anything in the filesystem. Used to debug.
Notes:
******
* "IPv6 interface" and "create tap" won't be put at all in the SlapOS Node configuration file if not explicitly written.
Examples:
*********
* Register computer named "mycomputer" to vifib::
slapos node register mycomputer
* Register computer named "mycomputer" to vifib using br0 as primary interface, tap0 as IPv6 interface and different local ipv4 subnet::
slapos node register mycomputer --interface-name br0 --ipv6-interface tap0 --ipv4-local-network 11.0.0.0/16
* Register computer named "mycomputer" to another SlapOS master accessible via https://www.myownslaposmaster.com, and SLAP webservice accessible via https://slap.myownslaposmaster.com (Note that this address should be the "slap" webservice URL, not web URL)::
slapos node register mycomputer --master-url https://slap.myownslaposmaster.com --master-url-web https://www.myownslaposmaster.com
XXX-Cedric : To be implemented
* Register computer named "mycomputer" to vifib, and ask to create tap interface to be able to host KVMs::
slapos node register mycomputer --create-tap
slapos node software
~~~~~~~~~~~~~~~~~~~~
Usage:
******
::
slapos node software [--logfile LOGFILE] [--verbose | -v] [--only_sr URL] [--all] [CONFIGURATION_FILE]
Run software installation/deletion.
Temporary note: equivalent of old slapgrid-sr.
# XXX: only_sr should be named ??? (process-only ?)
# XXX: add a "-vv", very verbose, option.
Parameters:
***********
--logfile LOGFILE If specified, will log as well output in the file located at FILE.
--only_sr URL Only process one specific Software Release that has been supplied on this Computer. If not supplied: do nothing.
--all Process all Software Releases, even already installed.
--verbose, -v Be more verbose.
Return values:
**************
(Among other standard Python return values)
0 Everything went fine
1 At least one software hasn't correctly been installed.
slapos node instance
~~~~~~~~~~~~~~~~~~~~
Usage:
******
::
slapos node instance [--logfile LOGFILE] [--verbose | -v] [--only_cp PARTITION] [--all] [CONFIGURATION_FILE]
Temporary note: equivalent of old slapgrid-cp.
Run instances deployment.
Parameters:
***********
--logfile LOGFILE If specified, will log as well output in the file located at FILE.
--only_cp PARTITION Only process one specific Computer Partition, if possible.
--all Force processing all Computer Partitions.
--verbose, -v Be more verbose.
Return values:
**************
(Among other standard Python return values)
0 Everything went fine
1 At least one instance hasn't correctly been processed.
2 At least one promise has failed.
slapos node report
~~~~~~~~~~~~~~~~~~
Usage:
******
::
slapos node report [--logfile LOGFILE] [--verbose | -v] [CONFIGURATION_FILE]
Run instance reports and garbage collection.
Temporary note: equivalent of old slapgrid-ur.
Parameters:
***********
--logfile LOGFILE If specified, will log as well output in the file located at FILE.
--verbose, -v Be more verbose.
Return values:
**************
(Among other standard Python return values)
0 Everything went fine
1 At least one instance hasn't correctly been processed.
slapos node <start|stop|restart|tail|status>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Usage:
slapos node <start|stop|restart|tail|status> <instance>:[process]
Start/Stop/Restart/Show stdout/stderr of instance and/or process.
Examples:
* Start all processes of slappart3:
slapos node start slappart3:
* Stop only apache in slappart1:
slapos node stop slappart1:apache
* Show stdout/stderr of mysqld in slappart2:
slapos node tail slappart2:mysqld
slapos node supervisorctl
~~~~~~~~~~~~~~~~~~~~~~~~~
Usage:
slapos node supervisorctl
Enter into supervisor console.
slapos node supervisord
~~~~~~~~~~~~~~~~~~~~~~~
Usage:
slapos node supervisord
Launch, if not already launched, supervisor daemon.
slapos node log
~~~~~~~~~~~~~~~
Note: Not yet implemented.
Usage:
slapos node log <software|instance|report>
Display log.
documentation/source/conf.py
View file @ 5f0924ff
...@@ -22,7 +22,13 @@ sys.path.append(os.path.abspath('../../')) ...@@ -22,7 +22,13 @@ sys.path.append(os.path.abspath('../../'))
# Add any Sphinx extension module names here, as strings. They can be extensions # Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones. # coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = ['sphinx.ext.autodoc', 'sphinx.ext.intersphinx', 'sphinx.ext.ifconfig', 'repoze.sphinx.autointerface'] extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.intersphinx',
'sphinx.ext.ifconfig',
'repoze.sphinx.autointerface',
'sphinxcontrib.programoutput'
]
# Add any paths that contain templates here, relative to this directory. # Add any paths that contain templates here, relative to this directory.
# templates_path = ['_templates'] # templates_path = ['_templates']
...@@ -45,9 +51,9 @@ copyright = u'2011, Vifib' ...@@ -45,9 +51,9 @@ copyright = u'2011, Vifib'
# built documents. # built documents.
# #
# The short X.Y version. # The short X.Y version.
version = '0.21-dev' from slapos.version import version
# The full version, including alpha/beta/rc tags. # The full version, including alpha/beta/rc tags.
release = '0.21-dev' release = version
# The language for content autogenerated by Sphinx. Refer to documentation # The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages. # for a list of supported languages.
......
documentation/source/slapos 0 → 120000
View file @ 5f0924ff
../../slapos/cli/entry.py
\ No newline at end of file
documentation/source/slapos.usage.rst 0 → 100644
View file @ 5f0924ff
=========================
SlapOS command line usage
=========================
Notes:
------
* Default SlapOS Master is http://www.slapos.org. It can be changed by altering configuration files or with the ``--master-url``
argument for commands that support it.
* Most commands take a configuration file parameter, provided as ``--cfg /path/to/file.cfg``.
If no such argument is provided:
* "node" commands read configuration from :file:`/etc/opt/slapos/slapos.cfg`, or the file referenced by the
``SLAPOS_CONFIGURATION`` environment variable.
* likewise, "client" commands (request, supply...) use :file:`~/.slapos/slapos.cfg`, or the ``SLAPOS_CLIENT_CONFIGURATION`` variable.
XXX TODO document 'alias' for software_url, software_group?, computer_group?
Common options
--------------
Without arguments, the ``slapos`` program lists all the available commands and common options.
.. program-output:: python slapos
The ``-q`` and ``-v`` options control the verbosity of console output (``-v``: DEBUG, default: INFO, ``-q``: WARNING).
Output to a logfile is not affected, and is the same as ``-v``.
SlapOS Client commands
----------------------
These commands are used by clients (as human beings or programs) to manage their own instances.
request
~~~~~~~
.. program-output:: python slapos help request
Examples
* Request a wordpress instance named "mybeautifulinstance" on Node named "COMP-12345"::
$ slapos request mybeautifulinstance wordpress --node id=COMP-12345
* Request a kvm instance named "mykvm" on Node named "COMP-12345", specifying nbd-host and nbd-ip parameters::
$ slapos request mykvm kvm --node id=COMP-12345 --configuration \
nbd-host=debian.nbd.vifib.net nbd-port=1024
XXX Change in slaplib: allow to fetch instance params without changing anything. i.e we should do "slapos request myalreadyrequestedinstance" to fetch connection parameters without erasing previously defined instance parameters.
..
search
~~~~~~
Note: Not yet implemented.
Usage:
slapos search <search parameters ex. computer region, instance reference, source_section, etc.>
Returns visible instances matching search parameters.
supply
~~~~~~
.. program-output:: python slapos help supply
Ask installation of a software on a specific node or group of nodes. Nodes will then be ready to accept instances of specified software.
Examples
* Ask installation of wordpress Software Release on COMP-12345::
$ slapos supply wordpress COMP-12345
remove
~~~~~~
.. program-output:: python slapos help remove
Ask Removal of a software from a specific node or group of nodes. Existing instances won't work anymore.
XXX "slapos autounsupply a.k.a slapos cleanup"
Examples
* Ask installation of wordpress Software Release on COMP-12345::
$ slapos supply wordpress COMP-12345
..
autosupply
~~~~~~~~~~
Note: Not yet implemented.
Usage:
slapos autosupply <software | software_group> <computer_guid | computer_group>
Like "slapos suppply", but on-demand. Software will be (re)installed only when at least one instance of this software is requested. When no instance of this software is deployed on the node, it will be uninstalled.
console
~~~~~~~
.. program-output:: python slapos help console
..
<stop|start|destroy>
~~~~~~~~~~~~~~~~~~~~
Note: Not yet implemented.
Usage:
slapos <stop|start|destroy> <instance reference>
Ask start/stop/destruction of selected instance.
Example:
* Ask to stop "mywordpressinstance"::
$ slapos stop mywordpressinstance
SlapOS Node commands
--------------------
This group of commands is used to control the current SlapOS Node. They are only useful to Node administrators.
node, node status
~~~~~~~~~~~~~~~~~
These are aliases for ``node supervisorctl status``.
It displays the status of the node, also running the supervisor daemon if needed.
.. program-output:: python slapos help node supervisorctl status
node register
~~~~~~~~~~~~~
.. program-output:: python slapos help node register
If login is not provided, asks for user's SlapOS Master account then password.
Node will register itself, if not already done, to the SlapOS Master defined in configuration file, and will generate SlapOS configuration file.
XXX-Cedric should be like this: If desired node name is already taken, will raise an error.
XXX-Cedric: --master-url-web url will disappear in REST API. Currently, "register" uses SlapOS master web URL to register computer, so it needs the web URL (like http://www.slapos.org)
If Node is already registered (slapos.cfg and certificate already present), issues a warning, backups original configuration and creates new one.
XXX-Cedric should check for IPv6 in selected interface
Parameters:
***********
--login LOGIN Your SlapOS Master login. If not provided, asks it interactively.
--password PASSWORD Your SlapOS Master password. If not provided, asks it interactively. NOTE: giving password as parameter should be avoided for security reasons.
--interface-name INTERFACE Use interface as primary interface. IP of Partitions will be added to it. Defaults to "eth0".
--master-url URL URL of SlapOS Master REST API. defaults to "https://slap.vifib.com".
--master-url-web URL URL of SlapOS Master web access. defaults to "https://www.vifib.com".
--partition-number NUMBER Number of partitions that will have your SlapOS Node. defaults to "10".
--ipv4-local-network NETWORK Subnetwork used to assign local IPv4 addresses. It should be a not used network in order to avoid conflicts. defaults to 10.0.0.0/16.
-t, --create-tap Will trigger creation of one virtual "tap" interface per Partition and attach it to primary interface. Requires primary interface to be a bridge. defaults to false. Needed to host virtual machines.
Notes:
******
* "IPv6 interface" and "create tap" won't be put at all in the SlapOS Node configuration file if not explicitly written.
Examples
* Register computer named "mycomputer" to SlapOS Master::
$ slapos node register mycomputer
* Register computer named "mycomputer" to SlapOS Master using br0 as primary interface, tap0 as IPv6 interface and different local ipv4 subnet::
$ slapos node register mycomputer --interface-name br0 --ipv6-interface tap0 \
--ipv4-local-network 11.0.0.0/16
* Register computer named "mycomputer" to another SlapOS master accessible via https://www.myownslaposmaster.com, and SLAP webservice accessible via https://slap.myownslaposmaster.com (Note that this address should be the "slap" webservice URL, not web URL)::
$ slapos node register mycomputer --master-url https://slap.myownslaposmaster.com \
--master-url-web https://www.myownslaposmaster.com
XXX-Cedric : To be implemented
* Register computer named "mycomputer" to SlapOS Master, and ask to create tap interface to be able to host KVMs::
$ slapos node register mycomputer --create-tap
node software
~~~~~~~~~~~~~
.. program-output:: python slapos help node software
Return values:
**************
(Among other standard Python return values)
* 0 Everything went fine
* 1 At least one software was not correctly installed.
node instance
~~~~~~~~~~~~~
.. program-output:: python slapos help node instance
Return values:
**************
(Among other standard Python return values)
* 0 Everything went fine
* 1 At least one instance was not correctly processed.
* 2 At least one promise has failed.
node report
~~~~~~~~~~~
.. program-output:: python slapos help node report
Run instance reports and garbage collection.
Return values:
**************
(Among other standard Python return values)
* 0 Everything went fine
* 1 At least one instance hasn't correctly been processed.
node start|stop|restart|tail|status
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
usage: slapos node <start|stop|restart|tail|status> [-h] [--cfg CFG] <instance>:[process]
Start/Stop/Restart/Show stdout/stderr of instance and/or process.
optional arguments:
-h, --help show this help message and exit
--cfg CFG SlapOS configuration file - defaults to
$SLAPOS_CONFIGURATION or /etc/opt/slapos/slapos.cfg
Examples
* Start all processes of slappart3::
$ slapos node start slappart3:
* Stop only apache in slappart1::
$ slapos node stop slappart1:apache
* Show stdout/stderr of mysqld in slappart2::
$ slapos node tail slappart2:mysqld
node supervisorctl
~~~~~~~~~~~~~~~~~~
.. program-output:: python slapos help node supervisorctl
node supervisord
~~~~~~~~~~~~~~~~
.. program-output:: python slapos help node supervisord
..
node log
~~~~~~~~
Note: Not yet implemented.
Usage:
slapos node log <software|instance|report>
Display log.
SlapOS Miscellaneous commands
-----------------------------
cache lookup
~~~~~~~~~~~~
.. program-output:: python slapos help cache lookup
Examples
* See if the wordpress Software Release is available in precompiled format for our distribution::
$ slapos cache lookup http://git.erp5.org/gitweb/slapos.git/blob_plain/refs/tags/slapos-0.156:/software/kvm/software.cfg
Software URL: http://git.erp5.org/gitweb/slapos.git/blob_plain/refs/tags/slapos-0.156:/software/kvm/software.cfg
MD5: 4410088e11f370503e9d78db4cfa4ec4
-------------
Available for:
distribution | version | id | compatible?
-----------------+--------------+----------------+-------------
CentOS | 6.3 | Final | no
Fedora | 17 | Beefy Miracle | no
Ubuntu | 12.04 | precise | yes
debian | 6.0.6 | | no
debian | 7.0 | | no
setup.py
View file @ 5f0924ff
...@@ -44,6 +44,7 @@ setup(name=name, ...@@ -44,6 +44,7 @@ setup(name=name,
# XML # XML
'zope.interface', # slap library implementes interfaces 'zope.interface', # slap library implementes interfaces
'zc.buildout', 'zc.buildout',
'cliff',
] + additional_install_requires, ] + additional_install_requires,
extra_requires={'docs': ('Sphinx', 'repoze.sphinx.autointerface'),}, extra_requires={'docs': ('Sphinx', 'repoze.sphinx.autointerface'),},
tests_require=[ tests_require=[
...@@ -54,21 +55,43 @@ setup(name=name, ...@@ -54,21 +55,43 @@ setup(name=name,
# accessing templates # accessing templates
entry_points={ entry_points={
'console_scripts': [ 'console_scripts': [
# One entry point to control them all
'slapos = slapos.entry:main',
'slapos-watchdog = slapos.grid.watchdog:main', 'slapos-watchdog = slapos.grid.watchdog:main',
'slapproxy = slapos.proxy:main', 'slapproxy = slapos.cli_legacy.proxy_start:main',
'slapproxy-query = slapos.proxy.query:main', 'slapos = slapos.cli.entry:main',
# Deprecated entry points # Deprecated entry points
'slapconsole = slapos.client:slapconsole', 'slapconsole = slapos.cli_legacy.console:console',
'slapformat = slapos.format:main', 'slapformat = slapos.cli_legacy.format:main',
'slapgrid = slapos.grid.slapgrid:run', 'slapgrid-sr = slapos.cli_legacy.slapgrid:runSoftwareRelease',
'slapgrid-sr = slapos.grid.slapgrid:runSoftwareRelease', 'slapgrid-cp = slapos.cli_legacy.slapgrid:runComputerPartition',
'slapgrid-cp = slapos.grid.slapgrid:runComputerPartition', 'slapgrid-ur = slapos.cli_legacy.slapgrid:runUsageReport',
'slapgrid-ur = slapos.grid.slapgrid:runUsageReport', 'slapgrid-supervisorctl = slapos.cli_legacy.svcbackend:supervisorctl',
'slapgrid-supervisorctl = slapos.grid.svcbackend:supervisorctl', 'slapgrid-supervisord = slapos.cli_legacy.svcbackend:supervisord',
'slapgrid-supervisord = slapos.grid.svcbackend:supervisord', 'bang = slapos.cli_legacy.bang:main',
'bang = slapos.bang:main', ],
'slapos.cli': [
# Utilities
'cache lookup = slapos.cli.cache:CacheLookupCommand',
# SlapOS Node commands
'node bang = slapos.cli.bang:BangCommand',
'node format = slapos.cli.format:FormatCommand',
'node register = slapos.cli.register:RegisterCommand',
'node supervisord = slapos.cli.supervisord:SupervisordCommand',
'node supervisorctl = slapos.cli.supervisorctl:SupervisorctlCommand',
'node status = slapos.cli.supervisorctl:SupervisorctlStatusCommand',
'node start = slapos.cli.supervisorctl:SupervisorctlStartCommand',
'node stop = slapos.cli.supervisorctl:SupervisorctlStopCommand',
'node restart = slapos.cli.supervisorctl:SupervisorctlRestartCommand',
'node tail = slapos.cli.supervisorctl:SupervisorctlTailCommand',
'node report = slapos.cli.slapgrid:ReportCommand',
'node software = slapos.cli.slapgrid:SoftwareCommand',
'node instance = slapos.cli.slapgrid:InstanceCommand',
# SlapOS client commands
'console = slapos.cli.console:ConsoleCommand',
'proxy start = slapos.cli.proxy_start:ProxyStartCommand',
'proxy show = slapos.cli.proxy_show:ProxyShowCommand',
'supply = slapos.cli.supply:SupplyCommand',
'remove = slapos.cli.remove:RemoveCommand',
'request = slapos.cli.request:RequestCommand',
] ]
}, },
test_suite="slapos.tests", test_suite="slapos.tests",
......
slapos/bang.py
View file @ 5f0924ff
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
# vim: set et sts=2:
############################################################################## ##############################################################################
# #
# Copyright (c) 2011, 2012 Vifib SARL and Contributors. # Copyright (c) 2011, 2012 Vifib SARL and Contributors.
...@@ -28,31 +29,17 @@ ...@@ -28,31 +29,17 @@
############################################################################## ##############################################################################
import slapos.slap.slap import slapos.slap.slap
import argparse
import ConfigParser
def main(*args):
parser = argparse.ArgumentParser() def do_bang(configp, message):
parser.add_argument("-m", "--message", default='', help="Message for bang.") computer_id = configp.get('slapos', 'computer_id')
parser.add_argument("configuration_file", nargs=1, type=argparse.FileType(), master_url = configp.get('slapos', 'master_url')
help="SlapOS configuration file.") if configp.has_option('slapos', 'key_file'):
if len(args) == 0: key_file = configp.get('slapos', 'key_file')
argument = parser.parse_args()
else:
argument = parser.parse_args(list(args))
configuration_file = argument.configuration_file[0]
message = argument.message
# Loads config (if config specified)
configuration = ConfigParser.SafeConfigParser()
configuration.readfp(configuration_file)
computer_id = configuration.get('slapos', 'computer_id')
master_url = configuration.get('slapos', 'master_url')
if configuration.has_option('slapos', 'key_file'):
key_file = configuration.get('slapos', 'key_file')
else: else:
key_file = None key_file = None
if configuration.has_option('slapos', 'cert_file'): if configp.has_option('slapos', 'cert_file'):
cert_file = configuration.get('slapos', 'cert_file') cert_file = configp.get('slapos', 'cert_file')
else: else:
cert_file = None cert_file = None
slap = slapos.slap.slap() slap = slapos.slap.slap()
......
slapos/cache.py
View file @ 5f0924ff
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
import ast import ast
import argparse
import ConfigParser
import hashlib import hashlib
import json import json
import re import re
...@@ -16,29 +14,20 @@ from slapos.grid.distribution import patched_linux_distribution ...@@ -16,29 +14,20 @@ from slapos.grid.distribution import patched_linux_distribution
def maybe_md5(s): def maybe_md5(s):
return re.match('[0-9a-f]{32}', s) return re.match('[0-9a-f]{32}', s)
def cache_lookup():
parser = argparse.ArgumentParser()
parser.add_argument("configuration_file", help="SlapOS configuration file")
parser.add_argument("software_url", help="Your software url or MD5 hash")
args = parser.parse_args()
configuration_parser = ConfigParser.SafeConfigParser() def do_lookup(configp, software_url):
configuration_parser.read(args.configuration_file) cache_dir = configp.get('networkcache', 'download-binary-dir-url')
configuration_parser.items('networkcache') if maybe_md5(software_url):
md5 = software_url
cache_dir = configuration_parser.get('networkcache', 'download-binary-dir-url')
if maybe_md5(args.software_url):
md5 = args.software_url
else: else:
md5 = hashlib.md5(args.software_url).hexdigest() md5 = hashlib.md5(software_url).hexdigest()
try: try:
response = urllib2.urlopen('%s/%s' % (cache_dir, md5)) response = urllib2.urlopen('%s/%s' % (cache_dir, md5))
except urllib2.HTTPError as e: except urllib2.HTTPError as e:
if e.code == 404: if e.code == 404:
print 'Object not in cache: %s' % args.software_url print 'Object not in cache: %s' % software_url
else: else:
print 'Error during cache lookup: %s (%s)' % (e.code, e.reason) print 'Error during cache lookup: %s (%s)' % (e.code, e.reason)
sys.exit(10) sys.exit(10)
...@@ -67,4 +56,3 @@ def cache_lookup(): ...@@ -67,4 +56,3 @@ def cache_lookup():
for os in ostable: for os in ostable:
compatible = 'yes' if networkcache.os_matches(os, linux_distribution) else 'no' compatible = 'yes' if networkcache.os_matches(os, linux_distribution) else 'no'
print '%-16s | %12s | %s | %s' % (os[0], os[1], os[2].center(14), compatible) print '%-16s | %12s | %s | %s' % (os[0], os[1], os[2].center(14), compatible)
slapos/cli/bang.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
from slapos.cli.config import ConfigCommand
from slapos.bang import do_bang
class BangCommand(ConfigCommand):
"""
request update on all partitions
"""
log = logging.getLogger('bang')
def get_parser(self, prog_name):
ap = super(BangCommand, self).get_parser(prog_name)
ap.add_argument('-m', '--message',
help='Message for bang')
return ap
def take_action(self, args):
configp = self.fetch_config(args)
do_bang(configp, args.message)
slapos/cli/cache.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
from slapos.cli.config import ConfigCommand
from slapos.cache import do_lookup
class CacheLookupCommand(ConfigCommand):
"""
perform a query to the networkcache
You can provide either a complete URL to the software release,
or a corresponding MD5 hash value.
The command will report which OS distribution/version have a binary
cache of the software release, and which ones are compatible
with the OS you are currently running.
"""
log = logging.getLogger('cache-lookup')
def get_parser(self, prog_name):
ap = super(CacheLookupCommand, self).get_parser(prog_name)
ap.add_argument('software_url',
help='Your software url or MD5 hash')
return ap
def take_action(self, args):
configp = self.fetch_config(args)
do_lookup(configp, args.software_url)
slapos/cli/command.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import argparse
import cliff
class Command(cliff.command.Command):
def get_parser(self, prog_name):
parser = argparse.ArgumentParser(
description=self.get_description(),
prog=prog_name,
formatter_class=argparse.RawDescriptionHelpFormatter
)
return parser
slapos/cli/config.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import ConfigParser
import os
from slapos.cli.command import Command
class ConfigError(Exception):
pass
class ConfigCommand(Command):
"Base class for commands that require a configuration file"
log = None
default_config_var = 'SLAPOS_CONFIGURATION'
# use this if default_config_var does not exist
default_config_path = '/etc/opt/slapos/slapos.cfg'
def get_parser(self, prog_name):
ap = super(ConfigCommand, self).get_parser(prog_name)
ap.add_argument('--cfg', help='SlapOS configuration file - ' +
'defaults to $%s ' % self.default_config_var +
'or %s' % self.default_config_path)
return ap
def _get_config(self, cfg_path, required=False):
"""
Returns a configuration object if file exists/readable/valid,
None otherwise.
Will raise an error instead of returning None if required is True.
Even if required is False, may still raise an exception from the
configparser if the configuration content is very broken.
We don't catch that exception as it will clearly show what is
wrong with the file.
"""
if not os.path.exists(cfg_path):
if required:
raise ConfigError('Configuration file does not exist: %s' % cfg_path)
else:
return None
configp = ConfigParser.SafeConfigParser()
if configp.read(cfg_path) != [cfg_path]:
# bad permission, etc.
if required:
raise ConfigError('Cannot parse configuration file: %s' % cfg_path)
else:
return None
return configp
def fetch_config(self, args):
if args.cfg:
cfg_path = args.cfg
else:
cfg_path = os.environ.get(self.default_config_var, self.default_config_path)
cfg_path = os.path.expanduser(cfg_path)
self.log.debug('Loading config: %s' % cfg_path)
return self._get_config(cfg_path, required=True)
class ClientConfigCommand(ConfigCommand):
default_config_var = 'SLAPOS_CLIENT_CONFIGURATION'
default_config_path = '~/.slapos/slapos-client.cfg'
slapos/cli/console.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
from slapos.cli.config import ClientConfigCommand
from slapos.client import init, do_console, ClientConfig
class ConsoleCommand(ClientConfigCommand):
"""
python console with slap library imported
You can play with the global "slap" object and
with the global "request" method.
examples :
>>> # Request instance
>>> request(kvm, "myuniquekvm")
>>> # Request software installation on owned computer
>>> supply(kvm, "mycomputer")
>>> # Fetch instance informations on already launched instance
>>> request(kvm, "myuniquekvm").getConnectionParameter("url")
"""
log = logging.getLogger('console')
def get_parser(self, prog_name):
ap = super(ConsoleCommand, self).get_parser(prog_name)
ap.add_argument('-u', '--master_url',
help='Url of SlapOS Master to use')
ap.add_argument('-k', '--key_file',
help='SSL Authorisation key file')
ap.add_argument('-c', '--cert_file',
help='SSL Authorisation certificate file')
return ap
def take_action(self, args):
configp = self.fetch_config(args)
conf = ClientConfig(args, configp)
local = init(conf)
do_console(local)
slapos/cli/entry.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
import sys
import cliff
import cliff.app
from cliff.app import LOG
import cliff.commandmanager
import slapos.version
class SlapOSCommandManager(cliff.commandmanager.CommandManager):
def find_command(self, argv):
"""Given an argument list, find a command and
return the processor and any remaining arguments.
"""
# a little cheating, 'slapos node' is not documented by the help command
if argv == ['node']:
argv = ['node', 'status']
search_args = argv[:]
name = ''
while search_args:
if search_args[0].startswith('-'):
raise ValueError('Invalid command %r' % search_args[0])
next_val = search_args.pop(0)
name = '%s %s' % (name, next_val) if name else next_val
if name in self.commands:
cmd_ep = self.commands[name]
cmd_factory = cmd_ep.load()
return (cmd_factory, name, search_args)
else:
print >>sys.stderr, ('The command %r does not exist or is not yet implemented.\n'
'Please have a look at http://community.slapos.org to read documentation or forum.\n'
'Please also make sure that SlapOS Node is up to date.' % (argv,))
sys.exit(5)
class SlapOSApp(cliff.app.App):
#
# self.options.verbose_level:
# -q -> 0 (WARNING)
# -> 1 (INFO)
# -v -> 2 (DEBUG)
# -vv -> 3 (...)
# etc.
#
log = logging.getLogger('slapos')
CONSOLE_MESSAGE_FORMAT = '%(message)s'
LOG_FILE_MESSAGE_FORMAT = '[%(asctime)s] %(levelname)-8s %(name)s %(message)s'
def __init__(self):
super(SlapOSApp, self).__init__(
description='SlapOS client %s' % slapos.version.version,
version=slapos.version.version,
command_manager=SlapOSCommandManager('slapos.cli'),
)
def build_option_parser(self, *args, **kw):
kw.setdefault('argparse_kwargs', {})
kw['argparse_kwargs']['conflict_handler'] = 'resolve'
parser = super(SlapOSApp, self).build_option_parser(*args, **kw)
# add two aliases for --log-file (for compatibility with old commands)
parser.add_argument(
'--log-file', '--logfile', '--log_file',
action='store',
default=None,
help='Specify a file to log output. Only console by default.',
)
# always show tracebacks on errors
parser.set_defaults(debug=True)
return parser
def initialize_app(self, argv):
if self.options.verbose_level > 2:
self.log.debug('initialize_app')
def prepare_to_run_command(self, cmd):
if self.options.verbose_level > 2:
self.log.debug('prepare_to_run_command %s', cmd.__class__.__name__)
def clean_up(self, cmd, result, err):
if self.options.verbose_level > 2:
self.log.debug('clean_up %s', cmd.__class__.__name__)
if err:
self.log.debug('got an error: %s', err)
def run(self, argv):
# same as cliff.App.run except that it won't re-raise
# a logged exception
try:
self.options, remainder = self.parser.parse_known_args(argv)
self.configure_logging()
self.interactive_mode = not remainder
self.initialize_app(remainder)
except Exception as err:
if hasattr(self, 'options'):
debug = self.options.debug
else:
debug = True
if debug:
LOG.exception(err)
# XXX change from cliff behaviour: avoid
# displaying the exception twice
# raise
else:
LOG.error(err)
return 1
result = 1
if self.interactive_mode:
result = self.interact()
else:
result = self.run_subcommand(remainder)
return result
def main(argv=sys.argv[1:]):
app = SlapOSApp()
if not argv:
argv = ['-h']
return app.run(argv)
if __name__ == '__main__':
sys.exit(main(sys.argv[1:]))
slapos/cli/format.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
import sys
from slapos.cli.config import ConfigCommand
from slapos.format import do_format, FormatConfig, tracing_monkeypatch, UsageError
class FormatCommand(ConfigCommand):
"""
create users, partitions and network configuration
"""
log = logging.getLogger('format')
def get_parser(self, prog_name):
ap = super(FormatCommand, self).get_parser(prog_name)
ap.add_argument('-x', '--computer_xml',
help="Path to file with computer's XML. If does not exists, will be created",
default=None)
ap.add_argument('--computer_json',
help="Path to a JSON version of the computer's XML (for development only).",
default=None)
ap.add_argument('-i', '--input_definition_file',
help="Path to file to read definition of computer instead of "
"declaration. Using definition file allows to disable "
"'discovery' of machine services and allows to define computer "
"configuration in fully controlled manner.")
ap.add_argument('-o', '--output_definition_file',
help="Path to file to write definition of computer from "
"declaration.")
ap.add_argument('-n', '--dry_run',
help="Don't actually do anything.",
default=False,
action="store_true")
ap.add_argument('--alter_user',
choices=['True', 'False'],
help="Shall slapformat alter user database [default: True]")
ap.add_argument('--alter_network',
choices=['True', 'False'],
help="Shall slapformat alter network configuration [default: True]")
ap.add_argument('--now',
help="Launch slapformat without delay",
default=False,
action="store_true")
return ap
def take_action(self, args):
configp = self.fetch_config(args)
conf = FormatConfig(logger=self.log)
conf.mergeConfig(args, configp)
if not self.app.options.log_file and conf.log_file:
# no log file is provided by argparser,
# we set up the one from config
file_handler = logging.FileHandler(conf.log_file)
formatter = logging.Formatter(self.app.LOG_FILE_MESSAGE_FORMAT)
file_handler.setFormatter(formatter)
self.log.addHandler(file_handler)
try:
conf.setConfig()
except UsageError as err:
sys.stderr.write(err.message + '\n')
sys.stderr.write("For help use --help\n")
sys.exit(1)
tracing_monkeypatch(conf)
try:
do_format(conf=conf)
except:
self.log.exception('Uncaught exception:')
raise
slapos/proxy/query.py → slapos/cli/proxy_show.py
View file @ 5f0924ff
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
# vim: set et sts=4:
import collections import collections
import ConfigParser import hashlib
from optparse import OptionParser, Option import logging
import sys
import lxml.etree import lxml.etree
import prettytable
import sqlite3 import sqlite3
from slapos.cli.config import ConfigCommand
from slapos.proxy import ProxyConfig
from slapos.proxy.db_version import DB_VERSION from slapos.proxy.db_version import DB_VERSION
class ProxyShowCommand(ConfigCommand):
class Parser(OptionParser):
""" """
Parse all arguments. display proxy instances and parameters
""" """
def __init__(self, usage=None, version=None):
""" log = logging.getLogger('proxy')
Initialize all options possibles.
""" def get_parser(self, prog_name):
OptionParser.__init__(self, usage=usage, version=version, ap = super(ProxyShowCommand, self).get_parser(prog_name)
option_list=[
Option("-u", "--database-uri", ap.add_argument('-u', '--database-uri',
type=str, help='URI for sqlite database')
help="URI for sqlite database"),
Option('--show-instances', ap.add_argument('--computers',
help="View instance information", help='view computer information',
default=False, action='store_true')
action="store_true"),
Option('--show-params', ap.add_argument('--software',
help="View published parameters", help='view software releases',
default=False, action='store_true')
action="store_true"),
Option('--show-network', ap.add_argument('--partitions',
help="View network information", help='view partitions',
default=False, action='store_true')
action="store_true"),
Option('--show-all', ap.add_argument('--slaves',
help="View all information", help='view slave instances',
default=False, action='store_true')
action="store_true"),
]) ap.add_argument('--params',
help='view published parameters',
def check_args(self): action='store_true')
"""
Check arguments ap.add_argument('--network',
""" help='view network settings',
(options, args) = self.parse_args() action='store_true')
if len(args) < 1:
self.error("Incorrect number of arguments") return ap
return options, args[0] def take_action(self, args):
configp = self.fetch_config(args)
class Config: conf = ProxyConfig(logger=self.log)
def setConfig(self, option_dict, configuration_file_path): conf.mergeConfig(args, configp)
""" conf.setConfig()
Set options given by parameters. do_show(conf=conf)
"""
# Set options parameters
for option, value in option_dict.__dict__.items():
setattr(self, option, value)
# Load configuration file
configuration_parser = ConfigParser.SafeConfigParser()
configuration_parser.read(configuration_file_path)
# Merges the arguments and configuration
for section in ("slapproxy", "slapos"):
configuration_dict = dict(configuration_parser.items(section))
for key in configuration_dict:
if not getattr(self, key, None):
setattr(self, key, configuration_dict[key])
if not self.database_uri:
raise ValueError('database-uri is required.')
tbl_computer = 'computer' + DB_VERSION
tbl_software = 'software' + DB_VERSION
tbl_partition = 'partition' + DB_VERSION tbl_partition = 'partition' + DB_VERSION
tbl_partition_network = 'partition_network' + DB_VERSION
tbl_slave = 'slave' + DB_VERSION
null_str = u"-"
def coalesce(*seq):
el = None
for el in seq:
if el is not None:
return el
return el
def print_table(qry, tablename, skip=None): def print_table(qry, tablename, skip=None):
...@@ -92,50 +76,33 @@ def print_table(qry, tablename, skip=None): ...@@ -92,50 +76,33 @@ def print_table(qry, tablename, skip=None):
skip = set() skip = set()
columns = [c[0] for c in qry.description if c[0] not in skip] columns = [c[0] for c in qry.description if c[0] not in skip]
rows = [] rows = []
for row in qry.fetchall(): for row in qry.fetchall():
line = {} rows.append([coalesce(row[col], '-') for col in columns])
for col in columns:
val = row[col]
if val is None:
val = null_str
line[col] = val.strip()
rows.append(line)
max_width = {col: len(col) for col in columns}
for row in rows:
for col in columns:
val = row[col]
max_width[col] = max(max_width[col], len(val) if val else 0)
hdr = [col.center(max_width[col]) for col in columns] pt = prettytable.PrettyTable(columns)
# https://code.google.com/p/prettytable/wiki/Tutorial
print for row in rows:
pt.add_row(row)
if rows: if rows:
print 'table %s:' % tablename, print 'table %s:' % tablename,
if skip:
print 'skipping %s' % ', '.join(skip)
else:
print
else: else:
print 'table %s: empty' % tablename print 'table %s: empty' % tablename
return return
if skip: print pt.get_string(border=True, padding_width=0, vrules=prettytable.NONE)
print 'skipping %s' % ', '.join(skip)
else:
print
print ' | '.join(hdr)
print '-+-'.join('-'*len(h) for h in hdr)
for row in rows:
cells = [row[col].ljust(max_width[col]) for col in columns]
print ' | '.join(cells)
def print_params(conn): def print_params(conn):
cur = conn.cursor() cur = conn.cursor()
print
qry = cur.execute("SELECT reference, partition_reference, software_type, connection_xml FROM %s" % tbl_partition) qry = cur.execute("SELECT reference, partition_reference, software_type, connection_xml FROM %s" % tbl_partition)
for row in qry.fetchall(): for row in qry.fetchall():
if not row['connection_xml']: if not row['connection_xml']:
...@@ -150,18 +117,19 @@ def print_params(conn): ...@@ -150,18 +117,19 @@ def print_params(conn):
if text and name in ('ssh-key', 'ssh-public-key'): if text and name in ('ssh-key', 'ssh-public-key'):
text = text[:20] + '...' + text[-20:] text = text[:20] + '...' + text[-20:]
print ' %s = %s' % (name, text) print ' %s = %s' % (name, text)
print
def print_computer_table(conn): def print_computer_table(conn):
tbl_computer = 'computer' + DB_VERSION
cur = conn.cursor() cur = conn.cursor()
qry = cur.execute("SELECT * FROM %s" % tbl_computer) qry = cur.execute("SELECT * FROM %s" % tbl_computer)
print_table(qry, tbl_computer) print_table(qry, tbl_computer)
def print_software_table(conn): def print_software_table(conn):
tbl_software = 'software' + DB_VERSION
cur = conn.cursor() cur = conn.cursor()
qry = cur.execute("SELECT * FROM %s" % tbl_software) qry = cur.execute("SELECT *, md5(url) as md5 FROM %s" % tbl_software)
print_table(qry, tbl_software) print_table(qry, tbl_software)
...@@ -170,21 +138,16 @@ def print_partition_table(conn): ...@@ -170,21 +138,16 @@ def print_partition_table(conn):
qry = cur.execute("SELECT * FROM %s WHERE slap_state<>'free'" % tbl_partition) qry = cur.execute("SELECT * FROM %s WHERE slap_state<>'free'" % tbl_partition)
print_table(qry, tbl_partition, skip=['xml', 'connection_xml', 'slave_instance_list']) print_table(qry, tbl_partition, skip=['xml', 'connection_xml', 'slave_instance_list'])
def print_slave_table(conn): def print_slave_table(conn):
tbl_slave = 'slave' + DB_VERSION
cur = conn.cursor() cur = conn.cursor()
qry = cur.execute("SELECT * FROM %s" % tbl_slave) qry = cur.execute("SELECT * FROM %s" % tbl_slave)
print_table(qry, tbl_slave, skip=['connection_xml']) print_table(qry, tbl_slave, skip=['connection_xml'])
def print_tables(conn):
print_computer_table(conn)
print_software_table(conn)
print_partition_table(conn)
print_slave_table(conn)
def print_network(conn): def print_network(conn):
print tbl_partition_network = 'partition_network' + DB_VERSION
cur = conn.cursor() cur = conn.cursor()
addr = collections.defaultdict(list) addr = collections.defaultdict(list)
qry = cur.execute(""" qry = cur.execute("""
...@@ -202,45 +165,40 @@ def print_network(conn): ...@@ -202,45 +165,40 @@ def print_network(conn):
print '%s: %s' % (partition_reference, ', '.join(addresses)) print '%s: %s' % (partition_reference, ', '.join(addresses))
def do_show(conf):
conn = sqlite3.connect(conf.database_uri)
def run(config):
conn = sqlite3.connect(config.database_uri)
conn.row_factory = sqlite3.Row conn.row_factory = sqlite3.Row
fn = [] conn.create_function('md5', 1, lambda s: hashlib.md5(s).hexdigest())
if config.show_all or config.show_instances: print_all = not any(
fn.append(print_tables) [
if config.show_all or config.show_params: conf.computers,
fn.append(print_params) conf.software,
if config.show_all or config.show_network: conf.partitions,
fn.append(print_network) conf.slaves,
conf.params,
if fn: conf.network,
for f in fn: ]
f(conn) )
else:
print 'usage: %s [ --show-params | --show-network | --show-instances | --show-all ]' % sys.argv[0] if print_all or conf.computers:
print_computer_table(conn)
print
if print_all or conf.software:
def main(): print_software_table(conn)
"Run default configuration." print
usage = "usage: %s [options] CONFIGURATION_FILE" % sys.argv[0] if print_all or conf.partitions:
print_partition_table(conn)
try: print
# Parse arguments if print_all or conf.slaves:
config = Config() print_slave_table(conn)
config.setConfig(*Parser(usage=usage).check_args()) print
if print_all or conf.params:
run(config) print_params(conn)
return_code = 0 print
except SystemExit, err: if print_all or conf.network:
# Catch exception raise by optparse print_network(conn)
return_code = err print
sys.exit(return_code)
slapos/cli/proxy_start.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
from slapos.cli.config import ConfigCommand
from slapos.proxy import do_proxy, ProxyConfig
class ProxyStartCommand(ConfigCommand):
"""
minimalist, stand-alone SlapOS Master
"""
log = logging.getLogger('proxy')
def get_parser(self, prog_name):
ap = super(ProxyStartCommand, self).get_parser(prog_name)
ap.add_argument('-u', '--database-uri',
help='URI for sqlite database')
return ap
def take_action(self, args):
configp = self.fetch_config(args)
conf = ProxyConfig(logger=self.log)
conf.mergeConfig(args, configp)
if not self.app.options.log_file and hasattr(conf, 'log_file'):
# no log file is provided by argparser,
# we set up the one from config
file_handler = logging.FileHandler(conf.log_file)
formatter = logging.Formatter(self.app.LOG_FILE_MESSAGE_FORMAT)
file_handler.setFormatter(formatter)
self.log.addHandler(file_handler)
conf.setConfig()
do_proxy(conf=conf)
slapos/cli/register.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
import sys
from slapos.cli.command import Command
from slapos.register.register import do_register, RegisterConfig
class RegisterCommand(Command):
"""
register a node in the SlapOS cloud
"""
log = logging.getLogger('register')
def get_parser(self, prog_name):
ap = super(RegisterCommand, self).get_parser(prog_name)
ap.add_argument('node_name',
help='Name of the node')
ap.add_argument('--interface-name',
help='Interface name to access internet',
default='eth0')
ap.add_argument('--master-url',
help='URL of SlapOS master',
default='https://slap.vifib.com')
ap.add_argument('--master-url-web',
help='URL of SlapOS Master webservice to register certificates',
default='https://www.slapos.org')
ap.add_argument('--partition-number',
help='Number of partition on computer',
default='10',
type=int)
ap.add_argument('--ipv4-local-network',
help='Base of ipv4 local network',
default='10.0.0.0/16')
ap.add_argument('--ipv6-interface',
help='Interface name to get ipv6',
default='')
ap.add_argument('--login',
help='User login on SlapOS Master webservice')
ap.add_argument('--password',
help='User password on SlapOs Master webservice')
ap.add_argument('-t', '--create-tap',
help='Will trigger creation of one virtual "tap" interface per '
'Partition and attach it to primary interface. Requires '
'primary interface to be a bridge. defaults to false. '
'Needed to host virtual machines.',
default=False,
action='store_true')
ap.add_argument('-n', '--dry-run',
help='Simulate the execution steps',
default=False,
action='store_true')
return ap
def take_action(self, args):
try:
conf = RegisterConfig(logger=self.log)
conf.setConfig(args)
return_code = do_register(conf)
except SystemExit as err:
return_code = err
sys.exit(return_code)
slapos/cli/remove.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
from slapos.cli.config import ClientConfigCommand
from slapos.client import init, do_remove, ClientConfig
class RemoveCommand(ClientConfigCommand):
"""
remove a Software from a node
"""
log = logging.getLogger('remove')
def get_parser(self, prog_name):
ap = super(RemoveCommand, self).get_parser(prog_name)
ap.add_argument('software_url',
help='Your software url')
ap.add_argument('node',
help="Target node")
return ap
def take_action(self, args):
configp = self.fetch_config(args)
conf = ClientConfig(args, configp)
local = init(conf)
do_remove(args.software_url, args.node, local)
slapos/cli/request.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
from slapos.cli.config import ClientConfigCommand
from slapos.client import init, do_request, ClientConfig
def parse_option_dict(options):
"""
Parse a list of option strings like foo=bar baz=qux and return a dictionary.
Will raise if keys are repeated.
"""
ret = {}
for option_pair in (options or []):
key, value = option_pair.split('=', 1)
if key in ret:
raise ValueError("Multiple values provided for the same key '%s'" % key)
ret[key] = value
return ret
class RequestCommand(ClientConfigCommand):
"""request an instance and get status and parameters of instance"""
log = logging.getLogger('request')
def get_parser(self, prog_name):
ap = super(RequestCommand, self).get_parser(prog_name)
ap.add_argument('reference',
help='Your instance reference')
ap.add_argument('software_url',
help='Your software url')
# XXX TODO can this be an MD5 hash?
# XXX TODO can we do a minimal check for correctness of this argument?
# the alternative is a silent failure for mistyped/obsolete/invalid URL
ap.add_argument('--node',
nargs='+',
help="Node request option 'option1=value1 option2=value2'")
ap.add_argument('--type',
help='Software type to be requested')
ap.add_argument('--state',
help='State of the requested instance')
ap.add_argument('--slave',
action='store_true',
help='Ask for a slave instance')
ap.add_argument('--parameters',
nargs='+',
help="Give your configuration 'option1=value1 option2=value2'")
return ap
def take_action(self, args):
args.node = parse_option_dict(args.node)
args.parameters = parse_option_dict(args.parameters)
configp = self.fetch_config(args)
conf = ClientConfig(args, configp)
local = init(conf)
do_request(conf, local)
slapos/cli/slapgrid.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
from slapos.cli.config import ConfigCommand
from slapos.grid.utils import setRunning, setFinished
from slapos.grid.slapgrid import (merged_options, check_missing_parameters, check_missing_files,
random_delay, create_slapgrid_object)
class SlapgridCommand(ConfigCommand):
log = None
method_name = NotImplemented
default_pidfile = NotImplemented
def get_parser(self, prog_name):
ap = super(SlapgridCommand, self).get_parser(prog_name)
# TODO move more options to the instance, software and report subclasses
ap.add_argument('--instance-root',
help='The instance root directory location.')
ap.add_argument('--software-root',
help='The software_root directory location.')
ap.add_argument('--master-url',
help='The master server URL. Mandatory.')
ap.add_argument('--computer-id',
help='The computer id defined in the server.')
ap.add_argument('--supervisord-socket',
help='The socket supervisor will use.')
ap.add_argument('--supervisord-configuration-path',
help='The location where supervisord configuration will be stored.')
ap.add_argument('--buildout', default=None,
help='Location of buildout binary.')
ap.add_argument('--pidfile',
help='The location where pidfile will be created. '
'Can be provided by configuration file, or defaults '
'to %s' % self.default_pidfile)
ap.add_argument('--key_file',
help='SSL Authorisation key file.')
ap.add_argument('--cert_file',
help='SSL Authorisation certificate file.')
ap.add_argument('--signature_private_key_file',
help='Signature private key file.')
ap.add_argument('--master_ca_file',
help='Root certificate of SlapOS master key.')
ap.add_argument('--certificate_repository_path',
help='Path to directory where downloaded certificates would be stored.')
ap.add_argument('--maximum-periodicity', type=int, default=None,
help='Periodicity at which buildout should be run in instance.')
ap.add_argument('--promise-timeout', type=int, default=3,
help='Promise timeout in seconds.')
ap.add_argument('--now', action='store_true',
help='Launch slapgrid without delay. Default behavior.')
return ap
def take_action(self, args):
configp = self.fetch_config(args)
options = merged_options(args, configp)
check_missing_parameters(options)
check_missing_files(options)
random_delay(options, logger=self.log)
slapgrid_object = create_slapgrid_object(options, logger=self.log)
pidfile = options.get('pidfile') or self.default_pidfile
if pidfile:
setRunning(logger=self.log, pidfile=pidfile)
try:
return getattr(slapgrid_object, self.method_name)()
finally:
if pidfile:
setFinished(pidfile)
class SoftwareCommand(SlapgridCommand):
"""run software installation/deletion"""
log = logging.getLogger('software')
method_name = 'processSoftwareReleaseList'
default_pidfile = '/opt/slapos/slapgrid-sr.pid'
def get_parser(self, prog_name):
ap = super(SoftwareCommand, self).get_parser(prog_name)
only = ap.add_mutually_exclusive_group()
only.add_argument('--all', action='store_true',
help='Process all Software Releases, even if already installed.')
only.add_argument('--only-sr', '--only',
help='Force the update of a single software release (can be full URL or MD5 hash), '
'even if is already installed. This option will make all other '
'sofware releases be ignored.')
return ap
class InstanceCommand(SlapgridCommand):
"""run instance deployment"""
log = logging.getLogger('instance')
method_name = 'processComputerPartitionList'
default_pidfile = '/opt/slapos/slapgrid-cp.pid'
def get_parser(self, prog_name):
ap = super(InstanceCommand, self).get_parser(prog_name)
only = ap.add_mutually_exclusive_group()
only.add_argument('--all', action='store_true',
help='Process all Computer Partitions.')
only.add_argument('--only-cp', '--only',
help='Update a single or a list of computer partitions '
'(ie.:slappartX, slappartY), '
'this option will make all other computer partitions be ignored.')
return ap
class ReportCommand(SlapgridCommand):
"""run instance reports and garbage collection"""
log = logging.getLogger('report')
method_name = 'agregateAndSendUsage'
default_pidfile = '/opt/slapos/slapgrid-ur.pid'
slapos/cli/supervisorctl.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import argparse
import logging
import os
from slapos.cli.config import ConfigCommand
from slapos.grid.svcbackend import launchSupervisord
import supervisor.supervisorctl
class SupervisorctlCommand(ConfigCommand):
"""enter into supervisor console, for process management"""
log = logging.getLogger('supervisorctl')
def get_parser(self, prog_name):
ap = super(SupervisorctlCommand, self).get_parser(prog_name)
ap.add_argument('supervisor_args',
nargs=argparse.REMAINDER,
help='parameters passed to supervisorctl')
return ap
def take_action(self, args):
configp = self.fetch_config(args)
instance_root = configp.get('slapos', 'instance_root')
configuration_file = os.path.join(instance_root, 'etc', 'supervisord.conf')
launchSupervisord(socket=os.path.join(instance_root, 'supervisord.socket'),
configuration_file=configuration_file,
logger=self.log)
supervisor.supervisorctl.main(args=['-c', configuration_file] + args.supervisor_args)
class SupervisorctlAliasCommand(SupervisorctlCommand):
def take_action(self, args):
args.supervisor_args = [self.alias] + args.supervisor_args
super(SupervisorctlAliasCommand, self).take_action(args)
class SupervisorctlStatusCommand(SupervisorctlAliasCommand):
"""alias for 'node supervisorctl status'"""
alias = 'status'
class SupervisorctlStartCommand(SupervisorctlAliasCommand):
"""alias for 'node supervisorctl start'"""
alias = 'start'
class SupervisorctlStopCommand(SupervisorctlAliasCommand):
"""alias for 'node supervisorctl stop'"""
alias = 'stop'
class SupervisorctlRestartCommand(SupervisorctlAliasCommand):
"""alias for 'node supervisorctl restart'"""
alias = 'restart'
class SupervisorctlTailCommand(SupervisorctlAliasCommand):
"""alias for 'node supervisorctl tail'"""
alias = 'tail'
slapos/cli/supervisord.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
import os
from slapos.cli.config import ConfigCommand
from slapos.grid.svcbackend import launchSupervisord
class SupervisordCommand(ConfigCommand):
"""launch, if not already running, supervisor daemon"""
log = logging.getLogger('supervisord')
def take_action(self, args):
configp = self.fetch_config(args)
instance_root = configp.get('slapos', 'instance_root')
launchSupervisord(socket=os.path.join(instance_root, 'supervisord.socket'),
configuration_file=os.path.join(instance_root, 'etc', 'supervisord.conf'),
logger=self.log)
slapos/cli/supply.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
from slapos.cli.config import ClientConfigCommand
from slapos.client import init, do_supply, ClientConfig
class SupplyCommand(ClientConfigCommand):
"""
supply a Software to a node
"""
log = logging.getLogger('supply')
def get_parser(self, prog_name):
ap = super(SupplyCommand, self).get_parser(prog_name)
ap.add_argument('software_url',
help='Your software url')
ap.add_argument('node',
help="Target node")
return ap
def take_action(self, args):
configp = self.fetch_config(args)
conf = ClientConfig(args, configp)
local = init(conf)
do_supply(args.software_url, args.node, local)
slapos/cli_legacy/bang.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
##############################################################################
#
# Copyright (c) 2011, 2012 Vifib SARL and Contributors.
# All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly advised to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import argparse
import ConfigParser
from slapos.bang import do_bang
def main(*args):
ap = argparse.ArgumentParser()
ap.add_argument('-m', '--message', default='', help='Message for bang.')
ap.add_argument('configuration_file', type=argparse.FileType(),
help='SlapOS configuration file.')
if args:
args = ap.parse_args(list(args))
else:
args = ap.parse_args()
configp = ConfigParser.SafeConfigParser()
configp.readfp(args.configuration_file)
do_bang(configp, args.message)
slapos/cli_legacy/cache.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import argparse
import ConfigParser
from slapos.cache import do_lookup
def cache_lookup():
ap = argparse.ArgumentParser()
ap.add_argument("configuration_file", help="SlapOS configuration file")
ap.add_argument("software_url", help="Your software url or MD5 hash")
args = ap.parse_args()
configp = ConfigParser.SafeConfigParser()
configp.read(args.configuration_file)
do_lookup(configp, args.software_url)
slapos/cli_legacy/console.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import argparse
import os
import textwrap
from slapos.client import ClientConfig, init, do_console
from slapos.cli_legacy.util import get_config_parser
def console():
description = textwrap.dedent("""\
slapconsole allows you interact with slap API. You can play with the global
"slap" object and with the global "request" method.
examples :
>>> # Request instance
>>> request(kvm, "myuniquekvm")
>>> # Request software installation on owned computer
>>> supply(kvm, "mycomputer")
>>> # Fetch instance informations on already launched instance
>>> request(kvm, "myuniquekvm").getConnectionParameter("url")""")
ap = argparse.ArgumentParser(description=description,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument('-u', '--master_url',
default=None,
help='Url of SlapOS Master to use.')
ap.add_argument('-k', '--key_file',
help="SSL Authorisation key file.")
ap.add_argument('-c', '--cert_file',
help="SSL Authorisation certificate file.")
ap.add_argument('configuration_file',
help='path to slapos.cfg')
args = ap.parse_args()
if not os.path.isfile(args.configuration_file):
ap.error("%s: Not found or not a regular file." % args.configuration_file)
configp = get_config_parser(args.configuration_file)
conf = ClientConfig(args, configp)
local = init(conf)
do_console(local)
slapos/entry.py → slapos/cli_legacy/entry.py
View file @ 5f0924ff
...@@ -30,19 +30,20 @@ import argparse ...@@ -30,19 +30,20 @@ import argparse
import ConfigParser import ConfigParser
import os import os
import sys import sys
from slapos.bang import main as bang
from slapos.client import slapconsole as console from slapos.cli_legacy.bang import main as bang
from slapos.client import request as request from slapos.cli_legacy.console import console
from slapos.client import remove as remove from slapos.cli_legacy.request import request
from slapos.client import supply as supply from slapos.cli_legacy.remove import remove
from slapos.format import main as format from slapos.cli_legacy.supply import supply
from slapos.cache import cache_lookup from slapos.cli_legacy.format import main as format
from slapos.grid.slapgrid import runComputerPartition as instance from slapos.cli_legacy.cache import cache_lookup
from slapos.grid.slapgrid import runSoftwareRelease as software from slapos.cli_legacy.slapgrid import runComputerPartition as instance
from slapos.grid.slapgrid import runUsageReport as report from slapos.cli_legacy.slapgrid import runSoftwareRelease as software
from slapos.grid.svcbackend import supervisord from slapos.cli_legacy.slapgrid import runUsageReport as report
from slapos.grid.svcbackend import supervisorctl from slapos.cli_legacy.svcbackend import supervisord
from slapos.register.register import main as register from slapos.cli_legacy.svcbackend import supervisorctl
from slapos.cli_legacy.register import main as register
from slapos.version import version from slapos.version import version
# Note: this whole file is a hack. We should better try dedicated library # Note: this whole file is a hack. We should better try dedicated library
...@@ -60,6 +61,7 @@ class EntryPointNotImplementedError(NotImplementedError): ...@@ -60,6 +61,7 @@ class EntryPointNotImplementedError(NotImplementedError):
def __init__(self, *args, **kw_args): def __init__(self, *args, **kw_args):
NotImplementedError.__init__(self, *args, **kw_args) NotImplementedError.__init__(self, *args, **kw_args)
def checkSlaposCfg(): def checkSlaposCfg():
""" """
Check if a slapos configuration file was given as a argument. Check if a slapos configuration file was given as a argument.
...@@ -71,12 +73,13 @@ def checkSlaposCfg(): ...@@ -71,12 +73,13 @@ def checkSlaposCfg():
for element in sys.argv: for element in sys.argv:
if '.cfg' in element: if '.cfg' in element:
if os.path.exists(element): if os.path.exists(element):
configuration = ConfigParser.SafeConfigParser() configp = ConfigParser.SafeConfigParser()
configuration.read(element) configp.read(element)
if configuration.has_section('slapos'): if configp.has_section('slapos'):
return True return True
return False return False
def checkOption(option): def checkOption(option):
""" """
Check if a given option is already in call line Check if a given option is already in call line
...@@ -88,11 +91,12 @@ def checkOption(option): ...@@ -88,11 +91,12 @@ def checkOption(option):
if key in element: if key in element:
return True return True
sys.argv.append(key) sys.argv.append(key)
if len(option) > 1 : if len(option) > 1:
sys.argv = sys.argv + option[1:] sys.argv = sys.argv + option[1:]
return True return True
def call(fun, config=False, option=None):
def call(fun, config_path=False, option=None):
""" """
Add missing options to sys.argv Add missing options to sys.argv
Add config if asked and it is missing Add config if asked and it is missing
...@@ -102,12 +106,13 @@ def call(fun, config=False, option=None): ...@@ -102,12 +106,13 @@ def call(fun, config=False, option=None):
option = [] option = []
for element in option: for element in option:
checkOption(element) checkOption(element)
if config: if config_path:
if not checkSlaposCfg(): if not checkSlaposCfg():
sys.argv = [sys.argv[0]] + [os.path.expanduser(config)] + sys.argv[1:] sys.argv = [sys.argv[0]] + [os.path.expanduser(config_path)] + sys.argv[1:]
fun() fun()
sys.exit(0) sys.exit(0)
def dispatch(command, is_node_command): def dispatch(command, is_node_command):
""" Dispatch to correct SlapOS module. """ Dispatch to correct SlapOS module.
Here we could use introspection to get rid of the big "if" statements, Here we could use introspection to get rid of the big "if" statements,
...@@ -124,34 +129,34 @@ def dispatch(command, is_node_command): ...@@ -124,34 +129,34 @@ def dispatch(command, is_node_command):
if command == 'register': if command == 'register':
call(register) call(register)
elif command == 'software': elif command == 'software':
call(software, config=GLOBAL_SLAPOS_CONFIGURATION, call(software, config_path=GLOBAL_SLAPOS_CONFIGURATION,
option=['--pidfile /opt/slapos/slapgrid-sr.pid']) option=['--pidfile /opt/slapos/slapgrid-sr.pid'])
elif command == 'instance': elif command == 'instance':
call(instance, config=GLOBAL_SLAPOS_CONFIGURATION, call(instance, config_path=GLOBAL_SLAPOS_CONFIGURATION,
option=['--pidfile /opt/slapos/slapgrid-cp.pid']) option=['--pidfile /opt/slapos/slapgrid-cp.pid'])
elif command == 'report': elif command == 'report':
call(report, config=GLOBAL_SLAPOS_CONFIGURATION, call(report, config_path=GLOBAL_SLAPOS_CONFIGURATION,
option=['--pidfile /opt/slapos/slapgrid-ur.pid']) option=['--pidfile /opt/slapos/slapgrid-ur.pid'])
elif command == 'bang': elif command == 'bang':
call(bang, config=GLOBAL_SLAPOS_CONFIGURATION) call(bang, config_path=GLOBAL_SLAPOS_CONFIGURATION)
elif command == 'format': elif command == 'format':
call(format, config=GLOBAL_SLAPOS_CONFIGURATION, option=['-c', '-v']) call(format, config_path=GLOBAL_SLAPOS_CONFIGURATION, option=['-c', '-v'])
elif command == 'supervisord': elif command == 'supervisord':
call(supervisord, config=GLOBAL_SLAPOS_CONFIGURATION) call(supervisord, config_path=GLOBAL_SLAPOS_CONFIGURATION)
elif command == 'supervisorctl': elif command == 'supervisorctl':
call(supervisorctl, config=GLOBAL_SLAPOS_CONFIGURATION) call(supervisorctl, config_path=GLOBAL_SLAPOS_CONFIGURATION)
elif command in ['start', 'stop', 'restart', 'status', 'tail']: elif command in ['start', 'stop', 'restart', 'status', 'tail']:
# Again, too hackish # Again, too hackish
sys.argv[-2:-2] = [command] sys.argv[-2:-2] = [command]
call(supervisorctl, config=GLOBAL_SLAPOS_CONFIGURATION) call(supervisorctl, config_path=GLOBAL_SLAPOS_CONFIGURATION)
else: else:
return False return False
elif command == 'request': elif command == 'request':
call(request, config=USER_SLAPOS_CONFIGURATION) call(request, config_path=USER_SLAPOS_CONFIGURATION)
elif command == 'supply': elif command == 'supply':
call(supply, config=USER_SLAPOS_CONFIGURATION) call(supply, config_path=USER_SLAPOS_CONFIGURATION)
elif command == 'remove': elif command == 'remove':
call(remove, config=USER_SLAPOS_CONFIGURATION) call(remove, config_path=USER_SLAPOS_CONFIGURATION)
elif command == 'start': elif command == 'start':
raise EntryPointNotImplementedError(command) raise EntryPointNotImplementedError(command)
elif command == 'stop': elif command == 'stop':
...@@ -159,12 +164,13 @@ def dispatch(command, is_node_command): ...@@ -159,12 +164,13 @@ def dispatch(command, is_node_command):
elif command == 'destroy': elif command == 'destroy':
raise EntryPointNotImplementedError(command) raise EntryPointNotImplementedError(command)
elif command == 'console': elif command == 'console':
call(console, config=USER_SLAPOS_CONFIGURATION) call(console, config_path=USER_SLAPOS_CONFIGURATION)
elif command == 'cache-lookup': elif command == 'cache-lookup':
call(cache_lookup, config=GLOBAL_SLAPOS_CONFIGURATION) call(cache_lookup, config_path=GLOBAL_SLAPOS_CONFIGURATION)
else: else:
return False return False
def main(): def main():
""" """
Main entry point of SlapOS Node. Used to dispatch commands to python Main entry point of SlapOS Node. Used to dispatch commands to python
...@@ -206,24 +212,23 @@ Node subcommands usage: ...@@ -206,24 +212,23 @@ Node subcommands usage:
# Parse arguments # Parse arguments
# XXX remove the "positional arguments" from help message # XXX remove the "positional arguments" from help message
parser = argparse.ArgumentParser(usage=usage) ap = argparse.ArgumentParser(usage=usage)
parser.add_argument('command') ap.add_argument('command')
parser.add_argument('argument_list', nargs=argparse.REMAINDER) ap.add_argument('argument_list', nargs=argparse.REMAINDER)
namespace = parser.parse_args() args = ap.parse_args()
# Set sys.argv for the sub-entry point that we will call # Set sys.argv for the sub-entry point that we will call
command_line = [namespace.command] command_line = [args.command]
command_line.extend(namespace.argument_list) command_line.extend(args.argument_list)
sys.argv = command_line sys.argv = command_line
try: try:
if not dispatch(namespace.command, is_node): if not dispatch(args.command, is_node):
parser.print_help() ap.print_help()
sys.exit(1) sys.exit(1)
except EntryPointNotImplementedError, exception: except EntryPointNotImplementedError, exception:
print ('The command %s does not exist or is not yet implemented. Please ' print ('The command %s does not exist or is not yet implemented. Please '
'have a look at http://community.slapos.org to read documentation or ' 'have a look at http://community.slapos.org to read documentation or '
'forum. Please also make sure that SlapOS Node is up to ' 'forum. Please also make sure that SlapOS Node is up to '
'date.' % exception) 'date.' % exception)
sys.exit(1) sys.exit(1)
slapos/cli_legacy/format.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
##############################################################################
#
# Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors.
# All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly advised to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import argparse
import ConfigParser
import logging
import sys
import os
from slapos.format import FormatConfig, UsageError, tracing_monkeypatch, do_format
def main(*args):
"Run default configuration."
ap = argparse.ArgumentParser()
ap.add_argument('-x', '--computer_xml',
help="Path to file with computer's XML. If does not exists, will be created",
default=None)
ap.add_argument('--computer_json',
help="Path to a JSON version of the computer's XML (for development only).",
default=None)
ap.add_argument('-l', '--log_file',
help="The path to the log file used by the script.")
ap.add_argument('-i', '--input_definition_file',
help="Path to file to read definition of computer instead of "
"declaration. Using definition file allows to disable "
"'discovery' of machine services and allows to define computer "
"configuration in fully controlled manner.")
ap.add_argument('-o', '--output_definition_file',
help="Path to file to write definition of computer from "
"declaration.")
ap.add_argument('-n', '--dry_run',
help="Don't actually do anything.",
default=False,
action="store_true")
ap.add_argument('-v', '--verbose',
default=False,
action="store_true",
help="Verbose output.")
# the console option is actually ignored and not used anymore.
ap.add_argument('-c', '--console',
default=False,
action="store_true",
help="Console output.")
ap.add_argument('--alter_user',
choices=['True', 'False'],
help="Shall slapformat alter user database [default: True]")
ap.add_argument('--alter_network',
choices=['True', 'False'],
help="Shall slapformat alter network configuration [default: True]")
ap.add_argument('--now',
help="Launch slapformat without delay",
default=False,
action="store_true")
ap.add_argument('configuration_file',
help='path to slapos.cfg')
if args:
args = ap.parse_args(list(args))
else:
args = ap.parse_args()
logger = logging.getLogger("slapformat")
logger.addHandler(logging.StreamHandler())
if args.verbose:
logger.setLevel(logging.DEBUG)
logger.debug("Verbose mode enabled.")
else:
logger.setLevel(logging.INFO)
conf = FormatConfig(logger=logger)
configp = ConfigParser.SafeConfigParser()
if configp.read(args.configuration_file) != [args.configuration_file]:
raise UsageError('Cannot find or parse configuration file: %s' % args.configuration_file)
conf.mergeConfig(args, configp)
if conf.log_file:
if not os.path.isdir(os.path.dirname(conf.log_file)):
# fallback to console only if directory for logs does not exists and
# continue to run
raise ValueError('Please create directory %r to store %r log file' % (
os.path.dirname(conf.log_file), conf.log_file))
else:
file_handler = logging.FileHandler(conf.log_file)
file_handler.setFormatter(logging.Formatter("%(asctime)s - "
"%(name)s - %(levelname)s - %(message)s"))
conf.logger.addHandler(file_handler)
conf.logger.info('Configured logging to file %r' % conf.log_file)
try:
conf.setConfig()
except UsageError as exc:
sys.stderr.write(exc.message + '\n')
sys.stderr.write("For help use --help\n")
sys.exit(1)
tracing_monkeypatch(conf)
try:
do_format(conf=conf)
except:
conf.logger.exception('Uncaught exception:')
raise
slapos/cli_legacy/proxy_start.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
# vim: set et sts=2:
import argparse
import ConfigParser
import logging
import os
import sys
from slapos.proxy import ProxyConfig, do_proxy
class UsageError(Exception):
pass
def main():
ap = argparse.ArgumentParser()
ap.add_argument('-l', '--log_file',
help='The path to the log file used by the script.')
ap.add_argument('-v', '--verbose',
action='store_true',
help='Verbose output.')
# XXX not used anymore, deprecated
ap.add_argument('-c', '--console',
action='store_true',
help='Console output.')
ap.add_argument('-u', '--database-uri',
help='URI for sqlite database')
ap.add_argument('configuration_file',
help='path to slapos.cfg')
args = ap.parse_args()
logger = logging.getLogger('slapproxy')
logger.addHandler(logging.StreamHandler())
if args.verbose:
logger.setLevel(logging.DEBUG)
else:
logger.setLevel(logging.INFO)
conf = ProxyConfig(logger=logger)
configp = ConfigParser.SafeConfigParser()
if configp.read(args.configuration_file) != [args.configuration_file]:
raise UsageError('Cannot find or parse configuration file: %s' % args.configuration_file)
conf.mergeConfig(args, configp)
if conf.log_file:
if not os.path.isdir(os.path.dirname(conf.log_file)):
raise ValueError('Please create directory %r to store %r log file' % (
os.path.dirname(conf.log_file), conf.log_file))
file_handler = logging.FileHandler(conf.log_file)
file_handler.setFormatter(logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s"))
logger.addHandler(file_handler)
logger.info('Configured logging to file %r' % conf.log_file)
conf.setConfig()
try:
do_proxy(conf=conf)
return_code = 0
except SystemExit as err:
return_code = err
sys.exit(return_code)
slapos/cli_legacy/register.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
# vim: set et sts=2:
##############################################################################
#
# Copyright (c) 2012 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly advised to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import argparse
import logging
import sys
from slapos.register.register import do_register, RegisterConfig
def main():
ap = argparse.ArgumentParser()
ap.add_argument('node_name',
help='Name of the node')
ap.add_argument('--interface-name',
help='Interface name to access internet',
default='eth0')
ap.add_argument('--master-url',
help='URL of SlapOS master',
default='https://slap.vifib.com')
ap.add_argument('--master-url-web',
help='URL of SlapOS Master webservice to register certificates',
default='https://www.slapos.org')
ap.add_argument('--partition-number',
help='Number of partition on computer',
default='10',
type=int)
ap.add_argument('--ipv4-local-network',
help='Base of ipv4 local network',
default='10.0.0.0/16')
ap.add_argument('--ipv6-interface',
help='Interface name to get ipv6',
default='')
ap.add_argument('--login',
help='User login on SlapOS Master webservice')
ap.add_argument('--password',
help='User password on SlapOs Master webservice')
ap.add_argument('-t', '--create-tap',
help='Will trigger creation of one virtual "tap" interface per '
'Partition and attach it to primary interface. Requires '
'primary interface to be a bridge. defaults to false. '
'Needed to host virtual machines.',
default=False,
action='store_true')
ap.add_argument('-n', '--dry-run',
help='Simulate the execution steps',
default=False,
action='store_true')
args = ap.parse_args()
if args.password and not args.login:
ap.error('Please enter your login with your password')
logger = logging.getLogger('Register')
handler = logging.StreamHandler()
logger.setLevel(logging.DEBUG)
handler.setLevel(logging.INFO)
handler.setFormatter(logging.Formatter('%(levelname)s - %(message)s'))
logger.addHandler(handler)
try:
conf = RegisterConfig(logger=logger)
conf.setConfig(args)
return_code = do_register(conf)
except SystemExit as exc:
return_code = exc
sys.exit(return_code)
slapos/cli_legacy/remove.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import argparse
from slapos.client import ClientConfig, init, do_remove
from slapos.cli_legacy.util import get_config_parser
def remove():
ap = argparse.ArgumentParser()
ap.add_argument('configuration_file',
help='SlapOS configuration file')
ap.add_argument('software_url',
help='Your software url')
ap.add_argument('node',
help='Target node')
args = ap.parse_args()
configp = get_config_parser(args.configuration_file)
conf = ClientConfig(args, configp)
local = init(conf)
do_remove(args.software_url, args.node, local)
slapos/cli_legacy/request.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import argparse
from slapos.client import ClientConfig, init, do_request
from slapos.cli_legacy.util import get_config_parser
def argToDict(element):
"""
convert a table of string 'key=value' to dict
"""
if element is not None:
element_dict = dict([arg.split('=') for arg in element])
return element_dict
def request():
"""Run when invoking slapos request. Request an instance."""
# Parse arguments and inititate needed parameters
# XXX-Cedric: move argument parsing to main entry point
ap = argparse.ArgumentParser()
ap.add_argument('configuration_file',
help='SlapOS configuration file.')
ap.add_argument('reference',
help='Your instance reference')
ap.add_argument('software_url',
help='Your software url')
ap.add_argument('--node',
nargs='*',
help='Node request option '
"'option1=value1 option2=value2'")
ap.add_argument('--type',
type=str,
help='Define software type to be requested')
ap.add_argument('--slave',
action='store_true', default=False,
help='Ask for a slave instance')
ap.add_argument('--configuration',
nargs='*',
help='Give your configuration '
"'option1=value1 option2=value2'")
args = ap.parse_args()
if args.configuration:
args.parameters = argToDict(args.configuration)
if args.node:
args.node = argToDict(args.node)
configp = get_config_parser(args.configuration_file)
conf = ClientConfig(args, configp)
local = init(conf)
do_request(conf, local)
slapos/cli_legacy/slapgrid.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
# vim: set et sts=2:
import argparse
import ConfigParser
import logging
import sys
from slapos.grid.utils import setRunning, setFinished
from slapos.grid.slapgrid import (merged_options, check_missing_parameters,
check_missing_files, random_delay, create_slapgrid_object)
def parse_arguments(*argument_tuple):
"""Parse arguments and return options dictionary merged with the config file."""
ap = argparse.ArgumentParser()
ap.add_argument('--instance-root',
help='The instance root directory location.')
ap.add_argument('--software-root',
help='The software_root directory location.')
ap.add_argument('--master-url',
help='The master server URL. Mandatory.')
ap.add_argument('--computer-id',
help='The computer id defined in the server.')
ap.add_argument('--supervisord-socket',
help='The socket supervisor will use.')
ap.add_argument('--supervisord-configuration-path',
help='The location where supervisord configuration will be stored.')
ap.add_argument('--buildout', default=None,
help='Location of buildout binary.')
ap.add_argument('--pidfile',
help='The location where pidfile will be created.')
ap.add_argument('--logfile',
help='The location where slapgrid logfile will be created.')
ap.add_argument('--key_file',
help='SSL Authorisation key file.')
ap.add_argument('--cert_file',
help='SSL Authorisation certificate file.')
ap.add_argument('--signature_private_key_file',
help='Signature private key file.')
ap.add_argument('--master_ca_file',
help='Root certificate of SlapOS master key.')
ap.add_argument('--certificate_repository_path',
help='Path to directory where downloaded certificates would be stored.')
ap.add_argument('-v', '--verbose', action='store_true',
help='Be verbose.')
ap.add_argument('--maximum-periodicity', type=int, default=None,
help='Periodicity at which buildout should be run in instance.')
ap.add_argument('--promise-timeout', type=int, default=3,
help='Promise timeout in seconds.')
ap.add_argument('--now', action='store_true',
help='Launch slapgrid without delay. Default behavior.')
ap.add_argument('--all', action='store_true',
help='Launch slapgrid to process all Software Releases '
'and/or Computer Partitions.')
ap.add_argument('--only-sr',
help='Force the update of a single software release (use url hash), '
'even if is already installed. This option will make all other '
'sofware releases be ignored.')
ap.add_argument('--only-cp',
help='Update a single or a list of computer partitions '
'(ie.:slappartX, slappartY), '
'this option will make all other computer partitions be ignored.')
ap.add_argument('configuration_file', type=argparse.FileType(),
help='SlapOS configuration file.')
# Deprecated options
ap.add_argument('-c', '--console', action='store_true',
help="Deprecated, doesn't do anything.")
ap.add_argument('--develop', action='store_true',
help='Deprecated, same as --all.')
ap.add_argument('--only_sr',
help='Deprecated, same as --only-sr.')
ap.add_argument('--only_cp',
help='Deprecated, same as --only-cp.')
ap.add_argument('--maximal_delay',
help='Deprecated. Will only work from configuration file in the future.')
if not argument_tuple:
args = ap.parse_args()
else:
args = ap.parse_args(list(argument_tuple))
return args
def setup_logger(options):
logger = logging.getLogger(__name__)
if options.get('logfile'):
handler = logging.FileHandler(options['logfile'])
else:
handler = logging.StreamHandler()
if options['verbose']:
handler.setLevel(logging.DEBUG)
else:
handler.setLevel(logging.INFO)
formatter = logging.Formatter(fmt='%(asctime)s %(name)-18s: '
'%(levelname)-8s %(message)s',
datefmt='%Y-%m-%dT%H:%M:%S')
handler.setFormatter(formatter)
logger.addHandler(handler)
return logger
def parseArgumentTupleAndReturnSlapgridObject(*argument_tuple):
"""Returns a new instance of slapgrid.Slapgrid created with argument+config parameters.
Also returns the pidfile path, and configures logger.
"""
args = parse_arguments(*argument_tuple)
configp = ConfigParser.SafeConfigParser()
configp.readfp(args.configuration_file)
options = merged_options(args, configp)
logger = setup_logger(options)
check_missing_parameters(options)
check_missing_files(options)
random_delay(options, logger=logger)
slapgrid_object = create_slapgrid_object(options, logger=logger)
return slapgrid_object, options.get('pidfile')
def realRun(argument_tuple, method):
slapgrid_object, pidfile = parseArgumentTupleAndReturnSlapgridObject(*argument_tuple)
if pidfile:
setRunning(logger=slapgrid_object.logger, pidfile=pidfile)
try:
return getattr(slapgrid_object, method)()
finally:
if pidfile:
setFinished(pidfile)
def runSoftwareRelease(*argument_tuple):
"""Hook for entry point to process Software Releases"""
sys.exit(realRun(argument_tuple, 'processSoftwareReleaseList'))
def runComputerPartition(*argument_tuple):
"""Hook for entry point to process Computer Partitions"""
sys.exit(realRun(argument_tuple, 'processComputerPartitionList'))
def runUsageReport(*argument_tuple):
"""Hook for entry point to process Usage Reports"""
sys.exit(realRun(argument_tuple, 'agregateAndSendUsage'))
slapos/cli_legacy/supply.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import argparse
from slapos.client import ClientConfig, init, do_supply
from slapos.cli_legacy.util import get_config_parser
def supply():
"""
Run when invoking slapos supply. Mostly argument parsing.
"""
ap = argparse.ArgumentParser()
ap.add_argument('configuration_file',
help='SlapOS configuration file')
ap.add_argument('software_url',
help='Your software url')
ap.add_argument('node',
help='Target node')
args = ap.parse_args()
configp = get_config_parser(args.configuration_file)
conf = ClientConfig(args, configp)
local = init(conf)
do_supply(args.software_url, args.node, local)
slapos/cli_legacy/svcbackend.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
# vim: set et sts=2:
##############################################################################
#
# Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors.
# All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly advised to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import logging
import os
from optparse import OptionParser
import ConfigParser
from slapos.grid.svcbackend import launchSupervisord
def getOptionDict(*argument_tuple):
usage = """
Typical usage:
* %prog CONFIGURATION_FILE [arguments passed to supervisor]
""".strip()
parser = OptionParser(usage=usage)
# Parses arguments
if argument_tuple:
(argument_option_instance, argument_list) = parser.parse_args(list(argument_tuple))
else:
# No arguments given to entry point : we parse sys.argv.
(argument_option_instance, argument_list) = parser.parse_args()
if not argument_list:
parser.error("Configuration file is obligatory. Consult documentation by calling with -h.")
configuration_file = argument_list[0]
if not os.path.exists(configuration_file):
parser.error("Could not read configuration file : %s" % configuration_file)
slapgrid_configuration = ConfigParser.SafeConfigParser()
slapgrid_configuration.read(configuration_file)
# Merges the two dictionnaries
option_dict = dict(slapgrid_configuration.items("slapos"))
# Supervisord configuration location
option_dict.setdefault('supervisord_configuration_path',
os.path.join(option_dict['instance_root'], 'etc', 'supervisord.conf'))
# Supervisord socket
option_dict.setdefault('supervisord_socket',
os.path.join(option_dict['instance_root'], 'supervisord.socket'))
return option_dict, argument_list[1:]
def supervisorctl(*argument_tuple):
logger = logging.getLogger('SVCBackend')
logger.setLevel(logging.INFO)
handler = logging.StreamHandler()
logger.addHandler(handler)
option_dict, args = getOptionDict(*argument_tuple)
import supervisor.supervisorctl
launchSupervisord(socket=option_dict['supervisord_socket'],
configuration_file=option_dict['supervisord_configuration_path'],
logger=logger)
supervisor.supervisorctl.main(args=['-c', option_dict['supervisord_configuration_path']] + args)
def supervisord(*argument_tuple):
logger = logging.getLogger('SVCBackend')
logger.setLevel(logging.INFO)
handler = logging.StreamHandler()
logger.addHandler(handler)
option_dict, _ = getOptionDict(*argument_tuple)
launchSupervisord(socket=option_dict['supervisord_socket'],
configuration_file=option_dict['supervisord_configuration_path'],
logger=logger)
slapos/cli_legacy/util.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import ConfigParser
import os
def get_config_parser(path):
configp = ConfigParser.SafeConfigParser()
path = os.path.expanduser(path)
if not os.path.isfile(path):
raise OSError('Specified configuration file %s does not exist. Exiting.' % path)
configp.read(path)
return configp
slapos/client.py
View file @ 5f0924ff
...@@ -27,121 +27,37 @@ ...@@ -27,121 +27,37 @@
# #
############################################################################## ##############################################################################
import argparse import atexit
import ConfigParser import ConfigParser
import pprint
from optparse import OptionParser, Option
import os import os
from slapos.slap import ResourceNotReady import pprint
import slapos.slap.slap
import sys
import atexit
class Parser(OptionParser):
"""
Parse all arguments.
"""
def __init__(self, usage=None, version=None):
"""
Initialize all options possibles.
"""
OptionParser.__init__(self, usage=usage, version=version,
option_list=[
Option("-u", "--master_url",
default=None,
action="store",
help="Url of SlapOS Master to use."),
Option("-k", "--key_file",
action="store",
help="SSL Authorisation key file."),
Option("-c", "--cert_file",
action="store",
help="SSL Authorisation certificate file.")
])
def check_args(self):
"""
Check arguments
"""
(options, args) = self.parse_args()
if len(args) == 0:
self.error("Incorrect number of arguments")
elif not os.path.isfile(args[0]):
self.error("%s: Not found or not a regular file." % args[0])
# Return options and only first element of args since there is only one.
return options, args[0]
def argToDict(element): import slapos.slap.slap
""" from slapos.slap import ResourceNotReady
convert a table of string 'key=value' to dict
"""
if element is not None:
element_dict = dict([arg.split('=') for arg in element])
return element_dict
def check_request_args():
"""
Parser for request
"""
parser = argparse.ArgumentParser()
parser.add_argument("configuration_file",
help="SlapOS configuration file.")
parser.add_argument("reference",
help="Your instance reference")
parser.add_argument("software_url",
help="Your software url")
parser.add_argument("--node",
nargs = '*',
help = "Node request option "
"'option1=value1 option2=value2'")
parser.add_argument("--type",
type = str,
help = "Define software type to be requested")
parser.add_argument("--slave",
action = "store_true", default=False,
help = "Ask for a slave instance")
parser.add_argument("--configuration",
nargs = '*',
help = "Give your configuration "
"'option1=value1 option2=value2'")
args = parser.parse_args()
# Convert to dict
if args.configuration is not None:
args.configuration = argToDict(args.configuration)
if args.node is not None:
args.node = argToDict(args.node)
return args
class ClientConfig(object):
state = None
class Config: def __init__(self, args, configp=None):
def __init__(self, option_dict, configuration_file_path=None): # XXX configp cannot possibly be optional
""" """
Set options given by parameters. Set options given by parameters.
""" """
# Set options parameters # Set options parameters
for option, value in option_dict.__dict__.items(): for key, value in args.__dict__.items():
setattr(self, option, value) setattr(self, key, value)
# Load configuration file
configuration_parser = ConfigParser.SafeConfigParser()
if configuration_file_path:
configuration_file_path = os.path.expanduser(configuration_file_path)
if not os.path.isfile(configuration_file_path):
raise OSError('Specified configuration file %s does not exist.'
' Exiting.' % configuration_file_path)
configuration_parser.read(configuration_file_path)
# Merges the arguments and configuration # Merges the arguments and configuration
try: try:
configuration_dict = dict(configuration_parser.items('slapconsole')) configuration_dict = dict(configp.items('slapconsole'))
except ConfigParser.NoSectionError: except ConfigParser.NoSectionError:
pass pass
else: else:
for key in configuration_dict: for key in configuration_dict:
if not getattr(self, key, None): if not getattr(self, key, None):
setattr(self, key, configuration_dict[key]) setattr(self, key, configuration_dict[key])
configuration_dict = dict(configuration_parser.items('slapos')) configuration_dict = dict(configp.items('slapos'))
master_url = configuration_dict.get('master_url', None) master_url = configuration_dict.get('master_url', None)
# Backward compatibility, if no key and certificate given in option # Backward compatibility, if no key and certificate given in option
# take one from slapos configuration # take one from slapos configuration
...@@ -158,67 +74,73 @@ class Config: ...@@ -158,67 +74,73 @@ class Config:
else: else:
setattr(self, 'master_url', master_url) setattr(self, 'master_url', master_url)
def init(config): if self.key_file:
self.key_file = os.path.expanduser(self.key_file)
if self.cert_file:
self.cert_file = os.path.expanduser(self.cert_file)
def init(conf):
"""Initialize Slap instance, connect to server and create """Initialize Slap instance, connect to server and create
aliases to common software releases""" aliases to common software releases"""
# XXX check certificate and key existence
slap = slapos.slap.slap() slap = slapos.slap.slap()
slap.initializeConnection(config.master_url, slap.initializeConnection(conf.master_url,
key_file=config.key_file, cert_file=config.cert_file) key_file=conf.key_file, cert_file=conf.cert_file)
local = globals().copy() local = globals().copy()
local['slap'] = slap local['slap'] = slap
# Create aliases as global variables # Create aliases as global variables
try: try:
alias = config.alias.split('\n') alias = conf.alias.split('\n')
except AttributeError: except AttributeError:
alias = [] alias = []
software_list = [] software_list = []
for software in alias: for software in alias:
if software is not '': if software:
name, url = software.split(' ') name, url = software.split(' ')
software_list.append(name) software_list.append(name)
local[name] = url local[name] = url
# Create global variable too see available aliases # Create global variable too see available aliases
local['software_list'] = software_list local['software_list'] = software_list
# Create global shortcut functions to request instance and software # Create global shortcut functions to request instance and software
def shorthandRequest(*args, **kwargs): def shorthandRequest(*args, **kwargs):
return slap.registerOpenOrder().request(*args, **kwargs) return slap.registerOpenOrder().request(*args, **kwargs)
def shorthandSupply(*args, **kwargs): def shorthandSupply(*args, **kwargs):
return slap.registerSupply().supply(*args, **kwargs) return slap.registerSupply().supply(*args, **kwargs)
local['request'] = shorthandRequest local['request'] = shorthandRequest
local['supply'] = shorthandSupply local['supply'] = shorthandSupply
return local return local
def request():
"""Run when invoking slapos request. Request an instance.""" def do_request(conf, local):
# Parse arguments and inititate needed parameters print("Requesting %s..." % conf.reference)
# XXX-Cedric: move argument parsing to main entry point if conf.software_url in local:
options = check_request_args() conf.software_url = local[conf.software_url]
config = Config(options, options.configuration_file)
local = init(config)
# Request instance
print("Requesting %s..." % config.reference)
if config.software_url in local:
config.software_url = local[config.software_url]
try: try:
partition = local['slap'].registerOpenOrder().request( partition = local['slap'].registerOpenOrder().request(
software_release = config.software_url, software_release = conf.software_url,
partition_reference = config.reference, partition_reference = conf.reference,
partition_parameter_kw = config.configuration, partition_parameter_kw = conf.parameters,
software_type = config.type, software_type = conf.type,
filter_kw = config.node, filter_kw = conf.node,
shared = config.slave state = conf.state,
shared = conf.slave
) )
print "Instance requested.\nState is : %s." % partition.getState() print "Instance requested.\nState is : %s." % partition.getState()
print "Connection parameters of instance are:" print "Connection parameters of instance are:"
pprint.pprint(partition.getConnectionParameterDict()) pprint.pprint(partition.getConnectionParameterDict())
print "You can rerun command to get up-to-date informations." print "You can rerun command to get up-to-date informations."
except ResourceNotReady: except ResourceNotReady:
print("Instance requested. Master is provisionning it. Please rerun in a " print("Instance requested. Master is provisioning it. Please rerun in a "
"couple of minutes to get connection informations.") "couple of minutes to get connection informations.")
exit(2) exit(2)
def _supply(software_url, computer_id, local, remove=False):
def do_supply(software_url, computer_id, local, remove=False):
""" """
Request installation of Software Release Request installation of Software Release
'software_url' on computer 'computer_id'. 'software_url' on computer 'computer_id'.
...@@ -243,58 +165,12 @@ def _supply(software_url, computer_id, local, remove=False): ...@@ -243,58 +165,12 @@ def _supply(software_url, computer_id, local, remove=False):
) )
print 'Done.' print 'Done.'
def supply():
"""
Run when invoking slapos supply. Mostly argument parsing.
"""
# XXX-Cedric: move argument parsing to main entry point
parser = argparse.ArgumentParser()
parser.add_argument("configuration_file",
help="SlapOS configuration file")
parser.add_argument("software_url",
help="Your software url")
parser.add_argument("node",
help="Target node")
args = parser.parse_args()
config = Config(args, args.configuration_file)
_supply(args.software_url, args.node, init(config))
def remove(): def do_remove(software_url, node, local):
""" do_supply(software_url, node, local, remove=True)
Run when invoking slapos remove. Mostly argument parsing.
"""
# XXX-Cedric: move argument parsing to main entry point
parser = argparse.ArgumentParser()
parser.add_argument("configuration_file",
help="SlapOS configuration file.")
parser.add_argument("software_url",
help="Your software url")
parser.add_argument("node",
help="Target node")
args = parser.parse_args()
config = Config(args, args.configuration_file)
_supply(args.software_url, args.node, init(config), remove=True)
def slapconsole():
"""Ran when invoking slapconsole"""
# Parse arguments
usage = """usage: %s [options] CONFIGURATION_FILE
slapconsole allows you interact with slap API. You can play with the global
"slap" object and with the global "request" method.
examples :
>>> # Request instance
>>> request(kvm, "myuniquekvm")
>>> # Request software installation on owned computer
>>> supply(kvm, "mycomputer")
>>> # Fetch instance informations on already launched instance
>>> request(kvm, "myuniquekvm").getConnectionParameter("url")""" % sys.argv[0]
config = Config(*Parser(usage=usage).check_args())
local = init(config)
def do_console(local):
# try to enable readline with completion and history # try to enable readline with completion and history
try: try:
import readline import readline
...@@ -316,4 +192,3 @@ examples : ...@@ -316,4 +192,3 @@ examples :
atexit.register(save_history) atexit.register(save_history)
__import__("code").interact(banner="", local=local) __import__("code").interact(banner="", local=local)
slapos/format.py
View file @ 5f0924ff
...@@ -28,8 +28,6 @@ ...@@ -28,8 +28,6 @@
# #
############################################################################## ##############################################################################
from optparse import OptionParser, Option
from xml_marshaller import xml_marshaller
import ConfigParser import ConfigParser
import errno import errno
import fcntl import fcntl
...@@ -41,7 +39,6 @@ import netifaces ...@@ -41,7 +39,6 @@ import netifaces
import os import os
import pwd import pwd
import random import random
import slapos.slap as slap
import shutil import shutil
import socket import socket
import struct import struct
...@@ -53,12 +50,10 @@ import traceback ...@@ -53,12 +50,10 @@ import traceback
import zipfile import zipfile
import lxml.etree import lxml.etree
from slapos.version import version import xml_marshaller.xml_marshaller
# set up logging from slapos.util import mkdir_p
logger = logging.getLogger("slapformat") import slapos.slap as slap
logger.setLevel(logging.INFO)
def prettify_xml(xml): def prettify_xml(xml):
...@@ -66,17 +61,14 @@ def prettify_xml(xml): ...@@ -66,17 +61,14 @@ def prettify_xml(xml):
return lxml.etree.tostring(root, pretty_print=True) return lxml.etree.tostring(root, pretty_print=True)
from slapos.util import mkdir_p
class OS(object): class OS(object):
"""Wrap parts of the 'os' module to provide logging of performed actions.""" """Wrap parts of the 'os' module to provide logging of performed actions."""
_os = os _os = os
def __init__(self, config): def __init__(self, conf):
self._dry_run = config.dry_run self._dry_run = conf.dry_run
self._verbose = config.verbose self._logger = conf.logger
self._logger = config.logger
add = self._addWrapper add = self._addWrapper
add('chown') add('chown')
add('chmod') add('chmod')
...@@ -85,11 +77,10 @@ class OS(object): ...@@ -85,11 +77,10 @@ class OS(object):
def _addWrapper(self, name): def _addWrapper(self, name):
def wrapper(*args, **kw): def wrapper(*args, **kw):
if self._verbose: arg_list = [repr(x) for x in args] + [
arg_list = [repr(x) for x in args] + [ '%s=%r' % (x, y) for x, y in kw.iteritems()
'%s=%r' % (x, y) for x, y in kw.iteritems() ]
] self._logger.debug('%s(%s)' % (name, ', '.join(arg_list)))
self._logger.debug('%s(%s)' % (name, ', '.join(arg_list)))
if not self._dry_run: if not self._dry_run:
getattr(self._os, name)(*args, **kw) getattr(self._os, name)(*args, **kw)
setattr(self, name, wrapper) setattr(self, name, wrapper)
...@@ -161,36 +152,35 @@ def netmaskToPrefixIPv6(netmask): ...@@ -161,36 +152,35 @@ def netmaskToPrefixIPv6(netmask):
netaddr.strategy.ipv6.str_to_int(netmask)] netaddr.strategy.ipv6.str_to_int(netmask)]
def _getDict(instance): def _getDict(obj):
""" """
Serialize an object instance into dictionaries. List and dict will remains Serialize an object into dictionaries. List and dict will remains
the same, basic type too. But encapsulated object will be returned as dict. the same, basic type too. But encapsulated object will be returned as dict.
Set, collections and other aren't handle for now. Set, collections and other aren't handle for now.
Args: Args:
instance: an object of any type. obj: an object of any type.
Returns: Returns:
A dictionary if the given object wasn't a list, a list otherwise. A dictionary if the given object wasn't a list, a list otherwise.
""" """
if isinstance(instance, list): if isinstance(obj, list):
return [_getDict(item) for item in instance] return [_getDict(item) for item in obj]
elif isinstance(instance, dict):
result = {}
for key in instance:
result[key] = _getDict(instance[key])
return result
if isinstance(obj, dict):
dikt = obj
else: else:
try: try:
dikt = instance.__dict__ dikt = obj.__dict__
except AttributeError: except AttributeError:
return instance return obj
result = {}
for key, value in dikt.iteritems(): return {
result[key] = _getDict(value) key: _getDict(value)
return result for key, value in dikt.iteritems()
# do not attempt to serialize logger: it is both useless and recursive.
if not isinstance(value, logging.Logger)
}
class Computer(object): class Computer(object):
...@@ -248,30 +238,30 @@ class Computer(object): ...@@ -248,30 +238,30 @@ class Computer(object):
# Can't find address # Can't find address
raise NoAddressOnInterface('No valid IPv6 found on %s.' % self.interface.name) raise NoAddressOnInterface('No valid IPv6 found on %s.' % self.interface.name)
def send(self, config): def send(self, conf):
""" """
Send a marshalled dictionary of the computer object serialized via_getDict. Send a marshalled dictionary of the computer object serialized via_getDict.
""" """
slap_instance = slap.slap() slap_instance = slap.slap()
connection_dict = {} connection_dict = {}
if config.key_file and config.cert_file: if conf.key_file and conf.cert_file:
connection_dict['key_file'] = config.key_file connection_dict['key_file'] = conf.key_file
connection_dict['cert_file'] = config.cert_file connection_dict['cert_file'] = conf.cert_file
slap_instance.initializeConnection(config.master_url, slap_instance.initializeConnection(conf.master_url,
**connection_dict) **connection_dict)
slap_computer = slap_instance.registerComputer(self.reference) slap_computer = slap_instance.registerComputer(self.reference)
if config.dry_run: if conf.dry_run:
return return
try: try:
slap_computer.updateConfiguration(xml_marshaller.dumps(_getDict(self))) slap_computer.updateConfiguration(xml_marshaller.xml_marshaller.dumps(_getDict(self)))
except slap.NotFoundError as error: except slap.NotFoundError as error:
raise slap.NotFoundError("%s\nERROR : This SlapOS node is not recognised by " raise slap.NotFoundError("%s\nERROR : This SlapOS node is not recognised by "
"SlapOS Master. Please make sure computer_id of slapos.cfg looks " "SlapOS Master. Please make sure computer_id of slapos.cfg looks "
"like 'COMP-123' and is correct.\nError is : 404 Not Found." % error) "like 'COMP-123' and is correct.\nError is : 404 Not Found." % error)
def dump(self, path_to_xml, path_to_json): def dump(self, path_to_xml, path_to_json, logger):
""" """
Dump the computer object to an xml file via xml_marshaller. Dump the computer object to an xml file via xml_marshaller.
...@@ -286,7 +276,7 @@ class Computer(object): ...@@ -286,7 +276,7 @@ class Computer(object):
with open(path_to_json, 'wb') as fout: with open(path_to_json, 'wb') as fout:
fout.write(json.dumps(computer_dict, sort_keys=True, indent=2)) fout.write(json.dumps(computer_dict, sort_keys=True, indent=2))
new_xml = xml_marshaller.dumps(computer_dict) new_xml = xml_marshaller.xml_marshaller.dumps(computer_dict)
new_pretty_xml = prettify_xml(new_xml) new_pretty_xml = prettify_xml(new_xml)
path_to_archive = path_to_xml + '.zip' path_to_archive = path_to_xml + '.zip'
...@@ -340,7 +330,7 @@ class Computer(object): ...@@ -340,7 +330,7 @@ class Computer(object):
A Computer object. A Computer object.
""" """
dumped_dict = xml_marshaller.loads(open(path_to_xml).read()) dumped_dict = xml_marshaller.xml_marshaller.loads(open(path_to_xml).read())
# Reconstructing the computer object from the xml # Reconstructing the computer object from the xml
computer = Computer( computer = Computer(
...@@ -684,12 +674,13 @@ class Tap(object): ...@@ -684,12 +674,13 @@ class Tap(object):
class Interface(object): class Interface(object):
"""Represent a network interface on the system""" """Represent a network interface on the system"""
def __init__(self, name, ipv4_local_network, ipv6_interface=None): def __init__(self, logger, name, ipv4_local_network, ipv6_interface=None):
""" """
Attributes: Attributes:
name: String, the name of the interface name: String, the name of the interface
""" """
self.logger = logger
self.name = str(name) self.name = str(name)
self.ipv4_local_network = ipv4_local_network self.ipv4_local_network = ipv4_local_network
self.ipv6_interface = ipv6_interface self.ipv6_interface = ipv6_interface
...@@ -853,7 +844,7 @@ class Interface(object): ...@@ -853,7 +844,7 @@ class Interface(object):
if self._addSystemAddress(addr, netmask, False): if self._addSystemAddress(addr, netmask, False):
return dict(addr=addr, netmask=netmask) return dict(addr=addr, netmask=netmask)
else: else:
logger.warning('Impossible to add old local IPv4 %s. Generating ' self.logger.warning('Impossible to add old local IPv4 %s. Generating '
'new IPv4 address.' % addr) 'new IPv4 address.' % addr)
return self._generateRandomIPv4Address(netmask) return self._generateRandomIPv4Address(netmask)
else: else:
...@@ -911,7 +902,7 @@ class Interface(object): ...@@ -911,7 +902,7 @@ class Interface(object):
# succeed, return it # succeed, return it
return dict(addr=addr, netmask=netmask) return dict(addr=addr, netmask=netmask)
else: else:
logger.warning('Impossible to add old public IPv6 %s. ' self.logger.warning('Impossible to add old public IPv6 %s. '
'Generating new IPv6 address.' % addr) 'Generating new IPv6 address.' % addr)
# Try 10 times to add address, raise in case if not possible # Try 10 times to add address, raise in case if not possible
...@@ -931,76 +922,8 @@ class Interface(object): ...@@ -931,76 +922,8 @@ class Interface(object):
raise AddressGenerationError(addr) raise AddressGenerationError(addr)
class Parser(OptionParser): def parse_computer_definition(conf, definition_path):
""" conf.logger.info('Using definition file %r' % definition_path)
Parse all arguments.
"""
def __init__(self, usage=None, version=version):
"""
Initialize all options possibles.
"""
OptionParser.__init__(self, usage=usage, version=version,
option_list=[
Option("-x", "--computer_xml",
help="Path to file with computer's XML. If does not exists, "
"will be created",
default=None,
type=str),
Option("--computer_json",
help="Path to a JSON version of the computer's XML (for development only).",
default=None,
type=str),
Option("-l", "--log_file",
help="The path to the log file used by the script.",
type=str),
Option("-i", "--input_definition_file",
help="Path to file to read definition of computer instead of "
"declaration. Using definition file allows to disable "
"'discovery' of machine services and allows to define computer "
"configuration in fully controlled manner.",
type=str),
Option("-o", "--output_definition_file",
help="Path to file to write definition of computer from "
"declaration.",
type=str),
Option("-n", "--dry_run",
help="Don't actually do anything.",
default=False,
action="store_true"),
Option("-v", "--verbose",
default=False,
action="store_true",
help="Verbose output."),
Option("-c", "--console",
default=False,
action="store_true",
help="Console output."),
Option('--alter_user', choices=['True', 'False'],
help="Shall slapformat alter user database [default: True]"),
Option('--alter_network', choices=['True', 'False'],
help="Shall slapformat alter network configuration [default: True]"),
Option('--now',
help="Launch slapformat without delay",
default=False,
action="store_true"),
])
def check_args(self, args):
"""
Check arguments
"""
if args:
(options, args) = self.parse_args(list(args))
else:
(options, args) = self.parse_args()
if len(args) != 1:
self.error("Incorrect number of arguments")
return options, args[0]
def parse_computer_definition(config, definition_path):
config.logger.info('Using definition file %r' % definition_path)
computer_definition = ConfigParser.RawConfigParser({ computer_definition = ConfigParser.RawConfigParser({
'software_user': 'slapsoft', 'software_user': 'slapsoft',
}) })
...@@ -1010,20 +933,20 @@ def parse_computer_definition(config, definition_path): ...@@ -1010,20 +933,20 @@ def parse_computer_definition(config, definition_path):
netmask = None netmask = None
if computer_definition.has_option('computer', 'address'): if computer_definition.has_option('computer', 'address'):
address, netmask = computer_definition.get('computer', 'address').split('/') address, netmask = computer_definition.get('computer', 'address').split('/')
if config.alter_network and config.interface_name is not None \ if conf.alter_network and conf.interface_name is not None \
and config.ipv4_local_network is not None: and conf.ipv4_local_network is not None:
interface = Interface(config.interface_name, config.ipv4_local_network, interface = Interface(conf.logger, conf.interface_name, conf.ipv4_local_network,
config.ipv6_interface) conf.ipv6_interface)
computer = Computer( computer = Computer(
reference=config.computer_id, reference=conf.computer_id,
interface=interface, interface=interface,
addr=address, addr=address,
netmask=netmask, netmask=netmask,
ipv6_interface=config.ipv6_interface, ipv6_interface=conf.ipv6_interface,
software_user=computer_definition.get('computer', 'software_user'), software_user=computer_definition.get('computer', 'software_user'),
) )
partition_list = [] partition_list = []
for partition_number in range(int(config.partition_amount)): for partition_number in range(int(conf.partition_amount)):
section = 'partition_%s' % partition_number section = 'partition_%s' % partition_number
user = User(computer_definition.get(section, 'user')) user = User(computer_definition.get(section, 'user'))
address_list = [] address_list = []
...@@ -1032,7 +955,7 @@ def parse_computer_definition(config, definition_path): ...@@ -1032,7 +955,7 @@ def parse_computer_definition(config, definition_path):
address_list.append(dict(addr=address, netmask=netmask)) address_list.append(dict(addr=address, netmask=netmask))
tap = Tap(computer_definition.get(section, 'network_interface')) tap = Tap(computer_definition.get(section, 'network_interface'))
partition = Partition(reference=computer_definition.get(section, 'pathname'), partition = Partition(reference=computer_definition.get(section, 'pathname'),
path=os.path.join(config.instance_root, path=os.path.join(conf.instance_root,
computer_definition.get(section, 'pathname')), computer_definition.get(section, 'pathname')),
user=user, user=user,
address_list=address_list, address_list=address_list,
...@@ -1042,48 +965,48 @@ def parse_computer_definition(config, definition_path): ...@@ -1042,48 +965,48 @@ def parse_computer_definition(config, definition_path):
return computer return computer
def parse_computer_xml(config, xml_path): def parse_computer_xml(conf, xml_path):
if os.path.exists(xml_path): if os.path.exists(xml_path):
config.logger.info('Loading previous computer data from %r' % xml_path) conf.logger.info('Loading previous computer data from %r' % xml_path)
computer = Computer.load(xml_path, computer = Computer.load(xml_path,
reference=config.computer_id, reference=conf.computer_id,
ipv6_interface=config.ipv6_interface) ipv6_interface=conf.ipv6_interface)
# Connect to the interface defined by the configuration # Connect to the interface defined by the configuration
computer.interface = Interface(config.interface_name, config.ipv4_local_network, computer.interface = Interface(conf.logger, conf.interface_name, conf.ipv4_local_network,
config.ipv6_interface) conf.ipv6_interface)
else: else:
# If no pre-existent configuration found, create a new computer object # If no pre-existent configuration found, create a new computer object
config.logger.warning('Creating new data computer with id %r' % config.computer_id) conf.logger.warning('Creating new data computer with id %r' % conf.computer_id)
computer = Computer( computer = Computer(
reference=config.computer_id, reference=conf.computer_id,
interface=Interface(config.interface_name, config.ipv4_local_network, interface=Interface(conf.logger, conf.interface_name, conf.ipv4_local_network,
config.ipv6_interface), conf.ipv6_interface),
addr=None, addr=None,
netmask=None, netmask=None,
ipv6_interface=config.ipv6_interface, ipv6_interface=conf.ipv6_interface,
software_user=config.software_user, software_user=conf.software_user,
) )
partition_amount = int(config.partition_amount) partition_amount = int(conf.partition_amount)
existing_partition_amount = len(computer.partition_list) existing_partition_amount = len(computer.partition_list)
if existing_partition_amount > partition_amount: if existing_partition_amount > partition_amount:
raise ValueError('Requested amount of computer partitions (%s) is lower ' raise ValueError('Requested amount of computer partitions (%s) is lower '
'then already configured (%s), cannot continue' % (partition_amount, 'then already configured (%s), cannot continue' % (partition_amount,
len(computer.partition_list))) len(computer.partition_list)))
config.logger.info('Adding %s new partitions' % conf.logger.info('Adding %s new partitions' %
(partition_amount-existing_partition_amount)) (partition_amount - existing_partition_amount))
for nb_iter in range(existing_partition_amount, partition_amount): for nb_iter in range(existing_partition_amount, partition_amount):
# add new ones # add new ones
user = User("%s%s" % (config.user_base_name, nb_iter)) user = User("%s%s" % (conf.user_base_name, nb_iter))
tap = Tap("%s%s" % (config.tap_base_name, nb_iter)) tap = Tap("%s%s" % (conf.tap_base_name, nb_iter))
path = os.path.join(config.instance_root, "%s%s" % ( path = os.path.join(conf.instance_root, "%s%s" % (
config.partition_base_name, nb_iter)) conf.partition_base_name, nb_iter))
computer.partition_list.append( computer.partition_list.append(
Partition( Partition(
reference="%s%s" % (config.partition_base_name, nb_iter), reference="%s%s" % (conf.partition_base_name, nb_iter),
path=path, path=path,
user=user, user=user,
address_list=None, address_list=None,
...@@ -1093,7 +1016,7 @@ def parse_computer_xml(config, xml_path): ...@@ -1093,7 +1016,7 @@ def parse_computer_xml(config, xml_path):
return computer return computer
def write_computer_definition(config, computer): def write_computer_definition(conf, computer):
computer_definition = ConfigParser.RawConfigParser() computer_definition = ConfigParser.RawConfigParser()
computer_definition.add_section('computer') computer_definition.add_section('computer')
if computer.address is not None and computer.netmask is not None: if computer.address is not None and computer.netmask is not None:
...@@ -1109,44 +1032,58 @@ def write_computer_definition(config, computer): ...@@ -1109,44 +1032,58 @@ def write_computer_definition(config, computer):
computer_definition.set(section, 'user', partition.user.name) computer_definition.set(section, 'user', partition.user.name)
computer_definition.set(section, 'network_interface', partition.tap.name) computer_definition.set(section, 'network_interface', partition.tap.name)
computer_definition.set(section, 'pathname', partition.reference) computer_definition.set(section, 'pathname', partition.reference)
computer_definition.write(open(config.output_definition_file, 'w')) computer_definition.write(open(conf.output_definition_file, 'w'))
config.logger.info('Stored computer definition in %r' % config.output_definition_file) conf.logger.info('Stored computer definition in %r' % conf.output_definition_file)
def random_delay(conf):
# Add delay between 0 and 1 hour
# XXX should be the contrary: now by default, and cron should have
# --maximal-delay=3600
if not conf.now:
duration = float(60 * 60) * random.random()
print("Sleeping for %s seconds. To disable this feature, " \
"use with --now parameter in manual." % duration)
time.sleep(duration)
def run(config): def do_format(conf):
if config.input_definition_file: random_delay(conf)
computer = parse_computer_definition(config, config.input_definition_file)
if conf.input_definition_file:
computer = parse_computer_definition(conf, conf.input_definition_file)
else: else:
# no definition file, figure out computer # no definition file, figure out computer
computer = parse_computer_xml(config, config.computer_xml) computer = parse_computer_xml(conf, conf.computer_xml)
computer.instance_root = config.instance_root computer.instance_root = conf.instance_root
computer.software_root = config.software_root computer.software_root = conf.software_root
config.logger.info('Updating computer') conf.logger.info('Updating computer')
address = computer.getAddress(config.create_tap) address = computer.getAddress(conf.create_tap)
computer.address = address['addr'] computer.address = address['addr']
computer.netmask = address['netmask'] computer.netmask = address['netmask']
if config.output_definition_file: if conf.output_definition_file:
write_computer_definition(config, computer) write_computer_definition(conf, computer)
computer.construct(alter_user=config.alter_user, computer.construct(alter_user=conf.alter_user,
alter_network=config.alter_network, alter_network=conf.alter_network,
create_tap=config.create_tap) create_tap=conf.create_tap)
if getattr(config, 'certificate_repository_path', None): if getattr(conf, 'certificate_repository_path', None):
mkdir_p(config.certificate_repository_path, mode=0o700) mkdir_p(conf.certificate_repository_path, mode=0o700)
# Dumping and sending to the erp5 the current configuration # Dumping and sending to the erp5 the current configuration
if not config.dry_run: if not conf.dry_run:
computer.dump(path_to_xml=config.computer_xml, computer.dump(path_to_xml=conf.computer_xml,
path_to_json=config.computer_json) path_to_json=conf.computer_json,
config.logger.info('Posting information to %r' % config.master_url) logger=conf.logger)
computer.send(config) conf.logger.info('Posting information to %r' % conf.master_url)
config.logger.info('slapformat successfully prepared computer.') computer.send(conf)
conf.logger.info('slapformat successfully prepared computer.')
class Config(object): class FormatConfig(object):
key_file = None key_file = None
cert_file = None cert_file = None
alter_network = None alter_network = None
...@@ -1155,14 +1092,14 @@ class Config(object): ...@@ -1155,14 +1092,14 @@ class Config(object):
computer_xml = None computer_xml = None
computer_json = None computer_json = None
input_definition_file = None input_definition_file = None
logger = None
log_file = None log_file = None
output_definition_file = None output_definition_file = None
verbose = None
dry_run = None dry_run = None
console = None
software_user = None software_user = None
def __init__(self, logger):
self.logger = logger
@staticmethod @staticmethod
def checkRequiredBinary(binary_list): def checkRequiredBinary(binary_list):
missing_binary_list = [] missing_binary_list = []
...@@ -1179,33 +1116,26 @@ class Config(object): ...@@ -1179,33 +1116,26 @@ class Config(object):
raise UsageError('Some required binaries are missing or not ' raise UsageError('Some required binaries are missing or not '
'functional: %s' % (','.join(missing_binary_list), )) 'functional: %s' % (','.join(missing_binary_list), ))
def setConfig(self, option_dict, configuration_file_path): def mergeConfig(self, args, configp):
""" """
Set options given by parameters. Set options given by parameters.
Must be executed before setting up the logger.
""" """
self.key_file = None self.key_file = None
self.cert_file = None self.cert_file = None
# set up logging # Set argument parameters
# XXX-Cedric: change code to use global logger for key, value in args.__dict__.items():
self.logger = logger setattr(self, key, value)
# Set options parameters
for option, value in option_dict.__dict__.items():
setattr(self, option, value)
# Load configuration file
configuration_parser = ConfigParser.SafeConfigParser()
if configuration_parser.read(configuration_file_path) != [configuration_file_path]:
raise UsageError('Cannot find or parse configuration file: %s' % configuration_file_path)
# Merges the arguments and configuration # Merges the arguments and configuration
for section in ("slapformat", "slapos"): for section in ("slapformat", "slapos"):
configuration_dict = dict(configuration_parser.items(section)) configuration_dict = dict(configp.items(section))
for key in configuration_dict: for key in configuration_dict:
if not getattr(self, key, None): if not getattr(self, key, None):
setattr(self, key, configuration_dict[key]) setattr(self, key, configuration_dict[key])
def setConfig(self):
# setup some nones # setup some nones
for parameter in ['interface_name', 'partition_base_name', 'user_base_name', for parameter in ['interface_name', 'partition_base_name', 'user_base_name',
'tap_base_name', 'ipv4_local_network', 'ipv6_interface']: 'tap_base_name', 'ipv4_local_network', 'ipv6_interface']:
...@@ -1234,10 +1164,6 @@ class Config(object): ...@@ -1234,10 +1164,6 @@ class Config(object):
if self.create_tap is None: if self.create_tap is None:
self.create_tap = True self.create_tap = True
# Configure logging
if self.console:
self.logger.addHandler(logging.StreamHandler())
# Convert strings to booleans # Convert strings to booleans
for o in ['alter_network', 'alter_user', 'create_tap']: for o in ['alter_network', 'alter_user', 'create_tap']:
attr = getattr(self, o) attr = getattr(self, o)
...@@ -1275,22 +1201,9 @@ class Config(object): ...@@ -1275,22 +1201,9 @@ class Config(object):
if root_needed and os.getuid() != 0: if root_needed and os.getuid() != 0:
message = "Root rights are needed" message = "Root rights are needed"
self.logger.error(message) self.logger.error(message)
sys.stderr.write(message+'\n') sys.stderr.write(message + '\n')
sys.exit() sys.exit()
if self.log_file:
if not os.path.isdir(os.path.dirname(self.log_file)):
# fallback to console only if directory for logs does not exists and
# continue to run
raise ValueError('Please create directory %r to store %r log file' % (
os.path.dirname(self.log_file), self.log_file))
else:
file_handler = logging.FileHandler(self.log_file)
file_handler.setFormatter(logging.Formatter("%(asctime)s - "
"%(name)s - %(levelname)s - %(message)s"))
self.logger.addHandler(file_handler)
self.logger.info('Configured logging to file %r' % self.log_file)
# Check mandatory options # Check mandatory options
for parameter in ('computer_id', 'instance_root', 'master_url', for parameter in ('computer_id', 'instance_root', 'master_url',
'software_root', 'computer_xml'): 'software_root', 'computer_xml'):
...@@ -1307,9 +1220,6 @@ class Config(object): ...@@ -1307,9 +1220,6 @@ class Config(object):
sys.exit(1) sys.exit(1)
self.logger.info("Started.") self.logger.info("Started.")
if self.verbose:
self.logger.setLevel(logging.DEBUG)
self.logger.debug("Verbose mode enabled.")
if self.dry_run: if self.dry_run:
self.logger.info("Dry-run mode enabled.") self.logger.info("Dry-run mode enabled.")
if self.create_tap: if self.create_tap:
...@@ -1325,22 +1235,22 @@ class Config(object): ...@@ -1325,22 +1235,22 @@ class Config(object):
self.output_definition_file = os.path.abspath(self.output_definition_file) self.output_definition_file = os.path.abspath(self.output_definition_file)
def tracing_monkeypatch(conf):
def tracing_monkeypatch(config):
"""Substitute os module and callAndRead function with tracing wrappers.""" """Substitute os module and callAndRead function with tracing wrappers."""
global os global os
global callAndRead global callAndRead
real_callAndRead = callAndRead real_callAndRead = callAndRead
os = OS(config) os = OS(conf)
if config.dry_run: if conf.dry_run:
def dry_callAndRead(argument_list, raise_on_error=True): def dry_callAndRead(argument_list, raise_on_error=True):
if argument_list == ['brctl', 'show']: if argument_list == ['brctl', 'show']:
return real_callAndRead(argument_list, raise_on_error) return real_callAndRead(argument_list, raise_on_error)
else: else:
return 0, '' return 0, ''
callAndRead = dry_callAndRead callAndRead = dry_callAndRead
def fake_getpwnam(user): def fake_getpwnam(user):
class result(object): class result(object):
pw_uid = 12345 pw_uid = 12345
...@@ -1350,40 +1260,7 @@ def tracing_monkeypatch(config): ...@@ -1350,40 +1260,7 @@ def tracing_monkeypatch(config):
else: else:
dry_callAndRead = real_callAndRead dry_callAndRead = real_callAndRead
if config.verbose: def logging_callAndRead(argument_list, raise_on_error=True):
def logging_callAndRead(argument_list, raise_on_error=True): conf.logger.debug(' '.join(argument_list))
config.logger.debug(' '.join(argument_list)) return dry_callAndRead(argument_list, raise_on_error)
return dry_callAndRead(argument_list, raise_on_error) callAndRead = logging_callAndRead
callAndRead = logging_callAndRead
def main(*args):
"Run default configuration."
# Parse arguments
usage = "usage: %s [options] CONFIGURATION_FILE" % sys.argv[0]
options, configuration_file_path = Parser(usage=usage).check_args(args)
config = Config()
try:
config.setConfig(options, configuration_file_path)
except UsageError as err:
sys.stderr.write(err.message + '\n')
sys.stderr.write("For help use --help\n")
sys.exit(1)
tracing_monkeypatch(config)
# Add delay between 0 and 1 hour
# XXX should be the contrary: now by default, and cron should have
# --maximal-delay=3600
if not config.now:
duration = float(60*60) * random.random()
print("Sleeping for %s seconds. To disable this feature, " \
"use with --now parameter in manual." % duration)
time.sleep(duration)
try:
run(config)
except:
config.logger.exception('Uncaught exception:')
raise
slapos/grid/SlapObject.py
View file @ 5f0924ff
...@@ -28,50 +28,59 @@ ...@@ -28,50 +28,59 @@
# #
############################################################################## ##############################################################################
import logging
import os import os
import shutil
import subprocess
import pkg_resources import pkg_resources
import pwd import pwd
import shutil
import stat import stat
import subprocess
import tarfile import tarfile
import tempfile import tempfile
import textwrap import textwrap
import utils
import xmlrpclib import xmlrpclib
from supervisor import xmlrpc from supervisor import xmlrpc
from slapos.grid.utils import (md5digest, getCleanEnvironment,
SlapPopen, dropPrivileges, updateFile)
from slapos.grid import utils # for mocked methods
from slapos.slap.slap import NotFoundError from slapos.slap.slap import NotFoundError
from svcbackend import getSupervisorRPC from slapos.grid.svcbackend import getSupervisorRPC
from exception import BuildoutFailedError, WrongPermissionError, \ from slapos.grid.exception import (BuildoutFailedError, WrongPermissionError,
PathDoesNotExistError PathDoesNotExistError)
from networkcache import download_network_cached, upload_network_cached from slapos.grid.networkcache import download_network_cached, upload_network_cached
from watchdog import getWatchdogID from slapos.grid.watchdog import getWatchdogID
REQUIRED_COMPUTER_PARTITION_PERMISSION = '0750' REQUIRED_COMPUTER_PARTITION_PERMISSION = 0o750
class Software(object): class Software(object):
"""This class is responsible for installing a software release""" """This class is responsible for installing a software release"""
def __init__(self, url, software_root, buildout, def __init__(self, url, software_root, buildout, logger,
signature_private_key_file=None, signature_certificate_list=None, signature_private_key_file=None, signature_certificate_list=None,
upload_cache_url=None, upload_dir_url=None, shacache_cert_file=None, upload_cache_url=None, upload_dir_url=None, shacache_cert_file=None,
shacache_key_file=None, shadir_cert_file=None, shadir_key_file=None, shacache_key_file=None, shadir_cert_file=None, shadir_key_file=None,
download_binary_cache_url=None, upload_binary_cache_url=None, download_binary_cache_url=None, upload_binary_cache_url=None,
download_binary_dir_url=None, upload_binary_dir_url=None, download_binary_dir_url=None, upload_binary_dir_url=None,
download_from_binary_cache_url_blacklist = [], download_from_binary_cache_url_blacklist=None,
upload_to_binary_cache_url_blacklist = []): upload_to_binary_cache_url_blacklist=None):
"""Initialisation of class parameters """Initialisation of class parameters
""" """
if download_from_binary_cache_url_blacklist is None:
download_from_binary_cache_url_blacklist = []
if upload_to_binary_cache_url_blacklist is None:
upload_to_binary_cache_url_blacklist = []
self.url = url self.url = url
self.software_root = software_root self.software_root = software_root
self.software_url_hash = utils.getSoftwareUrlHash(self.url) self.software_url_hash = md5digest(self.url)
self.software_path = os.path.join(self.software_root, self.software_path = os.path.join(self.software_root,
self.software_url_hash) self.software_url_hash)
self.buildout = buildout self.buildout = buildout
self.logger = logging.getLogger('BuildoutManager') self.logger = logger
self.signature_private_key_file = signature_private_key_file self.signature_private_key_file = signature_private_key_file
self.signature_certificate_list = signature_certificate_list self.signature_certificate_list = signature_certificate_list
self.upload_cache_url = upload_cache_url self.upload_cache_url = upload_cache_url
...@@ -116,15 +125,14 @@ class Software(object): ...@@ -116,15 +125,14 @@ class Software(object):
else: else:
self._install_from_buildout() self._install_from_buildout()
# Upload to binary cache if possible and allowed # Upload to binary cache if possible and allowed
if (self.software_root and self.url and self.software_url_hash \ if all([self.software_root, self.url, self.software_url_hash,
and self.upload_binary_cache_url \ self.upload_binary_cache_url, self.upload_binary_dir_url]):
and self.upload_binary_dir_url):
blacklisted = False blacklisted = False
for url in self.upload_to_binary_cache_url_blacklist: for url in self.upload_to_binary_cache_url_blacklist:
if self.url.startswith(url): if self.url.startswith(url):
blacklisted = True blacklisted = True
self.logger.info("Can't upload to binary cache: " self.logger.info("Can't upload to binary cache: "
"Software Release URL is blacklisted.") "Software Release URL is blacklisted.")
break break
if not blacklisted: if not blacklisted:
self.uploadSoftwareRelease(tarpath) self.uploadSoftwareRelease(tarpath)
...@@ -136,8 +144,8 @@ class Software(object): ...@@ -136,8 +144,8 @@ class Software(object):
it. If it fails, we notify the server. it. If it fails, we notify the server.
""" """
root_stat_info = os.stat(self.software_root) root_stat_info = os.stat(self.software_root)
os.environ = utils.getCleanEnvironment(pwd.getpwuid(root_stat_info.st_uid os.environ = getCleanEnvironment(logger=self.logger,
).pw_dir) home_path=pwd.getpwuid(root_stat_info.st_uid).pw_dir)
if not os.path.isdir(self.software_path): if not os.path.isdir(self.software_path):
os.mkdir(self.software_path) os.mkdir(self.software_path)
extends_cache = tempfile.mkdtemp() extends_cache = tempfile.mkdtemp()
...@@ -145,41 +153,44 @@ class Software(object): ...@@ -145,41 +153,44 @@ class Software(object):
# In case when running as root copy ownership, to simplify logic # In case when running as root copy ownership, to simplify logic
for path in [self.software_path, extends_cache]: for path in [self.software_path, extends_cache]:
path_stat_info = os.stat(path) path_stat_info = os.stat(path)
if root_stat_info.st_uid != path_stat_info.st_uid or\ if (root_stat_info.st_uid != path_stat_info.st_uid or
root_stat_info.st_gid != path_stat_info.st_gid: root_stat_info.st_gid != path_stat_info.st_gid):
os.chown(path, root_stat_info.st_uid, os.chown(path, root_stat_info.st_uid,
root_stat_info.st_gid) root_stat_info.st_gid)
try: try:
buildout_parameter_list = [ buildout_parameter_list = [
'buildout:extends-cache=%s' % extends_cache, 'buildout:extends-cache=%s' % extends_cache,
'buildout:directory=%s' % self.software_path,] 'buildout:directory=%s' % self.software_path
]
if self.signature_private_key_file or \ if (self.signature_private_key_file or
self.upload_cache_url or \ self.upload_cache_url or
self.upload_dir_url is not None: self.upload_dir_url):
buildout_parameter_list.append('buildout:networkcache-section=networkcache') buildout_parameter_list.append('buildout:networkcache-section=networkcache')
for buildout_option, value in ( for buildout_option, value in [
('%ssignature-private-key-file=%s', self.signature_private_key_file), ('%ssignature-private-key-file=%s', self.signature_private_key_file),
('%supload-cache-url=%s', self.upload_cache_url), ('%supload-cache-url=%s', self.upload_cache_url),
('%supload-dir-url=%s', self.upload_dir_url), ('%supload-dir-url=%s', self.upload_dir_url),
('%sshacache-cert-file=%s', self.shacache_cert_file), ('%sshacache-cert-file=%s', self.shacache_cert_file),
('%sshacache-key-file=%s', self.shacache_key_file), ('%sshacache-key-file=%s', self.shacache_key_file),
('%sshadir-cert-file=%s', self.shadir_cert_file), ('%sshadir-cert-file=%s', self.shadir_cert_file),
('%sshadir-key-file=%s', self.shadir_key_file), ('%sshadir-key-file=%s', self.shadir_key_file)
): ]:
if value: if value:
buildout_parameter_list.append( \ buildout_parameter_list.append(buildout_option % ('networkcache:', value))
buildout_option % ('networkcache:', value))
buildout_cfg = os.path.join(self.software_path, 'buildout.cfg') buildout_cfg = os.path.join(self.software_path, 'buildout.cfg')
self.createProfileIfMissing(buildout_cfg, self.url) self.createProfileIfMissing(buildout_cfg, self.url)
buildout_parameter_list.extend(['-c', buildout_cfg]) buildout_parameter_list.extend(['-c', buildout_cfg])
utils.bootstrapBuildout(self.software_path, self.buildout, utils.bootstrapBuildout(path=self.software_path,
additional_buildout_parametr_list=buildout_parameter_list) buildout=self.buildout,
utils.launchBuildout(self.software_path, logger=self.logger,
os.path.join(self.software_path, 'bin', 'buildout'), additional_buildout_parametr_list=buildout_parameter_list)
additional_buildout_parametr_list=buildout_parameter_list) utils.launchBuildout(path=self.software_path,
buildout_binary=os.path.join(self.software_path, 'bin', 'buildout'),
logger=self.logger,
additional_buildout_parametr_list=buildout_parameter_list)
finally: finally:
shutil.rmtree(extends_cache) shutil.rmtree(extends_cache)
...@@ -226,7 +237,7 @@ class Software(object): ...@@ -226,7 +237,7 @@ class Software(object):
if func == os.path.islink: if func == os.path.islink:
os.unlink(path) os.unlink(path)
else: else:
os.chmod(path, 0600) os.chmod(path, 0o600)
func(path) func(path)
try: try:
if os.path.exists(self.software_path): if os.path.exists(self.software_path):
...@@ -235,10 +246,8 @@ class Software(object): ...@@ -235,10 +246,8 @@ class Software(object):
else: else:
self.logger.info('Path %r does not exists, no need to remove.' % self.logger.info('Path %r does not exists, no need to remove.' %
self.software_path) self.software_path)
except IOError as error: except IOError as exc:
error_string = "I/O error while removing software (%s): %s" % (self.url, raise IOError("I/O error while removing software (%s): %s" % (self.url, exc))
error)
raise IOError(error_string)
class Partition(object): class Partition(object):
...@@ -256,10 +265,12 @@ class Partition(object): ...@@ -256,10 +265,12 @@ class Partition(object):
server_url, server_url,
software_release_url, software_release_url,
buildout, buildout,
logger,
certificate_repository_path=None, certificate_repository_path=None,
): ):
"""Initialisation of class parameters""" """Initialisation of class parameters"""
self.buildout = buildout self.buildout = buildout
self.logger = logger
self.software_path = software_path self.software_path = software_path
self.instance_path = instance_path self.instance_path = instance_path
self.run_path = os.path.join(self.instance_path, 'etc', 'run') self.run_path = os.path.join(self.instance_path, 'etc', 'run')
...@@ -268,7 +279,6 @@ class Partition(object): ...@@ -268,7 +279,6 @@ class Partition(object):
supervisord_partition_configuration_path supervisord_partition_configuration_path
self.supervisord_socket = supervisord_socket self.supervisord_socket = supervisord_socket
self.computer_partition = computer_partition self.computer_partition = computer_partition
self.logger = logging.getLogger('Partition')
self.computer_id = computer_id self.computer_id = computer_id
self.partition_id = partition_id self.partition_id = partition_id
self.server_url = server_url self.server_url = server_url
...@@ -284,19 +294,18 @@ class Partition(object): ...@@ -284,19 +294,18 @@ class Partition(object):
self._updateCertificate() self._updateCertificate()
def _updateCertificate(self): def _updateCertificate(self):
if not os.path.exists(self.key_file) or \ if not os.path.exists(self.key_file) or not os.path.exists(self.cert_file):
not os.path.exists(self.cert_file):
self.logger.info('Certificate and key not found, downloading to %r and ' self.logger.info('Certificate and key not found, downloading to %r and '
'%r' % (self.cert_file, self.key_file)) '%r' % (self.cert_file, self.key_file))
try: try:
partition_certificate = self.computer_partition.getCertificate() partition_certificate = self.computer_partition.getCertificate()
except NotFoundError: except NotFoundError:
raise NotFoundError('Partition %s is not known from SlapOS Master.' % \ raise NotFoundError('Partition %s is not known from SlapOS Master.' %
self.partition_id) self.partition_id)
open(self.key_file, 'w').write(partition_certificate['key']) open(self.key_file, 'w').write(partition_certificate['key'])
open(self.cert_file, 'w').write(partition_certificate['certificate']) open(self.cert_file, 'w').write(partition_certificate['certificate'])
for f in [self.key_file, self.cert_file]: for f in [self.key_file, self.cert_file]:
os.chmod(f, 0400) os.chmod(f, 0o400)
os.chown(f, *self.getUserGroupId()) os.chown(f, *self.getUserGroupId())
def getUserGroupId(self): def getUserGroupId(self):
...@@ -307,24 +316,24 @@ class Partition(object): ...@@ -307,24 +316,24 @@ class Partition(object):
return (uid, gid) return (uid, gid)
def addServiceToGroup(self, partition_id, def addServiceToGroup(self, partition_id,
runner_list, path, extension = ''): runner_list, path, extension=''):
uid, gid = self.getUserGroupId() uid, gid = self.getUserGroupId()
program_partition_template = pkg_resources.resource_stream(__name__, program_partition_template = pkg_resources.resource_stream(__name__,
'templates/program_partition_supervisord.conf.in').read() 'templates/program_partition_supervisord.conf.in').read()
for runner in runner_list: for runner in runner_list:
self.partition_supervisor_configuration += '\n' + \ self.partition_supervisor_configuration += '\n' + \
program_partition_template % dict( program_partition_template % {
program_id='_'.join([partition_id, runner]), 'program_id': '_'.join([partition_id, runner]),
program_directory=self.instance_path, 'program_directory': self.instance_path,
program_command=os.path.join(path, runner), 'program_command': os.path.join(path, runner),
program_name=runner+extension, 'program_name': runner + extension,
instance_path=self.instance_path, 'instance_path': self.instance_path,
user_id=uid, 'user_id': uid,
group_id=gid, 'group_id': gid,
# As supervisord has no environment to inherit, setup a minimalistic one # As supervisord has no environment to inherit, setup a minimalistic one
HOME=pwd.getpwuid(uid).pw_dir, 'HOME': pwd.getpwuid(uid).pw_dir,
USER=pwd.getpwuid(uid).pw_name, 'USER': pwd.getpwuid(uid).pw_name,
) }
def updateSymlink(self, sr_symlink, software_path): def updateSymlink(self, sr_symlink, software_path):
if os.path.lexists(sr_symlink): if os.path.lexists(sr_symlink):
...@@ -339,7 +348,7 @@ class Partition(object): ...@@ -339,7 +348,7 @@ class Partition(object):
""" Creates configuration file from template in software_path, then """ Creates configuration file from template in software_path, then
installs the software partition with the help of buildout installs the software partition with the help of buildout
""" """
self.logger.info("Installing Computer Partition %s..." \ self.logger.info("Installing Computer Partition %s..."
% self.computer_partition.getId()) % self.computer_partition.getId())
# Checks existence and permissions of Partition directory # Checks existence and permissions of Partition directory
# Note : Partitions have to be created and configured before running slapgrid # Note : Partitions have to be created and configured before running slapgrid
...@@ -351,15 +360,14 @@ class Partition(object): ...@@ -351,15 +360,14 @@ class Partition(object):
self.updateSymlink(sr_symlink, self.software_path) self.updateSymlink(sr_symlink, self.software_path)
instance_stat_info = os.stat(self.instance_path) instance_stat_info = os.stat(self.instance_path)
permission = oct(stat.S_IMODE(instance_stat_info.st_mode)) permission = stat.S_IMODE(instance_stat_info.st_mode)
if permission != REQUIRED_COMPUTER_PARTITION_PERMISSION: if permission != REQUIRED_COMPUTER_PARTITION_PERMISSION:
raise WrongPermissionError('Wrong permissions in %s : actual ' \ raise WrongPermissionError('Wrong permissions in %s: actual '
'permissions are : %s, wanted ' \ 'permissions are: 0%o, wanted are 0%o' %
'are %s' % (self.instance_path, permission,
(self.instance_path, permission, REQUIRED_COMPUTER_PARTITION_PERMISSION))
REQUIRED_COMPUTER_PARTITION_PERMISSION)) os.environ = getCleanEnvironment(logger=self.logger,
os.environ = utils.getCleanEnvironment(pwd.getpwuid( home_path=pwd.getpwuid(instance_stat_info.st_uid).pw_dir)
instance_stat_info.st_uid).pw_dir)
# Generates buildout part from template # Generates buildout part from template
template_location = os.path.join(self.software_path, 'instance.cfg') template_location = os.path.join(self.software_path, 'instance.cfg')
# Backward compatibility: "instance.cfg" file was named "template.cfg". # Backward compatibility: "instance.cfg" file was named "template.cfg".
...@@ -369,23 +377,23 @@ class Partition(object): ...@@ -369,23 +377,23 @@ class Partition(object):
self.logger.debug("Copying %r to %r" % (template_location, config_location)) self.logger.debug("Copying %r to %r" % (template_location, config_location))
try: try:
shutil.copy(template_location, config_location) shutil.copy(template_location, config_location)
except IOError, e: except IOError as exc:
# Template not found on SR, we notify user. # Template not found on SR, we notify user.
raise IOError('Software Release %s is not correctly installed.\n' raise IOError('Software Release %s is not correctly installed.\n%s' % (
'%s' % (self.software_release_url, e)) self.software_release_url, exc))
# fill generated buildout with additional information # fill generated buildout with additional information
buildout_text = open(config_location).read() buildout_text = open(config_location).read()
buildout_text += '\n\n' + pkg_resources.resource_string(__name__, buildout_text += '\n\n' + pkg_resources.resource_string(__name__,
'templates/buildout-tail.cfg.in') % dict( 'templates/buildout-tail.cfg.in') % {
computer_id=self.computer_id, 'computer_id': self.computer_id,
partition_id=self.partition_id, 'partition_id': self.partition_id,
server_url=self.server_url, 'server_url': self.server_url,
software_release_url=self.software_release_url, 'software_release_url': self.software_release_url,
key_file=self.key_file, 'key_file': self.key_file,
cert_file=self.cert_file 'cert_file': self.cert_file,
) }
open(config_location, 'w').write(buildout_text) open(config_location, 'w').write(buildout_text)
os.chmod(config_location, 0640) os.chmod(config_location, 0o640)
# Try to find the best possible buildout: # Try to find the best possible buildout:
# *) if software_root/bin/bootstrap exists use this one to bootstrap # *) if software_root/bin/bootstrap exists use this one to bootstrap
# locally # locally
...@@ -418,12 +426,14 @@ class Partition(object): ...@@ -418,12 +426,14 @@ class Partition(object):
self.logger.debug('Invoking %r in %r' % (' '.join(invocation_list), self.logger.debug('Invoking %r in %r' % (' '.join(invocation_list),
self.instance_path)) self.instance_path))
process_handler = utils.SlapPopen(invocation_list, process_handler = SlapPopen(invocation_list,
preexec_fn=lambda: utils.dropPrivileges(uid, gid), preexec_fn=lambda: dropPrivileges(uid, gid, logger=self.logger),
cwd=self.instance_path, cwd=self.instance_path,
env=utils.getCleanEnvironment(pwd.getpwuid(uid).pw_dir), env=getCleanEnvironment(logger=self.logger,
stdout=subprocess.PIPE, home_path=pwd.getpwuid(uid).pw_dir),
stderr=subprocess.STDOUT) stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
logger=self.logger)
if process_handler.returncode is None or process_handler.returncode != 0: if process_handler.returncode is None or process_handler.returncode != 0:
message = 'Failed to bootstrap buildout in %r.' % (self.instance_path) message = 'Failed to bootstrap buildout in %r.' % (self.instance_path)
self.logger.error(message) self.logger.error(message)
...@@ -432,18 +442,23 @@ class Partition(object): ...@@ -432,18 +442,23 @@ class Partition(object):
if not os.path.exists(buildout_binary): if not os.path.exists(buildout_binary):
# use own buildout generation # use own buildout generation
utils.bootstrapBuildout(self.instance_path, self.buildout, utils.bootstrapBuildout(path=self.instance_path,
['buildout:bin-directory=%s'% os.path.join(self.instance_path, buildout=self.buildout,
'sbin')]) logger=self.logger,
additional_buildout_parameter_list=
['buildout:bin-directory=%s' %
os.path.join(self.instance_path, 'sbin')])
buildout_binary = os.path.join(self.instance_path, 'sbin', 'buildout') buildout_binary = os.path.join(self.instance_path, 'sbin', 'buildout')
# Launches buildout # Launches buildout
utils.launchBuildout(self.instance_path, buildout_binary) utils.launchBuildout(path=self.instance_path,
buildout_binary=buildout_binary,
logger=self.logger)
# Generates supervisord configuration file from template # Generates supervisord configuration file from template
self.logger.info("Generating supervisord config file from template...") self.logger.info("Generating supervisord config file from template...")
# check if CP/etc/run exists and it is a directory # check if CP/etc/run exists and it is a directory
# iterate over each file in CP/etc/run # iterate over each file in CP/etc/run
# iterate over each file in CP/etc/service adding WatchdogID to their name # iterate over each file in CP/etc/service adding WatchdogID to their name
# if at least one is not 0750 raise -- partition has something funny # if at least one is not 0o750 raise -- partition has something funny
runner_list = [] runner_list = []
service_list = [] service_list = []
if os.path.exists(self.run_path): if os.path.exists(self.run_path):
...@@ -461,16 +476,17 @@ class Partition(object): ...@@ -461,16 +476,17 @@ class Partition(object):
partition_id = self.computer_partition.getId() partition_id = self.computer_partition.getId()
group_partition_template = pkg_resources.resource_stream(__name__, group_partition_template = pkg_resources.resource_stream(__name__,
'templates/group_partition_supervisord.conf.in').read() 'templates/group_partition_supervisord.conf.in').read()
self.partition_supervisor_configuration = group_partition_template % dict( self.partition_supervisor_configuration = group_partition_template % {
instance_id=partition_id, 'instance_id': partition_id,
program_list=','.join(['_'.join([partition_id, runner]) 'program_list': ','.join(['_'.join([partition_id, runner])
for runner in runner_list+service_list])) for runner in runner_list + service_list])
# Same method to add to service and run }
self.addServiceToGroup(partition_id, runner_list,self.run_path) # Same method to add to service and run
self.addServiceToGroup(partition_id, service_list,self.service_path, self.addServiceToGroup(partition_id, runner_list, self.run_path)
self.addServiceToGroup(partition_id, service_list, self.service_path,
extension=getWatchdogID()) extension=getWatchdogID())
utils.updateFile(self.supervisord_partition_configuration_path, updateFile(self.supervisord_partition_configuration_path,
self.partition_supervisor_configuration) self.partition_supervisor_configuration)
self.updateSupervisor() self.updateSupervisor()
def start(self): def start(self):
...@@ -481,21 +497,21 @@ class Partition(object): ...@@ -481,21 +497,21 @@ class Partition(object):
partition_id = self.computer_partition.getId() partition_id = self.computer_partition.getId()
try: try:
supervisor.startProcessGroup(partition_id, False) supervisor.startProcessGroup(partition_id, False)
except xmlrpclib.Fault, e: except xmlrpclib.Fault as exc:
if e.faultString.startswith('BAD_NAME:'): if exc.faultString.startswith('BAD_NAME:'):
self.logger.info("Nothing to start on %s..." % \ self.logger.info("Nothing to start on %s..." %
self.computer_partition.getId()) self.computer_partition.getId())
else: else:
self.logger.info("Requested start of %s..." % self.computer_partition.getId()) self.logger.info("Requested start of %s..." % self.computer_partition.getId())
def stop(self): def stop(self):
"""Asks supervisord to stop the instance.""" """Asks supervisord to stop the instance."""
supervisor = self.getSupervisorRPC()
partition_id = self.computer_partition.getId() partition_id = self.computer_partition.getId()
try: try:
supervisor = self.getSupervisorRPC()
supervisor.stopProcessGroup(partition_id, False) supervisor.stopProcessGroup(partition_id, False)
except xmlrpclib.Fault, e: except xmlrpclib.Fault as exc:
if e.faultString.startswith('BAD_NAME:'): if exc.faultString.startswith('BAD_NAME:'):
self.logger.info('Partition %s not known in supervisord, ignoring' % partition_id) self.logger.info('Partition %s not known in supervisord, ignoring' % partition_id)
else: else:
self.logger.info("Requested stop of %s..." % self.computer_partition.getId()) self.logger.info("Requested stop of %s..." % self.computer_partition.getId())
...@@ -503,7 +519,7 @@ class Partition(object): ...@@ -503,7 +519,7 @@ class Partition(object):
def destroy(self): def destroy(self):
"""Destroys the partition and makes it available for subsequent use." """Destroys the partition and makes it available for subsequent use."
""" """
self.logger.info("Destroying Computer Partition %s..." \ self.logger.info("Destroying Computer Partition %s..."
% self.computer_partition.getId()) % self.computer_partition.getId())
# Launches "destroy" binary if exists # Launches "destroy" binary if exists
destroy_executable_location = os.path.join(self.instance_path, 'sbin', destroy_executable_location = os.path.join(self.instance_path, 'sbin',
...@@ -511,12 +527,14 @@ class Partition(object): ...@@ -511,12 +527,14 @@ class Partition(object):
if os.path.exists(destroy_executable_location): if os.path.exists(destroy_executable_location):
uid, gid = self.getUserGroupId() uid, gid = self.getUserGroupId()
self.logger.debug('Invoking %r' % destroy_executable_location) self.logger.debug('Invoking %r' % destroy_executable_location)
process_handler = utils.SlapPopen([destroy_executable_location], process_handler = SlapPopen([destroy_executable_location],
preexec_fn=lambda: utils.dropPrivileges(uid, gid), preexec_fn=lambda: dropPrivileges(uid, gid, logger=self.logger),
cwd=self.instance_path, cwd=self.instance_path,
env=utils.getCleanEnvironment(pwd.getpwuid(uid).pw_dir), env=getCleanEnvironment(logger=self.logger,
stdout=subprocess.PIPE, home_path=pwd.getpwuid(uid).pw_dir),
stderr=subprocess.STDOUT) stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
logger=self.logger)
if process_handler.returncode is None or process_handler.returncode != 0: if process_handler.returncode is None or process_handler.returncode != 0:
message = 'Failed to destroy Computer Partition in %r.' % \ message = 'Failed to destroy Computer Partition in %r.' % \
self.instance_path self.instance_path
...@@ -542,10 +560,8 @@ class Partition(object): ...@@ -542,10 +560,8 @@ class Partition(object):
if os.path.exists(self.supervisord_partition_configuration_path): if os.path.exists(self.supervisord_partition_configuration_path):
os.remove(self.supervisord_partition_configuration_path) os.remove(self.supervisord_partition_configuration_path)
self.updateSupervisor() self.updateSupervisor()
except IOError as error: except IOError as exc:
error_string = "I/O error while freeing partition (%s): %s" \ raise IOError("I/O error while freeing partition (%s): %s" % (self.instance_path, exc))
% (self.instance_path, error)
raise IOError(error_string)
def fetchInformations(self): def fetchInformations(self):
"""Fetch usage informations with buildout, returns it. """Fetch usage informations with buildout, returns it.
...@@ -589,4 +605,3 @@ class Partition(object): ...@@ -589,4 +605,3 @@ class Partition(object):
supervisor.addProcessGroup(gname) supervisor.addProcessGroup(gname)
self.logger.info('Updated %r' % gname) self.logger.info('Updated %r' % gname)
self.logger.debug('Supervisord updated') self.logger.debug('Supervisord updated')
slapos/grid/slapgrid.py
View file @ 5f0924ff
...@@ -28,12 +28,6 @@ ...@@ -28,12 +28,6 @@
# #
############################################################################## ##############################################################################
import argparse
import ConfigParser
from exception import BuildoutFailedError
from hashlib import md5
from lxml import etree
import logging
import os import os
import pkg_resources import pkg_resources
import random import random
...@@ -45,30 +39,22 @@ import tempfile ...@@ -45,30 +39,22 @@ import tempfile
import time import time
import traceback import traceback
import warnings import warnings
if sys.version_info < (2, 6): if sys.version_info < (2, 6):
warnings.warn('Used python version (%s) is old and have problems with' warnings.warn('Used python version (%s) is old and has problems with'
' IPv6 connections' % sys.version.split('\n')[0]) ' IPv6 connections' % sys.version.split('\n')[0])
from lxml import etree
from slapos.slap.slap import NotFoundError from slapos.slap.slap import NotFoundError
from slapos.slap.slap import ServerError from slapos.slap.slap import ServerError
from SlapObject import Software, Partition, WrongPermissionError, \ from slapos.grid.exception import BuildoutFailedError
PathDoesNotExistError from slapos.grid.SlapObject import Software, Partition
from svcbackend import launchSupervisord from slapos.grid.svcbackend import launchSupervisord
from utils import createPrivateDirectory from slapos.grid.utils import (md5digest, createPrivateDirectory, dropPrivileges,
from utils import dropPrivileges SlapPopen, updateFile)
from utils import getSoftwareUrlHash import slapos.slap
from utils import setRunning
from utils import setFinished
from utils import SlapPopen
from utils import updateFile
from slapos import slap
MANDATORY_PARAMETER_LIST = [
'computer_id',
'instance_root',
'master_url',
'software_root',
]
# XXX: should be moved to SLAP library # XXX: should be moved to SLAP library
COMPUTER_PARTITION_DESTROYED_STATE = 'destroyed' COMPUTER_PARTITION_DESTROYED_STATE = 'destroyed'
...@@ -90,310 +76,156 @@ class _formatXMLError(Exception): ...@@ -90,310 +76,156 @@ class _formatXMLError(Exception):
pass pass
def parseArgumentTupleAndReturnSlapgridObject(*argument_tuple): def check_missing_parameters(options):
"""Parses arguments either from command line, from method parameters or from required = set([
config file. Then returns a new instance of slapgrid.Slapgrid with those 'computer_id',
parameters. Also returns the options dict and unused variable list, and 'instance_root',
configures logger. 'master_url',
""" 'software_root',
parser = argparse.ArgumentParser() ])
parser.add_argument("--instance-root",
help="The instance root directory location.") if 'key_file' in options:
parser.add_argument("--software-root", required.add('certificate_repository_path')
help="The software_root directory location.") required.add('cert_file')
parser.add_argument("--master-url", if 'cert_file' in options:
help="The master server URL. Mandatory.") required.add('certificate_repository_path')
parser.add_argument("--computer-id", required.add('key_file')
help="The computer id defined in the server.")
parser.add_argument("--supervisord-socket", missing = required.difference(options)
help="The socket supervisor will use.")
parser.add_argument("--supervisord-configuration-path", if missing:
help="The location where supervisord configuration will be stored.") raise RuntimeError('Missing mandatory parameters: %s' % ', '.join(sorted(missing)))
parser.add_argument("--buildout", default=None,
help="Location of buildout binary.")
parser.add_argument("--pidfile", def check_missing_files(options):
help="The location where pidfile will be created.") req_files = [
parser.add_argument("--logfile", options.get('key_file'),
help="The location where slapgrid logfile will be created.") options.get('cert_file'),
parser.add_argument("--key_file", help="SSL Authorisation key file.") options.get('master_ca_file'),
parser.add_argument("--cert_file", options.get('shacache-cert-file'),
help="SSL Authorisation certificate file.") options.get('shacache-key-file'),
parser.add_argument("--signature_private_key_file", options.get('shadir-cert-file'),
help="Signature private key file.") options.get('shadir-key-file'),
parser.add_argument("--master_ca_file", options.get('signature_private_key_file')
help="Root certificate of SlapOS master key.") ]
parser.add_argument("--certificate_repository_path",
help="Path to directory where downloaded certificates would be stored.") req_dirs = [
parser.add_argument("-c", "--console", action="store_true", default=False, options.get('certificate_repository_path')
help="Deprecated, doesn't do anything.") ]
parser.add_argument("-v", "--verbose", action="store_true", default=False,
help="Be verbose.") for f in req_files:
parser.add_argument("--maximum-periodicity", type=int, default=None, if f and not os.path.exists(f):
help="Periodicity at which buildout should be run in instance.") raise RuntimeError('File %r does not exist.' % f)
parser.add_argument("--promise-timeout", type=int, default=3,
help="Promise timeout in seconds.") for d in req_dirs:
parser.add_argument("--now", action="store_true", default=False, if d and not os.path.isdir(d):
help="Launch slapgrid without delay. Default behavior.") raise RuntimeError('Directory %r does not exist' % d)
parser.add_argument("--all", action="store_true", default=False,
help="Launch slapgrid to process all Softare Releases "
"and/or Computer Partitions.") def merged_options(args, configp):
parser.add_argument("--only-sr", options = dict(configp.items('slapos'))
help="Force the update of a single software release (use url hash), "
"even if is already installed. This option will make all others " if configp.has_section('networkcache'):
"sofware releases be ignored.") options.update(dict(configp.items('networkcache')))
parser.add_argument("--only-cp", for key, value in vars(args).iteritems():
help="Update a single or a list of computer partitions " if value is not None:
"(ie.:slappartX, slappartY)," options[key] = value
"this option will make all others computer partitions be ignored.")
if options.get('all'):
parser.add_argument("configuration_file", nargs=1, type=argparse.FileType(), options['develop'] = True
help="SlapOS configuration file.")
if options.get('maximum_periodicity') is not None:
# Deprecated options options['force_periodicity'] = True
parser.add_argument("--develop", action="store_true", default=False,
help="Deprecated, same as --all.")
parser.add_argument("--only_sr",
help="Deprecated, same as --only-sr.")
parser.add_argument("--only_cp",
help="Deprecated, same as --only-cp.")
parser.add_argument("--maximal_delay",
help="Deprecated. Will only work from configuration file in the future.")
# Parses arguments
if argument_tuple == ():
# No arguments given to entry point : we parse sys.argv.
argument_option_instance = parser.parse_args()
else:
argument_option_instance = \
parser.parse_args(list(argument_tuple))
# Parses arguments from config file, if needed, then merge previous arguments
option_dict = {}
configuration_file = argument_option_instance.configuration_file[0]
# Loads config (if config specified)
slapgrid_configuration = ConfigParser.SafeConfigParser()
slapgrid_configuration.readfp(configuration_file)
# Merges the two dictionnaries
option_dict = dict(slapgrid_configuration.items("slapos"))
if slapgrid_configuration.has_section("networkcache"):
option_dict.update(dict(slapgrid_configuration.items("networkcache")))
for argument_key, argument_value in vars(argument_option_instance
).iteritems():
if argument_value is not None:
option_dict.update({argument_key: argument_value})
# Configures logger.
if option_dict['verbose']:
level = logging.DEBUG
else:
level = logging.INFO
logging.basicConfig(format='%(asctime)s %(levelname)-8s %(message)s',
level=level,
datefmt='%Y-%m-%dT%H:%M:%S')
if option_dict.get('logfile'):
console = logging.FileHandler(option_dict['logfile'])
console.setLevel(level)
console.setFormatter(logging.Formatter(
'%(asctime)s %(name)-18s: %(levelname)-8s %(message)s'))
logging.getLogger('').addHandler(console)
missing_mandatory_parameter_list = []
for mandatory_parameter in MANDATORY_PARAMETER_LIST:
if not mandatory_parameter in option_dict:
missing_mandatory_parameter_list.append(mandatory_parameter)
if option_dict.get('all') is True:
option_dict['develop'] = True
if option_dict.get('maximum_periodicity') is not None:
option_dict['force_periodicity'] = True
repository_required = False
if 'key_file' in option_dict:
repository_required = True
if not 'cert_file' in option_dict:
missing_mandatory_parameter_list.append('cert_file')
if 'cert_file' in option_dict:
repository_required = True
if not 'key_file' in option_dict:
missing_mandatory_parameter_list.append('key_file')
if repository_required:
if 'certificate_repository_path' not in option_dict:
missing_mandatory_parameter_list.append('certificate_repository_path')
if len(missing_mandatory_parameter_list) > 0:
parser.error('Missing mandatory parameters:\n%s' % '\n'.join(
missing_mandatory_parameter_list))
key_file = option_dict.get('key_file')
cert_file = option_dict.get('cert_file')
master_ca_file = option_dict.get('master_ca_file')
signature_private_key_file = option_dict.get('signature_private_key_file')
mandatory_file_list = [key_file, cert_file, master_ca_file]
# signature_private_key_file is not mandatory, we must be able to run
# slapgrid scripts without this parameter.
if signature_private_key_file:
mandatory_file_list.append(signature_private_key_file)
for k in ['shacache-cert-file', 'shacache-key-file', 'shadir-cert-file',
'shadir-key-file']:
mandatory_file_list.append(option_dict.get(k, None))
for f in mandatory_file_list:
if f is not None:
if not os.path.exists(f):
parser.error('File %r does not exist.' % f)
certificate_repository_path = option_dict.get('certificate_repository_path')
if certificate_repository_path is not None:
if not os.path.isdir(certificate_repository_path):
parser.error('Directory %r does not exist' %
certificate_repository_path)
# Supervisord configuration location # Supervisord configuration location
if not option_dict.get('supervisord_configuration_path'): if not options.get('supervisord_configuration_path'):
option_dict['supervisord_configuration_path'] = \ options['supervisord_configuration_path'] = \
os.path.join(option_dict['instance_root'], 'etc', 'supervisord.conf') os.path.join(options['instance_root'], 'etc', 'supervisord.conf')
# Supervisord socket # Supervisord socket
if not option_dict.get('supervisord_socket'): if not options.get('supervisord_socket'):
option_dict['supervisord_socket'] = \ options['supervisord_socket'] = \
os.path.join(option_dict['instance_root'], 'supervisord.socket') os.path.join(options['instance_root'], 'supervisord.socket')
signature_certificate_list_string = \
option_dict.get('signature-certificate-list', None)
if signature_certificate_list_string is not None:
cert_marker = "-----BEGIN CERTIFICATE-----"
signature_certificate_list = [cert_marker + '\n' + q.strip() \
for q in signature_certificate_list_string.split(cert_marker) \
if q.strip()]
else:
signature_certificate_list = None
# Parse cache / binary cache options # Parse cache / binary cache options
# Backward compatibility about "binary-cache-url-blacklist" deprecated option # Backward compatibility about "binary-cache-url-blacklist" deprecated option
if option_dict.get("binary-cache-url-blacklist") and not \ if options.get("binary-cache-url-blacklist") and not \
option_dict.get("download-from-binary-cache-url-blacklist"): options.get("download-from-binary-cache-url-blacklist"):
option_dict["download-from-binary-cache-url-blacklist"] = \ options["download-from-binary-cache-url-blacklist"] = \
option_dict["binary-cache-url-blacklist"] options["binary-cache-url-blacklist"]
option_dict["download-from-binary-cache-url-blacklist"] = [ options["download-from-binary-cache-url-blacklist"] = [
url.strip() for url in option_dict.get( url.strip() for url in options.get(
"download-from-binary-cache-url-blacklist", "").split('\n') if url] "download-from-binary-cache-url-blacklist", "").split('\n') if url]
option_dict["upload-to-binary-cache-url-blacklist"] = [ options["upload-to-binary-cache-url-blacklist"] = [
url.strip() for url in option_dict.get( url.strip() for url in options.get(
"upload-to-binary-cache-url-blacklist", "").split('\n') if url] "upload-to-binary-cache-url-blacklist", "").split('\n') if url]
# Sleep for a random time to avoid SlapOS Master being DDOSed by an army of return options
# SlapOS Nodes configured with cron.
if option_dict["now"]:
# XXX-Cedric: deprecate "--now"
maximal_delay = 0
else:
maximal_delay = int(option_dict.get("maximal_delay", "0"))
if maximal_delay > 0:
duration = random.randint(1, maximal_delay)
logging.info("Sleeping for %s seconds. To disable this feature, " \
"check --now parameter in slapgrid help." % duration)
time.sleep(duration)
# Return new Slapgrid instance and options
return ([Slapgrid(software_root=option_dict['software_root'],
instance_root=option_dict['instance_root'],
master_url=option_dict['master_url'],
computer_id=option_dict['computer_id'],
supervisord_socket=option_dict['supervisord_socket'],
supervisord_configuration_path=option_dict[
'supervisord_configuration_path'],
key_file=key_file,
cert_file=cert_file,
master_ca_file=master_ca_file,
certificate_repository_path=certificate_repository_path,
signature_private_key_file=signature_private_key_file,
signature_certificate_list=signature_certificate_list,
download_binary_cache_url=\
option_dict.get('download-binary-cache-url', None),
upload_binary_cache_url=\
option_dict.get('upload-binary-cache-url', None),
download_from_binary_cache_url_blacklist=\
option_dict.get('download-from-binary-cache-url-blacklist', []),
upload_to_binary_cache_url_blacklist=\
option_dict.get('upload-to-binary-cache-url-blacklist', []),
upload_cache_url=option_dict.get('upload-cache-url', None),
download_binary_dir_url=\
option_dict.get('download-binary-dir-url', None),
upload_binary_dir_url=\
option_dict.get('upload-binary-dir-url', None),
upload_dir_url=option_dict.get('upload-dir-url', None),
buildout=option_dict.get('buildout'),
promise_timeout=option_dict['promise_timeout'],
shacache_cert_file=option_dict.get('shacache-cert-file', None),
shacache_key_file=option_dict.get('shacache-key-file', None),
shadir_cert_file=option_dict.get('shadir-cert-file', None),
shadir_key_file=option_dict.get('shadir-key-file', None),
develop=option_dict.get('develop', False),
software_release_filter_list=option_dict.get('only-sr',
# Try to fetch from deprecated argument
option_dict.get('only_sr', None)),
computer_partition_filter_list=option_dict.get('only-cp',
# Try to fetch from deprecated argument
option_dict.get('only_cp', None)),
force_periodicity = option_dict.get('force_periodicity', False),
maximum_periodicity = option_dict.get('maximum_periodicity', 86400),
),
option_dict])
def realRun(argument_tuple, method_list):
slapgrid_object, option_dict = \
parseArgumentTupleAndReturnSlapgridObject(*argument_tuple)
pidfile = option_dict.get('pidfile')
if pidfile:
setRunning(pidfile)
try:
failed = False
failed_promise = False
for method in method_list:
# Quite complicated way to figure out if everything went fine
return_value = getattr(slapgrid_object, method)()
if return_value == SLAPGRID_FAIL:
failed = True
if return_value == SLAPGRID_PROMISE_FAIL:
failed_promise = True
finally:
if pidfile:
setFinished(pidfile)
if failed:
sys.exit(SLAPGRID_FAIL)
if failed_promise:
sys.exit(SLAPGRID_PROMISE_FAIL)
sys.exit(SLAPGRID_SUCCESS)
def run(*argument_tuple):
"""Hooks for generic entry point to proces Software Releases (sr),
Computer Partitions (cp) and Usage Reports (ur)
Will run one by one each task (sr, cp, ur). If specified,
will run in the user wanted order.
"""
realRun(argument_tuple, ['processSoftwareReleaseList',
'processComputerPartitionList', 'agregateAndSendUsage'])
def runSoftwareRelease(*argument_tuple): def random_delay(options, logger):
"""Hook for entry point to process Software Releases only
""" """
realRun(argument_tuple, ['processSoftwareReleaseList']) Sleep for a random time to avoid SlapOS Master being DDOSed by an army of
SlapOS Nodes configured with cron.
def runComputerPartition(*argument_tuple):
"""Hook for entry point to process Computer Partitions only
""" """
realRun(argument_tuple, ['processComputerPartitionList']) if options['now']:
# XXX-Cedric: deprecate '--now'
return
maximal_delay = int(options.get('maximal_delay', '0'))
if maximal_delay:
duration = random.randint(1, maximal_delay)
logger.info('Sleeping for %s seconds. To disable this feature, ' \
'check --now parameter in slapgrid help.' % duration)
time.sleep(duration)
def runUsageReport(*argument_tuple): def create_slapgrid_object(options, logger):
"""Hook for entry point to process Usage Reports only signature_certificate_list = None
""" if 'signature-certificate-list' in options:
realRun(argument_tuple, ['agregateAndSendUsage']) cert_marker = '-----BEGIN CERTIFICATE-----'
signature_certificate_list = [
cert_marker + '\n' + q.strip()
for q in options['signature-certificate-list'].split(cert_marker)
if q.strip()
]
op = options
return Slapgrid(software_root=op['software_root'],
instance_root=op['instance_root'],
master_url=op['master_url'],
computer_id=op['computer_id'],
supervisord_socket=op['supervisord_socket'],
supervisord_configuration_path=op['supervisord_configuration_path'],
buildout=op.get('buildout'),
logger=logger,
force_periodicity = op.get('force_periodicity', False),
maximum_periodicity = op.get('maximum_periodicity', 86400),
key_file=op.get('key_file'),
cert_file=op.get('cert_file'),
signature_private_key_file=op.get('signature_private_key_file'),
signature_certificate_list=signature_certificate_list,
download_binary_cache_url=op.get('download-binary-cache-url'),
upload_binary_cache_url=op.get('upload-binary-cache-url'),
download_from_binary_cache_url_blacklist=\
op.get('download-from-binary-cache-url-blacklist', []),
upload_to_binary_cache_url_blacklist=\
op.get('upload-to-binary-cache-url-blacklist', []),
upload_cache_url=op.get('upload-cache-url'),
download_binary_dir_url=op.get('download-binary-dir-url'),
upload_binary_dir_url=op.get('upload-binary-dir-url'),
upload_dir_url=op.get('upload-dir-url'),
master_ca_file=op.get('master_ca_file'),
certificate_repository_path=op.get('certificate_repository_path'),
promise_timeout=op['promise_timeout'],
shacache_cert_file=op.get('shacache-cert-file'),
shacache_key_file=op.get('shacache-key-file'),
shadir_cert_file=op.get('shadir-cert-file'),
shadir_key_file=op.get('shadir-key-file'),
develop=op.get('develop', False),
# Try to fetch from deprecated argument
software_release_filter_list=op.get('only-sr', op.get('only_sr')),
# Try to fetch from deprecated argument
computer_partition_filter_list=op.get('only-cp', op.get('only_cp')))
class Slapgrid(object): class Slapgrid(object):
...@@ -412,6 +244,7 @@ class Slapgrid(object): ...@@ -412,6 +244,7 @@ class Slapgrid(object):
supervisord_socket, supervisord_socket,
supervisord_configuration_path, supervisord_configuration_path,
buildout, buildout,
logger,
force_periodicity=False, force_periodicity=False,
maximum_periodicity=86400, maximum_periodicity=86400,
key_file=None, key_file=None,
...@@ -465,17 +298,15 @@ class Slapgrid(object): ...@@ -465,17 +298,15 @@ class Slapgrid(object):
self.shacache_key_file = shacache_key_file self.shacache_key_file = shacache_key_file
self.shadir_cert_file = shadir_cert_file self.shadir_cert_file = shadir_cert_file
self.shadir_key_file = shadir_key_file self.shadir_key_file = shadir_key_file
# Configures logger self.logger = logger
self.logger = logging.getLogger('Slapgrid')
# Creates objects from slap module # Creates objects from slap module
self.slap = slap.slap() self.slap = slapos.slap.slap()
self.slap.initializeConnection(self.master_url, key_file=self.key_file, self.slap.initializeConnection(self.master_url, key_file=self.key_file,
cert_file=self.cert_file, master_ca_file=self.master_ca_file) cert_file=self.cert_file, master_ca_file=self.master_ca_file)
self.computer = self.slap.registerComputer(self.computer_id) self.computer = self.slap.registerComputer(self.computer_id)
# Defines all needed paths # Defines all needed paths
self.instance_etc_directory = os.path.join(self.instance_root, 'etc')
self.supervisord_configuration_directory = \ self.supervisord_configuration_directory = \
os.path.join(self.instance_etc_directory, 'supervisord.conf.d') os.path.join(self.instance_root, 'etc', 'supervisord.conf.d')
self.buildout = buildout self.buildout = buildout
self.promise_timeout = promise_timeout self.promise_timeout = promise_timeout
self.develop = develop self.develop = develop
...@@ -494,7 +325,7 @@ class Slapgrid(object): ...@@ -494,7 +325,7 @@ class Slapgrid(object):
def getWatchdogLine(self): def getWatchdogLine(self):
invocation_list = [WATCHDOG_PATH] invocation_list = [WATCHDOG_PATH]
invocation_list.append("--master-url '%s' " % self.master_url) invocation_list.append("--master-url '%s' " % self.master_url)
if self.certificate_repository_path is not None: if self.certificate_repository_path:
invocation_list.append("--certificate-repository-path '%s'" \ invocation_list.append("--certificate-repository-path '%s'" \
% self.certificate_repository_path) % self.certificate_repository_path)
invocation_list.append("--computer-id '%s'" % self.computer_id) invocation_list.append("--computer-id '%s'" % self.computer_id)
...@@ -506,64 +337,58 @@ class Slapgrid(object): ...@@ -506,64 +337,58 @@ class Slapgrid(object):
""" """
# Checks for software_root and instance_root existence # Checks for software_root and instance_root existence
if not os.path.isdir(self.software_root): if not os.path.isdir(self.software_root):
error = "%s does not exist." % self.software_root raise OSError('%s does not exist.' % self.software_root)
raise OSError(error)
if not os.path.isdir(self.instance_root): if not os.path.isdir(self.instance_root):
error = "%s does not exist." % self.instance_root raise OSError('%s does not exist.' % self.instance_root)
raise OSError(error)
# Creates everything needed # Creates everything needed
try:
# Creates instance_root structure # Creates instance_root structure
createPrivateDirectory(self.instance_etc_directory) createPrivateDirectory(os.path.join(self.instance_root, 'var'))
createPrivateDirectory(os.path.join(self.instance_root, 'var')) createPrivateDirectory(os.path.join(self.instance_root, 'var', 'log'))
createPrivateDirectory(os.path.join(self.instance_root, 'var', 'log')) createPrivateDirectory(os.path.join(self.instance_root, 'var', 'run'))
createPrivateDirectory(os.path.join(self.instance_root, 'var', 'run'))
createPrivateDirectory(self.supervisord_configuration_directory) createPrivateDirectory(os.path.join(self.instance_root, 'etc'))
# Creates supervisord configuration createPrivateDirectory(self.supervisord_configuration_directory)
updateFile(self.supervisord_configuration_path,
pkg_resources.resource_stream(__name__, # Creates supervisord configuration
'templates/supervisord.conf.in').read() % dict( updateFile(self.supervisord_configuration_path,
supervisord_configuration_directory=\ pkg_resources.resource_stream(__name__,
self.supervisord_configuration_directory, 'templates/supervisord.conf.in').read() % {
supervisord_socket=os.path.abspath(self.supervisord_socket), 'supervisord_configuration_directory': self.supervisord_configuration_directory,
supervisord_loglevel='info', 'supervisord_socket': os.path.abspath(self.supervisord_socket),
supervisord_logfile=os.path.abspath(os.path.join( 'supervisord_loglevel': 'info',
self.instance_root, 'var', 'log', 'supervisord.log')), 'supervisord_logfile': os.path.abspath(os.path.join(self.instance_root, 'var', 'log', 'supervisord.log')),
supervisord_logfile_maxbytes='50MB', 'supervisord_logfile_maxbytes': '50MB',
supervisord_nodaemon='false', 'supervisord_nodaemon': 'false',
supervisord_pidfile=os.path.abspath(os.path.join( 'supervisord_pidfile': os.path.abspath(os.path.join(self.instance_root, 'var', 'run', 'supervisord.pid')),
self.instance_root, 'var', 'run', 'supervisord.pid')), 'supervisord_logfile_backups': '10',
supervisord_logfile_backups='10', 'watchdog_command': self.getWatchdogLine(),
watchdog_command = self.getWatchdogLine(), })
))
except (WrongPermissionError, PathDoesNotExistError) as error:
raise error
def getComputerPartitionList(self): def getComputerPartitionList(self):
try: try:
computer_partition_list = self.computer.getComputerPartitionList() return self.computer.getComputerPartitionList()
except socket.error as error: except socket.error as exc:
self.logger.fatal(error) self.logger.fatal(exc)
raise raise
return computer_partition_list
def processSoftwareReleaseList(self): def processSoftwareReleaseList(self):
"""Will process each Software Release. """Will process each Software Release.
""" """
self.checkEnvironmentAndCreateStructure() self.checkEnvironmentAndCreateStructure()
logger = logging.getLogger('SoftwareReleases') self.logger.info('Processing software releases...')
logger.info("Processing software releases...")
# Boolean to know if every instance has correctly been deployed # Boolean to know if every instance has correctly been deployed
clean_run = True clean_run = True
for software_release in self.computer.getSoftwareReleaseList(): for software_release in self.computer.getSoftwareReleaseList():
state = software_release.getState() state = software_release.getState()
try: try:
software_release_uri = software_release.getURI() software_release_uri = software_release.getURI()
url_hash = md5(software_release_uri).hexdigest() url_hash = md5digest(software_release_uri)
software_path = os.path.join(self.software_root, url_hash) software_path = os.path.join(self.software_root, url_hash)
software = Software(url=software_release_uri, software = Software(url=software_release_uri,
software_root=self.software_root, software_root=self.software_root,
buildout=self.buildout, buildout=self.buildout,
logger=self.logger,
signature_private_key_file=self.signature_private_key_file, signature_private_key_file=self.signature_private_key_file,
signature_certificate_list=self.signature_certificate_list, signature_certificate_list=self.signature_certificate_list,
download_binary_cache_url=self.download_binary_cache_url, download_binary_cache_url=self.download_binary_cache_url,
...@@ -585,43 +410,40 @@ class Slapgrid(object): ...@@ -585,43 +410,40 @@ class Slapgrid(object):
if self.develop or (not os.path.exists(completed_tag) and \ if self.develop or (not os.path.exists(completed_tag) and \
len(self.software_release_filter_list) == 0) or \ len(self.software_release_filter_list) == 0) or \
url_hash in self.software_release_filter_list or \ url_hash in self.software_release_filter_list or \
url_hash in (md5(uri).hexdigest() for uri in self.software_release_filter_list): url_hash in (md5digest(uri) for uri in self.software_release_filter_list):
try: try:
software_release.building() software_release.building()
except NotFoundError: except NotFoundError:
pass pass
software.install() software.install()
file_descriptor = open(completed_tag, 'w') with open(completed_tag, 'w') as fout:
file_descriptor.write(time.asctime()) fout.write(time.asctime())
file_descriptor.close()
elif state == 'destroyed': elif state == 'destroyed':
if os.path.exists(software_path): if os.path.exists(software_path):
logger.info('Destroying %r...' % software_release_uri) self.logger.info('Destroying %r...' % software_release_uri)
software.destroy() software.destroy()
logger.info('Destroyed %r.' % software_release_uri) self.logger.info('Destroyed %r.' % software_release_uri)
# Send log before exiting # Send log before exiting
except (SystemExit, KeyboardInterrupt): except (SystemExit, KeyboardInterrupt):
exception = traceback.format_exc() software_release.error(traceback.format_exc(), logger=self.logger)
software_release.error(exception)
raise raise
# Buildout failed: send log but don't print it to output (already done) # Buildout failed: send log but don't print it to output (already done)
except BuildoutFailedError as exception: except BuildoutFailedError as exc:
clean_run = False clean_run = False
try: try:
software_release.error(exception) software_release.error(exc, logger=self.logger)
except (SystemExit, KeyboardInterrupt): except (SystemExit, KeyboardInterrupt):
raise raise
except Exception: except Exception:
exception = traceback.format_exc() self.logger.error('Problem while reporting error, continuing:\n%s' %
logger.error('Problem during reporting error, continuing:\n' + traceback.format_exc())
exception)
# For everything else: log it, send it, continue. # For everything else: log it, send it, continue.
except Exception: except Exception:
exception = traceback.format_exc() exc = traceback.format_exc()
logger.error(exception) self.logger.error(exc)
software_release.error(exception) software_release.error(exc, logger=self.logger)
clean_run = False clean_run = False
else: else:
if state == 'available': if state == 'available':
...@@ -634,7 +456,7 @@ class Slapgrid(object): ...@@ -634,7 +456,7 @@ class Slapgrid(object):
software_release.destroyed() software_release.destroyed()
except (NotFoundError, ServerError): except (NotFoundError, ServerError):
print traceback.format_exc() print traceback.format_exc()
logger.info("Finished software releases.") self.logger.info('Finished software releases.')
# Return success value # Return success value
if not clean_run: if not clean_run:
...@@ -644,12 +466,12 @@ class Slapgrid(object): ...@@ -644,12 +466,12 @@ class Slapgrid(object):
def _launchSupervisord(self): def _launchSupervisord(self):
launchSupervisord(self.supervisord_socket, launchSupervisord(self.supervisord_socket,
self.supervisord_configuration_path) self.supervisord_configuration_path,
logger=self.logger)
def _checkPromises(self, computer_partition): def _checkPromises(self, computer_partition):
self.logger.info("Checking promises...") self.logger.info("Checking promises...")
instance_path = os.path.join(self.instance_root, instance_path = os.path.join(self.instance_root, computer_partition.getId())
computer_partition.getId())
uid, gid = None, None uid, gid = None, None
stat_info = os.stat(instance_path) stat_info = os.stat(instance_path)
...@@ -662,11 +484,8 @@ class Slapgrid(object): ...@@ -662,11 +484,8 @@ class Slapgrid(object):
# Get the list of promises # Get the list of promises
promise_dir = os.path.join(instance_path, 'etc', 'promise') promise_dir = os.path.join(instance_path, 'etc', 'promise')
if os.path.exists(promise_dir) and os.path.isdir(promise_dir): if os.path.exists(promise_dir) and os.path.isdir(promise_dir):
cwd = instance_path
promises_list = os.listdir(promise_dir)
# Check whether every promise is kept # Check whether every promise is kept
for promise in promises_list: for promise in os.listdir(promise_dir):
promise_present = True promise_present = True
command = [os.path.join(promise_dir, promise)] command = [os.path.join(promise_dir, promise)]
...@@ -674,13 +493,13 @@ class Slapgrid(object): ...@@ -674,13 +493,13 @@ class Slapgrid(object):
promise = os.path.basename(command[0]) promise = os.path.basename(command[0])
self.logger.info("Checking promise %r.", promise) self.logger.info("Checking promise %r.", promise)
kw = dict(stdout=subprocess.PIPE, stderr=subprocess.PIPE,
stdin=subprocess.PIPE)
process_handler = subprocess.Popen(command, process_handler = subprocess.Popen(command,
preexec_fn=lambda: dropPrivileges(uid, gid), preexec_fn=lambda: dropPrivileges(uid, gid, logger=self.logger),
cwd=cwd, cwd=instance_path,
env=None if sys.platform == 'cygwin' else {}, **kw) env=None if sys.platform == 'cygwin' else {},
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE)
process_handler.stdin.flush() process_handler.stdin.flush()
process_handler.stdin.close() process_handler.stdin.close()
process_handler.stdin = None process_handler.stdin = None
...@@ -705,8 +524,6 @@ class Slapgrid(object): ...@@ -705,8 +524,6 @@ class Slapgrid(object):
""" """
Process a Computer Partition, depending on its state Process a Computer Partition, depending on its state
""" """
logger = logging.getLogger('ComputerPartitionProcessing')
computer_partition_id = computer_partition.getId() computer_partition_id = computer_partition.getId()
# Sanity checks before processing # Sanity checks before processing
...@@ -720,7 +537,7 @@ class Slapgrid(object): ...@@ -720,7 +537,7 @@ class Slapgrid(object):
(computer_partition_id not in self.computer_partition_filter_list): (computer_partition_id not in self.computer_partition_filter_list):
return return
logger.info('Processing Computer Partition %s...' % computer_partition_id) self.logger.info('Processing Computer Partition %s.' % computer_partition_id)
instance_path = os.path.join(self.instance_root, computer_partition_id) instance_path = os.path.join(self.instance_root, computer_partition_id)
...@@ -742,13 +559,16 @@ class Slapgrid(object): ...@@ -742,13 +559,16 @@ class Slapgrid(object):
# Try to process it anyway, it may need to be deleted. # Try to process it anyway, it may need to be deleted.
software_url = None software_url = None
try: try:
software_path = os.path.join(self.software_root, software_path = os.path.join(self.software_root, md5digest(software_url))
getSoftwareUrlHash(software_url))
except TypeError: except TypeError:
# Problem with instance: SR URI not set. # Problem with instance: SR URI not set.
# Try to process it anyway, it may need to be deleted. # Try to process it anyway, it may need to be deleted.
software_path = None software_path = None
self.logger.info(' Software URL: %s' % software_url)
self.logger.info(' Software path: %s' % software_path)
self.logger.info(' Instance path: %s' % instance_path)
periodicity = self.maximum_periodicity periodicity = self.maximum_periodicity
if software_path: if software_path:
# Get periodicity from periodicity file if not forced # Get periodicity from periodicity file if not forced
...@@ -759,8 +579,7 @@ class Slapgrid(object): ...@@ -759,8 +579,7 @@ class Slapgrid(object):
periodicity = int(open(periodicity_path).read()) periodicity = int(open(periodicity_path).read())
except ValueError: except ValueError:
os.remove(periodicity_path) os.remove(periodicity_path)
exception = traceback.format_exc() self.logger.error(traceback.format_exc())
logger.error(exception)
# Check if timestamp from server is more recent than local one. # Check if timestamp from server is more recent than local one.
# If not: it's not worth processing this partition (nothing has # If not: it's not worth processing this partition (nothing has
...@@ -786,8 +605,7 @@ class Slapgrid(object): ...@@ -786,8 +605,7 @@ class Slapgrid(object):
os.remove(timestamp_path) os.remove(timestamp_path)
except ValueError: except ValueError:
os.remove(timestamp_path) os.remove(timestamp_path)
exception = traceback.format_exc() self.logger.error(traceback.format_exc())
logger.error(exception)
local_partition = Partition( local_partition = Partition(
software_path=software_path, software_path=software_path,
...@@ -802,7 +620,8 @@ class Slapgrid(object): ...@@ -802,7 +620,8 @@ class Slapgrid(object):
server_url=self.master_url, server_url=self.master_url,
software_release_url=software_url, software_release_url=software_url,
certificate_repository_path=self.certificate_repository_path, certificate_repository_path=self.certificate_repository_path,
buildout=self.buildout) buildout=self.buildout,
logger=self.logger)
computer_partition_state = computer_partition.getState() computer_partition_state = computer_partition.getState()
if computer_partition_state == COMPUTER_PARTITION_STARTED_STATE: if computer_partition_state == COMPUTER_PARTITION_STARTED_STATE:
...@@ -826,15 +645,14 @@ class Slapgrid(object): ...@@ -826,15 +645,14 @@ class Slapgrid(object):
try: try:
computer_partition.stopped() computer_partition.stopped()
except (SystemExit, KeyboardInterrupt): except (SystemExit, KeyboardInterrupt):
exception = traceback.format_exc() computer_partition.error(traceback.format_exc(), logger=self.logger)
computer_partition.error(exception)
raise raise
except Exception: except Exception:
pass pass
else: else:
error_string = "Computer Partition %r has unsupported state: %s" % \ error_string = "Computer Partition %r has unsupported state: %s" % \
(computer_partition_id, computer_partition_state) (computer_partition_id, computer_partition_state)
computer_partition.error(error_string) computer_partition.error(error_string, logger=self.logger)
raise NotImplementedError(error_string) raise NotImplementedError(error_string)
# If partition has been successfully processed, write timestamp # If partition has been successfully processed, write timestamp
...@@ -849,7 +667,6 @@ class Slapgrid(object): ...@@ -849,7 +667,6 @@ class Slapgrid(object):
""" """
Try to filter valid partitions to be processed from free partitions. Try to filter valid partitions to be processed from free partitions.
""" """
logger = logging.getLogger('ComputerPartitionProcessing')
filtered_computer_partition_list = [] filtered_computer_partition_list = []
for computer_partition in computer_partition_list: for computer_partition in computer_partition_list:
try: try:
...@@ -880,32 +697,29 @@ class Slapgrid(object): ...@@ -880,32 +697,29 @@ class Slapgrid(object):
# Send log before exiting # Send log before exiting
except (SystemExit, KeyboardInterrupt): except (SystemExit, KeyboardInterrupt):
exception = traceback.format_exc() computer_partition.error(traceback.format_exc(), logger=self.logger)
computer_partition.error(exception)
raise raise
# Buildout failed: send log but don't print it to output (already done) # Buildout failed: send log but don't print it to output (already done)
except BuildoutFailedError, exception: except BuildoutFailedError as exc:
try: try:
computer_partition.error(exception) computer_partition.error(exc, logger=self.logger)
except (SystemExit, KeyboardInterrupt): except (SystemExit, KeyboardInterrupt):
raise raise
except Exception: except Exception:
exception = traceback.format_exc() self.logger.error('Problem during reporting error, continuing:\n%s' %
logger.error('Problem during reporting error, continuing:\n' + traceback.format_exc())
exception)
# For everything else: log it, send it, continue. # For everything else: log it, send it, continue.
except Exception as exception: except Exception as exc:
logger.error(traceback.format_exc()) self.logger.error(traceback.format_exc())
try: try:
computer_partition.error(exception) computer_partition.error(exc, logger=self.logger)
except (SystemExit, KeyboardInterrupt): except (SystemExit, KeyboardInterrupt):
raise raise
except Exception: except Exception:
exception = traceback.format_exc() self.logger.error('Problem during reporting error, continuing:\n%s' %
logger.error('Problem during reporting error, continuing:\n' + traceback.format_exc())
exception)
return filtered_computer_partition_list return filtered_computer_partition_list
...@@ -913,8 +727,7 @@ class Slapgrid(object): ...@@ -913,8 +727,7 @@ class Slapgrid(object):
""" """
Will start supervisord and process each Computer Partition. Will start supervisord and process each Computer Partition.
""" """
logger = logging.getLogger('ComputerPartitionProcessing') self.logger.info('Processing computer partitions...')
logger.info('Processing computer partitions...')
# Prepares environment # Prepares environment
self.checkEnvironmentAndCreateStructure() self.checkEnvironmentAndCreateStructure()
self._launchSupervisord() self._launchSupervisord()
...@@ -938,48 +751,44 @@ class Slapgrid(object): ...@@ -938,48 +751,44 @@ class Slapgrid(object):
# Send log before exiting # Send log before exiting
except (SystemExit, KeyboardInterrupt): except (SystemExit, KeyboardInterrupt):
exception = traceback.format_exc() computer_partition.error(traceback.format_exc(), logger=self.logger)
computer_partition.error(exception)
raise raise
except Slapgrid.PromiseError as exception: except Slapgrid.PromiseError as exc:
clean_run_promise = False clean_run_promise = False
try: try:
logger.error(exception) self.logger.error(exc)
computer_partition.error(exception) computer_partition.error(exc, logger=self.logger)
except (SystemExit, KeyboardInterrupt): except (SystemExit, KeyboardInterrupt):
raise raise
except Exception: except Exception:
exception = traceback.format_exc() self.logger.error('Problem during reporting error, continuing:\n%s' %
logger.error('Problem during reporting error, continuing:\n' + traceback.format_exc())
exception)
# Buildout failed: send log but don't print it to output (already done) # Buildout failed: send log but don't print it to output (already done)
except BuildoutFailedError, exception: except BuildoutFailedError as exc:
clean_run = False clean_run = False
try: try:
computer_partition.error(exception) computer_partition.error(exc, logger=self.logger)
except (SystemExit, KeyboardInterrupt): except (SystemExit, KeyboardInterrupt):
raise raise
except Exception: except Exception:
exception = traceback.format_exc() self.logger.error('Problem during reporting error, continuing:\n%s' %
logger.error('Problem during reporting error, continuing:\n' + traceback.format_exc())
exception)
# For everything else: log it, send it, continue. # For everything else: log it, send it, continue.
except Exception as exception: except Exception as exc:
clean_run = False clean_run = False
logger.error(traceback.format_exc()) self.logger.error(traceback.format_exc())
try: try:
computer_partition.error(exception) computer_partition.error(exc, logger=self.logger)
except (SystemExit, KeyboardInterrupt): except (SystemExit, KeyboardInterrupt):
raise raise
except Exception: except Exception:
exception = traceback.format_exc() self.logger.error('Problem during reporting error, continuing:\n%s' %
logger.error('Problem during reporting error, continuing:\n' + traceback.format_exc())
exception)
logger.info("Finished computer partitions.") self.logger.info('Finished computer partitions.')
# Return success value # Return success value
if not clean_run: if not clean_run:
...@@ -991,9 +800,6 @@ class Slapgrid(object): ...@@ -991,9 +800,6 @@ class Slapgrid(object):
def validateXML(self, to_be_validated, xsd_model): def validateXML(self, to_be_validated, xsd_model):
"""Validates a given xml file""" """Validates a given xml file"""
logger = logging.getLogger('XMLValidating')
#We retrieve the xsd model #We retrieve the xsd model
xsd_model = StringIO.StringIO(xsd_model) xsd_model = StringIO.StringIO(xsd_model)
xmlschema_doc = etree.parse(xsd_model) xmlschema_doc = etree.parse(xsd_model)
...@@ -1001,10 +807,10 @@ class Slapgrid(object): ...@@ -1001,10 +807,10 @@ class Slapgrid(object):
try: try:
document = etree.fromstring(to_be_validated) document = etree.fromstring(to_be_validated)
except (etree.XMLSyntaxError, etree.DocumentInvalid) as e: except (etree.XMLSyntaxError, etree.DocumentInvalid) as exc:
logger.info('Failed to parse this XML report : %s\n%s' % \ self.logger.info('Failed to parse this XML report : %s\n%s' % \
(to_be_validated, _formatXMLError(e))) (to_be_validated, _formatXMLError(exc)))
logger.error(_formatXMLError(e)) self.logger.error(_formatXMLError(exc))
return False return False
if xmlschema.validate(document): if xmlschema.validate(document):
...@@ -1033,17 +839,16 @@ class Slapgrid(object): ...@@ -1033,17 +839,16 @@ class Slapgrid(object):
for computer_partition_usage in computer_partition_usage_list: for computer_partition_usage in computer_partition_usage_list:
try: try:
root = etree.fromstring(computer_partition_usage.usage) root = etree.fromstring(computer_partition_usage.usage)
except UnicodeError, e: except UnicodeError as exc:
self.logger.info("Failed to read %s." % ( self.logger.info("Failed to read %s." % computer_partition_usage.usage)
computer_partition_usage.usage))
self.logger.error(UnicodeError) self.logger.error(UnicodeError)
raise UnicodeError("Failed to read %s: %s" % (computer_partition_usage.usage, e)) raise UnicodeError("Failed to read %s: %s" % (computer_partition_usage.usage, exc))
except (etree.XMLSyntaxError, etree.DocumentInvalid) as e: except (etree.XMLSyntaxError, etree.DocumentInvalid) as exc:
self.logger.info("Failed to parse %s." % (computer_partition_usage.usage)) self.logger.info("Failed to parse %s." % (computer_partition_usage.usage))
self.logger.error(e) self.logger.error(exc)
raise _formatXMLError(e) raise _formatXMLError(exc)
except Exception, e: except Exception as exc:
raise Exception("Failed to generate XML report: %s" % e) raise Exception("Failed to generate XML report: %s" % exc)
for movement in root.findall('movement'): for movement in root.findall('movement'):
xml.append('<movement>') xml.append('<movement>')
...@@ -1067,8 +872,7 @@ class Slapgrid(object): ...@@ -1067,8 +872,7 @@ class Slapgrid(object):
slap_computer_usage = self.slap.registerComputer(self.computer_id) slap_computer_usage = self.slap.registerComputer(self.computer_id)
computer_partition_usage_list = [] computer_partition_usage_list = []
logger = logging.getLogger('UsageReporting') self.logger.info('Aggregating and sending usage reports...')
logger.info("Aggregating and sending usage reports...")
#We retrieve XSD models #We retrieve XSD models
try: try:
...@@ -1110,14 +914,14 @@ class Slapgrid(object): ...@@ -1110,14 +914,14 @@ class Slapgrid(object):
script_list_to_run = os.listdir(report_path) script_list_to_run = os.listdir(report_path)
else: else:
script_list_to_run = [] script_list_to_run = []
#We now generate the pseudorandom name for the xml file #We now generate the pseudorandom name for the xml file
# and we add it in the invocation_list # and we add it in the invocation_list
f = tempfile.NamedTemporaryFile() f = tempfile.NamedTemporaryFile()
name_xml = '%s.%s' % ('slapreport', os.path.basename(f.name)) name_xml = '%s.%s' % ('slapreport', os.path.basename(f.name))
path_to_slapreport = os.path.join(instance_path, 'var', 'xml_report', path_to_slapreport = os.path.join(instance_path, 'var', 'xml_report',
name_xml) name_xml)
failed_script_list = [] failed_script_list = []
for script in script_list_to_run: for script in script_list_to_run:
invocation_list = [] invocation_list = []
...@@ -1127,7 +931,7 @@ class Slapgrid(object): ...@@ -1127,7 +931,7 @@ class Slapgrid(object):
#f = tempfile.NamedTemporaryFile() #f = tempfile.NamedTemporaryFile()
#name_xml = '%s.%s' % ('slapreport', os.path.basename(f.name)) #name_xml = '%s.%s' % ('slapreport', os.path.basename(f.name))
#path_to_slapreport = os.path.join(instance_path, 'var', name_xml) #path_to_slapreport = os.path.join(instance_path, 'var', name_xml)
invocation_list.append(path_to_slapreport) invocation_list.append(path_to_slapreport)
#Dropping privileges #Dropping privileges
uid, gid = None, None uid, gid = None, None
...@@ -1135,26 +939,26 @@ class Slapgrid(object): ...@@ -1135,26 +939,26 @@ class Slapgrid(object):
#stat sys call to get statistics informations #stat sys call to get statistics informations
uid = stat_info.st_uid uid = stat_info.st_uid
gid = stat_info.st_gid gid = stat_info.st_gid
kw = dict(stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
process_handler = SlapPopen(invocation_list, process_handler = SlapPopen(invocation_list,
preexec_fn=lambda: dropPrivileges(uid, gid), preexec_fn=lambda: dropPrivileges(uid, gid, logger=self.logger),
cwd=os.path.join(instance_path, 'etc', 'report'), cwd=os.path.join(instance_path, 'etc', 'report'),
env=None, **kw) env=None,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
logger=self.logger)
if process_handler.returncode is None: if process_handler.returncode is None:
process_handler.kill() process_handler.kill()
if process_handler.returncode != 0: if process_handler.returncode != 0:
clean_run = False clean_run = False
failed_script_list.append("Script %r failed." % script) failed_script_list.append("Script %r failed." % script)
logger.warning("Failed to run %r" % invocation_list) self.logger.warning('Failed to run %r' % invocation_list)
if len(failed_script_list): if len(failed_script_list):
computer_partition.error('\n'.join(failed_script_list)) computer_partition.error('\n'.join(failed_script_list), logger=self.logger)
# Whatever happens, don't stop processing other instances # Whatever happens, don't stop processing other instances
except Exception: except Exception:
computer_partition_id = computer_partition.getId() self.logger.info('Cannot run usage script(s) for %r: %s' % (
exception = traceback.format_exc() computer_partition.getId(),
issue = "Cannot run usage script(s) for %r: %s" % ( traceback.format_exc()))
computer_partition_id, exception)
logger.info(issue)
#Now we loop through the different computer partitions to report #Now we loop through the different computer partitions to report
report_usage_issue_cp_list = [] report_usage_issue_cp_list = []
...@@ -1170,22 +974,19 @@ class Slapgrid(object): ...@@ -1170,22 +974,19 @@ class Slapgrid(object):
filename_list = os.listdir(dir_reports) filename_list = os.listdir(dir_reports)
else: else:
filename_list = [] filename_list = []
#logger.debug('name List %s' % filename_list) #self.logger.debug('name List %s' % filename_list)
usage = ''
for filename in filename_list: for filename in filename_list:
file_path = os.path.join(dir_reports, filename) file_path = os.path.join(dir_reports, filename)
if os.path.exists(file_path): if os.path.exists(file_path):
usage_file = open(file_path, 'r') usage = open(file_path, 'r').read()
usage = usage_file.read()
usage_file.close()
#We check the validity of xml content of each reports #We check the validity of xml content of each reports
if not self.validateXML(usage, partition_consumption_model): if not self.validateXML(usage, partition_consumption_model):
logger.info('WARNING: The XML file %s generated by slapreport is ' self.logger.info('WARNING: The XML file %s generated by slapreport is '
'not valid - This report is left as is at %s where you can ' 'not valid - This report is left as is at %s where you can '
'inspect what went wrong ' % (filename, dir_reports)) 'inspect what went wrong ' % (filename, dir_reports))
# Warn the SlapOS Master that a partition generates corrupted xml # Warn the SlapOS Master that a partition generates corrupted xml
# report # report
else: else:
...@@ -1195,7 +996,7 @@ class Slapgrid(object): ...@@ -1195,7 +996,7 @@ class Slapgrid(object):
computer_partition_usage_list.append(computer_partition_usage) computer_partition_usage_list.append(computer_partition_usage)
filename_delete_list.append(filename) filename_delete_list.append(filename)
else: else:
logger.debug("Usage report %r not found, ignored" % file_path) self.logger.debug('Usage report %r not found, ignored' % file_path)
#After sending the aggregated file we remove all the valid xml reports #After sending the aggregated file we remove all the valid xml reports
for filename in filename_delete_list: for filename in filename_delete_list:
...@@ -1203,15 +1004,14 @@ class Slapgrid(object): ...@@ -1203,15 +1004,14 @@ class Slapgrid(object):
# Whatever happens, don't stop processing other instances # Whatever happens, don't stop processing other instances
except Exception: except Exception:
computer_partition_id = computer_partition.getId() self.logger.info('Cannot run usage script(s) for %r: %s' % (
exception = traceback.format_exc() computer_partition.getId(),
issue = "Cannot run usage script(s) for %r: %s" % ( traceback.format_exc()))
computer_partition_id, exception)
logger.info(issue)
for computer_partition_usage in computer_partition_usage_list: for computer_partition_usage in computer_partition_usage_list:
logger.info('computer_partition_usage_list : %s - %s' % \ self.logger.info('computer_partition_usage_list: %s - %s' % (
(computer_partition_usage.usage, computer_partition_usage.getId())) computer_partition_usage.usage,
computer_partition_usage.getId()))
#If there is, at least, one report #If there is, at least, one report
if computer_partition_usage_list != []: if computer_partition_usage_list != []:
...@@ -1219,22 +1019,21 @@ class Slapgrid(object): ...@@ -1219,22 +1019,21 @@ class Slapgrid(object):
#We generate the final XML report with asXML method #We generate the final XML report with asXML method
computer_consumption = self.asXML(computer_partition_usage_list) computer_consumption = self.asXML(computer_partition_usage_list)
logger.info('Final xml report : %s' % computer_consumption) self.logger.info('Final xml report: %s' % computer_consumption)
#We test the XML report before sending it #We test the XML report before sending it
if self.validateXML(computer_consumption, computer_consumption_model): if self.validateXML(computer_consumption, computer_consumption_model):
logger.info('XML file generated by asXML is valid') self.logger.info('XML file generated by asXML is valid')
slap_computer_usage.reportUsage(computer_consumption) slap_computer_usage.reportUsage(computer_consumption)
else: else:
logger.info('XML file generated by asXML is not valid !') self.logger.info('XML file generated by asXML is not valid !')
raise ValueError('XML file generated by asXML is not valid !') raise ValueError('XML file generated by asXML is not valid !')
except Exception: except Exception:
computer_partition_id = computer_partition.getId() issue = "Cannot report usage for %r: %s" % (
exception = traceback.format_exc() computer_partition.getId(),
issue = "Cannot report usage for %r: %s" % (computer_partition_id, traceback.format_exc())
exception) self.logger.info(issue)
logger.info(issue) computer_partition.error(issue, logger=self.logger)
computer_partition.error(issue)
report_usage_issue_cp_list.append(computer_partition_id) report_usage_issue_cp_list.append(computer_partition_id)
for computer_partition in computer_partition_list: for computer_partition in computer_partition_list:
...@@ -1243,11 +1042,11 @@ class Slapgrid(object): ...@@ -1243,11 +1042,11 @@ class Slapgrid(object):
computer_partition_id = computer_partition.getId() computer_partition_id = computer_partition.getId()
try: try:
software_url = computer_partition.getSoftwareRelease().getURI() software_url = computer_partition.getSoftwareRelease().getURI()
software_path = os.path.join(self.software_root, software_path = os.path.join(self.software_root, md5digest(software_url))
getSoftwareUrlHash(software_url))
except (NotFoundError, TypeError): except (NotFoundError, TypeError):
software_url = None software_url = None
software_path = None software_path = None
local_partition = Partition( local_partition = Partition(
software_path=software_path, software_path=software_path,
instance_path=os.path.join(self.instance_root, instance_path=os.path.join(self.instance_root,
...@@ -1263,42 +1062,40 @@ class Slapgrid(object): ...@@ -1263,42 +1062,40 @@ class Slapgrid(object):
software_release_url=software_url, software_release_url=software_url,
certificate_repository_path=self.certificate_repository_path, certificate_repository_path=self.certificate_repository_path,
buildout=self.buildout, buildout=self.buildout,
) logger=self.logger)
local_partition.stop() local_partition.stop()
try: try:
computer_partition.stopped() computer_partition.stopped()
except (SystemExit, KeyboardInterrupt): except (SystemExit, KeyboardInterrupt):
exception = traceback.format_exc() computer_partition.error(traceback.format_exc(), logger=self.logger)
computer_partition.error(exception)
raise raise
except Exception: except Exception:
pass pass
if computer_partition.getId() in report_usage_issue_cp_list: if computer_partition.getId() in report_usage_issue_cp_list:
logger.info('Ignoring destruction of %r, as not report usage was ' self.logger.info('Ignoring destruction of %r, as no report usage was sent' %
'sent' % computer_partition.getId()) computer_partition.getId())
continue continue
local_partition.destroy() local_partition.destroy()
except (SystemExit, KeyboardInterrupt): except (SystemExit, KeyboardInterrupt):
exception = traceback.format_exc() computer_partition.error(traceback.format_exc(), logger=self.logger)
computer_partition.error(exception)
raise raise
except Exception: except Exception:
clean_run = False clean_run = False
exception = traceback.format_exc() exc = traceback.format_exc()
computer_partition.error(exception) computer_partition.error(exc, logger=self.logger)
logger.error(exception) self.logger.error(exc)
try: try:
computer_partition.destroyed() computer_partition.destroyed()
except slap.NotFoundError: except NotFoundError:
logger.debug('Ignored slap error while trying to inform about ' self.logger.debug('Ignored slap error while trying to inform about '
'destroying not fully configured Computer Partition %r' % 'destroying not fully configured Computer Partition %r' %
computer_partition.getId()) computer_partition.getId())
except ServerError as server_error: except ServerError as server_error:
logger.debug('Ignored server error while trying to inform about ' self.logger.debug('Ignored server error while trying to inform about '
'destroying Computer Partition %r. Error is :\n%r' % 'destroying Computer Partition %r. Error is:\n%r' %
(computer_partition.getId(), server_error.args[0])) (computer_partition.getId(), server_error.args[0]))
logger.info("Finished usage reports.") self.logger.info('Finished usage reports.')
# Return success value # Return success value
if not clean_run: if not clean_run:
......
slapos/grid/svcbackend.py
View file @ 5f0924ff
...@@ -28,18 +28,16 @@ ...@@ -28,18 +28,16 @@
# #
############################################################################## ##############################################################################
from supervisor import xmlrpc
import time import time
from utils import SlapPopen
import logging
import os import os
import sys import sys
import xmlrpclib import xmlrpclib
from optparse import OptionParser
import ConfigParser
import socket as socketlib import socket as socketlib
import subprocess import subprocess
from supervisor import xmlrpc
from slapos.grid.utils import SlapPopen
def getSupervisorRPC(socket): def getSupervisorRPC(socket):
supervisor_transport = xmlrpc.SupervisorTransport('', '', supervisor_transport = xmlrpc.SupervisorTransport('', '',
...@@ -49,20 +47,19 @@ def getSupervisorRPC(socket): ...@@ -49,20 +47,19 @@ def getSupervisorRPC(socket):
return getattr(server_proxy, 'supervisor') return getattr(server_proxy, 'supervisor')
def launchSupervisord(socket, configuration_file): def launchSupervisord(socket, configuration_file, logger):
logger = logging.getLogger('SVCBackend')
supervisor = getSupervisorRPC(socket)
if os.path.exists(socket): if os.path.exists(socket):
trynum = 1 trynum = 1
while trynum < 6: while trynum < 6:
try: try:
supervisor = getSupervisorRPC(socket)
status = supervisor.getState() status = supervisor.getState()
except xmlrpclib.Fault as e: except xmlrpclib.Fault as e:
if e.faultCode == 6 and e.faultString == 'SHUTDOWN_STATE': if e.faultCode == 6 and e.faultString == 'SHUTDOWN_STATE':
logger.info('Supervisor in shutdown procedure, will check again later.') logger.info('Supervisor in shutdown procedure, will check again later.')
trynum += 1 trynum += 1
time.sleep(2 * trynum) time.sleep(2 * trynum)
except Exception: except Exception as e:
# In case if there is problem with connection, assume that supervisord # In case if there is problem with connection, assume that supervisord
# is not running and try to run it # is not running and try to run it
break break
...@@ -90,85 +87,34 @@ def launchSupervisord(socket, configuration_file): ...@@ -90,85 +87,34 @@ def launchSupervisord(socket, configuration_file):
env={}, env={},
executable=sys.executable, executable=sys.executable,
stdout=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT) stderr=subprocess.STDOUT,
result = supervisord_popen.communicate()[0] logger=logger)
if supervisord_popen.returncode == 0:
logger.info('Supervisord command invoked with: %s' % result)
try:
default_timeout = socketlib.getdefaulttimeout()
current_timeout = 1
trynum = 1
while trynum < 6:
try:
socketlib.setdefaulttimeout(current_timeout)
status = supervisor.getState()
if status['statename'] == 'RUNNING' and status['statecode'] == 1:
return
logger.warning('Wrong status name %(statename)r and code '
'%(statecode)r, trying again' % status)
trynum += 1
except Exception:
current_timeout = 5 * trynum
trynum += 1
else:
logger.info('Supervisord started correctly in try %s.' % trynum)
return
logger.warning('Issue while checking supervisord.')
finally:
socketlib.setdefaulttimeout(default_timeout)
else:
log_message = 'Supervisord unknown problem: %s' % result
logger.warning(log_message)
def getOptionDict(*argument_tuple):
usage = """
Typical usage:
* %prog CONFIGURATION_FILE [arguments passed to supervisor]
""".strip()
parser = OptionParser(usage=usage)
# Parses arguments
if argument_tuple:
(argument_option_instance, argument_list) = parser.parse_args(list(argument_tuple))
else:
# No arguments given to entry point : we parse sys.argv.
(argument_option_instance, argument_list) = parser.parse_args()
if not argument_list: result = supervisord_popen.communicate()[0]
parser.error("Configuration file is obligatory. Consult documentation by " if supervisord_popen.returncode:
"calling with -h.") logger.warning('Supervisord unknown problem: %s' % result)
configuration_file = argument_list[0] return
if not os.path.exists(configuration_file):
parser.error("Could not read configuration file : %s" % configuration_file)
slapgrid_configuration = ConfigParser.SafeConfigParser()
slapgrid_configuration.read(configuration_file)
# Merges the two dictionnaries
option_dict = dict(slapgrid_configuration.items("slapos"))
# Supervisord configuration location
option_dict.setdefault('supervisord_configuration_path',
os.path.join(option_dict['instance_root'], 'etc', 'supervisord.conf'))
# Supervisord socket
option_dict.setdefault('supervisord_socket',
os.path.join(option_dict['instance_root'], 'supervisord.socket'))
return option_dict, argument_list[1:]
def supervisorctl(*argument_tuple):
option_dict, args = getOptionDict(*argument_tuple)
import supervisor.supervisorctl
launchSupervisord(option_dict['supervisord_socket'],
option_dict['supervisord_configuration_path'])
supervisor.supervisorctl.main(args=['-c',
option_dict['supervisord_configuration_path']] + args)
def supervisord(*argument_tuple):
option_dict, _ = getOptionDict(*argument_tuple)
launchSupervisord(option_dict['supervisord_socket'],
option_dict['supervisord_configuration_path'])
try:
default_timeout = socketlib.getdefaulttimeout()
current_timeout = 1
trynum = 1
while trynum < 6:
try:
socketlib.setdefaulttimeout(current_timeout)
supervisor = getSupervisorRPC(socket)
status = supervisor.getState()
if status['statename'] == 'RUNNING' and status['statecode'] == 1:
return
logger.warning('Wrong status name %(statename)r and code '
'%(statecode)r, trying again' % status)
trynum += 1
except Exception as e:
current_timeout = 5 * trynum
trynum += 1
else:
logger.info('Supervisord started correctly in try %s.' % trynum)
return
logger.warning('Issue while checking supervisord.')
finally:
socketlib.setdefaulttimeout(default_timeout)
slapos/grid/utils.py
View file @ 5f0924ff
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
# vim: set et sts=2:
############################################################################## ##############################################################################
# #
# Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors. # Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors.
...@@ -27,21 +28,20 @@ ...@@ -27,21 +28,20 @@
# #
############################################################################## ##############################################################################
import logging import grp
import hashlib import hashlib
import os import os
import pkg_resources import pkg_resources
import pwd
import stat import stat
import subprocess import subprocess
import sys import sys
import pwd
import grp from slapos.grid.exception import BuildoutFailedError, WrongPermissionError
from exception import BuildoutFailedError, WrongPermissionError
from hashlib import md5
# Such umask by default will create paths with full permission # Such umask by default will create paths with full permission
# for user, non writable by group and not accessible by others # for user, non writable by group and not accessible by others
SAFE_UMASK = 027 SAFE_UMASK = 0o27
PYTHON_ENVIRONMENT_REMOVE_LIST = [ PYTHON_ENVIRONMENT_REMOVE_LIST = [
'PYTHONHOME', 'PYTHONHOME',
...@@ -94,6 +94,7 @@ class SlapPopen(subprocess.Popen): ...@@ -94,6 +94,7 @@ class SlapPopen(subprocess.Popen):
log. log.
""" """
def __init__(self, *args, **kwargs): def __init__(self, *args, **kwargs):
logger = kwargs.pop('logger')
kwargs.update(stdin=subprocess.PIPE) kwargs.update(stdin=subprocess.PIPE)
if sys.platform == 'cygwin' and kwargs.get('env') == {}: if sys.platform == 'cygwin' and kwargs.get('env') == {}:
kwargs['env'] = None kwargs['env'] = None
...@@ -102,25 +103,22 @@ class SlapPopen(subprocess.Popen): ...@@ -102,25 +103,22 @@ class SlapPopen(subprocess.Popen):
self.stdin.close() self.stdin.close()
self.stdin = None self.stdin = None
logger = logging.getLogger('SlapProcessManager')
# XXX-Cedric: this algorithm looks overkill for simple logging. # XXX-Cedric: this algorithm looks overkill for simple logging.
output_lines = [] output_lines = []
while True: while True:
line = self.stdout.readline() line = self.stdout.readline()
if line == '' and self.poll() != None: if line == '' and self.poll() is not None:
break break
output_lines.append(line) output_lines.append(line)
if line[-1:] == '\n': logger.info(line.rstrip('\n'))
line = line[:-1]
logger.info(line)
self.output = ''.join(output_lines) self.output = ''.join(output_lines)
def getSoftwareUrlHash(url):
return md5(url).hexdigest() def md5digest(url):
return hashlib.md5(url).hexdigest()
def getCleanEnvironment(home_path='/tmp'): def getCleanEnvironment(logger, home_path='/tmp'):
logger = logging.getLogger('CleanEnvironment')
changed_env = {} changed_env = {}
removed_env = [] removed_env = []
env = os.environ.copy() env = os.environ.copy()
...@@ -137,46 +135,41 @@ def getCleanEnvironment(home_path='/tmp'): ...@@ -137,46 +135,41 @@ def getCleanEnvironment(home_path='/tmp'):
return env return env
def setRunning(pid_file): def setRunning(logger, pidfile):
"""Creates a pidfile. If a pidfile already exists, we exit""" """Creates a pidfile. If a pidfile already exists, we exit"""
logger = logging.getLogger('Slapgrid') # XXX might use http://code.activestate.com/recipes/577911-context-manager-for-a-daemon-pid-file/
if os.path.exists(pid_file): if os.path.exists(pidfile):
# Pid file is present
try: try:
pid = int(open(pid_file, 'r').readline()) pid = int(open(pidfile, 'r').readline())
except ValueError: except ValueError:
pid = None pid = None
# XXX This could use psutil library. # XXX This could use psutil library.
if pid is not None and os.path.exists("/proc/%s" % pid): if pid and os.path.exists("/proc/%s" % pid):
# In case process is present, ignore.
logger.info('New slapos process started, but another slapos ' logger.info('New slapos process started, but another slapos '
'process is aleady running with pid %s, exiting.' % pid) 'process is aleady running with pid %s, exiting.' % pid)
sys.exit(10) sys.exit(10)
logger.info('Existing pid file %r was stale one, overwritten' % pid_file) logger.info('Existing pid file %r was stale, overwritten' % pidfile)
# Start new process # Start new process
write_pid(pid_file) write_pid(logger, pidfile)
def setFinished(pid_file): def setFinished(pidfile):
try: try:
os.remove(pid_file) os.remove(pidfile)
except OSError: except OSError:
pass pass
def write_pid(pid_file): def write_pid(logger, pidfile):
logger = logging.getLogger('Slapgrid')
pid = os.getpid()
try: try:
f = open(pid_file, 'w') with open(pidfile, 'w') as fout:
f.write('%s' % pid) fout.write('%s' % os.getpid())
f.close()
except (IOError, OSError): except (IOError, OSError):
logger.critical('slapgrid could not write pidfile %s' % pid_file) logger.critical('slapgrid could not write pidfile %s' % pidfile)
raise raise
def dropPrivileges(uid, gid): def dropPrivileges(uid, gid, logger):
"""Drop privileges to uid, gid if current uid is 0 """Drop privileges to uid, gid if current uid is 0
Do tests to check if dropping was successful and that no system call is able Do tests to check if dropping was successful and that no system call is able
...@@ -184,14 +177,13 @@ def dropPrivileges(uid, gid): ...@@ -184,14 +177,13 @@ def dropPrivileges(uid, gid):
Does nothing in case if uid and gid are not 0 Does nothing in case if uid and gid are not 0
""" """
logger = logging.getLogger('dropPrivileges')
# XXX-Cedric: remove format / just do a print, otherwise formatting is done # XXX-Cedric: remove format / just do a print, otherwise formatting is done
# twice # twice
current_uid, current_gid = os.getuid(), os.getgid() current_uid, current_gid = os.getuid(), os.getgid()
if uid == 0 or gid == 0: if uid == 0 or gid == 0:
raise OSError('Dropping privileges to uid = %r or ' \ raise OSError('Dropping privileges to uid = %r or ' \
'gid = %r is too dangerous' % (uid, gid)) 'gid = %r is too dangerous' % (uid, gid))
if not(current_uid == 0 and current_gid == 0): if current_uid or current_gid:
logger.debug('Running as uid = %r, gid = %r, dropping not needed and not ' logger.debug('Running as uid = %r, gid = %r, dropping not needed and not '
'possible' % (current_uid, current_gid)) 'possible' % (current_uid, current_gid))
return return
...@@ -209,12 +201,12 @@ def dropPrivileges(uid, gid): ...@@ -209,12 +201,12 @@ def dropPrivileges(uid, gid):
uid, gid, group_list) uid, gid, group_list)
new_uid, new_gid, new_group_list = os.getuid(), os.getgid(), os.getgroups() new_uid, new_gid, new_group_list = os.getuid(), os.getgid(), os.getgroups()
if not (new_uid == uid and new_gid == gid and set(new_group_list) == group_list): if not (new_uid == uid and new_gid == gid and set(new_group_list) == group_list):
raise OSError('%s new_uid = %r and new_gid = %r and ' \ raise OSError('%s new_uid = %r and new_gid = %r and '
'new_group_list = %r which is fatal.' 'new_group_list = %r which is fatal.'
% (message_pre, % (message_pre,
new_uid, new_uid,
new_gid, new_gid,
new_group_list)) new_group_list))
# assert that it is not possible to go back to running one # assert that it is not possible to go back to running one
try: try:
...@@ -236,18 +228,16 @@ def dropPrivileges(uid, gid): ...@@ -236,18 +228,16 @@ def dropPrivileges(uid, gid):
logger.debug('Succesfully dropped privileges to uid=%r gid=%r' % (uid, gid)) logger.debug('Succesfully dropped privileges to uid=%r gid=%r' % (uid, gid))
def bootstrapBuildout(path, buildout=None, def bootstrapBuildout(path, logger, buildout=None,
additional_buildout_parametr_list=None): additional_buildout_parametr_list=None):
if additional_buildout_parametr_list is None: if additional_buildout_parametr_list is None:
additional_buildout_parametr_list = [] additional_buildout_parametr_list = []
logger = logging.getLogger('BuildoutManager')
# Reads uid/gid of path, launches buildout with thoses privileges # Reads uid/gid of path, launches buildout with thoses privileges
stat_info = os.stat(path) stat_info = os.stat(path)
uid = stat_info.st_uid uid = stat_info.st_uid
gid = stat_info.st_gid gid = stat_info.st_gid
invocation_list = [sys.executable, '-S'] invocation_list = [sys.executable, '-S']
kw = dict()
if buildout is not None: if buildout is not None:
invocation_list.append(buildout) invocation_list.append(buildout)
invocation_list.extend(additional_buildout_parametr_list) invocation_list.extend(additional_buildout_parametr_list)
...@@ -274,25 +264,26 @@ def bootstrapBuildout(path, buildout=None, ...@@ -274,25 +264,26 @@ def bootstrapBuildout(path, buildout=None,
logger.debug('Set umask from %03o to %03o' % (umask, SAFE_UMASK)) logger.debug('Set umask from %03o to %03o' % (umask, SAFE_UMASK))
logger.debug('Invoking: %r in directory %r' % (' '.join(invocation_list), logger.debug('Invoking: %r in directory %r' % (' '.join(invocation_list),
path)) path))
kw.update(stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
process_handler = SlapPopen(invocation_list, process_handler = SlapPopen(invocation_list,
preexec_fn=lambda: dropPrivileges(uid, gid), preexec_fn=lambda: dropPrivileges(uid, gid, logger=logger),
cwd=path, **kw) cwd=path,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
logger=logger)
if process_handler.returncode is None or process_handler.returncode != 0: if process_handler.returncode is None or process_handler.returncode != 0:
message = 'Failed to run buildout profile in directory %r' % (path) message = 'Failed to run buildout profile in directory %r' % (path)
logger.error(message) logger.error(message)
raise BuildoutFailedError('%s:\n%s\n' % (message, process_handler.output)) raise BuildoutFailedError('%s:\n%s\n' % (message, process_handler.output))
except OSError as error: except OSError as exc:
raise BuildoutFailedError(error) raise BuildoutFailedError(exc)
finally: finally:
old_umask = os.umask(umask) old_umask = os.umask(umask)
logger.debug('Restore umask from %03o to %03o' % (old_umask, umask)) logger.debug('Restore umask from %03o to %03o' % (old_umask, umask))
def launchBuildout(path, buildout_binary, def launchBuildout(path, buildout_binary, logger,
additional_buildout_parametr_list=None): additional_buildout_parametr_list=None):
""" Launches buildout.""" """ Launches buildout."""
logger = logging.getLogger('BuildoutManager')
if additional_buildout_parametr_list is None: if additional_buildout_parametr_list is None:
additional_buildout_parametr_list = [] additional_buildout_parametr_list = []
# Reads uid/gid of path, launches buildout with thoses privileges # Reads uid/gid of path, launches buildout with thoses privileges
...@@ -300,9 +291,7 @@ def launchBuildout(path, buildout_binary, ...@@ -300,9 +291,7 @@ def launchBuildout(path, buildout_binary,
uid = stat_info.st_uid uid = stat_info.st_uid
gid = stat_info.st_gid gid = stat_info.st_gid
# Extract python binary to prevent shebang size limit # Extract python binary to prevent shebang size limit
file = open(buildout_binary, 'r') line = open(buildout_binary, 'r').readline()
line = file.readline()
file.close()
invocation_list = [] invocation_list = []
if line.startswith('#!'): if line.startswith('#!'):
line = line[2:] line = line[2:]
...@@ -316,42 +305,44 @@ def launchBuildout(path, buildout_binary, ...@@ -316,42 +305,44 @@ def launchBuildout(path, buildout_binary,
logger.debug('Set umask from %03o to %03o' % (umask, SAFE_UMASK)) logger.debug('Set umask from %03o to %03o' % (umask, SAFE_UMASK))
logger.debug('Invoking: %r in directory %r' % (' '.join(invocation_list), logger.debug('Invoking: %r in directory %r' % (' '.join(invocation_list),
path)) path))
kw = dict(stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
process_handler = SlapPopen(invocation_list, process_handler = SlapPopen(invocation_list,
preexec_fn=lambda: dropPrivileges(uid, gid), cwd=path, preexec_fn=lambda: dropPrivileges(uid, gid, logger=logger),
env=getCleanEnvironment(pwd.getpwuid(uid).pw_dir), **kw) cwd=path,
env=getCleanEnvironment(logger=logger,
home_path=path),
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
logger=logger)
if process_handler.returncode is None or process_handler.returncode != 0: if process_handler.returncode is None or process_handler.returncode != 0:
message = 'Failed to run buildout profile in directory %r' % (path) message = 'Failed to run buildout profile in directory %r' % (path)
logger.error(message) logger.error(message)
raise BuildoutFailedError('%s:\n%s\n' % (message, process_handler.output)) raise BuildoutFailedError('%s:\n%s\n' % (message, process_handler.output))
except OSError as error: except OSError as exc:
raise BuildoutFailedError(error) raise BuildoutFailedError(exc)
finally: finally:
old_umask = os.umask(umask) old_umask = os.umask(umask)
logger.debug('Restore umask from %03o to %03o' % (old_umask, umask)) logger.debug('Restore umask from %03o to %03o' % (old_umask, umask))
def updateFile(file_path, content, mode='0600'): def updateFile(file_path, content, mode=0o600):
"""Creates an executable with "content" as content.""" """Creates an executable with "content" as content."""
altered = False altered = False
if not (os.path.isfile(file_path)) or \ if not (os.path.isfile(file_path)) or \
not(hashlib.md5(open(file_path).read()).digest() ==\ not(hashlib.md5(open(file_path).read()).digest() ==\
hashlib.md5(content).digest()): hashlib.md5(content).digest()):
with open(file_path, 'w') as fout:
fout.write(content)
altered = True altered = True
file_file = open(file_path, 'w')
file_file.write(content)
file_file.flush()
file_file.close()
os.chmod(file_path, stat.S_IREAD | stat.S_IWRITE | stat.S_IEXEC) os.chmod(file_path, stat.S_IREAD | stat.S_IWRITE | stat.S_IEXEC)
if oct(stat.S_IMODE(os.stat(file_path).st_mode)) != mode: if stat.S_IMODE(os.stat(file_path).st_mode) != mode:
os.chmod(file_path, int(mode, 8)) os.chmod(file_path, mode)
altered = True altered = True
return altered return altered
def updateExecutable(executable_path, content): def updateExecutable(executable_path, content):
"""Creates an executable with "content" as content.""" """Creates an executable with "content" as content."""
return updateFile(executable_path, content, '0700') return updateFile(executable_path, content, 0o700)
def createPrivateDirectory(path): def createPrivateDirectory(path):
...@@ -359,8 +350,8 @@ def createPrivateDirectory(path): ...@@ -359,8 +350,8 @@ def createPrivateDirectory(path):
if not os.path.isdir(path): if not os.path.isdir(path):
os.mkdir(path) os.mkdir(path)
os.chmod(path, stat.S_IREAD | stat.S_IWRITE | stat.S_IEXEC) os.chmod(path, stat.S_IREAD | stat.S_IWRITE | stat.S_IEXEC)
permission = oct(stat.S_IMODE(os.stat(path).st_mode)) permission = stat.S_IMODE(os.stat(path).st_mode)
if permission not in ('0700'): if permission != 0o700:
raise WrongPermissionError('Wrong permissions in %s ' \ raise WrongPermissionError('Wrong permissions in %s: ' \
': is %s, should be 0700' 'is 0%o, should be 0700'
% (path, permission)) % (path, permission))
slapos/proxy/__init__.py
View file @ 5f0924ff
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
# vim: set et sts=2:
############################################################################## ##############################################################################
# #
# Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors. # Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors.
...@@ -27,113 +28,38 @@ ...@@ -27,113 +28,38 @@
# #
############################################################################## ##############################################################################
import os
import sys
from optparse import OptionParser, Option
import logging import logging
import logging.handlers
import ConfigParser
class Parser(OptionParser): class ProxyConfig(object):
""" def __init__(self, logger):
Parse all arguments. self.logger = logger
"""
def __init__(self, usage=None, version=None):
"""
Initialize all options possibles.
"""
OptionParser.__init__(self, usage=usage, version=version,
option_list=[
Option("-l", "--log_file",
help="The path to the log file used by the script.",
type=str),
Option("-v", "--verbose",
default=False,
action="store_true",
help="Verbose output."),
Option("-c", "--console",
default=False,
action="store_true",
help="Console output."),
Option("-u", "--database-uri",
type=str,
help="URI for sqlite database"),
])
def check_args(self): def mergeConfig(self, args, configp):
"""
Check arguments
"""
(options, args) = self.parse_args()
if len(args) != 1:
self.error("Incorrect number of arguments")
return options, args[0]
class Config:
def setConfig(self, option_dict, configuration_file_path):
"""
Set options given by parameters.
"""
# Set options parameters # Set options parameters
for option, value in option_dict.__dict__.items(): for option, value in args.__dict__.items():
setattr(self, option, value) setattr(self, option, value)
# Load configuration file
configuration_parser = ConfigParser.SafeConfigParser()
configuration_parser.read(configuration_file_path)
# Merges the arguments and configuration # Merges the arguments and configuration
for section in ("slapproxy", "slapos"): for section in ("slapproxy", "slapos"):
configuration_dict = dict(configuration_parser.items(section)) configuration_dict = dict(configp.items(section))
for key in configuration_dict: for key in configuration_dict:
if not getattr(self, key, None): if not getattr(self, key, None):
setattr(self, key, configuration_dict[key]) setattr(self, key, configuration_dict[key])
# set up logging
self.logger = logging.getLogger("slapproxy")
self.logger.setLevel(logging.INFO)
if self.console:
self.logger.addHandler(logging.StreamHandler())
def setConfig(self):
if not self.database_uri: if not self.database_uri:
raise ValueError('database-uri is required.') raise ValueError('database-uri is required.')
if self.log_file:
if not os.path.isdir(os.path.dirname(self.log_file)):
# fallback to console only if directory for logs does not exists and
# continue to run
raise ValueError('Please create directory %r to store %r log file' % (
os.path.dirname(self.log_file), self.log_file))
else:
file_handler = logging.FileHandler(self.log_file)
file_handler.setFormatter(logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s"))
self.logger.addHandler(file_handler)
self.logger.info('Configured logging to file %r' % self.log_file)
self.logger.info("Started.")
if self.verbose:
self.logger.setLevel(logging.DEBUG)
self.logger.debug("Verbose mode enabled.")
def run(config):
from views import app
app.config['computer_id'] = config.computer_id
app.config['DATABASE_URI'] = config.database_uri
app.run(host=config.host, port=int(config.port))
def main():
"Run default configuration."
usage = "usage: %s [options] CONFIGURATION_FILE" % sys.argv[0]
try: def do_proxy(conf):
# Parse arguments from slapos.proxy.views import app
config = Config() for handler in conf.logger.handlers:
config.setConfig(*Parser(usage=usage).check_args()) app.logger.addHandler(handler)
app.logger.setLevel(logging.INFO)
app.config['computer_id'] = conf.computer_id
app.config['DATABASE_URI'] = conf.database_uri
app.run(host=conf.host, port=int(conf.port))
run(config)
return_code = 0
except SystemExit, err:
# Catch exception raise by optparse
return_code = err
sys.exit(return_code)
slapos/register/register.py
View file @ 5f0924ff
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
# vim: set et sts=2:
############################################################################## ##############################################################################
# #
# Copyright (c) 2012 Vifib SARL and Contributors. All Rights Reserved. # Copyright (c) 2012 Vifib SARL and Contributors. All Rights Reserved.
...@@ -26,12 +27,11 @@ ...@@ -26,12 +27,11 @@
# #
############################################################################## ##############################################################################
# XXX dry_run will happily register a new node on the slapos master. Isn't it supposed to be no-op?
import base64
import ConfigParser import ConfigParser
from getpass import getpass import getpass
import logging
from optparse import OptionParser, Option
import os import os
import shutil import shutil
import stat import stat
...@@ -40,119 +40,40 @@ import tempfile ...@@ -40,119 +40,40 @@ import tempfile
import urllib2 import urllib2
class SlapError(Exception): def authenticate(request, login, password):
""" auth = '%s:%s' % (login, password)
Slap error authheader = 'Basic %s' % auth.encode('base64').rstrip()
""" request.add_header('Authorization', authheader)
def __init__(self, message):
self.msg = message
class UsageError(SlapError):
pass
class ExecError(SlapError):
pass
class Parser(OptionParser):
"""
Parse all arguments.
"""
def __init__(self, usage=None, version=None):
"""
Initialize all options possibles.
"""
OptionParser.__init__(self, usage=usage, version=version,
option_list=[
Option("--interface-name",
help="Interface name to access internet",
default='eth0',
type=str),
Option("--master-url",
help="URL of SlapOS master",
default='https://slap.vifib.com',
type=str),
Option("--master-url-web",
help="URL of SlapOS Master webservice to register certificates",
default='https://www.slapos.org',
type=str),
Option("--partition-number",
help="Number of partition on computer",
default='10',
type=int),
Option("--ipv4-local-network",
help="Base of ipv4 local network",
default='10.0.0.0/16',
type=str),
Option("--ipv6-interface",
help="Interface name to get ipv6",
default='',
type=str),
Option("--login",
help="User login on SlapOS Master webservice",
default=None,
type=str),
Option("--password",
help="User password on SlapOs Master webservice",
default=None,
type=str),
Option("-t", "--create-tap",
help="""Will trigger creation of one virtual "tap" interface per \
Partition and attach it to primary interface. Requires primary interface to be \
a bridge. defaults to false. Needed to host virtual machines.""",
default=False,
action="store_true"),
Option("-n", "--dry-run",
help="Simulate the execution steps",
default=False,
action="store_true"),
])
def check_args(self):
"""
Check arguments
"""
(options, args) = self.parse_args()
if len(args) != 1:
self.error("Incorrect number of arguments")
node_name = args[0]
if options.password != None and options.login == None :
self.error("Please enter your login with your password")
return options, node_name
def get_login():
"""Get user id and encode it for basic identification"""
login = raw_input("SlapOS Master Login: ")
password = getpass()
identification = base64.encodestring('%s:%s' % (login, password))[:-1]
return identification
def check_login(identification, master_url_web): def check_credentials(url, login, password):
"""Check if logged correctly on SlapOS Master""" """Check if logged correctly on SlapOS Master"""
request = urllib2.Request(master_url_web) request = urllib2.Request(url)
# Prepare header for basic authentification authenticate(request, login, password)
authheader = "Basic %s" % identification return 'Logout' in urllib2.urlopen(request).read()
request.add_header("Authorization", authheader)
home_page_url = urllib2.urlopen(request).read()
if 'Logout' in home_page_url:
return 1
else : return 0
def get_certificates(identification, node_name, master_url_web): def get_certificates(master_url_web, node_name, login, password, logger):
"""Download certificates from SlapOS Master""" """Download certificates from SlapOS Master"""
register_server_url = '/'.join([master_url_web, ("add-a-server/WebSection_registerNewComputer?dialog_id=WebSection_viewServerInformationDialog&dialog_method=WebSection_registerNewComputer&title={}&object_path=/erp5/web_site_module/hosting/add-a-server&update_method=&cancel_url=https%3A//www.vifib.net/add-a-server/WebSection_viewServerInformationDialog&Base_callDialogMethod=&field_your_title=Essai1&dialog_category=None&form_id=view".format(node_name))]) register_server_url = '/'.join([master_url_web, ("add-a-server/WebSection_registerNewComputer?dialog_id=WebSection_viewServerInformationDialog&dialog_method=WebSection_registerNewComputer&title={}&object_path=/erp5/web_site_module/hosting/add-a-server&update_method=&cancel_url=https%3A//www.vifib.net/add-a-server/WebSection_viewServerInformationDialog&Base_callDialogMethod=&field_your_title=Essai1&dialog_category=None&form_id=view".format(node_name))])
request = urllib2.Request(register_server_url) request = urllib2.Request(register_server_url)
# Prepare header for basic authentification authenticate(request, login, password)
authheader = "Basic %s" % identification
request.add_header("Authorization", authheader) try:
url = urllib2.urlopen(request) req = urllib2.urlopen(request)
page = url.read() except urllib2.HTTPError as exc:
return page # raise a readable exception if the computer name is already used,
# instead of an opaque 500 Internal Error.
# this will not work with the new API.
if exc.getcode() == 500:
error_body = exc.read()
if 'Certificate still active.' in error_body:
logger.error('The node name "%s" is already in use. Please change the name, or revoke the active certificate if you want to replace the node.' % node_name)
sys.exit(1)
raise
return req.read()
def parse_certificates(source): def parse_certificates(source):
...@@ -161,7 +82,7 @@ def parse_certificates(source): ...@@ -161,7 +82,7 @@ def parse_certificates(source):
c_end = source.find("</textarea>", c_start) c_end = source.find("</textarea>", c_start)
k_start = source.find("-----BEGIN PRIVATE KEY-----") k_start = source.find("-----BEGIN PRIVATE KEY-----")
k_end = source.find("</textarea>", k_start) k_end = source.find("</textarea>", k_start)
return [source[c_start:c_end], source[k_start:k_end]] return source[c_start:c_end], source[k_start:k_end]
def get_computer_name(certificate): def get_computer_name(certificate):
...@@ -170,50 +91,50 @@ def get_computer_name(certificate): ...@@ -170,50 +91,50 @@ def get_computer_name(certificate):
i = certificate.find("/email", k) i = certificate.find("/email", k)
return certificate[k:i] return certificate[k:i]
def save_former_config(config):
def save_former_config(conf):
"""Save former configuration if found""" """Save former configuration if found"""
# Check for config file in /etc/opt/slapos/ # Check for config file in /etc/opt/slapos/
if os.path.exists('/etc/opt/slapos/slapos.cfg'): if os.path.exists('/etc/opt/slapos/slapos.cfg'):
former_slapos_configuration = '/etc/opt/slapos' former = '/etc/opt/slapos'
else : former_slapos_configuration = 0 else:
if former_slapos_configuration: return
saved_slapos_configuration = former_slapos_configuration + '.old'
while True: saved = former + '.old'
if os.path.exists(saved_slapos_configuration): while True:
print "Slapos configuration detected in %s" % saved_slapos_configuration if os.path.exists(saved):
if saved_slapos_configuration[len(saved_slapos_configuration) - 1] != 'd' : print "Slapos configuration detected in %s" % saved
saved_slapos_configuration = saved_slapos_configuration[:len(saved_slapos_configuration) - 1] \ if saved[-1] != 'd':
+ str(int(saved_slapos_configuration[len(saved_slapos_configuration) - 1]) + 1 ) saved = saved[:-1] + str(int(saved[-1]) + 1)
else : else:
saved_slapos_configuration += ".1" saved += '.1'
else: break else:
config.logger.info("Former slapos configuration detected in %s moving to %s" % (former_slapos_configuration, saved_slapos_configuration)) break
shutil.move(former_slapos_configuration, saved_slapos_configuration) conf.logger.info("Former slapos configuration detected in %s moving to %s" % (former, saved))
shutil.move(former, saved)
def get_slapos_conf_example(): def get_slapos_conf_example():
""" """
Get slapos.cfg.example and return its path Get slapos.cfg.example and return its path
""" """
register_server_url = "http://git.erp5.org/gitweb/slapos.core.git/blob_plain/HEAD:/slapos.cfg.example" request = urllib2.Request('http://git.erp5.org/gitweb/slapos.core.git/blob_plain/HEAD:/slapos.cfg.example')
request = urllib2.Request(register_server_url) req = urllib2.urlopen(request)
url = urllib2.urlopen(request)
page = url.read()
_, path = tempfile.mkstemp() _, path = tempfile.mkstemp()
slapos_cfg_example = open(path,'w') with open(path, 'w') as fout:
slapos_cfg_example.write(page) fout.write(req.read())
slapos_cfg_example.close()
return path return path
def slapconfig(config): def slapconfig(conf):
"""Base Function to configure slapos in /etc/opt/slapos""" """Base Function to configure slapos in /etc/opt/slapos"""
dry_run = config.dry_run dry_run = conf.dry_run
# Create slapos configuration directory if needed # Create slapos configuration directory if needed
slap_configuration_directory = os.path.normpath(config.slapos_configuration) slap_conf_dir = os.path.normpath(conf.slapos_configuration)
# Make sure everybody can read slapos configuration directory: # Make sure everybody can read slapos configuration directory:
# Add +x to directories in path # Add +x to directories in path
directory = os.path.dirname(slap_configuration_directory) directory = os.path.dirname(slap_conf_dir)
while True: while True:
if os.path.dirname(directory) == directory: if os.path.dirname(directory) == directory:
break break
...@@ -221,100 +142,84 @@ def slapconfig(config): ...@@ -221,100 +142,84 @@ def slapconfig(config):
os.chmod(directory, os.stat(directory).st_mode | stat.S_IXGRP | stat.S_IRGRP | stat.S_IXOTH | stat.S_IROTH) os.chmod(directory, os.stat(directory).st_mode | stat.S_IXGRP | stat.S_IRGRP | stat.S_IXOTH | stat.S_IROTH)
directory = os.path.dirname(directory) directory = os.path.dirname(directory)
if not os.path.exists(slap_configuration_directory): if not os.path.exists(slap_conf_dir):
config.logger.info ("Creating directory: %s" % slap_configuration_directory) conf.logger.info("Creating directory: %s" % slap_conf_dir)
if not dry_run: if not dry_run:
os.mkdir(slap_configuration_directory, 0711) os.mkdir(slap_conf_dir, 0o711)
user_certificate_repository_path = os.path.join(slap_configuration_directory,'ssl') user_certificate_repository_path = os.path.join(slap_conf_dir, 'ssl')
if not os.path.exists(user_certificate_repository_path): if not os.path.exists(user_certificate_repository_path):
config.logger.info ("Creating directory: %s" % user_certificate_repository_path) conf.logger.info("Creating directory: %s" % user_certificate_repository_path)
if not dry_run: if not dry_run:
os.mkdir(user_certificate_repository_path, 0711) os.mkdir(user_certificate_repository_path, 0o711)
key_file = os.path.join(user_certificate_repository_path, 'key') key_file = os.path.join(user_certificate_repository_path, 'key')
cert_file = os.path.join(user_certificate_repository_path, 'certificate') cert_file = os.path.join(user_certificate_repository_path, 'certificate')
for (src, dst) in [(config.key, key_file), (config.certificate, for src, dst in [
cert_file)]: (conf.key, key_file),
config.logger.info ("Copying to %r, and setting minimum privileges" % dst) (conf.certificate, cert_file)
]:
conf.logger.info("Copying to %r, and setting minimum privileges" % dst)
if not dry_run: if not dry_run:
destination = open(dst,'w') with open(dst, 'w') as destination:
destination.write(''.join(src)) destination.write(''.join(src))
destination.close() os.chmod(dst, 0o600)
os.chmod(dst, 0600)
os.chown(dst, 0, 0) os.chown(dst, 0, 0)
certificate_repository_path = os.path.join(slap_configuration_directory, 'ssl', 'partition_pki') certificate_repository_path = os.path.join(slap_conf_dir, 'ssl', 'partition_pki')
if not os.path.exists(certificate_repository_path): if not os.path.exists(certificate_repository_path):
config.logger.info ("Creating directory: %s" % certificate_repository_path) conf.logger.info("Creating directory: %s" % certificate_repository_path)
if not dry_run: if not dry_run:
os.mkdir(certificate_repository_path, 0711) os.mkdir(certificate_repository_path, 0o711)
# Put slapos configuration file # Put slapos configuration file
slap_configuration_file_location = os.path.join(slap_configuration_directory, slap_conf_file = os.path.join(slap_conf_dir, 'slapos.cfg')
'slapos.cfg') conf.logger.info("Creating slap configuration: %s" % slap_conf_file)
config.logger.info ("Creating slap configuration: %s"
% slap_configuration_file_location)
# Get example configuration file # Get example configuration file
slapos_cfg_example = get_slapos_conf_example() slapos_cfg_example = get_slapos_conf_example()
configuration_example_parser = ConfigParser.RawConfigParser() new_configp = ConfigParser.RawConfigParser()
configuration_example_parser.read(slapos_cfg_example) new_configp.read(slapos_cfg_example)
os.remove(slapos_cfg_example) os.remove(slapos_cfg_example)
# prepare slapos section for section, key, value in [
slaposconfig = dict( ('slapos', 'computer_id', conf.computer_id),
computer_id=config.computer_id, master_url=config.master_url, ('slapos', 'master_url', conf.master_url),
key_file=key_file, cert_file=cert_file, ('slapos', 'key_file', key_file),
certificate_repository_path=certificate_repository_path) ('slapos', 'cert_file', cert_file),
for key in slaposconfig: ('slapos', 'certificate_repository_path', certificate_repository_path),
configuration_example_parser.set('slapos', key, slaposconfig[key]) ('slapformat', 'interface_name', conf.interface_name),
('slapformat', 'ipv4_local_network', conf.ipv4_local_network),
# prepare slapformat ('slapformat', 'partition_amount', conf.partition_number),
slapformatconfig = dict( ('slapformat', 'create_tap', conf.create_tap)
interface_name=config.interface_name, ]:
ipv4_local_network=config.ipv4_local_network, new_configp.set(section, key, value)
partition_amount=config.partition_number,
create_tap=config.create_tap if conf.ipv6_interface:
) new_configp.set('slapformat', 'ipv6_interface', conf.ipv6_interface)
for key in slapformatconfig :
configuration_example_parser.set('slapformat', key, slapformatconfig[key])
if not config.ipv6_interface == '':
configuration_example_parser.set('slapformat',
'ipv6_interface',
config.ipv6_interface)
if not dry_run: if not dry_run:
slap_configuration_file = open(slap_configuration_file_location, "w") with open(slap_conf_file, 'w') as fout:
configuration_example_parser.write(slap_configuration_file) new_configp.write(fout)
conf.logger.info("SlapOS configuration: DONE")
config.logger.info ("SlapOS configuration: DONE") class RegisterConfig(object):
"""
Class containing all parameters needed for configuration
"""
def __init__(self, logger):
self.logger = logger
# Class containing all parameters needed for configuration def setConfig(self, options):
class Config:
def setConfig(self, option_dict, node_name):
""" """
Set options given by parameters. Set options given by parameters.
""" """
# Set options parameters # Set options parameters
for option, value in option_dict.__dict__.items(): for option, value in options.__dict__.items():
setattr(self, option, value) setattr(self, option, value)
self.node_name = node_name
# Define logger for register
self.logger = logging.getLogger('Register')
self.logger.setLevel(logging.DEBUG)
# create console handler and set level to debug
self.ch = logging.StreamHandler()
self.ch.setLevel(logging.INFO)
# create formatter
self.formatter = logging.Formatter('%(levelname)s - %(message)s')
# add formatter to ch
self.ch.setFormatter(self.formatter)
# add ch to logger
self.logger.addHandler(self.ch)
def COMPConfig(self, slapos_configuration, computer_id, certificate, key): def COMPConfig(self, slapos_configuration, computer_id, certificate, key):
self.slapos_configuration = slapos_configuration self.slapos_configuration = slapos_configuration
...@@ -328,66 +233,46 @@ class Config: ...@@ -328,66 +233,46 @@ class Config:
self.logger.debug("Number of partition: %s" % self.partition_number) self.logger.debug("Number of partition: %s" % self.partition_number)
self.logger.info("Using Interface %s" % self.interface_name) self.logger.info("Using Interface %s" % self.interface_name)
self.logger.debug("Ipv4 sub network: %s" % self.ipv4_local_network) self.logger.debug("Ipv4 sub network: %s" % self.ipv4_local_network)
self.logger.debug("Ipv6 Interface: %s" %self.ipv6_interface) self.logger.debug("Ipv6 Interface: %s" % self.ipv6_interface)
def register(config): def gen_auth(conf):
ask = True
if conf.login:
if conf.password:
yield conf.login, conf.password
ask = False
else:
yield conf.login, getpass.getpass()
while ask:
yield raw_input('SlapOS Master Login: '), getpass.getpass()
def do_register(conf):
"""Register new computer on SlapOS Master and generate slapos.cfg""" """Register new computer on SlapOS Master and generate slapos.cfg"""
# Get User identification and check them
if config.login == None : for login, password in gen_auth(conf):
while True : if check_credentials(conf.master_url_web, login, password):
print ("Please enter your SlapOS Master login") break
user_id = get_login() conf.logger.warning('Wrong login/password')
if check_login(user_id, config.master_url_web):
break
config.logger.warning ("Wrong login/password")
else: else:
if config.password == None : return 1
config.password = getpass()
user_id = base64.encodestring('%s:%s' % (config.login, config.password))[:-1]
if not check_login(user_id, config.master_url_web):
config.logger.error ("Wrong login/password")
return 1
# Get source code of page having certificate and key # Get source code of page having certificate and key
certificate_key = get_certificates(user_id, config.node_name, config.master_url_web) certificate_key = get_certificates(conf.master_url_web, conf.node_name, login, password, conf.logger)
# Parse certificate and key and get computer id # Parse certificate and key and get computer id
certificate_key = parse_certificates(certificate_key) certificate, key = parse_certificates(certificate_key)
certificate = certificate_key[0]
key = certificate_key[1]
COMP = get_computer_name(certificate) COMP = get_computer_name(certificate)
# Getting configuration parameters # Getting configuration parameters
slapos_configuration = '/etc/opt/slapos/' conf.COMPConfig(slapos_configuration='/etc/opt/slapos/',
config.COMPConfig(slapos_configuration=slapos_configuration, computer_id=COMP,
computer_id=COMP, certificate=certificate,
certificate = certificate, key=key)
key = key
)
# Save former configuration # Save former configuration
if not config.dry_run: if not conf.dry_run:
save_former_config(config) save_former_config(conf)
# Prepare Slapos Configuration # Prepare Slapos Configuration
slapconfig(config) slapconfig(conf)
print "Node has successfully been configured as %s." % COMP print "Node has successfully been configured as %s." % COMP
return 0 return 0
def main():
"Run default configuration."
usage = "usage: slapos node %s NODE_NAME [options] " % sys.argv[0]
try:
# Parse arguments
config = Config()
config.setConfig(*Parser(usage=usage).check_args())
return_code = register(config)
except UsageError, err:
print >> sys.stderr, err.msg
print >> sys.stderr, "For help use --help"
return_code = 16
except ExecError, err:
print >> sys.stderr, err.msg
return_code = 16
except SystemExit, err:
# Catch exception raise by optparse
return_code = err
sys.exit(return_code)
slapos/slap/slap.py
View file @ 5f0924ff
...@@ -34,8 +34,6 @@ __all__ = ["slap", "ComputerPartition", "Computer", "SoftwareRelease", ...@@ -34,8 +34,6 @@ __all__ = ["slap", "ComputerPartition", "Computer", "SoftwareRelease",
"Supply", "OpenOrder", "NotFoundError", "Unauthorized", "Supply", "OpenOrder", "NotFoundError", "Unauthorized",
"ResourceNotReady", "ServerError"] "ResourceNotReady", "ServerError"]
from interface import slap as interface
from xml_marshaller import xml_marshaller
import httplib import httplib
import logging import logging
import socket import socket
...@@ -43,9 +41,16 @@ import ssl ...@@ -43,9 +41,16 @@ import ssl
import traceback import traceback
import urllib import urllib
import urlparse import urlparse
import zope.interface import zope.interface
from interface import slap as interface
from xml_marshaller import xml_marshaller
fallback_logger = logging.getLogger(__name__)
fallback_handler = logging.StreamHandler()
fallback_logger.setLevel(logging.INFO)
fallback_logger.addHandler(fallback_handler)
log = logging.getLogger(__name__)
DEFAULT_SOFTWARE_TYPE = 'RootSoftwareInstance' DEFAULT_SOFTWARE_TYPE = 'RootSoftwareInstance'
...@@ -144,7 +149,7 @@ class SoftwareRelease(SlapDocument): ...@@ -144,7 +149,7 @@ class SoftwareRelease(SlapDocument):
else: else:
return self._software_release return self._software_release
def error(self, error_log): def error(self, error_log, logger=None):
try: try:
# Does not follow interface # Does not follow interface
self._connection_helper.POST('/softwareReleaseError', { self._connection_helper.POST('/softwareReleaseError', {
...@@ -152,8 +157,7 @@ class SoftwareRelease(SlapDocument): ...@@ -152,8 +157,7 @@ class SoftwareRelease(SlapDocument):
'computer_id' : self.getComputerId(), 'computer_id' : self.getComputerId(),
'error_log': error_log}) 'error_log': error_log})
except Exception: except Exception:
exception = traceback.format_exc() (logger or fallback_logger).error(traceback.format_exc())
log.error(exception)
def available(self): def available(self):
self._connection_helper.POST('/availableSoftwareRelease', { self._connection_helper.POST('/availableSoftwareRelease', {
...@@ -417,15 +421,14 @@ class ComputerPartition(SlapRequester): ...@@ -417,15 +421,14 @@ class ComputerPartition(SlapRequester):
'computer_partition_id': self.getId(), 'computer_partition_id': self.getId(),
}) })
def error(self, error_log): def error(self, error_log, logger=None):
try: try:
self._connection_helper.POST('/softwareInstanceError', { self._connection_helper.POST('/softwareInstanceError', {
'computer_id': self._computer_id, 'computer_id': self._computer_id,
'computer_partition_id': self.getId(), 'computer_partition_id': self.getId(),
'error_log': error_log}) 'error_log': error_log})
except Exception: except Exception:
exception = traceback.format_exc() (logger or fallback_logger).error(traceback.format_exc())
log.error(exception)
def bang(self, message): def bang(self, message):
self._connection_helper.POST('/softwareInstanceBang', { self._connection_helper.POST('/softwareInstanceBang', {
...@@ -660,7 +663,7 @@ class slap: ...@@ -660,7 +663,7 @@ class slap:
returns SoftwareRelease class object returns SoftwareRelease class object
""" """
return SoftwareRelease(software_release=software_release, return SoftwareRelease(software_release=software_release,
connection_helper=self._connection_helper, connection_helper=self._connection_helper
) )
def registerComputer(self, computer_guid): def registerComputer(self, computer_guid):
......
slapos/tests/entry.py
View file @ 5f0924ff
...@@ -28,7 +28,7 @@ ...@@ -28,7 +28,7 @@
import os import os
import shutil import shutil
import slapos.entry as entry from slapos.cli_legacy import entry
import sys import sys
import tempfile import tempfile
import unittest import unittest
...@@ -126,15 +126,15 @@ class TestCall (BasicMixin, unittest.TestCase): ...@@ -126,15 +126,15 @@ class TestCall (BasicMixin, unittest.TestCase):
options = ["--logfile /opt/slapos/logfile", options = ["--logfile /opt/slapos/logfile",
"--pidfile /opt/slapos/pidfile"] "--pidfile /opt/slapos/pidfile"]
config = '/etc/opt/slapos/slapos.cfg' config_path = '/etc/opt/slapos/slapos.cfg'
try: try:
entry.call(fun, config=config, option=options) entry.call(fun, config_path=config_path, option=options)
except SystemExit, e: except SystemExit, e:
self.assertEqual(e[0], 0) self.assertEqual(e[0], 0)
self.assertNotEqual(original_sysargv, sys.argv) self.assertNotEqual(original_sysargv, sys.argv)
for x in options: for x in options:
self.assertTrue(x in " ".join(sys.argv)) self.assertTrue(x in " ".join(sys.argv))
self.assertEqual(config, sys.argv[1]) self.assertEqual(config_path, sys.argv[1])
def test_config_and_missing_option_are_added(self): def test_config_and_missing_option_are_added(self):
""" """
...@@ -151,16 +151,16 @@ class TestCall (BasicMixin, unittest.TestCase): ...@@ -151,16 +151,16 @@ class TestCall (BasicMixin, unittest.TestCase):
return 0 return 0
options = [default_present_option, missing_option] options = [default_present_option, missing_option]
config = '/etc/opt/slapos/slapos.cfg' config_path = '/etc/opt/slapos/slapos.cfg'
try: try:
entry.call(fun, config=config, option=options) entry.call(fun, config_path=config_path, option=options)
except SystemExit, e: except SystemExit, e:
self.assertEqual(e[0], 0) self.assertEqual(e[0], 0)
self.assertNotEqual(original_sysargv, sys.argv) self.assertNotEqual(original_sysargv, sys.argv)
for x in (missing_option, present_option): for x in (missing_option, present_option):
self.assertTrue(x in " ".join(sys.argv)) self.assertTrue(x in " ".join(sys.argv))
self.assertFalse(default_present_option in " ".join(sys.argv)) self.assertFalse(default_present_option in " ".join(sys.argv))
self.assertEqual(config, sys.argv[1]) self.assertEqual(config_path, sys.argv[1])
def test_present_config_and_option_are_not_added(self): def test_present_config_and_option_are_not_added(self):
""" """
...@@ -178,9 +178,9 @@ class TestCall (BasicMixin, unittest.TestCase): ...@@ -178,9 +178,9 @@ class TestCall (BasicMixin, unittest.TestCase):
return 0 return 0
options = [default_present_option] options = [default_present_option]
config = '/etc/opt/slapos/slapos.cfg' config_path = '/etc/opt/slapos/slapos.cfg'
try: try:
entry.call(fun, config=config, option=options) entry.call(fun, config_path=config_path, option=options)
except SystemExit, e: except SystemExit, e:
self.assertEqual(e[0], 0) self.assertEqual(e[0], 0)
......
slapos/tests/slap.py
View file @ 5f0924ff
...@@ -24,12 +24,15 @@ ...@@ -24,12 +24,15 @@
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
# #
############################################################################## ##############################################################################
import xml_marshaller
import httplib
import os
import unittest import unittest
import urlparse import urlparse
import httplib
import slapos.slap import slapos.slap
import os import xml_marshaller
class UndefinedYetException(Exception): class UndefinedYetException(Exception):
"""To catch exceptions which are not yet defined""" """To catch exceptions which are not yet defined"""
......
slapos/tests/slapformat.py
View file @ 5f0924ff
...@@ -244,7 +244,9 @@ class TestComputer(SlapformatMixin): ...@@ -244,7 +244,9 @@ class TestComputer(SlapformatMixin):
@unittest.skip("Not implemented") @unittest.skip("Not implemented")
def test_construct_empty_prepared(self): def test_construct_empty_prepared(self):
computer = slapos.format.Computer('computer', computer = slapos.format.Computer('computer',
interface=slapos.format.Interface('bridge', '127.0.0.1/16')) interface=slapos.format.Interface(logger=self.test_result,
name='bridge',
ipv4_local_network='127.0.0.1/16'))
computer.instance_root = '/instance_root' computer.instance_root = '/instance_root'
computer.software_root = '/software_root' computer.software_root = '/software_root'
computer.construct() computer.construct()
...@@ -263,7 +265,9 @@ class TestComputer(SlapformatMixin): ...@@ -263,7 +265,9 @@ class TestComputer(SlapformatMixin):
def test_construct_empty_prepared_no_alter_user(self): def test_construct_empty_prepared_no_alter_user(self):
computer = slapos.format.Computer('computer', computer = slapos.format.Computer('computer',
interface=slapos.format.Interface('bridge', '127.0.0.1/16')) interface=slapos.format.Interface(logger=self.test_result,
name='bridge',
ipv4_local_network='127.0.0.1/16'))
computer.instance_root = '/instance_root' computer.instance_root = '/instance_root'
computer.software_root = '/software_root' computer.software_root = '/software_root'
computer.construct(alter_user=False) computer.construct(alter_user=False)
...@@ -279,7 +283,9 @@ class TestComputer(SlapformatMixin): ...@@ -279,7 +283,9 @@ class TestComputer(SlapformatMixin):
@unittest.skip("Not implemented") @unittest.skip("Not implemented")
def test_construct_empty_prepared_no_alter_network(self): def test_construct_empty_prepared_no_alter_network(self):
computer = slapos.format.Computer('computer', computer = slapos.format.Computer('computer',
interface=slapos.format.Interface('bridge', '127.0.0.1/16')) interface=slapos.format.Interface(logger=self.test_result,
name='bridge',
ipv4_local_network='127.0.0.1/16'))
computer.instance_root = '/instance_root' computer.instance_root = '/instance_root'
computer.software_root = '/software_root' computer.software_root = '/software_root'
computer.construct(alter_network=False) computer.construct(alter_network=False)
...@@ -298,7 +304,9 @@ class TestComputer(SlapformatMixin): ...@@ -298,7 +304,9 @@ class TestComputer(SlapformatMixin):
def test_construct_empty_prepared_no_alter_network_user(self): def test_construct_empty_prepared_no_alter_network_user(self):
computer = slapos.format.Computer('computer', computer = slapos.format.Computer('computer',
interface=slapos.format.Interface('bridge', '127.0.0.1/16')) interface=slapos.format.Interface(logger=self.test_result,
name='bridge',
ipv4_local_network='127.0.0.1/16'))
computer.instance_root = '/instance_root' computer.instance_root = '/instance_root'
computer.software_root = '/software_root' computer.software_root = '/software_root'
computer.construct(alter_network=False, alter_user=False) computer.construct(alter_network=False, alter_user=False)
...@@ -315,7 +323,9 @@ class TestComputer(SlapformatMixin): ...@@ -315,7 +323,9 @@ class TestComputer(SlapformatMixin):
@unittest.skip("Not implemented") @unittest.skip("Not implemented")
def test_construct_prepared(self): def test_construct_prepared(self):
computer = slapos.format.Computer('computer', computer = slapos.format.Computer('computer',
interface=slapos.format.Interface('bridge', '127.0.0.1/16')) interface=slapos.format.Interface(logger=self.test_result,
name='bridge',
ipv4_local_network='127.0.0.1/16'))
computer.instance_root = '/instance_root' computer.instance_root = '/instance_root'
computer.software_root = '/software_root' computer.software_root = '/software_root'
partition = slapos.format.Partition('partition', '/part_path', partition = slapos.format.Partition('partition', '/part_path',
...@@ -359,7 +369,9 @@ class TestComputer(SlapformatMixin): ...@@ -359,7 +369,9 @@ class TestComputer(SlapformatMixin):
def test_construct_prepared_no_alter_user(self): def test_construct_prepared_no_alter_user(self):
computer = slapos.format.Computer('computer', computer = slapos.format.Computer('computer',
interface=slapos.format.Interface('bridge', '127.0.0.1/16')) interface=slapos.format.Interface(logger=self.test_result,
name='bridge',
ipv4_local_network='127.0.0.1/16'))
computer.instance_root = '/instance_root' computer.instance_root = '/instance_root'
computer.software_root = '/software_root' computer.software_root = '/software_root'
partition = slapos.format.Partition('partition', '/part_path', partition = slapos.format.Partition('partition', '/part_path',
...@@ -400,7 +412,9 @@ class TestComputer(SlapformatMixin): ...@@ -400,7 +412,9 @@ class TestComputer(SlapformatMixin):
@unittest.skip("Not implemented") @unittest.skip("Not implemented")
def test_construct_prepared_no_alter_network(self): def test_construct_prepared_no_alter_network(self):
computer = slapos.format.Computer('computer', computer = slapos.format.Computer('computer',
interface=slapos.format.Interface('bridge', '127.0.0.1/16')) interface=slapos.format.Interface(logger=self.test_result,
name='bridge',
ipv4_local_network='127.0.0.1/16'))
computer.instance_root = '/instance_root' computer.instance_root = '/instance_root'
computer.software_root = '/software_root' computer.software_root = '/software_root'
partition = slapos.format.Partition('partition', '/part_path', partition = slapos.format.Partition('partition', '/part_path',
...@@ -440,7 +454,9 @@ class TestComputer(SlapformatMixin): ...@@ -440,7 +454,9 @@ class TestComputer(SlapformatMixin):
def test_construct_prepared_no_alter_network_user(self): def test_construct_prepared_no_alter_network_user(self):
computer = slapos.format.Computer('computer', computer = slapos.format.Computer('computer',
interface=slapos.format.Interface('bridge', '127.0.0.1/16')) interface=slapos.format.Interface(logger=self.test_result,
name='bridge',
ipv4_local_network='127.0.0.1/16'))
computer.instance_root = '/instance_root' computer.instance_root = '/instance_root'
computer.software_root = '/software_root' computer.software_root = '/software_root'
partition = slapos.format.Partition('partition', '/part_path', partition = slapos.format.Partition('partition', '/part_path',
......
slapos/tests/slapgrid.py
View file @ 5f0924ff
...@@ -25,25 +25,31 @@ ...@@ -25,25 +25,31 @@
# #
############################################################################## ##############################################################################
from slapos.grid import slapgrid
import httplib import httplib
import logging import logging
import os import os
from random import random import random
import shutil import shutil
import signal import signal
import slapos.slap.slap
import slapos.grid.utils
from slapos.grid.watchdog import Watchdog, getWatchdogID
import socket import socket
import sys import sys
import tempfile import tempfile
import textwrap import textwrap
import time import time
import unittest import unittest2
import urlparse import urlparse
import xml_marshaller import xml_marshaller
import slapos.slap.slap
import slapos.grid.utils
from slapos.grid import slapgrid
from slapos.cli_legacy.slapgrid import parseArgumentTupleAndReturnSlapgridObject
from slapos.grid.utils import md5digest
from slapos.grid.watchdog import Watchdog, getWatchdogID
dummylogger = logging.getLogger()
WATCHDOG_TEMPLATE = """#!%(python_path)s -S WATCHDOG_TEMPLATE = """#!%(python_path)s -S
...@@ -112,10 +118,15 @@ class BasicMixin: ...@@ -112,10 +118,15 @@ class BasicMixin:
'supervisord') 'supervisord')
self.usage_report_periodicity = 1 self.usage_report_periodicity = 1
self.buildout = None self.buildout = None
self.grid = slapgrid.Slapgrid(self.software_root, self.instance_root, self.grid = slapgrid.Slapgrid(self.software_root,
self.master_url, self.computer_id, self.supervisord_socket, self.instance_root,
self.supervisord_configuration_path, self.master_url,
self.buildout, develop=develop) self.computer_id,
self.supervisord_socket,
self.supervisord_configuration_path,
self.buildout,
develop=develop,
logger=logging.getLogger())
# monkey patch buildout bootstrap # monkey patch buildout bootstrap
def dummy(*args, **kw): def dummy(*args, **kw):
pass pass
...@@ -142,7 +153,7 @@ class BasicMixin: ...@@ -142,7 +153,7 @@ class BasicMixin:
shutil.rmtree(self._tempdir, True) shutil.rmtree(self._tempdir, True)
class TestBasicSlapgridCP(BasicMixin, unittest.TestCase): class TestBasicSlapgridCP(BasicMixin, unittest2.TestCase):
def test_no_software_root(self): def test_no_software_root(self):
self.assertRaises(OSError, self.grid.processComputerPartitionList) self.assertRaises(OSError, self.grid.processComputerPartitionList)
...@@ -308,7 +319,7 @@ class ComputerForTest: ...@@ -308,7 +319,7 @@ class ComputerForTest:
return (200, {}, '') return (200, {}, '')
elif method == 'POST' and 'url' in parsed_qs: elif method == 'POST' and 'url' in parsed_qs:
# XXX hardcoded to first sofwtare release! # XXX hardcoded to first software release!
software = self.software_list[0] software = self.software_list[0]
software.sequence.append(parsed_url.path) software.sequence.append(parsed_url.path)
if parsed_url.path == 'buildingSoftwareRelease': if parsed_url.path == 'buildingSoftwareRelease':
...@@ -381,11 +392,11 @@ class InstanceForTest: ...@@ -381,11 +392,11 @@ class InstanceForTest:
os.mkdir(certificate_repository_path) os.mkdir(certificate_repository_path)
self.cert_file = os.path.join(certificate_repository_path, self.cert_file = os.path.join(certificate_repository_path,
"%s.crt" % self.name) "%s.crt" % self.name)
self.certificate = str(random()) self.certificate = str(random.random())
open(self.cert_file, 'w').write(self.certificate) open(self.cert_file, 'w').write(self.certificate)
self.key_file = os.path.join(certificate_repository_path, self.key_file = os.path.join(certificate_repository_path,
"%s.key" % self.name) "%s.key" % self.name)
self.key = str(random()) self.key = str(random.random())
open(self.key_file, 'w').write(self.key) open(self.key_file, 'w').write(self.key)
class SoftwareForTest: class SoftwareForTest:
...@@ -400,8 +411,7 @@ class SoftwareForTest: ...@@ -400,8 +411,7 @@ class SoftwareForTest:
self.software_root = software_root self.software_root = software_root
self.name = 'http://sr%s/' % name self.name = 'http://sr%s/' % name
self.sequence = [] self.sequence = []
self.software_hash = \ self.software_hash = md5digest(self.name)
slapos.grid.utils.getSoftwareUrlHash(self.name)
self.srdir = os.path.join(self.software_root, self.software_hash) self.srdir = os.path.join(self.software_root, self.software_hash)
self.requested_state = 'available' self.requested_state = 'available'
os.mkdir(self.srdir) os.mkdir(self.srdir)
...@@ -442,7 +452,7 @@ touch worked"""): ...@@ -442,7 +452,7 @@ touch worked"""):
class TestSlapgridCPWithMaster(MasterMixin, unittest.TestCase): class TestSlapgridCPWithMaster(MasterMixin, unittest2.TestCase):
def test_nothing_to_do(self): def test_nothing_to_do(self):
...@@ -736,7 +746,7 @@ exit 1 ...@@ -736,7 +746,7 @@ exit 1
self.assertEqual('stopped', instance.state) self.assertEqual('stopped', instance.state)
class TestSlapgridCPWithMasterWatchdog(MasterMixin, unittest.TestCase): class TestSlapgridCPWithMasterWatchdog(MasterMixin, unittest2.TestCase):
def setUp(self): def setUp(self):
MasterMixin.setUp(self) MasterMixin.setUp(self)
...@@ -911,7 +921,7 @@ class TestSlapgridCPWithMasterWatchdog(MasterMixin, unittest.TestCase): ...@@ -911,7 +921,7 @@ class TestSlapgridCPWithMasterWatchdog(MasterMixin, unittest.TestCase):
self.assertEqual(computer.sequence,[]) self.assertEqual(computer.sequence,[])
class TestSlapgridCPPartitionProcessing (MasterMixin, unittest.TestCase): class TestSlapgridCPPartitionProcessing (MasterMixin, unittest2.TestCase):
def test_partition_timestamp(self): def test_partition_timestamp(self):
computer = ComputerForTest(self.software_root,self.instance_root) computer = ComputerForTest(self.software_root,self.instance_root)
...@@ -1298,7 +1308,7 @@ echo %s; echo %s; exit 42""" % (line1, line2)) ...@@ -1298,7 +1308,7 @@ echo %s; echo %s; exit 42""" % (line1, line2))
self.assertTrue("Failed to run buildout" in instance.error_log) self.assertTrue("Failed to run buildout" in instance.error_log)
class TestSlapgridUsageReport(MasterMixin, unittest.TestCase): class TestSlapgridUsageReport(MasterMixin, unittest2.TestCase):
""" """
Test suite about slapgrid-ur Test suite about slapgrid-ur
""" """
...@@ -1500,7 +1510,7 @@ class TestSlapgridUsageReport(MasterMixin, unittest.TestCase): ...@@ -1500,7 +1510,7 @@ class TestSlapgridUsageReport(MasterMixin, unittest.TestCase):
class TestSlapgridSoftwareRelease(MasterMixin, unittest.TestCase): class TestSlapgridSoftwareRelease(MasterMixin, unittest2.TestCase):
def test_one_software_buildout_fail_is_correctly_logged(self): def test_one_software_buildout_fail_is_correctly_logged(self):
""" """
1. We set up a software using a corrupted buildout 1. We set up a software using a corrupted buildout
...@@ -1521,7 +1531,7 @@ echo %s; echo %s; exit 42""" % (line1, line2)) ...@@ -1521,7 +1531,7 @@ echo %s; echo %s; exit 42""" % (line1, line2))
self.assertTrue(line2 in software.error_log) self.assertTrue(line2 in software.error_log)
self.assertTrue("Failed to run buildout" in software.error_log) self.assertTrue("Failed to run buildout" in software.error_log)
class SlapgridInitialization(unittest.TestCase): class SlapgridInitialization(unittest2.TestCase):
""" """
"Abstract" class setting setup and teardown for TestSlapgridArgumentTuple "Abstract" class setting setup and teardown for TestSlapgridArgumentTuple
and TestSlapgridConfigurationFile. and TestSlapgridConfigurationFile.
...@@ -1571,7 +1581,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization): ...@@ -1571,7 +1581,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization):
""" """
Raises if the argument list if empty and without configuration file. Raises if the argument list if empty and without configuration file.
""" """
parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject parser = parseArgumentTupleAndReturnSlapgridObject
# XXX: SystemExit is too generic exception, it is only known that # XXX: SystemExit is too generic exception, it is only known that
# something is wrong # something is wrong
self.assertRaises(SystemExit, parser, *()) self.assertRaises(SystemExit, parser, *())
...@@ -1581,7 +1591,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization): ...@@ -1581,7 +1591,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization):
Check if we can have the slapgrid object returned with the minimum Check if we can have the slapgrid object returned with the minimum
arguments. arguments.
""" """
parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject parser = parseArgumentTupleAndReturnSlapgridObject
return_list = parser(*self.default_arg_tuple) return_list = parser(*self.default_arg_tuple)
self.assertEquals(2, len(return_list)) self.assertEquals(2, len(return_list))
...@@ -1589,19 +1599,19 @@ class TestSlapgridArgumentTuple(SlapgridInitialization): ...@@ -1589,19 +1599,19 @@ class TestSlapgridArgumentTuple(SlapgridInitialization):
""" """
Raises if the signature_private_key_file does not exists. Raises if the signature_private_key_file does not exists.
""" """
parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject parser = parseArgumentTupleAndReturnSlapgridObject
argument_tuple = ("--signature_private_key_file", "/non/exists/path") + \ argument_tuple = ("--signature_private_key_file", "/non/exists/path") + \
self.default_arg_tuple self.default_arg_tuple
# XXX: SystemExit is too generic exception, it is only known that self.assertRaisesRegexp(RuntimeError,
# something is wrong "File '/non/exists/path' does not exist.",
self.assertRaises(SystemExit, parser, *argument_tuple) parser, *argument_tuple)
def test_signature_private_key_file(self): def test_signature_private_key_file(self):
""" """
Check if the signature private key argument value is available on Check if the signature private key argument value is available on
slapgrid object. slapgrid object.
""" """
parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject parser = parseArgumentTupleAndReturnSlapgridObject
argument_tuple = ("--signature_private_key_file", argument_tuple = ("--signature_private_key_file",
self.signature_key_file_descriptor.name) + \ self.signature_key_file_descriptor.name) + \
self.default_arg_tuple self.default_arg_tuple
...@@ -1613,7 +1623,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization): ...@@ -1613,7 +1623,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization):
""" """
Check if giving --all triggers "develop" option. Check if giving --all triggers "develop" option.
""" """
parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject parser = parseArgumentTupleAndReturnSlapgridObject
argument_tuple = ("--all",) + self.default_arg_tuple argument_tuple = ("--all",) + self.default_arg_tuple
slapgrid_object = parser(*argument_tuple)[0] slapgrid_object = parser(*argument_tuple)[0]
self.assertTrue(slapgrid_object.develop) self.assertTrue(slapgrid_object.develop)
...@@ -1623,7 +1633,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization): ...@@ -1623,7 +1633,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization):
Check if not giving --all neither --develop triggers "develop" Check if not giving --all neither --develop triggers "develop"
option to be False. option to be False.
""" """
parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject parser = parseArgumentTupleAndReturnSlapgridObject
argument_tuple = self.default_arg_tuple argument_tuple = self.default_arg_tuple
slapgrid_object = parser(*argument_tuple)[0] slapgrid_object = parser(*argument_tuple)[0]
self.assertFalse(slapgrid_object.develop) self.assertFalse(slapgrid_object.develop)
...@@ -1633,7 +1643,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization): ...@@ -1633,7 +1643,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization):
Check if not giving --maximum-periodicity triggers "force_periodicity" Check if not giving --maximum-periodicity triggers "force_periodicity"
option to be false. option to be false.
""" """
parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject parser = parseArgumentTupleAndReturnSlapgridObject
argument_tuple = self.default_arg_tuple argument_tuple = self.default_arg_tuple
slapgrid_object = parser(*argument_tuple)[0] slapgrid_object = parser(*argument_tuple)[0]
self.assertFalse(slapgrid_object.force_periodicity) self.assertFalse(slapgrid_object.force_periodicity)
...@@ -1642,7 +1652,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization): ...@@ -1642,7 +1652,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization):
""" """
Check if giving --maximum-periodicity triggers "force_periodicity" option. Check if giving --maximum-periodicity triggers "force_periodicity" option.
""" """
parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject parser = parseArgumentTupleAndReturnSlapgridObject
argument_tuple = ("--maximum-periodicity","40") + self.default_arg_tuple argument_tuple = ("--maximum-periodicity","40") + self.default_arg_tuple
slapgrid_object = parser(*argument_tuple)[0] slapgrid_object = parser(*argument_tuple)[0]
self.assertTrue(slapgrid_object.force_periodicity) self.assertTrue(slapgrid_object.force_periodicity)
...@@ -1666,7 +1676,7 @@ upload-to-binary-cache-url-blacklist = ...@@ -1666,7 +1676,7 @@ upload-to-binary-cache-url-blacklist =
http://2/bla http://2/bla
""" % dict(fake_file=self.fake_file_descriptor.name)) """ % dict(fake_file=self.fake_file_descriptor.name))
self.slapos_config_descriptor.seek(0) self.slapos_config_descriptor.seek(0)
slapgrid_object = slapgrid.parseArgumentTupleAndReturnSlapgridObject( slapgrid_object = parseArgumentTupleAndReturnSlapgridObject(
*self.default_arg_tuple)[0] *self.default_arg_tuple)[0]
self.assertEqual( self.assertEqual(
slapgrid_object.upload_to_binary_cache_url_blacklist, slapgrid_object.upload_to_binary_cache_url_blacklist,
...@@ -1694,7 +1704,7 @@ download-from-binary-cache-url-blacklist = ...@@ -1694,7 +1704,7 @@ download-from-binary-cache-url-blacklist =
http://2/bla http://2/bla
""" % dict(fake_file=self.fake_file_descriptor.name)) """ % dict(fake_file=self.fake_file_descriptor.name))
self.slapos_config_descriptor.seek(0) self.slapos_config_descriptor.seek(0)
slapgrid_object = slapgrid.parseArgumentTupleAndReturnSlapgridObject( slapgrid_object = parseArgumentTupleAndReturnSlapgridObject(
*self.default_arg_tuple)[0] *self.default_arg_tuple)[0]
self.assertEqual( self.assertEqual(
slapgrid_object.upload_to_binary_cache_url_blacklist, slapgrid_object.upload_to_binary_cache_url_blacklist,
...@@ -1726,7 +1736,7 @@ download-from-binary-cache-url-blacklist = ...@@ -1726,7 +1736,7 @@ download-from-binary-cache-url-blacklist =
http://4/bla http://4/bla
""" % dict(fake_file=self.fake_file_descriptor.name)) """ % dict(fake_file=self.fake_file_descriptor.name))
self.slapos_config_descriptor.seek(0) self.slapos_config_descriptor.seek(0)
slapgrid_object = slapgrid.parseArgumentTupleAndReturnSlapgridObject( slapgrid_object = parseArgumentTupleAndReturnSlapgridObject(
*self.default_arg_tuple)[0] *self.default_arg_tuple)[0]
self.assertEqual( self.assertEqual(
slapgrid_object.upload_to_binary_cache_url_blacklist, slapgrid_object.upload_to_binary_cache_url_blacklist,
...@@ -1755,7 +1765,7 @@ binary-cache-url-blacklist = ...@@ -1755,7 +1765,7 @@ binary-cache-url-blacklist =
http://2/bla http://2/bla
""" % dict(fake_file=self.fake_file_descriptor.name)) """ % dict(fake_file=self.fake_file_descriptor.name))
self.slapos_config_descriptor.seek(0) self.slapos_config_descriptor.seek(0)
slapgrid_object = slapgrid.parseArgumentTupleAndReturnSlapgridObject( slapgrid_object = parseArgumentTupleAndReturnSlapgridObject(
*self.default_arg_tuple)[0] *self.default_arg_tuple)[0]
self.assertEqual( self.assertEqual(
slapgrid_object.upload_to_binary_cache_url_blacklist, slapgrid_object.upload_to_binary_cache_url_blacklist,
...@@ -1767,7 +1777,7 @@ binary-cache-url-blacklist = ...@@ -1767,7 +1777,7 @@ binary-cache-url-blacklist =
) )
class TestSlapgridCPWithMasterPromise(MasterMixin, unittest.TestCase): class TestSlapgridCPWithMasterPromise(MasterMixin, unittest2.TestCase):
def test_one_failing_promise(self): def test_one_failing_promise(self):
computer = ComputerForTest(self.software_root,self.instance_root) computer = ComputerForTest(self.software_root,self.instance_root)
instance = computer.instance_list[0] instance = computer.instance_list[0]
...@@ -1822,7 +1832,7 @@ exit 127""" % {'worked_file': worked_file}) ...@@ -1822,7 +1832,7 @@ exit 127""" % {'worked_file': worked_file})
slapos.grid.slapgrid.SLAPGRID_PROMISE_FAIL) slapos.grid.slapgrid.SLAPGRID_PROMISE_FAIL)
self.assertTrue(os.path.isfile(worked_file)) self.assertTrue(os.path.isfile(worked_file))
self.assertEqual(instance.error_log, 'Error') self.assertEqual(instance.error_log[-5:], 'Error')
self.assertTrue(instance.error) self.assertTrue(instance.error)
self.assertIsNone(instance.state) self.assertIsNone(instance.state)
......
slapos/tests/slapobject.py
View file @ 5f0924ff
...@@ -25,12 +25,14 @@ ...@@ -25,12 +25,14 @@
# #
############################################################################## ##############################################################################
import logging
import os
import unittest
from slapos.grid import SlapObject from slapos.grid import SlapObject
from slapos.grid import utils from slapos.grid import utils
from slapos.grid import networkcache from slapos.grid import networkcache
from slapos.tests.slapgrid import BasicMixin from slapos.tests.slapgrid import BasicMixin
import os
import unittest
class FakeCallAndRead: class FakeCallAndRead:
...@@ -93,6 +95,7 @@ class TestSoftwareSlapObject(BasicMixin, unittest.TestCase): ...@@ -93,6 +95,7 @@ class TestSoftwareSlapObject(BasicMixin, unittest.TestCase):
url='http://example.com/software.cfg', url='http://example.com/software.cfg',
software_root=self.software_root, software_root=self.software_root,
buildout=self.buildout, buildout=self.buildout,
logger=logging.getLogger(),
signature_private_key_file='/signature/private/key_file', signature_private_key_file='/signature/private/key_file',
upload_cache_url='http://example.com/uploadcache', upload_cache_url='http://example.com/uploadcache',
upload_dir_url='http://example.com/uploaddir', upload_dir_url='http://example.com/uploaddir',
...@@ -129,7 +132,8 @@ class TestSoftwareSlapObject(BasicMixin, unittest.TestCase): ...@@ -129,7 +132,8 @@ class TestSoftwareSlapObject(BasicMixin, unittest.TestCase):
software = SlapObject.Software( software = SlapObject.Software(
url='http://example.com/software.cfg', url='http://example.com/software.cfg',
software_root=self.software_root, software_root=self.software_root,
buildout=self.buildout) buildout=self.buildout,
logger=logging.getLogger())
software.install() software.install()
...@@ -161,6 +165,7 @@ class TestSoftwareSlapObject(BasicMixin, unittest.TestCase): ...@@ -161,6 +165,7 @@ class TestSoftwareSlapObject(BasicMixin, unittest.TestCase):
url='http://example.com/software.cfg', url='http://example.com/software.cfg',
software_root=self.software_root, software_root=self.software_root,
buildout=self.buildout, buildout=self.buildout,
logger=logging.getLogger(),
signature_private_key_file='/signature/private/key_file', signature_private_key_file='/signature/private/key_file',
upload_cache_url='http://example.com/uploadcache', upload_cache_url='http://example.com/uploadcache',
upload_dir_url='http://example.com/uploaddir', upload_dir_url='http://example.com/uploaddir',
...@@ -192,6 +197,7 @@ class TestSoftwareSlapObject(BasicMixin, unittest.TestCase): ...@@ -192,6 +197,7 @@ class TestSoftwareSlapObject(BasicMixin, unittest.TestCase):
url='http://example.com/software.cfg', url='http://example.com/software.cfg',
software_root=self.software_root, software_root=self.software_root,
buildout=self.buildout, buildout=self.buildout,
logger=logging.getLogger(),
signature_private_key_file='/signature/private/key_file', signature_private_key_file='/signature/private/key_file',
upload_cache_url='http://example.com/uploadcache', upload_cache_url='http://example.com/uploadcache',
upload_dir_url='http://example.com/uploaddir', upload_dir_url='http://example.com/uploaddir',
......
slapos/tests/slapproxy.py
View file @ 5f0924ff
...@@ -27,6 +27,7 @@ ...@@ -27,6 +27,7 @@
# #
############################################################################## ##############################################################################
import ConfigParser
import os import os
import logging import logging
import shutil import shutil
...@@ -86,14 +87,16 @@ database_uri = %(tempdir)s/lib/proxy.db ...@@ -86,14 +87,16 @@ database_uri = %(tempdir)s/lib/proxy.db
""" """
Set config for slapproxy and start it Set config for slapproxy and start it
""" """
config = slapos.proxy.Config() conf = slapos.proxy.ProxyConfig(logger=logging.getLogger())
config.setConfig(*(ProxyOption(self.proxy_db), configp = ConfigParser.SafeConfigParser()
self.slapos_cfg)) configp.read(self.slapos_cfg)
conf.mergeConfig(ProxyOption(self.proxy_db), configp)
conf.setConfig()
views.app.config['TESTING'] = True views.app.config['TESTING'] = True
views.app.config['computer_id'] = self.computer_id views.app.config['computer_id'] = self.computer_id
views.app.config['DATABASE_URI'] = self.proxy_db views.app.config['DATABASE_URI'] = self.proxy_db
views.app.config['HOST'] = config.host views.app.config['HOST'] = conf.host
views.app.config['port'] = config.port views.app.config['port'] = conf.port
self.app = views.app.test_client() self.app = views.app.test_client()
def add_free_partition (self, partition_amount): def add_free_partition (self, partition_amount):
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7