Commit 4975b455 authored by Priscila Manhaes's avatar Priscila Manhaes

updated to master

parent 6d694550
lamp
=====
the lamp recipe help you to deploy simply a php based application on slapos. This recipe is
able to setup mariadb, apache and apache-php for your php application, is also capable to
configure your software during installation to ensure a full compatibility.
How to use?
-----------
just add this part in your software.cfg to use the lamp.simple module
[instance-recipe]
egg = slapos.cookbook
module = lamp.simple
you also need to extend lamp.cfg
extends =
http://git.erp5.org/gitweb/slapos.git/blob_plain/refs/tags/slapos-0.50:/stack/lamp.cfg
lamp.runner
=====
When you install some software (such as prestashop) you need to remove or rename folder, with slapos you can not
access to the www-data directory. to do this, you need to tell to lamp recipe to remove or/and it when software
will be instantiated. Some software requires more than rename or delete a folder (manualy create database etc...)
in this case you need to write a python script and lamp recipe must run it when installing your software.
How to use?
-----------
CONDITION
--------
the action (move, rename, launch script) only starts when the condition is filled.
in instance.cfg, add
file_token = path_of_file
and the action will begin when path_of_www-data/path_of_file will be created
you can also use database to check condition. add
table_name = name_of_table
constraint = sql_where_condition
name_of_table is the full or partial name(in some cases we can not know the prefix used to create tables) of table
into mariadb databse for example table_name = admin. if you use
name_of_table = **, the action will begin when database is ready.
constraint is the sql_condition to use when search entry into name_of_table for example constraint = `admin_id`=1
you can no use file_token and table_name at the same time, otherwise file_token will be used in priority. attention
to the conditions that will never be satisfied.
ACTION
-------
the action start when condition is true
1- delete file or folder
into instance.cfg, use
delete = file_or_folder1, file_or_folder2, file_or_folder3 ...
for example delete = admin
2- rename file or folder
into instance.cfg, use
rename = old_name1 => new_name1, old_name2 => new_name2, ... you can also use
rename = old_name1, old_name2 => new_name2, ... in this case old_name1 will be rename and the new name will be chose
by joining old_name1 and mysql_user: this should give
rename = old_name1 => old_name1-mysql_user, old_name2 => new_name2, ...
3- launch python script
use script = ${configure-script:location}/${configure-script:filename} into instance.cfg, add part configure-script
into software.cfg
parts = configure-script
[configure-script]
recipe = hexagonit.recipe.download
location = ${buildout:parts-directory}/${:_buildout_section_name_}
url = url_of_script_name.py
filename = script_name.py
download-only = True
the script_name.py should contain a main module, sys.argv is passed to the main. you can write script_name.py like this
....
def setup(args):
base_url, htdocs, renamed, mysql_user, mysql_password, mysql_database, mysql_host = args
.......
if __name__ == '__main__':
setup(sys.argv[1:])
base_url: is the url of php software
htdocs: is the path of www-data directory
mysql_user, mysql_password, mysql_database, mysql_host: is the mariadb parameters
mkdirectory
===========
mkdirectory loops on its options and create the directory joined
.. Note::
Use a slash ``/`` as directory separator. Don't use system dependent separator.
The slash will be parsed and replace by the operating system right separator.
Only use relative directory to the buildout root directory.
The created directory won't be added to path list.
##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import os
import hashlib
import ConfigParser
from slapos.recipe.librecipe import GenericBaseRecipe
class Recipe(GenericBaseRecipe):
def setPath(self):
self.ca_dir = self.options['ca-dir']
self.request_directory = self.options['requests-directory']
self.ca_private = self.options['ca-private']
self.ca_certs = self.options['ca-certs']
self.ca_newcerts = self.options['ca-newcerts']
self.ca_crl = self.options['ca-crl']
self.ca_key_ext = '.key'
self.ca_crt_ext = '.crt'
def install(self):
path_list = []
# XXX: We gotta find better a way to get these options
ca_country_code = 'XX'
ca_email = 'xx@example.com'
ca_state = 'State',
ca_city = 'City'
ca_company = 'Company'
# XXX: end
self.setPath()
config = dict(ca_dir=self.ca_dir, request_dir=self.request_directory)
for f in ['crlnumber', 'serial']:
if not os.path.exists(os.path.join(self.ca_dir, f)):
open(os.path.join(self.ca_dir, f), 'w').write('01')
if not os.path.exists(os.path.join(self.ca_dir, 'index.txt')):
open(os.path.join(self.ca_dir, 'index.txt'), 'w').write('')
openssl_configuration = os.path.join(self.ca_dir, 'openssl.cnf')
config.update(
working_directory=self.ca_dir,
country_code=ca_country_code,
state=ca_state,
city=ca_city,
company=ca_company,
email_address=ca_email,
)
self.createFile(openssl_configuration, self.substituteTemplate(
self.getTemplateFilename('openssl.cnf.ca.in'), config))
ca_wrapper = self.createPythonScript(
self.options['wrapper'],
'%s.certificate_authority.runCertificateAuthority' % __name__,
dict(
openssl_configuration=openssl_configuration,
openssl_binary=self.options['openssl-binary'],
certificate=os.path.join(self.ca_dir, 'cacert.pem'),
key=os.path.join(self.ca_private, 'cakey.pem'),
crl=self.ca_crl,
request_dir=self.request_directory
)
)
path_list.append(ca_wrapper)
return path_list
class Request(Recipe):
def _options(self, options):
if 'name' not in options:
options['name'] = self.name
def install(self):
self.setPath()
key_file = self.options['key-file']
cert_file = self.options['cert-file']
name = self.options['name']
hash_ = hashlib.sha512(name).hexdigest()
key = os.path.join(self.ca_private, hash_ + self.ca_key_ext)
certificate = os.path.join(self.ca_certs, hash_ + self.ca_crt_ext)
parser = ConfigParser.RawConfigParser()
parser.add_section('certificate')
parser.set('certificate', 'name', name)
parser.set('certificate', 'key_file', key)
parser.set('certificate', 'certificate_file', certificate)
parser.write(open(os.path.join(self.request_directory, hash_), 'w'))
for link in [key_file, cert_file]:
if os.path.islink(link):
os.unlink(link)
elif os.path.exists(link):
raise OSError("%r file should be a symbolic link.")
os.symlink(key, key_file)
os.symlink(certificate, cert_file)
wrapper = self.createPythonScript(
self.options['wrapper'],
'slapos.recipe.librecipe.execute.execute_wait',
[ [self.options['executable']],
[certificate, key] ],
)
return [key_file, cert_file, wrapper]
import os
import subprocess
import time
import ConfigParser
def popenCommunicate(command_list, input=None):
subprocess_kw = dict(stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
if input is not None:
subprocess_kw.update(stdin=subprocess.PIPE)
popen = subprocess.Popen(command_list, **subprocess_kw)
result = popen.communicate(input)[0]
if popen.returncode is None:
popen.kill()
if popen.returncode != 0:
raise ValueError('Issue during calling %r, result was:\n%s' % (
command_list, result))
return result
class CertificateAuthority:
def __init__(self, key, certificate, openssl_binary,
openssl_configuration, request_dir):
self.key = key
self.certificate = certificate
self.openssl_binary = openssl_binary
self.openssl_configuration = openssl_configuration
self.request_dir = request_dir
def checkAuthority(self):
file_list = [ self.key, self.certificate ]
ca_ready = True
for f in file_list:
if not os.path.exists(f):
ca_ready = False
break
if ca_ready:
return
for f in file_list:
if os.path.exists(f):
os.unlink(f)
try:
# no CA, let us create new one
popenCommunicate([self.openssl_binary, 'req', '-nodes', '-config',
self.openssl_configuration, '-new', '-x509', '-extensions',
'v3_ca', '-keyout', self.key, '-out', self.certificate,
'-days', '10950'], 'Automatic Certificate Authority\n')
except:
try:
for f in file_list:
if os.path.exists(f):
os.unlink(f)
except:
# do not raise during cleanup
pass
raise
def _checkCertificate(self, common_name, key, certificate):
file_list = [key, certificate]
ready = True
for f in file_list:
if not os.path.exists(f):
ready = False
break
if ready:
return False
for f in file_list:
if os.path.exists(f):
os.unlink(f)
csr = certificate + '.csr'
try:
popenCommunicate([self.openssl_binary, 'req', '-config',
self.openssl_configuration, '-nodes', '-new', '-keyout',
key, '-out', csr, '-days', '3650'],
common_name + '\n')
try:
popenCommunicate([self.openssl_binary, 'ca', '-batch', '-config',
self.openssl_configuration, '-out', certificate,
'-infiles', csr])
finally:
if os.path.exists(csr):
os.unlink(csr)
except:
try:
for f in file_list:
if os.path.exists(f):
os.unlink(f)
except:
# do not raise during cleanup
pass
raise
else:
return True
def checkRequestDir(self):
for request_file in os.listdir(self.request_dir):
parser = ConfigParser.RawConfigParser()
parser.readfp(open(os.path.join(self.request_dir, request_file), 'r'))
if self._checkCertificate(parser.get('certificate', 'name'),
parser.get('certificate', 'key_file'), parser.get('certificate',
'certificate_file')):
print 'Created certificate %r' % parser.get('certificate', 'name')
def runCertificateAuthority(ca_conf):
ca = CertificateAuthority(ca_conf['key'], ca_conf['certificate'],
ca_conf['openssl_binary'], ca_conf['openssl_configuration'],
ca_conf['request_dir'])
while True:
ca.checkAuthority()
ca.checkRequestDir()
# XXX
# Antoine: I really don't like that at all. It wastes useful CPU time.
# I think it would be a greater idea to use pyinotify
# <http://pyinotify.sourceforge.net/>
# Or we could use select() with socket as well.
time.sleep(60)
# end XXX
This diff is collapsed.
##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
from slapos.recipe.librecipe import BaseSlapRecipe
import os
import subprocess
import pkg_resources
import zc.buildout
import zc.recipe.egg
import sys
class Recipe(BaseSlapRecipe):
def getTemplateFilename(self, template_name):
return pkg_resources.resource_filename(__name__,
'template/%s' % template_name)
def _install(self):
self.path_list = []
self.requirements, self.ws = self.egg.working_set()
document_root = self.createDataDirectory('www')
apache_config = self.installApache(document_root)
self.setConnectionUrl(scheme='webdavs',
host=apache_config['ip'],
port=apache_config['port'],
auth=(apache_config['user'],
apache_config['password']))
return self.path_list
def installApache(self, document_root, ip=None, port=None):
if ip is None:
ip=self.getGlobalIPv6Address()
if port is None:
port = '9080'
htpasswd_config = self.createHtpasswd()
ssl_config = self.createCertificate(size=2048)
apache_config = dict(
pid_file=os.path.join(self.run_directory, 'httpd.pid'),
lock_file=os.path.join(self.run_directory, 'httpd.lock'),
davlock_db=os.path.join(self.run_directory, 'davdb.lock'),
ip=ip,
port=port,
error_log=os.path.join(self.log_directory, 'httpd-error.log'),
access_log=os.path.join(self.log_directory, 'httpd-access.log'),
document_root=document_root,
modules_dir=self.options['apache_modules_dir'],
mime_types=self.options['apache_mime_file'],
server_root=self.work_directory,
email_address='admin@vifib.net',
htpasswd_file=htpasswd_config['htpasswd_file'],
ssl_certificate=ssl_config['certificate'],
ssl_key=ssl_config['key'],
)
httpd_config_file = self.createConfigurationFile('httpd.conf',
self.substituteTemplate(self.getTemplateFilename('httpd.conf.in'),
apache_config))
self.path_list.append(httpd_config_file)
apache_runner = zc.buildout.easy_install.scripts(
[('httpd', 'slapos.recipe.librecipe.execute', 'execute')],
self.ws, sys.executable, self.wrapper_directory,
arguments=[self.options['apache_binary'],
'-f', httpd_config_file,
'-DFOREGROUND',
]
)[0]
self.path_list.append(apache_runner)
return dict(ip=apache_config['ip'],
port=apache_config['port'],
user=htpasswd_config['user'],
password=htpasswd_config['password']
)
def createHtpasswd(self):
htpasswd = self.createConfigurationFile('htpasswd', '')
self.path_list.append(htpasswd)
password = self.generatePassword()
user = 'user'
subprocess.check_call([self.options['apache_htpasswd'],
'-bc', htpasswd,
user, password
])
return dict(htpasswd_file=htpasswd,
user=user,
password=password)
def createCertificate(self, size=1024, subject='/C=FR/L=Marcq-en-Baroeul/O=Nexedi'):
key_file = os.path.join(self.etc_directory, 'httpd.key')
self.path_list.append(key_file)
certificate_file = os.path.join(self.etc_directory, 'httpd.crt')
self.path_list.append(certificate_file)
subprocess.check_call([self.options['openssl_binary'],
'req', '-x509', '-nodes',
'-newkey', 'rsa:%s' % size,
'-subj', str(subject),
'-out', certificate_file,
'-keyout', key_file
])
return dict(key=key_file,
certificate=certificate_file)
ServerRoot "%(server_root)s"
Listen [%(ip)s]:%(port)s
# Needed modules
LoadModule authn_file_module "%(modules_dir)s/mod_authn_file.so"
LoadModule authz_host_module "%(modules_dir)s/mod_authz_host.so"
LoadModule authz_user_module "%(modules_dir)s/mod_authz_user.so"
LoadModule auth_basic_module "%(modules_dir)s/mod_auth_basic.so"
LoadModule auth_digest_module "%(modules_dir)s/mod_auth_digest.so"
LoadModule log_config_module "%(modules_dir)s/mod_log_config.so"
LoadModule headers_module "%(modules_dir)s/mod_headers.so"
LoadModule setenvif_module "%(modules_dir)s/mod_setenvif.so"
LoadModule ssl_module "%(modules_dir)s/mod_ssl.so"
LoadModule mime_module "%(modules_dir)s/mod_mime.so"
LoadModule dav_module "%(modules_dir)s/mod_dav.so"
LoadModule dav_fs_module "%(modules_dir)s/mod_dav_fs.so"
LoadModule dir_module "%(modules_dir)s/mod_dir.so"
ServerAdmin %(email_address)s
# Quiet Server header (if not, Apache give its life history)
# It's safer
ServerTokens ProductOnly
DocumentRoot "%(document_root)s"
PidFile "%(pid_file)s"
LockFile "%(lock_file)s"
DavLockDB "%(davlock_db)s"
<Directory />
Options FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
</Directory>
<Directory %(document_root)s>
Options Indexes MultiViews
AllowOverride None
Order allow,deny
Allow from all
Dav On
# Security Rules to avoid DDoS Attacks
DavDepthInfinity Off
LimitXMLRequestBody 0
# Cross-Origin Resources Sharing
Header always set Access-Control-Max-Age "0"
Header always set Access-Control-Allow-Origin "*"
Header always set Access-Control-Allow-Methods "OPTIONS, GET, HEAD, POST, PUT, DELETE, PROPFIND"
Header always set Access-Control-Allow-Headers "Content-Type, X-Requested-With, X-HTTP-Method-Override, Accept, Authorization, Depth"
SetEnvIf Origin "(.+)" ORIGIN=$1
Header always set Access-Control-Allow-Origin %%{ORIGIN}e
AuthType Basic
AuthName "WebDAV Storage"
AuthUserFile "%(htpasswd_file)s"
<LimitExcept OPTIONS>
Require valid-user
</LimitExcept>
</Directory>
ErrorLog "%(error_log)s"
LogLevel warn
LogFormat "%%h %%l %%u %%t \"%%r\" %%>s %%b \"%%{Referer}i\" \"%%{User-Agent}i\"" combined
LogFormat "%%h %%l %%u %%t \"%%r\" %%>s %%b" common
CustomLog "%(access_log)s" common
DefaultType text/plain
TypesConfig "%(mime_types)s"
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
SSLEngine on
SSLCertificateFile "%(ssl_certificate)s"
SSLCertificateKeyFile "%(ssl_key)s"
##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import os
from slapos.recipe.librecipe import GenericBaseRecipe
class Recipe(GenericBaseRecipe):
def install(self):
self.logger.info("Installing dcron...")
path_list = []
cronstamps = self.options['cronstamps']
cron_d = self.options['cron-entries']
crontabs = self.options['crontabs']
catcher = self.options['catcher']
binary = self.options['binary']
script = self.createPythonScript(binary,
'slapos.recipe.librecipe.execute.execute',
[self.options['dcrond-binary'].strip(), '-s', cron_d, '-c', crontabs,
'-t', cronstamps, '-f', '-l', '5', '-M', catcher]
)
path_list.append(script)
self.logger.debug('Main cron executable created at : %r', script)
self.logger.info("dcron successfully installed.")
return path_list
class Part(GenericBaseRecipe):
def _options(self, options):
if 'name' not in options:
options['name'] = self.name
def install(self):
cron_d = self.options['cron-entries']
filename = os.path.join(cron_d, 'name')
with open(filename, 'w') as part:
part.write('%(frequency)s %(command)s\n' % {
'frequency': self.options['frequency'],
'command': self.options['command'],
})
return [filename]
##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
from slapos.recipe.librecipe import GenericBaseRecipe
class Recipe(GenericBaseRecipe):
def install(self):
remote_url = self.options['remote_backup']
backup_directory = self.options['directory']
wrapper = self.createPythonScript(
self.options['wrapper'],
'slapos.recipe.librecipe.execute.execute',
[self.options['duplicity_binary'], '--no-encryption',
backup_directory, remote_url]
)
return [wrapper]
This diff is collapsed.
This diff is collapsed.
...@@ -4,3 +4,4 @@ SSLCertificateKeyFile %(login_key)s ...@@ -4,3 +4,4 @@ SSLCertificateKeyFile %(login_key)s
SSLRandomSeed startup builtin SSLRandomSeed startup builtin
SSLRandomSeed connect builtin SSLRandomSeed connect builtin
SSLProxyEngine On
...@@ -34,9 +34,9 @@ RequestHeader unset REMOTE_USER ...@@ -34,9 +34,9 @@ RequestHeader unset REMOTE_USER
# Log configuration # Log configuration
ErrorLog "%(error_log)s" ErrorLog "%(error_log)s"
LogFormat "%%h %%{REMOTE_USER}i %%l %%u %%t \"%%r\" %%>s %%b \"%%{Referer}i\" \"%%{User-Agent}i\"" combined # Default apache log format with request time in microsecond at the end
LogFormat "%%h %%{REMOTE_USER}i %%l %%u %%t \"%%r\" %%>s %%b" common LogFormat "%%h %%l %%u %%t \"%%r\" %%>s %%b \"%%{Referer}i\" \"%%{User-Agent}i\" %%D" combined
CustomLog "%(access_log)s" common CustomLog "%(access_log)s" combined
# Directory protection # Directory protection
<Directory /> <Directory />
......
...@@ -9,10 +9,25 @@ defaults ...@@ -9,10 +9,25 @@ defaults
retries 1 retries 1
option redispatch option redispatch
maxconn 2000 maxconn 2000
timeout server 3000s # it is useless to have timeout much bigger than the one of apache.
timeout queue 5s # By default apache use 300s, so we set slightly more in order to
timeout connect 10s # make sure that apache will first stop the connection.
timeout client 3600s timeout server 305s
# Stop waiting in queue for a zope to become available.
# If no zope can be reached after one minute, consider the request will
# never succeed.
timeout queue 60s
# The connection should be immediate on LAN,
# so we should not set more than 5 seconds, and it could be already too much
timeout connect 5s
# As requested in haproxy doc, make this "at least equal to timeout server".
timeout client 305s
# Use "option httpclose" to not preserve client & server persistent connections
# while handling every incoming request individually, dispatching them one after
# another to servers, in HTTP close mode. This is really needed when haproxy
# is configured with maxconn to 1, without this options browser are unable
# to render a page
option httpclose
listen %(name)s %(ip)s:%(port)s listen %(name)s %(ip)s:%(port)s
cookie SERVERID insert cookie SERVERID insert
......
CREATE DATABASE IF NOT EXISTS %(mysql_database)s; CREATE DATABASE IF NOT EXISTS %(mysql_database)s;
GRANT ALL PRIVILEGES ON %(mysql_database)s.* TO %(mysql_user)s@'%%' IDENTIFIED BY '%(mysql_password)s'; GRANT ALL PRIVILEGES ON %(mysql_database)s.* TO %(mysql_user)s@'%%' IDENTIFIED BY '%(mysql_password)s';
GRANT ALL PRIVILEGES ON %(mysql_database)s.* TO %(mysql_user)s@'localhost' IDENTIFIED BY '%(mysql_password)s';
%(file_list)s { %(file_list)s {
daily daily
dateext dateext
rotate 30 rotate 3650
compress compress
notifempty notifempty
sharedscripts sharedscripts
......
This diff is collapsed.
# ZEO configuration file generated by SlapOS # ZEO configuration file generated by SlapOS
<zeo> <zeo>
address %(zeo_ip)s:%(zeo_port)s address %(zeo_ip)s:%(zeo_port)s
read-only false
invalidation-queue-size 100
pid-filename %(zeo_pid)s pid-filename %(zeo_pid)s
</zeo> </zeo>
...@@ -10,6 +8,7 @@ ...@@ -10,6 +8,7 @@
<eventlog> <eventlog>
<logfile> <logfile>
dateformat
path %(zeo_event_log)s path %(zeo_event_log)s
</logfile> </logfile>
</eventlog> </eventlog>
<zodb_db %(storage_name)s> <zodb_db %(storage_name)s>
cache-size %(zodb_cache_size)d
mount-point %(mount_point)s mount-point %(mount_point)s
<zeoclient> <zeoclient>
cache-size %(zeo_client_cache_size)s
server %(address)s server %(address)s
storage %(storage_name)s storage %(storage_name)s
name %(storage_name)s name %(storage_name)s
......
<zodb_db root> <zodb_db root>
cache-size %(zodb_cache_size)d
<filestorage> <filestorage>
path %(zodb_root_path)s path %(zodb_root_path)s
</filestorage> </filestorage>
......
...@@ -7,10 +7,8 @@ instancehome $INSTANCE ...@@ -7,10 +7,8 @@ instancehome $INSTANCE
# Used products # Used products
%(products)s %(products)s
# Environment override # Environment is setup in running wrapper script
<environment> # Reason: zope.conf is read too late for some componets
%(environment)s
</environment>
# No need to debug # No need to debug
debug-mode off debug-mode off
...@@ -34,11 +32,13 @@ lock-filename %(lock-filename)s ...@@ -34,11 +32,13 @@ lock-filename %(lock-filename)s
# Logging configuration # Logging configuration
<eventlog> <eventlog>
<logfile> <logfile>
dateformat
path %(event_log)s path %(event_log)s
</logfile> </logfile>
</eventlog> </eventlog>
<logger access> <logger access>
<logfile> <logfile>
dateformat
path %(z2_log)s path %(z2_log)s
</logfile> </logfile>
</logger> </logger>
......
...@@ -37,6 +37,9 @@ import hashlib ...@@ -37,6 +37,9 @@ import hashlib
class Recipe(BaseSlapRecipe): class Recipe(BaseSlapRecipe):
# To avoid magic numbers
VNC_BASE_PORT = 5900
def _install(self): def _install(self):
""" """
Set the connection dictionnary for the computer partition and create a list Set the connection dictionnary for the computer partition and create a list
...@@ -54,15 +57,30 @@ class Recipe(BaseSlapRecipe): ...@@ -54,15 +57,30 @@ class Recipe(BaseSlapRecipe):
self.ca_conf = self.installCertificateAuthority() self.ca_conf = self.installCertificateAuthority()
self.key_path, self.certificate_path = self.requestCertificate('noVNC') self.key_path, self.certificate_path = self.requestCertificate('noVNC')
# Install the socket_connection_attempt script
catcher = zc.buildout.easy_install.scripts(
[('check_port_listening', __name__ + 'socket_connection_attempt', 'connection_attempt')],
self.ws,
sys.executable,
self.bin_directory,
)
# Save the check_port_listening script path
check_port_listening_script = catcher[0]
# Get the port_listening_promise template path, and save it
self.port_listening_promise_path = pkg_resources.resource_filename(
__name__, 'template/port_listening_promise.in')
self.port_listening_promise_conf = dict(
check_port_listening_script=check_port_listening_script,
)
kvm_conf = self.installKvm(vnc_ip = self.getLocalIPv4Address()) kvm_conf = self.installKvm(vnc_ip = self.getLocalIPv4Address())
vnc_port = 5900 + kvm_conf['vnc_display'] vnc_port = Recipe.VNC_BASE_PORT + kvm_conf['vnc_display']
noVNC_conf = self.installNoVnc(source_ip = self.getGlobalIPv6Address(), noVNC_conf = self.installNoVnc(source_ip = self.getGlobalIPv6Address(),
source_port = 6080, source_port = 6080,
target_ip = kvm_conf['vnc_ip'], target_ip = kvm_conf['vnc_ip'],
target_port = vnc_port, target_port = vnc_port)
python_path = kvm_conf['python_path'])
self.linkBinary() self.linkBinary()
self.computer_partition.setConnectionDict(dict( self.computer_partition.setConnectionDict(dict(
...@@ -137,8 +155,7 @@ class Recipe(BaseSlapRecipe): ...@@ -137,8 +155,7 @@ class Recipe(BaseSlapRecipe):
# Instanciate KVM # Instanciate KVM
kvm_template_location = pkg_resources.resource_filename( kvm_template_location = pkg_resources.resource_filename(
__name__, os.path.join( __name__, 'template/kvm_run.in')
'template', 'kvm_run.in'))
kvm_runner_path = self.createRunningWrapper("kvm", kvm_runner_path = self.createRunningWrapper("kvm",
self.substituteTemplate(kvm_template_location, self.substituteTemplate(kvm_template_location,
...@@ -148,9 +165,7 @@ class Recipe(BaseSlapRecipe): ...@@ -148,9 +165,7 @@ class Recipe(BaseSlapRecipe):
# Instanciate KVM controller # Instanciate KVM controller
kvm_controller_template_location = pkg_resources.resource_filename( kvm_controller_template_location = pkg_resources.resource_filename(
__name__, os.path.join( __name__, 'template/kvm_controller_run.in')
'template',
'kvm_controller_run.in' ))
kvm_controller_runner_path = self.createRunningWrapper("kvm_controller", kvm_controller_runner_path = self.createRunningWrapper("kvm_controller",
self.substituteTemplate(kvm_controller_template_location, self.substituteTemplate(kvm_controller_template_location,
...@@ -165,10 +180,20 @@ class Recipe(BaseSlapRecipe): ...@@ -165,10 +180,20 @@ class Recipe(BaseSlapRecipe):
##slapreport_runner_path = self.instanciate_wrapper("slapreport", ##slapreport_runner_path = self.instanciate_wrapper("slapreport",
# [database_path, python_path]) # [database_path, python_path])
# Add VNC promise
self.port_listening_promise_conf.update(
hostname=kvm_conf['vnc_ip'],
port=Recipe.VNC_BASE_PORT + kvm_conf['vnc_display'],
)
self.createPromiseWrapper("vnc_promise",
self.substituteTemplate(self.port_listening_promise_path,
self.port_listening_promise_conf,
)
)
return kvm_conf return kvm_conf
def installNoVnc(self, source_ip, source_port, target_ip, target_port, def installNoVnc(self, source_ip, source_port, target_ip, target_port):
python_path):
""" """
Create noVNC configuration dictionnary and instanciate Websockify proxy Create noVNC configuration dictionnary and instanciate Websockify proxy
...@@ -184,11 +209,17 @@ class Recipe(BaseSlapRecipe): ...@@ -184,11 +209,17 @@ class Recipe(BaseSlapRecipe):
noVNC_conf['source_ip'] = source_ip noVNC_conf['source_ip'] = source_ip
noVNC_conf['source_port'] = source_port noVNC_conf['source_port'] = source_port
# Install numpy.
# XXX-Cedric : this looks like a hack. Do we have better solution, knowing
# That websockify is not an egg?
numpy = zc.buildout.easy_install.install(['numpy'], self.options['eggs-directory'])
environment = dict(PYTHONPATH='%s' % numpy.entries[0])
# Instanciate Websockify # Instanciate Websockify
websockify_runner_path = zc.buildout.easy_install.scripts([('websockify', websockify_runner_path = zc.buildout.easy_install.scripts([('websockify',
'slapos.recipe.librecipe.execute', 'execute_wait')], self.ws, 'slapos.recipe.librecipe.execute', 'executee_wait')], self.ws,
sys.executable, self.wrapper_directory, arguments=[ sys.executable, self.wrapper_directory, arguments=[
[python_path.strip(), [sys.executable.strip(),
self.options['websockify_path'], self.options['websockify_path'],
'--web', '--web',
self.options['noVNC_location'], self.options['noVNC_location'],
...@@ -197,11 +228,22 @@ class Recipe(BaseSlapRecipe): ...@@ -197,11 +228,22 @@ class Recipe(BaseSlapRecipe):
'--ssl-only', '--ssl-only',
'%s:%s' % (source_ip, source_port), '%s:%s' % (source_ip, source_port),
'%s:%s' % (target_ip, target_port)], '%s:%s' % (target_ip, target_port)],
[self.certificate_path, self.key_path]] [self.certificate_path, self.key_path],
environment]
)[0] )[0]
self.path_list.append(websockify_runner_path) self.path_list.append(websockify_runner_path)
# Add noVNC promise
self.port_listening_promise_conf.update(hostname=noVNC_conf['source_ip'],
port=noVNC_conf['source_port'],
)
self.createPromiseWrapper("novnc_promise",
self.substituteTemplate(self.port_listening_promise_path,
self.port_listening_promise_conf,
)
)
return noVNC_conf return noVNC_conf
def linkBinary(self): def linkBinary(self):
......
import socket
import sys
def connection_attempt():
try:
hostname, port = sys.argv[1:3]
except ValueError:
print >> sys.stderr, """Bad command line.
Usage: %s hostname|ip port""" % sys.argv[0]
sys.exit(1)
connection_okay = False
try:
s = socket.create_connection((hostname, port))
connection_okay = True
s.close()
except (socket.error, socket.timeout):
connection_okay = False
if not connection_okay:
print >> sys.stderr, "%(port)s on %(ip)s isn't listening" % {
'port': port, 'ip': hostname
}
sys.exit(127)
#!/usr/bin/env sh
"%(check_port_listening_script)s" "%(hostname)s" "%(port)s"
exit $?
This diff is collapsed.
import os
import sys
import time
def runApache(args):
sleep = 60
conf = args[0]
while True:
ready = True
for f in conf.get('required_path_list', []):
if not os.path.exists(f):
print 'File %r does not exists, sleeping for %s' % (f, sleep)
ready = False
if ready:
break
time.sleep(sleep)
apache_wrapper_list = [conf['binary'], '-f', conf['config'], '-DFOREGROUND']
apache_wrapper_list.extend(sys.argv[1:])
sys.stdout.flush()
sys.stderr.flush()
os.execl(apache_wrapper_list[0], *apache_wrapper_list)
import os
import subprocess
import time
import sys
def runMysql(args):
sleep = 60
conf = args[0]
mysqld_wrapper_list = [conf['mysqld_binary'], '--defaults-file=%s' %
conf['configuration_file']]
# we trust mysql_install that if mysql directory is available mysql was
# correctly initalised
if not os.path.isdir(os.path.join(conf['data_directory'], 'mysql')):
while True:
# XXX: Protect with proper root password
popen = subprocess.Popen([conf['mysql_install_binary'],
'--skip-name-resolve', '--no-defaults', '--datadir=%s' %
conf['data_directory']],
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
result = popen.communicate()[0]
if popen.returncode is None or popen.returncode != 0:
print "Failed to initialise server.\nThe error was: %s" % result
print "Waiting for %ss and retrying" % sleep
time.sleep(sleep)
else:
print "Mysql properly initialised"
break
else:
print "MySQL already initialised"
print "Starting %r" % mysqld_wrapper_list[0]
sys.stdout.flush()
sys.stderr.flush()
os.execl(mysqld_wrapper_list[0], *mysqld_wrapper_list)
def updateMysql(args):
conf = args[0]
sleep = 30
is_succeed = False
while True:
if not is_succeed:
mysql_upgrade_list = [conf['mysql_upgrade_binary'], '--no-defaults', '--user=root', '--socket=%s' % conf['socket']]
mysql_upgrade = subprocess.Popen(mysql_upgrade_list, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
result = mysql_upgrade.communicate()[0]
if mysql_upgrade.returncode is None:
mysql_upgrade.kill()
if mysql_upgrade.returncode != 0 and not 'is already upgraded' in result:
print "Command %r failed with result:\n%s" % (mysql_upgrade_list, result)
print 'Sleeping for %ss and retrying' % sleep
else:
if mysql_upgrade.returncode == 0:
print "MySQL database upgraded with result:\n%s" % result
else:
print "No need to upgrade MySQL database"
mysql_script = conf.get('mysql_script')
if mysql_script:
mysql_list = [conf['mysql_binary'].strip(), '--no-defaults', '-B', '--user=root', '--socket=%s' % conf['socket']]
mysql = subprocess.Popen(mysql_list, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
result = mysql.communicate(conf['mysql_script'])[0]
if mysql.returncode is None:
mysql.kill()
if mysql.returncode != 0:
print 'Command %r failed with:\n%s' % (mysql_list, result)
print 'Sleeping for %ss and retrying' % sleep
else:
is_succeed = True
print 'SlapOS initialisation script succesfully applied on database.'
sys.stdout.flush()
sys.stderr.flush()
time.sleep(sleep)
import sys
import subprocess
def executeRunner(args):
"""Start the instance configure. this may run a python script, move or/and rename
file or directory when dondition is filled. the condition may be when file exist or when an entry
exist into database.
"""
arguments, delete, rename, chmod, data = args
if delete != []:
print "Calling lampconfigure with 'delete' arguments"
result = subprocess.Popen(arguments + delete)
result.wait()
if rename != []:
for parameters in rename:
print "Calling lampconfigure with 'rename' arguments"
result = subprocess.Popen(arguments + parameters)
result.wait()
if chmod != []:
print "Calling lampconfigure with 'chmod' arguments"
result = subprocess.Popen(arguments + chmod)
result.wait()
if data != []:
print "Calling lampconfigure with 'run' arguments"
result = subprocess.Popen(arguments + data)
result.wait()
return
# Apache static configuration
# Automatically generated
# Basic server configuration
PidFile "%(pid_file)s"
LockFile "%(lock_file)s"
Listen %(ip)s:%(port)s
PHPINIDir %(php_ini_dir)s
ServerAdmin someone@email
DefaultType text/plain
TypesConfig conf/mime.types
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz
AddType application/x-httpd-php .php .phtml .php5 .php4
AddType application/x-httpd-php-source .phps
# Log configuration
ErrorLog "%(error_log)s"
LogLevel warn
LogFormat "%%h %%{REMOTE_USER}i %%l %%u %%t \"%%r\" %%>s %%b \"%%{Referer}i\" \"%%{User-Agent}i\"" combined
LogFormat "%%h %%{REMOTE_USER}i %%l %%u %%t \"%%r\" %%>s %%b" common
CustomLog "%(access_log)s" common
# Directory protection
<Directory />
Options FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
</Directory>
<Directory %(document_root)s>
Options FollowSymLinks
AllowOverride All
Order allow,deny
Allow from all
</Directory>
DocumentRoot %(document_root)s
DirectoryIndex index.html index.php
# List of modules
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule version_module modules/mod_version.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule mime_module modules/mod_mime.so
LoadModule dav_module modules/mod_dav.so
LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule headers_module modules/mod_headers.so
LoadModule dir_module modules/mod_dir.so
LoadModule php5_module modules/libphp5.so
LoadModule alias_module modules/mod_alias.so
LoadModule env_module modules/mod_env.so
LoadModule autoindex_module modules/mod_autoindex.so
# ERP5 buildout my.cnf template based on my-huge.cnf shipped with mysql
# The MySQL server
[mysqld]
# ERP5 by default requires InnoDB storage. MySQL by default fallbacks to using
# different engine, like MyISAM. Such behaviour generates problems only, when
# tables requested as InnoDB are silently created with MyISAM engine.
#
# Loud fail is really required in such case.
sql-mode="NO_ENGINE_SUBSTITUTION"
skip-show-database
port = %(tcp_port)s
bind-address = %(ip)s
socket = %(socket)s
datadir = %(data_directory)s
pid-file = %(pid_file)s
log-error = %(error_log)s
log-slow-file = %(slow_query_log)s
long_query_time = 5
max_allowed_packet = 128M
query_cache_size = 32M
plugin-load = ha_innodb_plugin.so
# The following are important to configure and depend a lot on to the size of
# your database and the available resources.
#innodb_buffer_pool_size = 4G
#innodb_log_file_size = 256M
#innodb_log_buffer_size = 8M
# Some dangerous settings you may want to uncomment if you only want
# performance or less disk access. Useful for unit tests.
#innodb_flush_log_at_trx_commit = 0
#innodb_flush_method = nosync
#innodb_doublewrite = 0
#sync_frm = 0
# Uncomment the following if you need binary logging, which is recommended
# on production instances (either for replication or incremental backups).
#log-bin=mysql-bin
# Force utf8 usage
collation_server = utf8_unicode_ci
character_set_server = utf8
skip-character-set-client-handshake
[mysql]
no-auto-rehash
socket = %(socket)s
[mysqlhotcopy]
interactive-timeout
CREATE DATABASE IF NOT EXISTS %(database)s;
GRANT ALL PRIVILEGES ON %(database)s.* TO %(user)s@localhost IDENTIFIED BY %(password)r;
GRANT ALL PRIVILEGES ON %(database)s.* TO %(user)s@'%%' IDENTIFIED BY %(password)r;
GRANT SHOW DATABASES ON *.* TO %(user)s@localhost IDENTIFIED BY %(password)r;
GRANT SHOW DATABASES ON *.* TO %(user)s@'%%' IDENTIFIED BY %(password)r;
FLUSH PRIVILEGES;
EXIT
[PHP]
engine = On
safe_mode = Off
expose_php = Off
error_reporting = E_ALL & ~(E_DEPRECATED|E_NOTICE|E_WARNING)
display_errors = On
display_startup_errors = Off
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = Off
ignore_repeated_source = Off
session.save_path = "%(tmp_directory)s"
session.auto_start = 0
date.timezone = Europe/Paris
file_uploads = On
upload_max_filesize = 16M
post_max_size = 16M
magic_quotes_gpc=Off
foreground = yes
output = %(log)s
pid = %(pid_file)s
syslog = no
[service]
client = yes
accept = %(local_host)s:%(local_port)s
connect = %(remote_host)s:%(remote_port)s
...@@ -33,9 +33,16 @@ from hashlib import md5 ...@@ -33,9 +33,16 @@ from hashlib import md5
import stat import stat
import netaddr import netaddr
import time import time
import re
import urlparse
# Use to do from slapos.recipe.librecipe import GenericBaseRecipe
from generic import GenericBaseRecipe
from genericslap import GenericSlapRecipe
class BaseSlapRecipe: class BaseSlapRecipe:
"""Base class for all slap.recipe.*""" """Base class for all slap.recipe.*"""
def __init__(self, buildout, name, options): def __init__(self, buildout, name, options):
"""Default initialisation""" """Default initialisation"""
self.name = name self.name = name
...@@ -60,6 +67,7 @@ class BaseSlapRecipe: ...@@ -60,6 +67,7 @@ class BaseSlapRecipe:
'xml_report') 'xml_report')
self.destroy_script_location = os.path.join(self, self.work_directory, self.destroy_script_location = os.path.join(self, self.work_directory,
'sbin', 'destroy') 'sbin', 'destroy')
self.promise_directory = os.path.join(self.etc_directory, 'promise')
# default directory structure information # default directory structure information
self.default_directory_list = [ self.default_directory_list = [
...@@ -71,6 +79,7 @@ class BaseSlapRecipe: ...@@ -71,6 +79,7 @@ class BaseSlapRecipe:
self.etc_directory, # CP/etc - configuration container self.etc_directory, # CP/etc - configuration container
self.wrapper_directory, # CP/etc/run - for wrappers self.wrapper_directory, # CP/etc/run - for wrappers
self.wrapper_report_directory, # CP/etc/report - for report wrappers self.wrapper_report_directory, # CP/etc/report - for report wrappers
self.promise_directory, # CP/etc/promise - for promise checking scripts
self.var_directory, # CP/var - partition "internal" container for logs, self.var_directory, # CP/var - partition "internal" container for logs,
# and another metadata # and another metadata
self.wrapper_xml_report_directory, # CP/var/xml_report - for xml_report wrappers self.wrapper_xml_report_directory, # CP/var/xml_report - for xml_report wrappers
...@@ -81,16 +90,19 @@ class BaseSlapRecipe: ...@@ -81,16 +90,19 @@ class BaseSlapRecipe:
# SLAP related information # SLAP related information
slap_connection = buildout['slap_connection'] slap_connection = buildout['slap_connection']
self.computer_id=slap_connection['computer_id'] self.computer_id = slap_connection['computer_id']
self.computer_partition_id=slap_connection['partition_id'] self.computer_partition_id = slap_connection['partition_id']
self.server_url=slap_connection['server_url'] self.server_url = slap_connection['server_url']
self.software_release_url=slap_connection['software_release_url'] self.software_release_url = slap_connection['software_release_url']
self.key_file=slap_connection.get('key_file') self.key_file = slap_connection.get('key_file')
self.cert_file=slap_connection.get('cert_file') self.cert_file = slap_connection.get('cert_file')
# setup egg to give possibility to generate scripts # setup egg to give possibility to generate scripts
self.egg = zc.recipe.egg.Egg(buildout, options['recipe'], options) self.egg = zc.recipe.egg.Egg(buildout, options['recipe'], options)
# Hook options
self._options(options)
# setup auto uninstall/install # setup auto uninstall/install
self._setupAutoInstallUninstall() self._setupAutoInstallUninstall()
...@@ -243,3 +255,55 @@ class BaseSlapRecipe: ...@@ -243,3 +255,55 @@ class BaseSlapRecipe:
def _install(self): def _install(self):
"""Hook which shall be implemented in children class""" """Hook which shall be implemented in children class"""
raise NotImplementedError('Shall be implemented by subclass') raise NotImplementedError('Shall be implemented by subclass')
def _options(self, options):
"""Hook which can be implemented in children class"""
pass
def createPromiseWrapper(self, promise_name, file_content):
"""Create a promise wrapper.
This wrapper aim to check if the software release is doing its job.
Return the promise file path.
"""
promise_path = os.path.join(self.promise_directory, promise_name)
self._writeExecutable(promise_path, file_content)
return promise_path
def setConnectionUrl(self, *args, **kwargs):
url = self._unparseUrl(*args, **kwargs)
self.setConnectionDict(dict(url=url))
def _unparseUrl(self, scheme, host, path='', params='', query='',
fragment='', port=None, auth=None):
"""Join a url with auth, host, and port.
* auth can be either a login string or a tuple (login, password).
* if the host is an ipv6 address, brackets will be added to surround it.
"""
# XXX-Antoine: I didn't find any standard module to join an url with
# login, password, ipv6 host and port.
# So instead of copy and past in every recipe I factorized it right here.
netloc = ''
if auth is not None:
auth = tuple(auth)
netloc = str(auth[0]) # Login
if len(auth) > 1:
netloc += ':%s' % auth[1] # Password
netloc += '@'
# host is an ipv6 address whithout brackets
if ':' in host and not re.match(r'^\[.*\]$', host):
netloc += '[%s]' % host
else:
netloc += str(host)
if port is not None:
netloc += ':%s' % port
url = urlparse.urlunparse((scheme, netloc, path, params, query, fragment))
return url
...@@ -23,6 +23,8 @@ def execute_wait(args): ...@@ -23,6 +23,8 @@ def execute_wait(args):
ready = False ready = False
if ready: if ready:
break break
# XXX: It's the same as ../ca/certificate_authoritiy.py
# We should use pyinotify as well. Or select() on socket.
time.sleep(sleep) time.sleep(sleep)
os.execv(exec_list[0], exec_list + sys.argv[1:]) os.execv(exec_list[0], exec_list + sys.argv[1:])
...@@ -39,6 +41,25 @@ def executee(args): ...@@ -39,6 +41,25 @@ def executee(args):
env[k] = v env[k] = v
os.execve(exec_list[0], exec_list + sys.argv[1:], env) os.execve(exec_list[0], exec_list + sys.argv[1:], env)
def executee_wait(args):
"""Portable execution with process replacement and environment manipulation"""
exec_list = list(args[0])
file_list = list(args[1])
environment = args[2]
env = os.environ.copy()
for k,v in environment.iteritems():
env[k] = v
sleep = 60
while True:
ready = True
for f in file_list:
if not os.path.exists(f):
print 'File %r does not exists, sleeping for %s' % (f, sleep)
ready = False
if ready:
break
time.sleep(sleep)
os.execve(exec_list[0], exec_list + sys.argv[1:], env)
def sig_handler(signal, frame): def sig_handler(signal, frame):
print 'Received signal %r, killing children and exiting' % signal print 'Received signal %r, killing children and exiting' % signal
......
##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import logging
import os
import sys
import inspect
import pkg_resources
import zc.buildout
class GenericBaseRecipe(object):
TRUE_VALUES = ['y', 'yes', '1', 'true']
def __init__(self, buildout, name, options):
"""Recipe initialisation"""
self.name = name
self.options = options
self.buildout = buildout
self.logger = logging.getLogger(name)
self._options(options) # Options Hook
self._ws = self.getWorkingSet()
def update(self):
"""By default update method does the same thing than install"""
return self.install()
def install(self):
"""Install method of the recipe. This must be overriden in child
classes """
raise NotImplementedError("install method is not implemented.")
def getWorkingSet(self):
"""If you want do override the default working set"""
egg = zc.recipe.egg.Egg(self.buildout, 'slapos.cookbook',
self.options.copy())
requirements, ws = egg.working_set()
return ws
def _options(self, options):
"""Options Hook method. This method can be overriden in child classes"""
return
def createFile(self, name, content, mode=0600):
"""Create a file with content
The parent directory should exists, else it would raise IOError"""
with open(name, 'w') as fileobject:
fileobject.write(content)
os.chmod(fileobject.name, mode)
return os.path.abspath(name)
def createExecutable(self, name, content, mode=0700):
return self.createFile(name, content, mode)
def createPythonScript(self, name, absolute_function, arguments=''):
"""Create a python script using zc.buildout.easy_install.scripts
* function should look like 'module.function', or only 'function'
if it is a builtin function."""
absolute_function = tuple(absolute_function.rsplit('.', 1))
if len(absolute_function) == 1:
absolute_function = ('__builtin__',) + absolute_function
if len(absolute_function) != 2:
raise ValueError("A non valid function was given")
module, function = absolute_function
path, filename = os.path.split(os.path.abspath(name))
script = zc.buildout.easy_install.scripts(
[(filename, module, function)], self._ws, sys.executable,
path, arguments=arguments)[0]
return script
def substituteTemplate(self, template_location, mapping_dict):
"""Read from file template_location an substitute content with
mapping_dict douing a dummy python format."""
with open(template_location, 'r') as template:
return template.read() % mapping_dict
def getTemplateFilename(self, template_name):
caller = inspect.stack()[1]
caller_frame = caller[0]
name = caller_frame.f_globals['__name__']
return pkg_resources.resource_filename(name,
'template/%s' % template_name)
def generatePassword(self, len_=32):
# TODO: implement a real password generator which remember the last
# call.
return "insecure"
def isTrueValue(self, value):
return str(value).lower() in GenericBaseRecipe.TRUE_VALUES
def optionIsTrue(self, optionname, default=None):
return self.isTrueValue(self.options[optionname])
##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import logging
from slapos import slap
import zc.buildout
import zc.recipe.egg
import time
import re
import urlparse
class GenericSlapRecipe(object):
"""Base class for all slap.recipe.*"""
def __init__(self, buildout, name, options):
"""Default initialisation"""
self.name = name
options['eggs'] = 'slapos.cookbook'
self.options = options
self.logger = logging.getLogger(self.name)
self.slap = slap.slap()
# SLAP related information
slap_connection = buildout['slap-connection']
self.computer_id = slap_connection['computer-id']
self.computer_partition_id = slap_connection['partition-id']
self.server_url = slap_connection['server-url']
self.software_release_url = slap_connection['software-release-url']
self.key_file = slap_connection.get('key-file')
self.cert_file = slap_connection.get('cert-file')
# setup egg to give possibility to generate scripts
self.egg = zc.recipe.egg.Egg(buildout, options['recipe'], options)
# Hook options
self._options(options)
# setup auto uninstall/install
self._setupAutoInstallUninstall()
def _setupAutoInstallUninstall(self):
"""By default SlapOS recipes are reinstalled each time"""
# Note: It is possible to create in future subclass which will do no-op in
# this method
self.options['slapos-timestamp'] = str(time.time())
def install(self):
self.slap.initializeConnection(self.server_url, self.key_file,
self.cert_file)
self.computer_partition = self.slap.registerComputerPartition(
self.computer_id,
self.computer_partition_id)
self.request = self.computer_partition.request
self.setConnectionDict = self.computer_partition.setConnectionDict
self.parameter_dict = self.computer_partition.getInstanceParameterDict()
# call children part of install
path_list = self._install()
return path_list
update = install
def _install(self):
"""Hook which shall be implemented in children class"""
raise NotImplementedError('Shall be implemented by subclass')
def _options(self, options):
"""Hook which can be implemented in children class"""
pass
def setConnectionUrl(self, *args, **kwargs):
url = self._unparseUrl(*args, **kwargs)
self.setConnectionDict(dict(url=url))
def _unparseUrl(self, scheme, host, path='', params='', query='',
fragment='', port=None, auth=None):
"""Join a url with auth, host, and port.
* auth can be either a login string or a tuple (login, password).
* if the host is an ipv6 address, brackets will be added to surround it.
"""
# XXX-Antoine: I didn't find any standard module to join an url with
# login, password, ipv6 host and port.
# So instead of copy and past in every recipe I factorized it right here.
netloc = ''
if auth is not None:
auth = tuple(auth)
netloc = str(auth[0]) # Login
if len(auth) > 1:
netloc += ':%s' % auth[1] # Password
netloc += '@'
# host is an ipv6 address whithout brackets
if ':' in host and not re.match(r'^\[.*\]$', host):
netloc += '[%s]' % host
else:
netloc += str(host)
if port is not None:
netloc += ':%s' % port
url = urlparse.urlunparse((scheme, netloc, path, params, query, fragment))
return url
##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import os
from slapos.recipe.librecipe import GenericBaseRecipe
class Recipe(GenericBaseRecipe):
def _options(self, options):
if 'name' not in options:
options['name'] = self.name
def install(self):
path_list = []
logrotate_backup = self.options['backup']
logrotate_d = self.options['logrotate-entries']
logrotate_conf_file = self.options['conf']
logrotate_conf = []
logrotate_conf.append("include %s" % logrotate_d)
logrotate_conf.append("olddir %s" % logrotate_backup)
logrotate_conf.append("dateext")
frequency = 'daily'
if 'frequency' in self.options:
frequency = self.options['frequency']
logrotate_conf.append(frequency)
num_rotate = 30
if 'num-rotate' in self.options:
num_rotate = self.options['num-rotate']
logrotate_conf.append("rotate %s" % num_rotate)
logrotate_conf.append("compress")
logrotate_conf.append("compresscmd %s" % self.options['gzip-binary'])
logrotate_conf.append("compressoptions -9")
logrotate_conf.append("uncompresscmd %s" % self.options['gunzip-binary'])
logrotate_conf_file = self.createFile(logrotate_conf_file, '\n'.join(logrotate_conf))
logrotate_conf.append(logrotate_conf_file)
state_file = self.options['state-file']
logrotate = self.createPythonScript(
self.options['wrapper'],
'slapos.recipe.librecipe.exceute.execute',
[self.options['logrotate-binary'], '-s', state_file, logrotate_conf_file, ]
)
path_list.append(logrotate)
return path_list
class Part(GenericBaseRecipe):
def _options(self, options):
if 'name' not in options:
options['name'] = self.name
def install(self):
logrotate_d = self.options['logrotate-entries']
part_path = os.path.join(logrotate_d, self.options['name'])
conf = []
if 'frequency' in self.options:
conf.append(self.options['frequency'])
if 'num-rotate' in self.options:
conf.append('rotate %s' % self.options['num-rotate'])
if 'post' in self.options:
conf.append("postrotate\n%s\nendscript" % self.options['post'])
if 'pre' in self.options:
conf.append("prerotate\n%s\nendscript" % self.options['pre'])
if self.optionIsTrue('sharedscripts', False):
conf.append("sharedscripts")
if self.optionIsTrue('notifempty', False):
conf.append('notifempty')
if self.optionIsTrue('create', True):
conf.append('create')
log = self.options['log']
self.createFile(os.path.join(logrotate_d, self.options['name']),
"%(logfiles)s {\n%(conf)s\n}" % {
'logfiles': log,
'conf': '\n'.join(conf),
}
)
return [part_path]
##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import os
from slapos.recipe.librecipe import GenericBaseRecipe
class Recipe(GenericBaseRecipe):
def _options(self, options):
self.directory = options.copy()
del self.directory['recipe']
str_mode = '0700'
if 'mode' in self.directory:
str_mode = self.directory['mode']
del self.directory['mode']
self.mode = int(str_mode, 8)
def install(self):
for directory in self.directory.values():
path = directory
if not os.path.exists(path):
os.mkdir(path, self.mode)
elif not os.path.isdir(path):
raise OSError("%s path exits, but it's not a directory.")
return []
This diff is collapsed.
import subprocess
import os
# Replace mysqldump | gzip > tmpdump && mv -f tmpdump dumpfile
def do_backup(kwargs):
mysqldump_cmd = kwargs['mysqldump']
gzip_bin = kwargs['gzip']
tmpdump = kwargs['tmpdump']
dumpfile = kwargs['dumpfile']
# mysqldump | gzip > tmpdump
with open(tmpdump, 'w') as output:
mysqldump = subprocess.Popen(mysqldump_cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
gzip = subprocess.Popen([gzip_bin],
stdin=mysqldump.stdout,
stdout=output,
stderr=subprocess.STDOUT)
mysqldump.stdout.close()
if gzip.wait() != 0:
raise ValueError("Gzip return a non zero value.")
os.rename(tmpdump, dumpfile)
import os
import sys
import time
def catdatefile(args):
directory = args[0]
try:
suffix = args[1]
except IndexError:
suffix = '.log'
f = open(os.path.join(directory,
time.strftime('%Y-%m-%d.%H:%M.%s') + suffix), 'aw')
for line in sys.stdin.read():
f.write(line)
f.close()
...@@ -4,9 +4,8 @@ import time ...@@ -4,9 +4,8 @@ import time
import sys import sys
def runMysql(args): def runMysql(conf):
sleep = 60 sleep = 60
conf = args[0]
mysqld_wrapper_list = [conf['mysqld_binary'], '--defaults-file=%s' % mysqld_wrapper_list = [conf['mysqld_binary'], '--defaults-file=%s' %
conf['configuration_file']] conf['configuration_file']]
# we trust mysql_install that if mysql directory is available mysql was # we trust mysql_install that if mysql directory is available mysql was
...@@ -16,8 +15,8 @@ def runMysql(args): ...@@ -16,8 +15,8 @@ def runMysql(args):
# XXX: Protect with proper root password # XXX: Protect with proper root password
# XXX: Follow http://dev.mysql.com/doc/refman/5.0/en/default-privileges.html # XXX: Follow http://dev.mysql.com/doc/refman/5.0/en/default-privileges.html
popen = subprocess.Popen([conf['mysql_install_binary'], popen = subprocess.Popen([conf['mysql_install_binary'],
'--skip-name-resolve', '--no-defaults', '--datadir=%s' % '--skip-name-resolve', '--skip-host-cache', '--no-defaults',
conf['data_directory']], '--datadir=%s' % conf['data_directory']],
stdout=subprocess.PIPE, stderr=subprocess.STDOUT) stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
result = popen.communicate()[0] result = popen.communicate()[0]
if popen.returncode is None or popen.returncode != 0: if popen.returncode is None or popen.returncode != 0:
...@@ -35,8 +34,7 @@ def runMysql(args): ...@@ -35,8 +34,7 @@ def runMysql(args):
os.execl(mysqld_wrapper_list[0], *mysqld_wrapper_list) os.execl(mysqld_wrapper_list[0], *mysqld_wrapper_list)
def updateMysql(args): def updateMysql(conf):
conf = args[0]
sleep = 30 sleep = 30
is_succeed = False is_succeed = False
while True: while True:
......
import sys
import os
import time
import subprocess
def import_remote_dump(kwargs):
# Get data from kwargs
lock_file = kwargs['lock_file']
database = kwargs['database']
mysql_binary = kwargs['mysql_binary']
mysql_socket = kwargs['mysql_socket']
duplicity_binary = kwargs['duplicity_binary']
remote_backup = kwargs['remote_backup']
local_directory = kwargs['local_directory']
dump_name = kwargs['dump_name']
zcat_binary = kwargs['zcat_binary']
# The script start really here
if os.path.exists(lock_file):
sys.exit(127)
while subprocess.call([mysql_binary, '--socket=%s' % mysql_socket,
'-u', 'root', '-e', 'use %s;' % database]) != 0:
time.sleep(10)
subprocess.check_call([duplicity_binary, 'restore', '--no-encryption',
remote_backup, local_directory])
zcat = subprocess.Popen([zcat_binary, os.path.join(local_directory,
dump_name)],
stdout=subprocess.PIPE)
mysql = subprocess.Popen([mysql_binary, '--socket=%s' % mysql_socket,
'-D', database, '-u', 'root'],
stdin=zcat.stdout)
zcat.stdout.close()
returncode = mysql.poll()
if returncode == 0:
open(lock_file, 'w').close() # Just a touch
sys.exit(returncode)
##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import zc.buildout
from slapos.recipe.librecipe import GenericSlapRecipe
class Recipe(GenericSlapRecipe):
def _options(self, options):
self.useparts = True
if 'url' in options:
self.useparts = False
self.url = options['url']
else:
self.urlparts = {}
if 'scheme' not in options:
raise zc.buildout.UserError("No scheme specified.")
else:
self.urlparts.update(scheme=options['scheme'])
if 'host' not in options:
raise zc.buildout.UserError("No host specified.")
else:
self.urlparts.update(host=options['host'])
def _install(self):
if self.useparts:
for option in ['path', 'params', 'query', 'fragment', 'port']:
if option in self.options:
self.urlparts[option] = self.options[option]
if 'user' in self.options:
self.urlparts.update(auth=(self.options['user'],))
if 'password' in self.options:
self.urlparts.update(auth=(self.options['user'],
self.options['password']))
self.setConnectionUrl(**self.urlparts)
else:
self.setConnectionDict(dict(url=self.url))
return []
##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import logging
import os
from slapos import slap as slapmodule
class Recipe(object):
def parseMultiValues(self, string):
return dict([ [str(column).strip() for column in line.split('=', 1)]
for line in str(string).splitlines() if '=' in line])
def __init__(self, buildout, name, options):
self.logger = logging.getLogger(name)
slap = slapmodule.slap()
slap_connection = buildout['slap_connection']
self.software_release_url = slap_connection['software_release_url']
# XXX: Dirty network interation stuff
slap.initializeConnection(slap_connection['server_url'],
slap_connection.get('key_file'),
slap_connection.get('cert_file'),
)
computer_partition = slap.registerComputerPartition(
slap_connection['computer_id'], slap_connection['partition_id'])
self.request = computer_partition.request
if 'software-url' not in options:
options['software-url'] = self.software_release_url
if 'name' not in options:
options['name'] = name
self.return_parameters = []
if 'return' in options:
self.return_parameters = [str(parameter).strip()
for parameter in options['return'].splitlines()]
else:
self.logger.warning("No parameter to return to main instance."
"Be careful about that...")
software_type = 'RootInstanceSoftware'
if 'software-type' in options:
software_type = options['software-type']
filter_kw = {}
if 'sla' in options:
filter_kw = self.parseMultiValues(options['sla'])
partition_parameter_kw = {}
if 'config' in options:
partition_parameter_kw = self.parseMultiValues(options['config'])
instance = self.request(options['software-url'], software_type,
options['name'], partition_parameter_kw=partition_parameter_kw,
filter_kw=filter_kw)
result = {}
for param in self.return_parameters:
result[param] = instance.getConnectionParameter(param)
# Return the connections parameters in options dict
for key, value in result.items():
options['connection-%s' % key] = value
def install(self):
return []
update = install
##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import shutil
import os
import sys
import time
from slapos.recipe.librecipe import GenericBaseRecipe
def log(args):
directory, suffix = args
filename = time.strftime('%Y-%m-%d.%H:%M.%s') + suffix
with open(os.path.join(directory, filename), 'aw') as logfile:
shutil.copyfileobj(sys.stdin, logfile)
class Recipe(GenericBaseRecipe):
def install(self):
self.logger.info("Simple logger installation")
binary = self.options['binary']
output = self.options['output']
suffix = self.options.get('suffix', '.log')
script = self.createPythonScript(binary,
'slapos.recipe.simplelogger.log',
arguments=[output, suffix])
self.logger.debug("Logger script created at : %r", script)
self.logger.info("Simple logger installed.")
return [script]
##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import os
import sys
import copy
from ConfigParser import ConfigParser
import subprocess
import slapos.slap
import netaddr
import logging
import zc.buildout
class Recipe:
def __init__(self, buildout, name, options):
self.buildout = buildout
self.options = options
self.name = name
self.logger = logging.getLogger(self.name)
def _getIpAddress(self, test_method):
"""Internal helper method to fetch ip address"""
if not 'ip_list' in self.parameter_dict:
raise AttributeError
for name, ip in self.parameter_dict['ip_list']:
if test_method(ip):
return ip
raise AttributeError
def getLocalIPv4Address(self):
"""Returns local IPv4 address available on partition"""
# XXX: Lack checking for locality of address
return self._getIpAddress(netaddr.valid_ipv4)
def getGlobalIPv6Address(self):
"""Returns global IPv6 address available on partition"""
# XXX: Lack checking for globality of address
return self._getIpAddress(netaddr.valid_ipv6)
def install(self):
slap = slapos.slap.slap()
slap_connection = self.buildout['slap_connection']
computer_id = slap_connection['computer_id']
computer_partition_id = slap_connection['partition_id']
server_url = slap_connection['server_url']
key_file = slap_connection.get('key_file')
cert_file = slap_connection.get('cert_file')
slap.initializeConnection(server_url, key_file, cert_file)
self.computer_partition = slap.registerComputerPartition(
computer_id,
computer_partition_id)
self.parameter_dict = self.computer_partition.getInstanceParameterDict()
software_type = self.parameter_dict['slap_software_type']
if software_type not in self.options:
if 'default' in self.options:
software_type = 'default'
else:
raise zc.buildout.UserError("This software type isn't mapped. And"
"there's no default software type.")
instance_file_path = self.options[software_type]
if not os.path.exists(instance_file_path):
raise zc.buildout.UserError("The specified buildout config file does not"
"exist.")
buildout = ConfigParser()
with open(instance_file_path) as instance_path:
buildout.readfp(instance_path)
buildout.set('buildout', 'installed',
'.installed-%s.cfg' % software_type)
buildout.add_section('slap-parameter')
for parameter, value in self.parameter_dict.items():
buildout.set('slap-parameter', parameter, value)
buildout.add_section('slap-network-information')
buildout.set('slap-network-information', 'local-ipv4',
self.getLocalIPv4Address())
buildout.set('slap-network-information', 'global-ipv6',
self.getGlobalIPv6Address())
# Copy/paste slap_connection
buildout.add_section('slap-connection')
for key, value in self.buildout['slap_connection'].iteritems():
# XXX: Waiting for SlapBaseRecipe to use dash instead of underscores
buildout.set('slap-connection', key.replace('_', '-'), value)
work_directory = os.path.abspath(self.buildout['buildout'][
'directory'])
buildout_filename = os.path.join(work_directory,
'buildout-%s.cfg' % software_type)
with open(buildout_filename, 'w') as buildout_file:
buildout.write(buildout_file)
# XXX-Antoine: We gotta find a better way to do this. I tried to check
# out how slapgrid-cp was running buildout. But it is worse than that.
command_line_args = copy.copy(sys.argv) + ['-c', buildout_filename]
self.logger.info("Invoking commandline : '%s'",
' '.join(command_line_args))
subprocess.check_call(command_line_args, cwd=work_directory,
env=os.environ.copy())
return []
update = install
##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import itertools
import zc.buildout
from slapos.recipe.librecipe import GenericBaseRecipe
class Recipe(GenericBaseRecipe):
def _options(self, options):
self.types = ['local', 'remote']
self.datas = ['address', 'port']
for type_ in self.types:
for data in self.datas:
opt = '%s-%s' % (type_, data)
if opt not in options:
raise zc.buildout.UserError("No %s for %s connections." % (data, type_))
self.isClient = self.optionIsTrue('client', default=False)
if self.isClient:
self.logger.info("Client mode")
else:
self.logger.info("Server mode")
if 'name' not in options:
options['name'] = self.name
def install(self):
path_list = []
conf = {}
gathered_options = ['%s-%s' % option
for option in itertools.product(self.types,
self.datas)]
for option in gathered_options:
# XXX: Because the options are using dash and the template uses
# underscore
conf[option.replace('-', '_')] = self.options[option]
pid_file = self.options['pid-file']
conf.update(pid_file=pid_file)
path_list.append(pid_file)
log_file = self.options['log-file']
conf.update(log=log_file)
if self.isClient:
template = self.getTemplateFilename('client.conf.in')
else:
template = self.getTemplateFilename('server.conf.in')
key = self.options['key-file']
cert = self.options['cert-file']
conf.update(key=key, cert=cert)
conf_file = self.createFile(
self.options['config-file'],
self.substituteTemplate(template, conf))
path_list.append(conf_file)
wrapper = self.createPythonScript(
self.options['wrapper'],
'slapos.recipe.librecipe.execute.execute',
[self.options['stunnel-binary'], conf_file]
)
path_list.append(wrapper)
return path_list
foreground = yes
output = %(log)s
pid = %(pid_file)s
syslog = no
[service]
client = yes
accept = %(local_host)s:%(local_port)s
connect = %(remote_host)s:%(remote_port)s
foreground = yes
output = %(log)s
pid = %(pid_file)s
syslog = no
key = %(key)s
cert = %(cert)s
[service]
accept = %(remote_address)s:%(remote_port)s
connect = %(local_address)s:%(local_port)s
...@@ -34,58 +34,6 @@ class Recipe(slapos.recipe.erp5.Recipe): ...@@ -34,58 +34,6 @@ class Recipe(slapos.recipe.erp5.Recipe):
default_bt5_list = [] default_bt5_list = []
def installKeyAuthorisationApache(self, ip, port, backend, key, certificate,
ca_conf, key_auth_path='/erp5/portal_slap'):
ssl_template = """SSLEngine on
SSLVerifyClient require
RequestHeader set REMOTE_USER %%{SSL_CLIENT_S_DN_CN}s
SSLCertificateFile %(key_auth_certificate)s
SSLCertificateKeyFile %(key_auth_key)s
SSLCACertificateFile %(ca_certificate)s
SSLCARevocationPath %(ca_crl)s"""
apache_conf = self._getApacheConfigurationDict('key_auth_apache', ip, port)
apache_conf['ssl_snippet'] = ssl_template % dict(
key_auth_certificate=certificate,
key_auth_key=key,
ca_certificate=ca_conf['ca_certificate'],
ca_crl=ca_conf['ca_crl']
)
prefix = 'ssl_key_auth_apache'
rewrite_rule_template = \
"RewriteRule (.*) http://%(backend)s%(key_auth_path)s$1 [L,P]"
path_template = pkg_resources.resource_string('slapos.recipe.erp5',
'template/apache.zope.conf.path.in')
path = path_template % dict(path='/')
d = dict(
path=path,
backend=backend,
backend_path='/',
port=apache_conf['port'],
vhname=path.replace('/', ''),
key_auth_path=key_auth_path,
)
rewrite_rule = rewrite_rule_template % d
apache_conf.update(**dict(
path_enable=path,
rewrite_rule=rewrite_rule
))
apache_config_file = self.createConfigurationFile(prefix + '.conf',
pkg_resources.resource_string('slapos.recipe.erp5',
'template/apache.zope.conf.in') % apache_conf)
self.path_list.append(apache_config_file)
self.path_list.extend(zc.buildout.easy_install.scripts([(
'key_auth_apache',
'slapos.recipe.erp5.apache', 'runApache')], self.ws,
sys.executable, self.wrapper_directory, arguments=[
dict(
required_path_list=[certificate, key, ca_conf['ca_certificate'],
ca_conf['ca_crl']],
binary=self.options['httpd_binary'],
config=apache_config_file
)
]))
return 'https://%(ip)s:%(port)s' % apache_conf
def _getZeoClusterDict(self): def _getZeoClusterDict(self):
site_path = '/erp5/' site_path = '/erp5/'
return { return {
...@@ -119,8 +67,8 @@ SSLCARevocationPath %(ca_crl)s""" ...@@ -119,8 +67,8 @@ SSLCARevocationPath %(ca_crl)s"""
self.getTemplateFilename('zope-zeo-snippet.conf.in'), dict( self.getTemplateFilename('zope-zeo-snippet.conf.in'), dict(
storage_name=storage_dict['storage_name'], storage_name=storage_dict['storage_name'],
address='%s:%s' % (storage_dict['ip'], storage_dict['port']), address='%s:%s' % (storage_dict['ip'], storage_dict['port']),
mount_point=mount_point mount_point=mount_point, zodb_cache_size=self.zodb_cache_size,
))) zeo_client_cache_size=self.zeo_client_cache_size)))
tidstorage_config = dict(host=self.getLocalIPv4Address(), port='6001') tidstorage_config = dict(host=self.getLocalIPv4Address(), port='6001')
zodb_configuration_string = '\n'.join(zodb_configuration_list) zodb_configuration_string = '\n'.join(zodb_configuration_list)
zope_port = 12000 zope_port = 12000
...@@ -150,9 +98,15 @@ SSLCARevocationPath %(ca_crl)s""" ...@@ -150,9 +98,15 @@ SSLCARevocationPath %(ca_crl)s"""
login_url_list) login_url_list)
apache_login = self.installBackendApache(self.getGlobalIPv6Address(), 15000, apache_login = self.installBackendApache(self.getGlobalIPv6Address(), 15000,
login_haproxy, backend_key, backend_certificate) login_haproxy, backend_key, backend_certificate)
# Install Frontend
frontend_domain_name = self.parameter_dict.get("domain_name", 'vifib')
frontend_key, frontend_certificate = \
self.requestCertificate(frontend_domain_name)
apache_frontend_login = self.installFrontendZopeApache( apache_frontend_login = self.installFrontendZopeApache(
self.getGlobalIPv6Address(), 4443, 'vifib', '/', self.getGlobalIPv6Address(), 4443, frontend_domain_name, '/',
apache_login, '/', backend_key, backend_certificate) apache_login, '', frontend_key, frontend_certificate)
# Four Web Service Nodes (Machine access) # Four Web Service Nodes (Machine access)
service_url_list = [] service_url_list = []
for i in (1, 2, 3, 4): for i in (1, 2, 3, 4):
...@@ -166,9 +120,9 @@ SSLCARevocationPath %(ca_crl)s""" ...@@ -166,9 +120,9 @@ SSLCARevocationPath %(ca_crl)s"""
key_auth_key, key_auth_certificate = self.requestCertificate( key_auth_key, key_auth_certificate = self.requestCertificate(
'Key Based Access') 'Key Based Access')
apache_keyauth = self.installKeyAuthorisationApache( apache_keyauth = self.installKeyAuthorisationApache(False, 15500,
self.getLocalIPv4Address(), 15500, service_haproxy, key_auth_key, service_haproxy, key_auth_key, key_auth_certificate, ca_conf,
key_auth_certificate, ca_conf, key_auth_path=self.key_auth_path) key_auth_path=self.key_auth_path)
memcached_conf = self.installMemcached(ip=self.getLocalIPv4Address(), memcached_conf = self.installMemcached(ip=self.getLocalIPv4Address(),
port=11000) port=11000)
kumo_conf = self.installKumo(self.getLocalIPv4Address()) kumo_conf = self.installKumo(self.getLocalIPv4Address())
...@@ -179,7 +133,7 @@ SSLCARevocationPath %(ca_crl)s""" ...@@ -179,7 +133,7 @@ SSLCARevocationPath %(ca_crl)s"""
# Connect direct to Zope to create the instance. # Connect direct to Zope to create the instance.
self.installERP5Site(user, password, service_url_list[-1], mysql_conf, self.installERP5Site(user, password, service_url_list[-1], mysql_conf,
conversion_server_conf, memcached_conf, kumo_conf, conversion_server_conf, memcached_conf, kumo_conf,
self.site_id, self.default_bt5_list) self.site_id, self.default_bt5_list, ca_conf)
self.setConnectionDict(dict( self.setConnectionDict(dict(
front_end_url=apache_frontend_login, front_end_url=apache_frontend_login,
...@@ -191,12 +145,6 @@ SSLCARevocationPath %(ca_crl)s""" ...@@ -191,12 +145,6 @@ SSLCARevocationPath %(ca_crl)s"""
kumo_url=kumo_conf['kumo_address'], kumo_url=kumo_conf['kumo_address'],
conversion_server_url='%(conversion_server_ip)s:%(conversion_server_port)s' % conversion_server_url='%(conversion_server_ip)s:%(conversion_server_port)s' %
conversion_server_conf, conversion_server_conf,
# openssl binary might be removed, as soon as CP environment will be
# fully controlled
openssl_binary=self.options['openssl_binary'],
# As soon as there would be Vifib ERP5 configuration and possibility to
# call it over the network this can be removed
certificate_authority_path=ca_conf['certificate_authority_path'],
# as installERP5Site is not trusted (yet) and this recipe is production # as installERP5Site is not trusted (yet) and this recipe is production
# ready expose more information # ready expose more information
mysql_url='%(mysql_database)s@%(ip)s:%(tcp_port)s %(mysql_user)s %(mysql_password)s' % mysql_conf, mysql_url='%(mysql_database)s@%(ip)s:%(tcp_port)s %(mysql_user)s %(mysql_password)s' % mysql_conf,
...@@ -213,13 +161,14 @@ SSLCARevocationPath %(ca_crl)s""" ...@@ -213,13 +161,14 @@ SSLCARevocationPath %(ca_crl)s"""
user, password = self.installERP5() user, password = self.installERP5()
zodb_dir = os.path.join(self.data_root_directory, 'zodb') zodb_dir = os.path.join(self.data_root_directory, 'zodb')
self._createDirectory(zodb_dir) self._createDirectory(zodb_dir)
zodb_root_path = os.path.join(zodb_dir, 'root.fs') zodb_root_path = os.path.join(zodb_dir, 'main.fs')
ip = self.getLocalIPv4Address() ip = self.getLocalIPv4Address()
zope_port = '18080' zope_port = '18080'
zope_access = self.installZope(ip, zope_port, 'zope_development', zope_access = self.installZope(ip, zope_port, 'zope_development',
zodb_configuration_string=self.substituteTemplate( zodb_configuration_string=self.substituteTemplate(
self.getTemplateFilename('zope-zodb-snippet.conf.in'), self.getTemplateFilename('zope-zodb-snippet.conf.in'),
dict(zodb_root_path=zodb_root_path)), dict(zodb_root_path=zodb_root_path,
zodb_cache_size=self.zodb_cache_size)),
thread_amount=8, with_timerservice=True) thread_amount=8, with_timerservice=True)
service_haproxy = self.installHaproxy(ip, 15000, 'service', service_haproxy = self.installHaproxy(ip, 15000, 'service',
self.site_check_path, [zope_access]) self.site_check_path, [zope_access])
...@@ -238,7 +187,7 @@ SSLCARevocationPath %(ca_crl)s""" ...@@ -238,7 +187,7 @@ SSLCARevocationPath %(ca_crl)s"""
self.linkBinary() self.linkBinary()
self.installERP5Site(user, password, zope_access, mysql_conf, self.installERP5Site(user, password, zope_access, mysql_conf,
conversion_server_conf, memcached_conf, kumo_conf, conversion_server_conf, memcached_conf, kumo_conf,
self.site_id, self.default_bt5_list) self.site_id, self.default_bt5_list, ca_conf)
self.setConnectionDict(dict( self.setConnectionDict(dict(
development_zope='http://%s:%s/' % (ip, zope_port), development_zope='http://%s:%s/' % (ip, zope_port),
...@@ -249,12 +198,6 @@ SSLCARevocationPath %(ca_crl)s""" ...@@ -249,12 +198,6 @@ SSLCARevocationPath %(ca_crl)s"""
kumo_url=kumo_conf['kumo_address'], kumo_url=kumo_conf['kumo_address'],
conversion_server_url='%(conversion_server_ip)s:%(conversion_server_port)s' % conversion_server_url='%(conversion_server_ip)s:%(conversion_server_port)s' %
conversion_server_conf, conversion_server_conf,
# openssl binary might be removed, as soon as CP environment will be
# fully controlled
openssl_binary=self.options['openssl_binary'],
# As soon as there would be Vifib ERP5 configuration and possibility to
# call it over the network this can be removed
certificate_authority_path=ca_conf['certificate_authority_path'],
# as installERP5Site is not trusted (yet) and this recipe is production # as installERP5Site is not trusted (yet) and this recipe is production
# ready expose more information # ready expose more information
mysql_url='%(mysql_database)s@%(ip)s:%(tcp_port)s %(mysql_user)s %(mysql_password)s' % mysql_conf, mysql_url='%(mysql_database)s@%(ip)s:%(tcp_port)s %(mysql_user)s %(mysql_password)s' % mysql_conf,
...@@ -267,6 +210,9 @@ SSLCARevocationPath %(ca_crl)s""" ...@@ -267,6 +210,9 @@ SSLCARevocationPath %(ca_crl)s"""
self.path_list = [] self.path_list = []
self.requirements, self.ws = self.egg.working_set() self.requirements, self.ws = self.egg.working_set()
# self.cron_d is a directory, where cron jobs can be registered # self.cron_d is a directory, where cron jobs can be registered
self.zodb_cache_size = int(self.options.get('zodb_cache_size', 5000))
self.zeo_client_cache_size = self.options.get('zeo_client_cache_size',
'20MB')
self.cron_d = self.installCrond() self.cron_d = self.installCrond()
self.logrotate_d, self.logrotate_backup = self.installLogrotate() self.logrotate_d, self.logrotate_backup = self.installLogrotate()
self.killpidfromfile = zc.buildout.easy_install.scripts( self.killpidfromfile = zc.buildout.easy_install.scripts(
...@@ -276,8 +222,6 @@ SSLCARevocationPath %(ca_crl)s""" ...@@ -276,8 +222,6 @@ SSLCARevocationPath %(ca_crl)s"""
if self.parameter_dict.get("flavour", "default") == 'configurator': if self.parameter_dict.get("flavour", "default") == 'configurator':
self.default_bt5_list = self.options.get("configurator_bt5_list", '').split() self.default_bt5_list = self.options.get("configurator_bt5_list", '').split()
if self.parameter_dict.get('development', 'false').lower() == 'true':
return self.installDevelopment()
if self.parameter_dict.get('production', 'false').lower() == 'true': if self.parameter_dict.get('production', 'false').lower() == 'true':
return self.installProduction() return self.installProduction()
raise NotImplementedError('Flavour of instance have to be given.') return self.installDevelopment()
...@@ -35,11 +35,12 @@ import zc.buildout ...@@ -35,11 +35,12 @@ import zc.buildout
class Recipe(BaseSlapRecipe): class Recipe(BaseSlapRecipe):
def _install(self): def _install(self):
self.requirements, self.ws = self.egg.working_set()
parameter_dict = self.computer_partition.getInstanceParameterDict() parameter_dict = self.computer_partition.getInstanceParameterDict()
ipv4 = self.getLocalIPv4Address(parameter_dict) ipv4 = self.getLocalIPv4Address()
ipv6 = self.getGlobalIPv6Address(parameter_dict) ipv6 = self.getGlobalIPv6Address()
self.install_mysql_server_configuration(self.getLocalIPv4Address(parameter_dict)) self.install_mysql_server_configuration(ipv4)
port = '8900' port = '8900'
tomcat_home = os.path.join(self.data_root_directory, 'tomcat') tomcat_home = os.path.join(self.data_root_directory, 'tomcat')
...@@ -56,8 +57,8 @@ class Recipe(BaseSlapRecipe): ...@@ -56,8 +57,8 @@ class Recipe(BaseSlapRecipe):
shtuil.rmtree(dst) shtuil.rmtree(dst)
raise raise
shutil.copy(self.options['hsql_location'].strip(), os.path.join(tomcat_lib, shutil.copy(self.options['jdbc_location'].strip(), os.path.join(tomcat_lib,
'hsqldb.jar')) 'jdbc.jar'))
# headless mode # headless mode
self._writeFile(os.path.join(tomcat_home, 'bin', 'setenv.sh'), '''#!/bin/sh self._writeFile(os.path.join(tomcat_home, 'bin', 'setenv.sh'), '''#!/bin/sh
export JAVA_OPTS="${JAVA_OPTS} -Djava.awt.headless=true" export JAVA_OPTS="${JAVA_OPTS} -Djava.awt.headless=true"
......
...@@ -49,7 +49,7 @@ ...@@ -49,7 +49,7 @@
Uncomment if you want to use MySQL and comment out other database configurations. Uncomment if you want to use MySQL and comment out other database configurations.
We need to set the sql_mode to a less strict value, see XWIKI-1945 We need to set the sql_mode to a less strict value, see XWIKI-1945
--> -->
<property name="connection.url">jdbc:mysql://%(mysql_ip)s:%(mysql_port)s/xwiki?useServerPrepStmts=false&amp;useUnicode=true&amp;characterEncoding=UTF-8&amp;sessionVariables=sql_mode=''</property> <property name="connection.url">jdbc:mysql://%(mysql_ip)s:%(mysql_port)s/xwiki?useServerPrepStmts=false&amp;useUnicode=true&amp;characterEncoding=UTF-8&amp;sessionVariables=&amp;sql_mode=''</property>
<property name="connection.username">xwiki</property> <property name="connection.username">xwiki</property>
<property name="connection.password">xwiki</property> <property name="connection.password">xwiki</property>
<property name="connection.driver_class">com.mysql.jdbc.Driver</property> <property name="connection.driver_class">com.mysql.jdbc.Driver</property>
......
...@@ -30,9 +30,6 @@ query_cache_size = 32M ...@@ -30,9 +30,6 @@ query_cache_size = 32M
# Try number of CPU's*2 for thread_concurrency # Try number of CPU's*2 for thread_concurrency
thread_concurrency = 8 thread_concurrency = 8
# Disable Federated by default
skip-federated
# Replication Master Server (default) # Replication Master Server (default)
# binary logging is required for replication # binary logging is required for replication
log-bin=mysql-bin log-bin=mysql-bin
......
This diff is collapsed.
...@@ -229,3 +229,5 @@ ListenIP=%(ip)s ...@@ -229,3 +229,5 @@ ListenIP=%(ip)s
# Mandatory: no # Mandatory: no
# Default: # Default:
# UserParameter= # UserParameter=
%(user_parameter_string)s
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment