Commit 836309f4 by Saurabh Committed by Julien Muchembled

Published parameters as simple storage for generated passwords and NEO cluster name

For performance reasons, the root partition requests subpartitions during
initialization of sections, whereas such processing should normally be done
during the update/install phase.

The consequence is that partitions may be requested whereas they depend on
sections that fail (usually just temporarily, because of missing returned
parameters in the first runs).

For example, the request of zope partitions depends on the generation of
passwords:

1. password generated (__init__)
2. zope partitions requested (__init__)
3. password saved (install)

As long as a failure happens between 2 and 3, zope parameters are always
updated with a different password.

In the case of NEO, the instanciation of zope partitions currently succeeds even
if the list of master nodes is missing (note that there is a minor bug to fix
here: whenever a NEO storage is not the main one, zope processes may start too
early, and the user may have to restart zopes manually). The 'inituser_done'
file is created but zope processes fail to start if NEO is used as main storage,
and all this happens before the password was saved in the root partition
([neo-0-final] failing to install because 'admins' parameter returned yet).

This was never an issue with ZEO because zopes start successfully at the same
time the 'inituser_done' file is created.

One way to solve this could have been to introduce a dummy dependency between
[neo-0-final] and any other section generating a password. Quite ugly and we
also found non-optimal to use a non-backuped file in the root partition to save
such information, whereas we need anyway to publish them for the user.

Therefore, we introduce a new 'publish-early' recipe for accessing and
publishing desired parameters before any request of partitions. Of course,
these must not be dropped by the usual [publish] section, and to avoid having
to repeating them all manually, we have also added a '-extends' option to the
'publish' recipe.

We use the same technique to autogenerate and configure cluster name for NEO,
which helps us in minimizing the number of params one has to pass for
requesting NEO.

In the 'generate.password' recipe, the 'storage-path' can now be empty, when
there's no need to save the generated password in a file.
1 parent c475889a
......@@ -167,6 +167,7 @@ setup(name=name,
'proactive = slapos.recipe.proactive:Recipe',
'publish = slapos.recipe.publish:Recipe',
'publish.serialised = slapos.recipe.publish:Serialised',
'publish-early = slapos.recipe.publish_early:Recipe',
'publishsection = slapos.recipe.publish:PublishSection',
'publishurl = slapos.recipe.publishurl:Recipe',
'readline = slapos.recipe.readline:Recipe',
......
......@@ -48,6 +48,9 @@ class Recipe(object):
- storage-path: plain-text persistent storage for password,
that can only be accessed by the user
(default: ${buildout:parts-directory}/${:_buildout_section_name_})
If storage-path is empty, the recipe does not save the password, which is
fine it is saved by other means, e.g. using the publish-early recipe.
"""
def __init__(self, buildout, name, options):
......@@ -57,16 +60,17 @@ class Recipe(object):
except KeyError:
self.storage_path = options['storage-path'] = os.path.join(
buildout['buildout']['parts-directory'], name)
try:
with open(self.storage_path) as f:
passwd = f.read()
except IOError as e:
if e.errno != errno.ENOENT:
raise
passwd = None
passwd = None
if self.storage_path:
try:
with open(self.storage_path) as f:
passwd = f.read()
except IOError as e:
if e.errno != errno.ENOENT:
raise
self.update = self.install
if not passwd:
passwd = self.generatePassword(int(options_get('bytes', '8')))
self.update = self.install
self.passwd = passwd
# Password must not go into .installed file, for 2 reasons:
# security of course but also to prevent buildout to always reinstall.
......@@ -88,7 +92,7 @@ class Recipe(object):
os.write(fd, self.passwd)
finally:
os.close(fd)
return self.storage_path
return self.storage_path
def update(self):
return ()
......@@ -32,15 +32,19 @@ CONNECTION_PARAMETER_STRING = 'connection-'
class Recipe(GenericSlapRecipe):
def _install(self):
publish_dict = dict()
options = self.options.copy()
del options['recipe']
slave_reference = options.pop('-slave-reference', None)
for k, v in options.iteritems():
if k[:1] == '-':
continue
publish_dict[k] = v
self._setConnectionDict(publish_dict, slave_reference)
publish_dict = {}
done = set()
extends = [self.name]
while extends:
name = extends.pop()
done.add(name)
for k, v in self.buildout[name].iteritems():
if k[:1] == '-':
if k == '-extends':
extends += set(v.split()) - done
elif k != 'recipe':
publish_dict[k] = v
self._setConnectionDict(publish_dict, self.options.get('-slave-reference'))
return []
def _setConnectionDict(self, publish_dict, slave_reference=None):
......
##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import slapos.slap
from slapos.recipe.librecipe import unwrap, wrap
from slapos.recipe.librecipe import GenericSlapRecipe
class Recipe(GenericSlapRecipe):
"""
Early initialization of published parameters.
The '-init' option defines parameters that should be published before
requesting any partitions, and how they are initialized.
Example:
[publish-early]
recipe = slapos.cookbook:publish-early
-init =
foo gen-foo:x
bar gen-bar:y
bar = z
[gen-foo]
...
[publish]
recipe = slapos.cookbook:publish.serialised
-extends = publish-early
...
${publish-early:foo} is initialized with the value of the published
parameter 'foo', or ${gen-foo:x} if it hasn't been published yet
(and in this case, it is published immediately as a way to save the value).
${publish-early:bar} is forced to 'z' (${gen-bar:y} ignored):
a line like 'bar = z' is usually rendered conditionally with Jinja2.
"""
def __init__(self, buildout, name, options):
GenericSlapRecipe.__init__(self, buildout, name, options)
published_dict = None
publish = False
for line in options['-init'].splitlines():
if line:
k, v = line.split()
if k not in options:
if published_dict is None:
self.slap.initializeConnection(self.server_url, self.key_file,
self.cert_file)
computer_partition = self.slap.registerComputerPartition(
self.computer_id, self.computer_partition_id)
published_dict = unwrap(
computer_partition.getConnectionParameterDict())
publish_dict = {}
try:
publish_dict[k] = published_dict[k]
except KeyError:
section, key = v.split(":")
publish_dict[k] = self.buildout[section][key]
publish = True
if publish:
computer_partition.setConnectionDict(wrap(publish_dict))
options.update(publish_dict)
install = update = lambda self: None
......@@ -38,7 +38,7 @@ parts =
{% set section_id_list = [] -%}
[{{ prefix }}request-common]
<= request-common-base
config-cluster = {{ dumps(parameter_dict['cluster']) }}
config-cluster = {{ parameter_dict['cluster'] }}
{% set replicas = parameter_dict.get('replicas', 0) -%}
config-partitions = {{ dumps(parameter_dict.get('partitions', 12)) }}
config-replicas = {{ dumps(replicas) }}
......
......@@ -74,7 +74,7 @@ context =
[root-common]
<= download-base-neo
md5sum = f3182000925308bc694d1dab46d8142b
md5sum = 26193dbb132d340c8ba919a616449a17
[instance-neo-admin]
<= download-base-neo
......
......@@ -278,8 +278,7 @@ context =
[template-erp5]
<= download-base
filename = instance-erp5.cfg.in
md5sum = df944bcd8d95d4ce667de91c54ff455b
md5sum = 70820c1608edfc923d57ec2d92aed012
[template-zeo]
<= download-base
......@@ -289,7 +288,7 @@ md5sum = 9670cf63099e2c520017a23defff51a4
[template-zope]
<= download-base
filename = instance-zope.cfg.in
md5sum = 38551c47a3ee3f4e40d2176301e99482
md5sum = 35f05cb0f8c8dc9c52bc02984bde49f5
link-binary =
${aspell:location}/bin/aspell
${dmtx-utils:location}/bin/dmtxwrite
......
......@@ -45,11 +45,13 @@ config-{{ k }} = {{ dumps(v) }}
{% do storage_dict.setdefault(zodb['type'], {}).__setitem__(name, zodb.pop('server')) -%}
{% endif -%}
{% endfor -%}
{% do assert(len(mountpoints) == len(zodb_dict)) %}
{% do assert(len(mountpoints) == len(zodb_dict)) -%}
{% set neo = [] -%}
{% for server_type, server_dict in storage_dict.iteritems() -%}
{% if server_type == 'neo' -%}
{% set ((name, server_dict),) = server_dict.items() -%}
{% do zodb_dict[name].setdefault('storage-dict', {}).__setitem__('name', server_dict['cluster']) -%}
{% do neo.append(server_dict.get('cluster')) -%}
{% do server_dict.__setitem__('cluster', '${publish-early:neo-cluster}') -%}
{{ root_common.request_neo(server_dict, 'zodb-neo', 'neo-') }}
{% else -%}
{{ assert(server_type == 'zeo', server_type) -}}
......@@ -58,21 +60,38 @@ config-{{ k }} = {{ dumps(v) }}
{% endif -%}
{% endfor -%}
[inituser-password]
{% set inituser_password = slapparameter_dict.get('inituser-password') -%}
{% if inituser_password -%}
passwd = {{ dumps(inituser_password) }}
{% else -%}
recipe = slapos.cookbook:generate.password
{% endif -%}
[publish-early]
recipe = slapos.cookbook:publish-early
-init =
inituser-password gen-password:passwd
deadlock-debugger-password gen-deadlock-debugger-password:passwd
{%- if neo %}
neo-cluster gen-neo-cluster:name
{%- if neo[0] %}
neo-cluster = {{ neo[0] }}
{%- endif %}
{%- endif %}
{%- set inituser_password = slapparameter_dict.get('inituser-password') %}
{%- if inituser_password %}
inituser-password = {{ inituser_password }}
{%- endif %}
{%- set deadlock_debugger_password = slapparameter_dict.get('deadlock-debugger-password') -%}
{%- if deadlock_debugger_password %}
deadlock-debugger-password = {{ deadlock_debugger_password }}
{%- endif %}
[deadlock-debugger-password]
{% set deadlock_debugger_password = slapparameter_dict.get('deadlock-debugger-password') -%}
{% if deadlock_debugger_password -%}
passwd = {{ dumps(deadlock_debugger_password) }}
{% else -%}
[gen-password]
recipe = slapos.cookbook:generate.password
{% endif -%}
storage-path =
[gen-deadlock-debugger-password]
<= gen-password
[gen-neo-cluster-base]
<= gen-password
[gen-neo-cluster]
name = neo-${gen-neo-cluster-base:passwd}
[request-zope-base]
<= request-common
......@@ -82,12 +101,12 @@ return =
config-bt5 = {{ dumps(slapparameter_dict.get('bt5', 'erp5_full_text_myisam_catalog erp5_configurator_standard erp5_configurator_maxma_demo erp5_configurator_ung erp5_configurator_run_my_doc')) }}
config-bt5-repository-url = {{ dumps(slapparameter_dict.get('bt5-repository-url', local_bt5_repository)) }}
config-cloudooo-url = ${request-cloudooo:connection-url}
config-deadlock-debugger-password = ${deadlock-debugger-password:passwd}
config-deadlock-debugger-password = ${publish-early:deadlock-debugger-password}
config-developer-list = {{ dumps(slapparameter_dict.get('developer-list', [inituser_login])) }}
config-hosts-dict = {{ dumps(slapparameter_dict.get('hosts-dict', {})) }}
config-hostalias-dict = {{ dumps(slapparameter_dict.get('hostalias-dict', {})) }}
config-inituser-login = {{ dumps(inituser_login) }}
config-inituser-password = ${inituser-password:passwd}
config-inituser-password = ${publish-early:inituser-password}
config-kumofs-url = ${request-memcached-persistent:connection-url}
config-memcached-url = ${request-memcached-volatile:connection-url}
config-mysql-test-url-list = ${request-mariadb:connection-test-database-list}
......@@ -98,6 +117,7 @@ config-timezone = {{ dumps(slapparameter_dict.get('timezone', 'UTC')) }}
config-zodb-dict = {{ dumps(zodb_dict) }}
{% for server_type, server_dict in storage_dict.iteritems() -%}
{% if server_type == 'neo' -%}
config-neo-cluster = ${publish-early:neo-cluster}
config-neo-name = {{ server_dict.keys()[0] }}
config-neo-masters = ${neo-0-final:connection-masters}
{% else -%}
......@@ -191,12 +211,11 @@ return = site_url
[publish]
recipe = slapos.cookbook:publish.serialised
-extends = publish-early
{% if 'neo' in storage_dict -%}
neo-masters = ${neo-0-final:connection-masters}
neo-admins = ${neo-0-final:connection-admins}
{% endif -%}
deadlock-debugger-password = ${deadlock-debugger-password:passwd}
inituser-password = ${inituser-password:passwd}
{#
Pick any published hosts-dict, they are expected to be identical - and there is
no way to check here.
......
......@@ -251,6 +251,7 @@ node-id = {{ dumps(node_id_base ~ '-' ~ index) }}
{% do log_list.append(log) -%}
{% do zodb['storage-dict'].__setitem__('logfile', log) -%}
{% if db_name == slapparameter_dict.get('neo-name') -%}
{% do zodb['storage-dict'].__setitem__('name', slapparameter_dict['neo-cluster']) -%}
{% do zodb['storage-dict'].__setitem__('master_nodes', slapparameter_dict['neo-masters']) -%}
{% endif -%}
{% endif -%}
......
Styling with Markdown is supported
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!