Automatic ERP5 Site creation

If the following configuration section is present:

<product-config initsite>
  owner <user-id>
  id <site-id>
  <key> <value>
</product-config>

the ERP5 product automatically creates a site at startup if there's none.
This is done by "extending" AppInitializer (OFS.Application).

<key> is any string parameter accepted by manage_addERP5Site, including new
ones to configure Cloudooo and install bt5. Cloudooo is often required by the
configurator: apart from that, it does not deserve a field on the addERP5Site
form.

This deprecates erp5_promise, at least in its current form, for several reasons:

- Since 7fe14b20, alarms aren't run often
  enough.

- For the configuration of external services like memcached, using an alarm or
  anything not instantaneous to update them is wrong because it poses a risk to
  connect to wrong instances. erp5_promise does not handle the catalog because
  it could not and we have another mechanism for that (userhosts), so let's
  handle other services like the catalog.

- The list of bt5 to have installed is only useful at site creation:
  - either (usually) to install configurators, which in turn decide of the list
    to have installed,
  - or a value provided the user, but SlapOS parameters aren't a place to
    duplicate the ERP5 UI.

- With previous commit, the list of bt5 repositories does not depend anymore
  on the location of the software built by SlapOS, and we also don't want to
  duplicate the UI, so the only case where it would remain useful is when the
  list is changed by the software itself, which almost never happens.
  We'll reimplement the promise as a constraint when necessary.

We end up with the minimum functionality to install a configurator
automatically, and the user does not have to wait anymore for an alarm to be
triggered.

The default site preference in erp5_core does not come anymore with a default
value that was specific to SlapOS. It was not enough for our needs and such
setting should anyway go in a system preference.

product/ERP5/ERP5Site.py

@@ -17,7 +17,7 @@
 
 import threading
 from weakref import ref as weakref
-from OFS.Application import Application
+from OFS.Application import Application, AppInitializer
 from Products.ERP5Type import Globals
 from Products.ERP5Type.Globals import package_home
 
@@ -68,9 +68,12 @@ def manage_addERP5Site(self,
                        email_from_address='postmaster@localhost',
                        email_from_name='Portal Administrator',
                        validate_email=0,
-                       erp5_catalog_storage='',
+                       erp5_catalog_storage='erp5_mysql_innodb_catalog',
                        erp5_sql_connection_string='test test',
                        cmf_activity_sql_connection_string='test test',
+                       bt5_repository_url='',
+                       bt5='',
+                       cloudooo_url='',
                        light_install=0,
                        reindex=1,
                        sql_reset=0,
@@ -86,6 +89,9 @@ def manage_addERP5Site(self,
                  erp5_catalog_storage,
                  erp5_sql_connection_string,
                  cmf_activity_sql_connection_string,
+                 bt5_repository_url,
+                 bt5,
+                 cloudooo_url,
                  create_activities=create_activities,
                  light_install=light_install,
                  reindex=reindex,
@@ -258,6 +264,24 @@ class ERP5Site(FolderMixIn, CMFSite, CacheCookieMixin):
     request.RESPONSE.realm = None
     return super(ERP5Site, self).__before_publishing_traverse__(self2, request)
 
+  def _initSystemPreference(self, cloudooo_url):
+    """
+    Post-addERP5Site code to make sure that cloudoo is configured,
+    which is often required by the configurator.
+    """
+    preference_tool = self.portal_preferences
+    id = 'default_system_preference'
+    portal_type = 'System Preference'
+    for pref in preference_tool.objectValues(portal_type=portal_type):
+      if pref.getPreferenceState() == 'global':
+        break
+    else:
+      from Products.ERP5Form.PreferenceTool import Priority
+      pref = preference_tool.newContent(id, portal_type,
+        priority=Priority.SITE, title='Default ' + portal_type)
+      pref.enable()
+    pref.setPreferredDocumentConversionServerUrl(cloudooo_url)
+
   def _createInitialSiteManager(self):
     # This section of code is inspired by
     # Products.CMFDefault.upgrade.to21.upgrade_root_site_manager(),
@@ -1844,6 +1868,9 @@ class ERP5Generator(PortalGenerator):
              erp5_catalog_storage,
              erp5_sql_connection_string,
              cmf_activity_sql_connection_string,
+             bt5_repository_url,
+             bt5,
+             cloudooo_url,
              create_activities=True,
              reindex=1,
              **kw):
@@ -1878,6 +1905,20 @@ class ERP5Generator(PortalGenerator):
 
     p._v_bootstrapping = False
 
+    # XXX: Is it useful to wait for indexing ?
+    after_method_id = 'immediateReindexObject'
+    if bt5_repository_url:
+      p.portal_templates.repository_dict = dict.fromkeys(
+        bt5_repository_url.split())
+      if bt5:
+        method_id = 'upgradeSite'
+        getattr(p.portal_templates.activate(after_method_id=after_method_id),
+                method_id)(bt5.split(), update_catalog=True)
+        after_method_id = method_id
+    if cloudooo_url:
+      p.portal_activities.activateObject(p, after_method_id=after_method_id,
+        )._initSystemPreference(cloudooo_url=cloudooo_url)
+
     return p
 
   @classmethod
@@ -2281,3 +2322,81 @@ class ERP5Generator(PortalGenerator):
       template_tool.installBusinessTemplateListFromRepository(
           ['erp5_promise'], activate=True, install_dependency=True)
       p.portal_alarms.subscribe()
+
+
+# Zope offers no mechanism to extend AppInitializer so let's monkey-patch.
+AppInitializer_initialize = AppInitializer.initialize.__func__
+def initialize(self):
+  AppInitializer.initialize = AppInitializer_initialize
+  self.initialize()
+  try:
+    kw = getConfiguration().product_config['initsite']
+  except KeyError:
+    return
+  meta_type = ERP5Site.meta_type
+  for _ in self.getApp().objectIds(meta_type):
+    return
+
+  # We defer the call to manage_addERP5Site via ZServer.PubCore because:
+  # - we use ZPublisher so that get_request() works
+  #   (see new_publish in Localizer.patches)
+  # - we want errors to be logged correctly
+  #   (see Zope2.zpublisher_exception_hook in Zope2.App.startup)
+  import inspect, time
+  from AccessControl.SecurityManagement import newSecurityManager
+  from cStringIO import StringIO
+  from Products.ZMySQLDA.db import DB, OperationalError
+  from ZPublisher import HTTPRequest, HTTPResponse
+  from ZServer.PubCore import handle
+  def addERP5Site():
+    default_kw = inspect.getcallargs(manage_addERP5Site, None, '')
+    db = (kw.get('erp5_sql_connection_string') or
+      default_kw['erp5_sql_connection_string'])
+    # The lock is to avoid that multiple zopes try to create a site when
+    # they're started at the same time, because this is a quite long operation
+    # (-> high probably of conflict with a lot of wasted CPU).
+    # Theoretically, the same precaution should be taken for previous initial
+    # commits, but they're small operations. At worst, SlapOS restarts zopes
+    # automatically.
+    # A ZODB lock would be better but unless we extend ZODB, this is only
+    # possible with NEO, by voting a change to oid 0 in a parallel transaction
+    # (and abort at the end): thanks to locking at object-level, this would not
+    # block the main transaction.
+    app = last = None
+    while 1:
+      try:
+        with DB(db).lock():
+          transaction.begin()
+          app = request['PARENTS'][0] = request['PARENTS'][0]()
+          for _ in app.objectIds(meta_type):
+            return
+          uf = app.acl_users
+          user = uf.getUser(kw['owner'])
+          if not user.has_role('Manager'):
+            response.unauthorized()
+          newSecurityManager(None, user.__of__(uf))
+          manage_addERP5Site(app.__of__(RequestContainer(REQUEST=request)),
+            **{k: kw.get(k, v) for k, v in default_kw.iteritems()
+                               if isinstance(v, str)})
+          transaction.get().note('Created ' + meta_type)
+          transaction.commit()
+          break
+      except OperationalError as e:
+        if app is not None:
+          raise
+        # MySQL server not ready (down or user not created).
+        e = str(e)
+        if last != e:
+          last = e
+          LOG('ERP5Site', WARNING,
+              'MySQL error while trying to create ERP5 site. Retrying...',
+              error=1)
+        time.sleep(5)
+  response = HTTPResponse.HTTPResponse(stdout=StringIO())
+  response._finish = lambda: None
+  request = HTTPRequest.HTTPRequest(
+    StringIO(), dict(REQUEST_METHOD='GET', SERVER_URL=''), response)
+  request.traverse = lambda *args, **kw: addERP5Site
+  handle('Zope2', request, response)
+
+AppInitializer.initialize = initialize

product/ERP5/bootstrap/erp5_core/PathTemplateItem/portal_preferences/default_site_preference.xml

@@ -322,10 +322,6 @@ It\'s the lowest priority one; ie. managers can create higher priority preferenc
             <key> <string>preferred_money_quantity_field_width</string> </key>
             <value> <int>10</int> </value>
         </item>
-        <item>
-            <key> <string>preferred_document_conversion_server_url</string> </key>
-            <value> <string>http://erp5-cloudooo:2020</string> </value>
-        </item>
         <item>
             <key> <string>preferred_quantity_field_width</string> </key>
             <value> <int>5</int> </value>

product/ERP5/dtml/addERP5Site.dtml

@@ -13,7 +13,7 @@
   Form style based on Aleksandar Vacić's work,
   ditributed under Creative Commons Paternity Licence:
     * http://creativecommons.org/licenses/by/2.0/
-    * http://www.aplus.co.yu/css/forms/?css=1
+    * http://aplus.rs/lab/forms/
 -->
 
 <style type="text/css">
@@ -91,11 +91,11 @@ label {
     </div>
     <div>
       <label class="form-label" for="erp5_sql_connection_string">ERP5 Database</label>
-      <input class="form-element" name="erp5_sql_connection_string" id="erp5_sql_connection_string" type="text" size="40" value="test test"/>
+      <input class="form-element" name="erp5_sql_connection_string" id="erp5_sql_connection_string" type="text" size="60" value="test test"/>
     </div>
     <div>
       <label class="form-label" for="cmf_activity_sql_connection_string">CMF Activity Database</label>
-      <input class="form-element" name="cmf_activity_sql_connection_string" id="cmf_activity_sql_connection_string" type="text" size="40" value="test test"/>
+      <input class="form-element" name="cmf_activity_sql_connection_string" id="cmf_activity_sql_connection_string" type="text" size="60" value="test test"/>
     </div>
     <div>
       <label class="form-label" for="sql_reset">Drop any existing table</label>
@@ -108,6 +108,19 @@ label {
     <br/>
   </fieldset>
 
+  <fieldset>
+    <legend class="form-title">Business Templates</legend>
+    <div title="Business Template names, or (TODO) URL pointing to a file in BT5 or ZEXP format.">
+      <label class="form-label" for="bt5">Install BT5</label>
+      <input class="form-element" name="bt5" id="bt5" type="text" size="60"/>
+    </div>
+    <div>
+      <label class="form-label" for="bt5_repository_url">Repositories</label>
+      <textarea class="form-element" name="bt5_repository_url" id="bt5_repository_url" cols="60" rows="4"></textarea>
+    </div>
+    <br/>
+  </fieldset>
+
   <input style="float:right" class="form-element" type="submit" name="submit" value="Create a new ERP5 Site instance"/>
 
 </form>