Skip to content

erp5
erp5
Commit b111b6e3 authored by Rafael Monnerat's avatar Rafael Monnerat
Browse files
Options

erp5_disaster_recovery: Initial public release 

See merge request nexedi/erp5!1900
parents 38483df3 751c2516
Pipeline #33502 failed with stage
    in 0 seconds
    Hide whitespace changes
    Inline Side-by-side
    Showing with 697 additions and 0 deletions
    bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery.xml 0 → 100644
    View file @ b111b6e3
    <?xml version="1.0"?>
    <ZopeData>
    <record id="1" aka="AAAAAAAAAAE=">
    <pickle>
    <global name="Folder" module="OFS.Folder"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>_objects</string> </key>
    <value>
    <tuple/>
    </value>
    </item>
    <item>
    <key> <string>id</string> </key>
    <value> <string>erp5_disaster_recovery</string> </value>
    </item>
    <item>
    <key> <string>title</string> </key>
    <value> <string></string> </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    </ZopeData>
    bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_checkDeletedDocumentList.py 0 → 100644
    View file @ b111b6e3
    portal = context.getPortalObject()
    document_list = portal.portal_catalog(
    limit=limit,
    uid={'query': min_uid, 'range': 'nlt'},
    sort_on=(('uid', 'ASC'),),
    )
    result_count = len(document_list)
    if result_count:
    if result_count == limit:
    portal.portal_activities.activate(activity='SQLQueue', priority=3).ERP5Site_checkDeletedDocumentList(document_list[-1].uid, limit, packet_size)
    column_list = [(x.path, x.uid) for x in document_list]
    for i in xrange(0, result_count, packet_size):
    portal.portal_activities.activate(activity='SQLQueue').ERP5Site_unindexDeletedDocumentList(column_list[i:i+packet_size])
    bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_checkDeletedDocumentList.xml 0 → 100644
    View file @ b111b6e3
    <?xml version="1.0"?>
    <ZopeData>
    <record id="1" aka="AAAAAAAAAAE=">
    <pickle>
    <global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>_bind_names</string> </key>
    <value>
    <object>
    <klass>
    <global name="_reconstructor" module="copy_reg"/>
    </klass>
    <tuple>
    <global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
    <global name="object" module="__builtin__"/>
    <none/>
    </tuple>
    <state>
    <dictionary>
    <item>
    <key> <string>_asgns</string> </key>
    <value>
    <dictionary>
    <item>
    <key> <string>name_container</string> </key>
    <value> <string>container</string> </value>
    </item>
    <item>
    <key> <string>name_context</string> </key>
    <value> <string>context</string> </value>
    </item>
    <item>
    <key> <string>name_m_self</string> </key>
    <value> <string>script</string> </value>
    </item>
    <item>
    <key> <string>name_subpath</string> </key>
    <value> <string>traverse_subpath</string> </value>
    </item>
    </dictionary>
    </value>
    </item>
    </dictionary>
    </state>
    </object>
    </value>
    </item>
    <item>
    <key> <string>_params</string> </key>
    <value> <string>min_uid, limit, packet_size</string> </value>
    </item>
    <item>
    <key> <string>id</string> </key>
    <value> <string>ERP5Site_checkDeletedDocumentList</string> </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    </ZopeData>
    bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_checkLatestModifiedDocumentList.py 0 → 100644
    View file @ b111b6e3
    from Products.ZSQLCatalog.SQLCatalog import Query
    portal = context.getPortalObject()
    document_list = portal.portal_catalog(
    limit=limit,
    uid={'query': min_uid, 'range': 'nlt'},
    indexation_timestamp=Query(**{'indexation_timestamp': (before, now), 'range': 'minngt'}),
    sort_on=(('uid', 'ASC'),),
    )
    result_count = len(document_list)
    if result_count:
    if result_count == limit:
    portal.portal_activities.activate(activity='SQLQueue', priority=3).ERP5Site_checkLatestModifiedDocumentList(document_list[-1].uid, limit, packet_size, before, now)
    column_list = [(x.path, x.uid) for x in document_list]
    for i in xrange(0, result_count, packet_size):
    portal.portal_activities.activate(activity='SQLQueue').ERP5Site_reindexOrUnindexDocumentList(column_list[i:i+packet_size])
    bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_checkLatestModifiedDocumentList.xml 0 → 100644
    View file @ b111b6e3
    <?xml version="1.0"?>
    <ZopeData>
    <record id="1" aka="AAAAAAAAAAE=">
    <pickle>
    <global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>_bind_names</string> </key>
    <value>
    <object>
    <klass>
    <global name="_reconstructor" module="copy_reg"/>
    </klass>
    <tuple>
    <global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
    <global name="object" module="__builtin__"/>
    <none/>
    </tuple>
    <state>
    <dictionary>
    <item>
    <key> <string>_asgns</string> </key>
    <value>
    <dictionary>
    <item>
    <key> <string>name_container</string> </key>
    <value> <string>container</string> </value>
    </item>
    <item>
    <key> <string>name_context</string> </key>
    <value> <string>context</string> </value>
    </item>
    <item>
    <key> <string>name_m_self</string> </key>
    <value> <string>script</string> </value>
    </item>
    <item>
    <key> <string>name_subpath</string> </key>
    <value> <string>traverse_subpath</string> </value>
    </item>
    </dictionary>
    </value>
    </item>
    </dictionary>
    </state>
    </object>
    </value>
    </item>
    <item>
    <key> <string>_params</string> </key>
    <value> <string>min_uid, limit, packet_size, before, now</string> </value>
    </item>
    <item>
    <key> <string>id</string> </key>
    <value> <string>ERP5Site_checkLatestModifiedDocumentList</string> </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    </ZopeData>
    bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_recoverFromRestoration.py 0 → 100644
    View file @ b111b6e3
    portal = context.getPortalObject()
    now = DateTime()
    before = now - int(days_before)
    strfstring = '%Y-%m-%d %H:%M:%S'
    portal.ERP5Site_checkLatestModifiedDocumentList(0, 1000, 100,
    before.strftime(strfstring), now.strftime(strfstring))
    # Force reindexation of recently created document
    # This expect module to use HBTree
    # The following category could be consider as "non-optimistic", but
    # it is minimal compared to a whole reindex and it is safe to prevent
    # Minor inconsistencies.
    for module_id in portal.objectIds(("ERP5 Folder",)):
    if module_id.endswith("_module"):
    portal[module_id].recurseCallMethod(
    'recursiveReindexObject',
    max_depth=1,
    min_depth=1,
    max_retry=0,
    activity_count=100,
    min_id=before.strftime("%Y%m%d"),
    )
    portal.ERP5Site_checkDeletedDocumentList(0, 1000, 100)
    # Whenever we trust that the catalog is consistent and more recent them the
    # ZODB, it isn't required to trigger the whole reindexation of the site.
    # If the Mariadb is eventually inconsitent or older them the current ZODB,
    # It's required reindex the whole site.
    if not optimistic:
    for module_id in [
    'portal_preferences',
    'portal_categories',
    'portal_alarms',
    'portal_simulation']+portal.objectIds(("ERP5 Folder",)):
    portal[module_id].recurseCallMethod(
    'immediateReindexObject',
    min_depth=1,
    max_depth=10000,
    activate_kw=dict(
    group_method_id='portal_catalog/catalogObjectList',
    alternate_method_id='alternateReindexObject',
    group_method_cost=1,
    priority=6,
    ),
    max_retry=0,
    activity_count=100,
    )
    return "OK"
    bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_recoverFromRestoration.xml 0 → 100644
    View file @ b111b6e3
    <?xml version="1.0"?>
    <ZopeData>
    <record id="1" aka="AAAAAAAAAAE=">
    <pickle>
    <global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>_bind_names</string> </key>
    <value>
    <object>
    <klass>
    <global name="_reconstructor" module="copy_reg"/>
    </klass>
    <tuple>
    <global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
    <global name="object" module="__builtin__"/>
    <none/>
    </tuple>
    <state>
    <dictionary>
    <item>
    <key> <string>_asgns</string> </key>
    <value>
    <dictionary>
    <item>
    <key> <string>name_container</string> </key>
    <value> <string>container</string> </value>
    </item>
    <item>
    <key> <string>name_context</string> </key>
    <value> <string>context</string> </value>
    </item>
    <item>
    <key> <string>name_m_self</string> </key>
    <value> <string>script</string> </value>
    </item>
    <item>
    <key> <string>name_subpath</string> </key>
    <value> <string>traverse_subpath</string> </value>
    </item>
    </dictionary>
    </value>
    </item>
    </dictionary>
    </state>
    </object>
    </value>
    </item>
    <item>
    <key> <string>_params</string> </key>
    <value> <string>optimistic=True, days_before=1</string> </value>
    </item>
    <item>
    <key> <string>id</string> </key>
    <value> <string>ERP5Site_recoverFromRestoration</string> </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    </ZopeData>
    bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_reindexOrUnindexDocumentList.py 0 → 100644
    View file @ b111b6e3
    from erp5.component.module.Log import log
    for path, uid in column_list:
    try:
    ob = context.restrictedTraverse(path)
    except KeyError:
    log("object not found", path)
    context.portal_catalog.activate(activity='SQLQueue').uncatalog_object(uid=uid)
    else:
    ob.reindexObject()
    bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_reindexOrUnindexDocumentList.xml 0 → 100644
    View file @ b111b6e3
    <?xml version="1.0"?>
    <ZopeData>
    <record id="1" aka="AAAAAAAAAAE=">
    <pickle>
    <global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>_bind_names</string> </key>
    <value>
    <object>
    <klass>
    <global name="_reconstructor" module="copy_reg"/>
    </klass>
    <tuple>
    <global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
    <global name="object" module="__builtin__"/>
    <none/>
    </tuple>
    <state>
    <dictionary>
    <item>
    <key> <string>_asgns</string> </key>
    <value>
    <dictionary>
    <item>
    <key> <string>name_container</string> </key>
    <value> <string>container</string> </value>
    </item>
    <item>
    <key> <string>name_context</string> </key>
    <value> <string>context</string> </value>
    </item>
    <item>
    <key> <string>name_m_self</string> </key>
    <value> <string>script</string> </value>
    </item>
    <item>
    <key> <string>name_subpath</string> </key>
    <value> <string>traverse_subpath</string> </value>
    </item>
    </dictionary>
    </value>
    </item>
    </dictionary>
    </state>
    </object>
    </value>
    </item>
    <item>
    <key> <string>_params</string> </key>
    <value> <string>column_list</string> </value>
    </item>
    <item>
    <key> <string>id</string> </key>
    <value> <string>ERP5Site_reindexOrUnindexDocumentList</string> </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    </ZopeData>
    bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_unindexDeletedDocumentList.py 0 → 100644
    View file @ b111b6e3
    from erp5.component.module.Log import log
    for path, uid in column_list:
    try:
    _ = context.restrictedTraverse(path)
    except KeyError:
    log("object not found", path)
    context.portal_catalog.activate(activity='SQLQueue').uncatalog_object(uid=uid)
    bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_unindexDeletedDocumentList.xml 0 → 100644
    View file @ b111b6e3
    <?xml version="1.0"?>
    <ZopeData>
    <record id="1" aka="AAAAAAAAAAE=">
    <pickle>
    <global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>_bind_names</string> </key>
    <value>
    <object>
    <klass>
    <global name="_reconstructor" module="copy_reg"/>
    </klass>
    <tuple>
    <global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
    <global name="object" module="__builtin__"/>
    <none/>
    </tuple>
    <state>
    <dictionary>
    <item>
    <key> <string>_asgns</string> </key>
    <value>
    <dictionary>
    <item>
    <key> <string>name_container</string> </key>
    <value> <string>container</string> </value>
    </item>
    <item>
    <key> <string>name_context</string> </key>
    <value> <string>context</string> </value>
    </item>
    <item>
    <key> <string>name_m_self</string> </key>
    <value> <string>script</string> </value>
    </item>
    <item>
    <key> <string>name_subpath</string> </key>
    <value> <string>traverse_subpath</string> </value>
    </item>
    </dictionary>
    </value>
    </item>
    </dictionary>
    </state>
    </object>
    </value>
    </item>
    <item>
    <key> <string>_params</string> </key>
    <value> <string>column_list</string> </value>
    </item>
    <item>
    <key> <string>id</string> </key>
    <value> <string>ERP5Site_unindexDeletedDocumentList</string> </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    </ZopeData>
    bt5/erp5_disaster_recovery/TestTemplateItem/portal_components/test.erp5.testDisasterRecovery.py 0 → 100644
    View file @ b111b6e3
    ##############################################################################
    #
    # Copyright (c) 2002-2024 Nexedi SA and Contributors. All Rights Reserved.
    #
    # WARNING: This program as such is intended to be used by professional
    # programmers who take the whole responsibility of assessing all potential
    # consequences resulting from its eventual inadequacies and bugs
    # End users who are looking for a ready-to-use solution with commercial
    # guarantees and support are strongly adviced to contract a Free Software
    # Service Company
    #
    # This program is Free Software; you can redistribute it and/or
    # modify it under the terms of the GNU General Public License
    # as published by the Free Software Foundation; either version 2
    # of the License, or (at your option) any later version.
    #
    # This program is distributed in the hope that it will be useful,
    # but WITHOUT ANY WARRANTY; without even the implied warranty of
    # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
    # GNU General Public License for more details.
    #
    # You should have received a copy of the GNU General Public License
    # along with this program; if not, write to the Free Software
    # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
    #
    ##############################################################################
    from Products.ERP5Type.tests.ERP5TypeTestCase import ERP5TypeTestCase
    class TestDisasterRecovery(ERP5TypeTestCase):
    """
    Test Disaster Recover
    """
    def test_missing_catalog(self):
    document = self.portal.person_module.newContent(
    portal_type='Person',
    title='Person for Disaster Recovery'
    )
    self.tic()
    _document = self.portal.portal_catalog.getResultValue(
    uid=document.getUid()
    )
    self.assertNotEqual(_document, None)
    self.assertEqual(document.getUid(), _document.getUid())
    self.portal.portal_catalog.uncatalog_object(uid=document.getUid())
    self.tic()
    _document = self.portal.portal_catalog.getResultValue(
    uid=document.getUid()
    )
    self.assertEqual(_document, None)
    self.portal.ERP5Site_recoverFromRestoration()
    self.tic()
    _document = self.portal.portal_catalog.getResultValue(
    uid=document.getUid()
    )
    self.assertNotEqual(_document, None)
    self.assertEqual(document.getUid(), _document.getUid())
    def test_catalog_but_deleted(self):
    if self.portal.person_module.getIdGenerator() != '_generatePerDayId':
    self.portal.person_module.setIdGenerator('_generatePerDayId')
    document = self.portal.person_module.newContent(
    portal_type='Person',
    title='Person for Disaster Recovery'
    )
    self.tic()
    _document = self.portal.portal_catalog.getResultValue(
    uid=document.getUid()
    )
    self.assertNotEqual(_document, None)
    self.assertEqual(document.getUid(), _document.getUid())
    # Force remove the object w/o trigger updates on catalog
    self.portal.person_module._objects = tuple([
    i for i in self.portal.person_module._objects
    if i['id'] != document.getId()])
    self.portal.person_module._delOb(document.getId())
    self.tic()
    connection = self.getSQLConnection()
    doc_list = connection.manage_test(
    "select * from catalog where path = '/%s/%s'" % (
    self.portal.getId(), document.getRelativeUrl()))
    self.assertEqual(len(doc_list), 1)
    doc_list = connection.manage_test("select * from catalog where uid = %s" % document.getUid())
    self.assertEqual(len(doc_list), 1)
    self.portal.ERP5Site_recoverFromRestoration()
    self.tic()
    ac = connection.manage_test("select * from catalog where uid = %s" % document.getUid())
    self.assertEqual(len(ac), 0)
    _document = self.portal.portal_catalog(uid=document.getUid())
    self.assertEqual(len(_document), 0)
    def test_cataloged_is_inconsistent(self):
    document = self.portal.person_module.newContent(
    portal_type='Person',
    title='Person for Disaster Recovery %s' % (
    str(self.portal.portal_ids.generateNewId(
    id_group=('erp5_disaster_recovery_test_id')))))
    self.tic()
    _document = self.portal.portal_catalog.getResultValue(
    uid=document.getUid()
    )
    self.assertNotEqual(_document, None)
    self.assertEqual(document.getUid(), _document.getUid())
    connection = self.getSQLConnection()
    connection.manage_test("update catalog set title = 'modified title' where uid = %s" % document.getUid())
    connection.manage_test("commit")
    self.commit()
    _document = self.portal.portal_catalog(title='modified title')
    self.assertEqual(len(_document), 1)
    _document = self.portal.portal_catalog(title=document.getTitle())
    self.assertEqual(len(_document), 0)
    self.portal.ERP5Site_recoverFromRestoration()
    self.tic()
    _document = self.portal.portal_catalog(title='modified title')
    self.assertEqual(len(_document), 0)
    _document = self.portal.portal_catalog(title=document.getTitle())
    self.assertEqual(len(_document), 1)
    _document = self.portal.portal_catalog(title=document.getTitle())
    self.assertEqual(_document[0].getUid(), document.getUid())
    bt5/erp5_disaster_recovery/TestTemplateItem/portal_components/test.erp5.testDisasterRecovery.xml 0 → 100644
    View file @ b111b6e3
    <?xml version="1.0"?>
    <ZopeData>
    <record id="1" aka="AAAAAAAAAAE=">
    <pickle>
    <global name="Test Component" module="erp5.portal_type"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>default_reference</string> </key>
    <value> <string>testDisasterRecovery</string> </value>
    </item>
    <item>
    <key> <string>description</string> </key>
    <value>
    <none/>
    </value>
    </item>
    <item>
    <key> <string>id</string> </key>
    <value> <string>test.erp5.testDisasterRecovery</string> </value>
    </item>
    <item>
    <key> <string>sid</string> </key>
    <value>
    <none/>
    </value>
    </item>
    <item>
    <key> <string>text_content_error_message</string> </key>
    <value>
    <tuple/>
    </value>
    </item>
    <item>
    <key> <string>text_content_warning_message</string> </key>
    <value>
    <tuple/>
    </value>
    </item>
    <item>
    <key> <string>version</string> </key>
    <value> <string>erp5</string> </value>
    </item>
    <item>
    <key> <string>workflow_history</string> </key>
    <value>
    <persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
    </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    <record id="2" aka="AAAAAAAAAAI=">
    <pickle>
    <global name="PersistentMapping" module="Persistence.mapping"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>data</string> </key>
    <value>
    <dictionary>
    <item>
    <key> <string>component_validation_workflow</string> </key>
    <value>
    <persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
    </value>
    </item>
    </dictionary>
    </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    <record id="3" aka="AAAAAAAAAAM=">
    <pickle>
    <global name="WorkflowHistoryList" module="Products.ERP5Type.Workflow"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>_log</string> </key>
    <value>
    <list>
    <dictionary>
    <item>
    <key> <string>action</string> </key>
    <value> <string>validate</string> </value>
    </item>
    <item>
    <key> <string>validation_state</string> </key>
    <value> <string>validated</string> </value>
    </item>
    </dictionary>
    </list>
    </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    </ZopeData>
    bt5/erp5_disaster_recovery/bt/dependency_list 0 → 100644
    View file @ b111b6e3
    erp5_core
    \ No newline at end of file
    bt5/erp5_disaster_recovery/bt/description 0 → 100644
    View file @ b111b6e3
    Business template to help to sync mariadb and Datafs consistency, after a restoration.
    \ No newline at end of file
    bt5/erp5_disaster_recovery/bt/template_skin_id_list 0 → 100644
    View file @ b111b6e3
    erp5_disaster_recovery
    \ No newline at end of file
    bt5/erp5_disaster_recovery/bt/template_test_id_list 0 → 100644
    View file @ b111b6e3
    test.erp5.testDisasterRecovery
    \ No newline at end of file
    bt5/erp5_disaster_recovery/bt/test_dependency_list 0 → 100644
    View file @ b111b6e3
    erp5_base
    erp5_full_text_mroonga_catalog
    \ No newline at end of file
    bt5/erp5_disaster_recovery/bt/title 0 → 100644
    View file @ b111b6e3
    erp5_disaster_recovery
    \ No newline at end of file
    bt5/erp5_disaster_recovery/bt/version 0 → 100644
    View file @ b111b6e3
    0.1
    \ No newline at end of file
    Markdown is supported
    0% or
    You are about to add 0 people to the discussion. Proceed with caution.
    Finish editing this message first!
    Please register or to comment
    GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7