Skip to content

S
slapos.core
Commit 3f50b761 authored by Ophélie Gagnard's avatar Ophélie Gagnard Committed by Ophélie Gagnard
Browse files
Options

slapos_metadata_transform_agent: Add comparison script. 

Add the script computing the diff between a scan and the reference data base.
Add Alarm_checkNodeStatus which detects new Data Array to be processed and calls the right script.
parent cee4771e
Show whitespace changes
Inline Side-by-side
Showing with 518 additions and 2 deletions
master/bt5/slapos_metadata_transform_agent/PathTemplateItem/portal_alarms/slapos_check_node_status.xml 0 → 100644
View file @ 3f50b761
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Alarm" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>active_sense_method_id</string> </key>
<value> <string>Alarm_checkNodeStatus</string> </value>
</item>
<item>
<key> <string>automatic_solve</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>slapos_check_node_status</string> </value>
</item>
<item>
<key> <string>periodicity_hour</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>periodicity_hour_frequency</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>periodicity_minute</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>periodicity_month</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>periodicity_month_day</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>periodicity_start_date</string> </key>
<value>
<object>
<klass>
<global name="DateTime" module="DateTime.DateTime"/>
</klass>
<tuple>
<none/>
</tuple>
<state>
<tuple>
<float>1609462860.0</float>
<string>GMT</string>
</tuple>
</state>
</object>
</value>
</item>
<item>
<key> <string>periodicity_week</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Alarm</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Slapos Check Node Status</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_metadata_transform_agent/PortalTypeBaseCategoryTemplateItem/base_category_list.xml
View file @ 3f50b761
<base_category_list> <base_category_list>
<portal_type id="Data Array">
<item>causality</item>
<item>predecessor</item>
</portal_type>
<portal_type id="Data Product"> <portal_type id="Data Product">
<item>publication_section</item> <item>publication_section</item>
</portal_type> </portal_type>
......
master/bt5/slapos_metadata_transform_agent/SkinTemplateItem/portal_skins/slapos_metadata_transform_agent/Alarm_checkNodeStatus.py 0 → 100644
View file @ 3f50b761
from DateTime import DateTime
portal = context.getPortalObject()
portal.portal_catalog.searchAndActivate(
portal_type="Data Array",
simulation_state="converted",
method_id='DataArray_generateDiffND',
activate_kw={'tag': tag},
)
context.activate(after_tag=tag).getId()
master/bt5/slapos_metadata_transform_agent/SkinTemplateItem/portal_skins/slapos_metadata_transform_agent/Alarm_checkNodeStatus.xml 0 → 100644
View file @ 3f50b761
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>tag, fixit,**kw</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Alarm_checkNodeStatus</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_metadata_transform_agent/SkinTemplateItem/portal_skins/slapos_metadata_transform_agent/DataArray_generateDiffND.py 0 → 100644
View file @ 3f50b761
import numpy as np
from Products.ZSQLCatalog.SQLCatalog import AndQuery, Query
# Do nothing for reference image, just change state
if 'file_system_image/reference_image' in context.getPublicationSectionList():
context.processFile()
return
for publication_section in context.getPublicationSectionList():
if 'distribution' in publication_section:
current_node_distribution = publication_section
reference_distribution = None
for predecessor in context.getPredecessorValueList(
portal_type='Data Array'
):
publication_section_list = predecessor.getPublicationSectionList()
if 'file_system_image/reference_image' in publication_section_list:
for publication_section in publication_section_list:
if 'distribution' in publication_section:
distribution_property = context.portal_categories.restrictedTraverse('publication_section/%s' % publication_section)
parent_distribution = distribution_property.getParentValue()
distribution_list = [x for x in parent_distribution.objectValues(sort_on='int_index')]
index = distribution_list.index(distribution_property) + 1
if index >= len(distribution_list):
context.setPublicationSectionList(context.getPublicationSectionList() + ['publication_section/file_system_image/diff_end'])
causality = context.getCausalityValue(portal_type='Data Product')
if causality and context.portal_workflow.isTransitionPossible(causality, 'invalidate'):
causality.invalidate(comment='Server has file modified')
context.processFile()
return
reference_distribution = distribution_list[index].getRelativeUrl()
if not reference_distribution:
reference_distribution = 'publication_section/%s' % current_node_distribution
query = AndQuery(
Query(portal_type = ["Data Array"]),
Query(simulation_state = ("converted", "processed")),
AndQuery(Query(publication_section_relative_url = 'publication_section/file_system_image/reference_image'),
Query(publication_section_relative_url = reference_distribution)))
reference_image = context.portal_catalog.getResultValue(query=query)
def getDiff(a1, a2):
return np.setdiff1d(a1[:], a2[:])
if reference_image:
diff_array = getDiff(context.getArray(), reference_image.getArray())
if len(diff_array) != 0:
new_data_array = context.data_array_module.newContent(portal_type='Data Array')
new_data_array.setArray(diff_array)
new_data_array.edit(
predecessor_value_list=[context, reference_image],
publication_section="file_system_image/node_image",
causality = context.getCausality()
)
new_data_array.convertFile()
else:
causality = context.getCausalityValue(portal_type='Data Product')
if causality and context.portal_workflow.isTransitionPossible(causality, 'validate'):
causality.validate(comment='Server is ok')
else:
causality = context.getCausalityValue(portal_type='Data Product')
if causality and context.portal_workflow.isTransitionPossible(causality, 'invalidate'):
causality.invalidate(comment='Server has file modified')
context.processFile()
master/bt5/slapos_metadata_transform_agent/SkinTemplateItem/portal_skins/slapos_metadata_transform_agent/DataArray_generateDiffND.xml 0 → 100644
View file @ 3f50b761
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>**kw</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>DataArray_generateDiffND</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_metadata_transform_agent/SkinTemplateItem/portal_skins/slapos_metadata_transform_agent/DataArray_view/my_causality_title.xml 0 → 100644
View file @ 3f50b761
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ProxyField" module="Products.ERP5Form.ProxyField"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>delegated_list</string> </key>
<value>
<list>
<string>portal_type</string>
<string>title</string>
</list>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>my_causality_title</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string>my_relation_field</string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string>Base_viewFieldLibrary</string> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value>
<list>
<tuple>
<string>Data Product</string>
<string>Data Product</string>
</tuple>
</list>
</value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Data Product</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_metadata_transform_agent/SkinTemplateItem/portal_skins/slapos_metadata_transform_agent/DataArray_view/my_predecessor_title_list.xml 0 → 100644
View file @ 3f50b761
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ProxyField" module="Products.ERP5Form.ProxyField"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>delegated_list</string> </key>
<value>
<list>
<string>height</string>
<string>portal_type</string>
<string>title</string>
<string>view_separator</string>
</list>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>my_predecessor_title_list</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string>my_multi_relation_field</string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string>Base_viewFieldLibrary</string> </value>
</item>
<item>
<key> <string>height</string> </key>
<value> <int>5</int> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value>
<list>
<tuple>
<string>Data Array</string>
<string>Data Array</string>
</tuple>
</list>
</value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Predecessors</string> </value>
</item>
<item>
<key> <string>view_separator</string> </key>
<value> <string encoding="cdata"><![CDATA[
<br />
]]></string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_metadata_transform_agent/bt/template_path_list
View file @ 3f50b761
...@@ -2,6 +2,7 @@ data_operation_module/convert_raw_data_to_array_data_operation ...@@ -2,6 +2,7 @@ data_operation_module/convert_raw_data_to_array_data_operation
data_product_module/convert_data_stream_to_data_array data_product_module/convert_data_stream_to_data_array
organisation_module/meta_destination organisation_module/meta_destination
organisation_module/server_node organisation_module/server_node
portal_alarms/slapos_check_node_status
portal_callables/MetadataIngestionPolicy_parseTag portal_callables/MetadataIngestionPolicy_parseTag
portal_callables/Metadata_convertDataStreamToArray portal_callables/Metadata_convertDataStreamToArray
portal_categories/publication_section/file_system_image portal_categories/publication_section/file_system_image
......
master/bt5/slapos_metadata_transform_agent/bt/template_portal_type_base_category_list
View file @ 3f50b761
Data Array | causality
Data Array | predecessor
Data Product | publication_section Data Product | publication_section
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7