Skip to content

S
slapos.core
Commit 87a831cf authored by Ophélie Gagnard's avatar Ophélie Gagnard Committed by Xiaowu Zhang
Browse files
Options

slapos_abyss: add bt5 to manage compute node file state 

this bt5 is used to monitoring compute node state

compute node upload file data into data stream of slapos master

then data stream is transformed into data array which can be easily processed

for more infos: https://www.erp5.com/rapidspace-Server.Check.Process
parent cfb59437
Hide whitespace changes
Inline Side-by-side
Showing with 8294 additions and 0 deletions
master/bt5/slapos_abyss/ActionTemplateItem/portal_types/Compute%20Node/create_data_supply.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ActionInformation" module="Products.CMFCore.ActionInformation"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>action</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
<item>
<key> <string>categories</string> </key>
<value>
<tuple>
<string>action_type/object_jio_action</string>
</tuple>
</value>
</item>
<item>
<key> <string>category</string> </key>
<value> <string>object_jio_action</string> </value>
</item>
<item>
<key> <string>condition</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>icon</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>create_data_supply</string> </value>
</item>
<item>
<key> <string>permissions</string> </key>
<value>
<tuple>
<string>View</string>
</tuple>
</value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Action Information</string> </value>
</item>
<item>
<key> <string>priority</string> </key>
<value> <float>6.0</float> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Create Data Supply</string> </value>
</item>
<item>
<key> <string>visible</string> </key>
<value> <int>1</int> </value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="Expression" module="Products.CMFCore.Expression"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>text</string> </key>
<value> <string>string:${object_url}/ComputeNode_createDataSupply</string> </value>
</item>
</dictionary>
</pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle>
<global name="Expression" module="Products.CMFCore.Expression"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>text</string> </key>
<value> <string>python:portal.Base_checkPermission(\'data_supply_module\', \'Add portal content\')</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/ActionTemplateItem/portal_types/Compute%20Node/create_data_transformation.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ActionInformation" module="Products.CMFCore.ActionInformation"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>action</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
<item>
<key> <string>categories</string> </key>
<value>
<tuple>
<string>action_type/object_jio_action</string>
</tuple>
</value>
</item>
<item>
<key> <string>category</string> </key>
<value> <string>object_jio_action</string> </value>
</item>
<item>
<key> <string>condition</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>icon</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>create_data_transformation</string> </value>
</item>
<item>
<key> <string>permissions</string> </key>
<value>
<tuple>
<string>View</string>
</tuple>
</value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Action Information</string> </value>
</item>
<item>
<key> <string>priority</string> </key>
<value> <float>7.0</float> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Create Data Transformation</string> </value>
</item>
<item>
<key> <string>visible</string> </key>
<value> <int>1</int> </value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="Expression" module="Products.CMFCore.Expression"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>text</string> </key>
<value> <string>string:${object_url}/ComputeNode_createDataTransformation</string> </value>
</item>
</dictionary>
</pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle>
<global name="Expression" module="Products.CMFCore.Expression"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>text</string> </key>
<value> <string>python:portal.Base_checkPermission(\'data_transformation_module\', \'Add portal content\')</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/ActionTemplateItem/portal_types/Compute%20Node/jump_to_data_array.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ActionInformation" module="Products.CMFCore.ActionInformation"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>action</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
<item>
<key> <string>categories</string> </key>
<value>
<tuple>
<string>action_type/object_jio_jump</string>
</tuple>
</value>
</item>
<item>
<key> <string>category</string> </key>
<value> <string>object_jio_jump</string> </value>
</item>
<item>
<key> <string>condition</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>icon</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>jump_to_data_array</string> </value>
</item>
<item>
<key> <string>permissions</string> </key>
<value>
<tuple>
<string>View</string>
</tuple>
</value>
</item>
<item>
<key> <string>priority</string> </key>
<value> <float>9.0</float> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Data Array</string> </value>
</item>
<item>
<key> <string>visible</string> </key>
<value> <int>1</int> </value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="Expression" module="Products.CMFCore.Expression"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>text</string> </key>
<value> <string encoding="cdata"><![CDATA[
string:${object_url}/Base_jumpToRelatedObject?base_category=causality&portal_type=Data+Array
]]></string> </value>
</item>
</dictionary>
</pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle>
<global name="Expression" module="Products.CMFCore.Expression"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>text</string> </key>
<value> <string>python: portal.Base_checkPermission(\'data_array_module\', \'View\')</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/ActionTemplateItem/portal_types/Compute%20Node/jump_to_data_stream.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ActionInformation" module="Products.CMFCore.ActionInformation"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>action</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
<item>
<key> <string>categories</string> </key>
<value>
<tuple>
<string>action_type/object_jio_jump</string>
</tuple>
</value>
</item>
<item>
<key> <string>category</string> </key>
<value> <string>object_jio_jump</string> </value>
</item>
<item>
<key> <string>condition</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>icon</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>jump_to_data_stream</string> </value>
</item>
<item>
<key> <string>permissions</string> </key>
<value>
<tuple>
<string>View</string>
</tuple>
</value>
</item>
<item>
<key> <string>priority</string> </key>
<value> <float>8.0</float> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Data Stream</string> </value>
</item>
<item>
<key> <string>visible</string> </key>
<value> <int>1</int> </value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="Expression" module="Products.CMFCore.Expression"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>text</string> </key>
<value> <string encoding="cdata"><![CDATA[
string:${object_url}/Base_jumpToRelatedObject?base_category=source&portal_type=Data+Stream
]]></string> </value>
</item>
</dictionary>
</pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle>
<global name="Expression" module="Products.CMFCore.Expression"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>text</string> </key>
<value> <string>python: portal.Base_checkPermission(\'data_stream_module\', \'View\')</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/ActionTemplateItem/portal_types/Compute%20Node/operating_system_file_info_view.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ActionInformation" module="Products.CMFCore.ActionInformation"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>action</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
<item>
<key> <string>categories</string> </key>
<value>
<tuple>
<string>action_type/object_view</string>
</tuple>
</value>
</item>
<item>
<key> <string>category</string> </key>
<value> <string>object_view</string> </value>
</item>
<item>
<key> <string>condition</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>icon</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>operating_system_file_info_view</string> </value>
</item>
<item>
<key> <string>permissions</string> </key>
<value>
<tuple>
<string>View</string>
</tuple>
</value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Action Information</string> </value>
</item>
<item>
<key> <string>priority</string> </key>
<value> <float>6.0</float> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Operating System File Info</string> </value>
</item>
<item>
<key> <string>visible</string> </key>
<value> <int>1</int> </value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="Expression" module="Products.CMFCore.Expression"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>text</string> </key>
<value> <string>string:${object_url}/ComputeNode_viewOperatingSystemFileInfo</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/ActionTemplateItem/portal_types/Data%20Array/mapped_content.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ActionInformation" module="Products.CMFCore.ActionInformation"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>action</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
<item>
<key> <string>categories</string> </key>
<value>
<tuple>
<string>action_type/object_view</string>
</tuple>
</value>
</item>
<item>
<key> <string>category</string> </key>
<value> <string>object_view</string> </value>
</item>
<item>
<key> <string>condition</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>icon</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>mapped_content</string> </value>
</item>
<item>
<key> <string>permissions</string> </key>
<value>
<tuple>
<string>View</string>
</tuple>
</value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Action Information</string> </value>
</item>
<item>
<key> <string>priority</string> </key>
<value> <float>3.0</float> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Mapped Content</string> </value>
</item>
<item>
<key> <string>visible</string> </key>
<value> <int>1</int> </value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="Expression" module="Products.CMFCore.Expression"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>text</string> </key>
<value> <string>string:${object_url}/DataArray_viewMappedContent</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/PathTemplateItem/data_operation_module/convert_data_stream_to_data_array.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Data Operation" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_Access_contents_information_Permission</string> </key>
<value>
<tuple>
<string>Assignee</string>
<string>Assignor</string>
<string>Associate</string>
<string>Auditor</string>
<string>Manager</string>
</tuple>
</value>
</item>
<item>
<key> <string>_Add_portal_content_Permission</string> </key>
<value>
<tuple>
<string>Assignee</string>
<string>Assignor</string>
<string>Associate</string>
<string>Manager</string>
</tuple>
</value>
</item>
<item>
<key> <string>_Modify_portal_content_Permission</string> </key>
<value>
<tuple>
<string>Assignee</string>
<string>Assignor</string>
<string>Associate</string>
<string>Manager</string>
</tuple>
</value>
</item>
<item>
<key> <string>_View_Permission</string> </key>
<value>
<tuple>
<string>Assignee</string>
<string>Assignor</string>
<string>Associate</string>
<string>Auditor</string>
<string>Manager</string>
</tuple>
</value>
</item>
<item>
<key> <string>aggregated_portal_type</string> </key>
<value>
<tuple>
<string>Data Array</string>
</tuple>
</value>
</item>
<item>
<key> <string>default_reference</string> </key>
<value> <string>data_operation_convert_data_stream_to_data_arrayarray</string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>convert_data_stream_to_data_array</string> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Data Operation</string> </value>
</item>
<item>
<key> <string>script_id</string> </key>
<value> <string>DataOperation_convertDataStreamToArray</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Convert Data Stream to Data Array</string> </value>
</item>
<item>
<key> <string>version</string> </key>
<value> <string>001</string> </value>
</item>
<item>
<key> <string>workflow_history</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary>
<item>
<key> <string>edit_workflow</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value>
</item>
<item>
<key> <string>validation_workflow</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAQ=</string> </persistent>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle>
<global name="WorkflowHistoryList" module="Products.ERP5Type.Workflow"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_log</string> </key>
<value>
<list>
<dictionary>
<item>
<key> <string>action</string> </key>
<value> <string>edit</string> </value>
</item>
<item>
<key> <string>actor</string> </key>
<value> <string>zope</string> </value>
</item>
<item>
<key> <string>comment</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>error_message</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>serial</string> </key>
<value> <string>1004.11839.46151.35686</string> </value>
</item>
<item>
<key> <string>state</string> </key>
<value> <string>current</string> </value>
</item>
<item>
<key> <string>time</string> </key>
<value>
<object>
<klass>
<global name="DateTime" module="DateTime.DateTime"/>
</klass>
<tuple>
<none/>
</tuple>
<state>
<tuple>
<float>1667898254.17</float>
<string>UTC</string>
</tuple>
</state>
</object>
</value>
</item>
</dictionary>
</list>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="4" aka="AAAAAAAAAAQ=">
<pickle>
<global name="WorkflowHistoryList" module="Products.ERP5Type.Workflow"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_log</string> </key>
<value>
<list>
<dictionary>
<item>
<key> <string>action</string> </key>
<value> <string>validate</string> </value>
</item>
<item>
<key> <string>actor</string> </key>
<value> <string>zope</string> </value>
</item>
<item>
<key> <string>comment</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>error_message</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>time</string> </key>
<value>
<object>
<klass>
<global name="DateTime" module="DateTime.DateTime"/>
</klass>
<tuple>
<none/>
</tuple>
<state>
<tuple>
<float>1628167369.17</float>
<string>UTC</string>
</tuple>
</state>
</object>
</value>
</item>
<item>
<key> <string>validation_state</string> </key>
<value> <string>validated</string> </value>
</item>
</dictionary>
</list>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/PathTemplateItem/data_product_module/convert_data_stream_to_data_array.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Data Product" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_Access_contents_information_Permission</string> </key>
<value>
<tuple>
<string>Assignee</string>
<string>Assignor</string>
<string>Associate</string>
<string>Auditor</string>
<string>Author</string>
<string>Manager</string>
<string>Owner</string>
</tuple>
</value>
</item>
<item>
<key> <string>_Add_portal_content_Permission</string> </key>
<value>
<tuple>
<string>Assignee</string>
<string>Assignor</string>
<string>Associate</string>
<string>Author</string>
<string>Manager</string>
<string>Owner</string>
</tuple>
</value>
</item>
<item>
<key> <string>_Modify_portal_content_Permission</string> </key>
<value>
<tuple>
<string>Assignee</string>
<string>Assignor</string>
<string>Associate</string>
<string>Author</string>
<string>Manager</string>
<string>Owner</string>
</tuple>
</value>
</item>
<item>
<key> <string>_View_Permission</string> </key>
<value>
<tuple>
<string>Assignee</string>
<string>Assignor</string>
<string>Associate</string>
<string>Auditor</string>
<string>Author</string>
<string>Manager</string>
<string>Owner</string>
</tuple>
</value>
</item>
<item>
<key> <string>aggregated_portal_type</string> </key>
<value>
<tuple>
<string>Data Array</string>
</tuple>
</value>
</item>
<item>
<key> <string>default_reference</string> </key>
<value> <string>test_server_array</string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>convert_data_stream_to_data_array</string> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Data Product</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>convert_data_stream_to_data_array</string> </value>
</item>
<item>
<key> <string>workflow_history</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary>
<item>
<key> <string>edit_workflow</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value>
</item>
<item>
<key> <string>validation_workflow</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAQ=</string> </persistent>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle>
<global name="WorkflowHistoryList" module="Products.ERP5Type.Workflow"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_log</string> </key>
<value>
<list>
<dictionary>
<item>
<key> <string>action</string> </key>
<value> <string>edit</string> </value>
</item>
<item>
<key> <string>actor</string> </key>
<value> <string>zope</string> </value>
</item>
<item>
<key> <string>comment</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>error_message</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>serial</string> </key>
<value> <string>995.40446.56556.44492</string> </value>
</item>
<item>
<key> <string>state</string> </key>
<value> <string>current</string> </value>
</item>
<item>
<key> <string>time</string> </key>
<value>
<object>
<klass>
<global name="DateTime" module="DateTime.DateTime"/>
</klass>
<tuple>
<none/>
</tuple>
<state>
<tuple>
<float>1635927912.83</float>
<string>UTC</string>
</tuple>
</state>
</object>
</value>
</item>
</dictionary>
</list>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="4" aka="AAAAAAAAAAQ=">
<pickle>
<global name="WorkflowHistoryList" module="Products.ERP5Type.Workflow"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_log</string> </key>
<value>
<list>
<dictionary>
<item>
<key> <string>action</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>actor</string> </key>
<value> <string>zope</string> </value>
</item>
<item>
<key> <string>comment</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>error_message</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>time</string> </key>
<value>
<object>
<klass>
<global name="DateTime" module="DateTime.DateTime"/>
</klass>
<tuple>
<none/>
</tuple>
<state>
<tuple>
<float>1634747092.36</float>
<string>UTC</string>
</tuple>
</state>
</object>
</value>
</item>
<item>
<key> <string>validation_state</string> </key>
<value> <string>draft</string> </value>
</item>
</dictionary>
</list>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/PathTemplateItem/portal_alarms/slapos_process_data_array.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Alarm" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>active_sense_method_id</string> </key>
<value> <string>Alarm_processDataArray</string> </value>
</item>
<item>
<key> <string>automatic_solve</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>slapos_process_data_array</string> </value>
</item>
<item>
<key> <string>periodicity_hour</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>periodicity_hour_frequency</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>periodicity_minute</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>periodicity_minute_frequency</string> </key>
<value> <int>15</int> </value>
</item>
<item>
<key> <string>periodicity_month</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>periodicity_month_day</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>periodicity_start_date</string> </key>
<value>
<object>
<klass>
<global name="DateTime" module="DateTime.DateTime"/>
</klass>
<tuple>
<none/>
</tuple>
<state>
<tuple>
<float>1609462860.0</float>
<string>GMT</string>
</tuple>
</state>
</object>
</value>
</item>
<item>
<key> <string>periodicity_week</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Alarm</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Slapos process data array</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/PathTemplateItem/portal_callables/DataArray_processDataArray.py 0 → 100644
View file @ 87a831cf
import numpy as np
from Products.ZSQLCatalog.SQLCatalog import AndQuery, Query
# for database image, archive previous version, then change state
current_publication_section_list = context.getPublicationSectionList()
if "file_system_image/process_state/converted" in current_publication_section_list:
current_publication_section_list.remove("file_system_image/process_state/converted")
current_publication_section_list.append("file_system_image/process_state/processed")
if 'file_system_image/database_image' in current_publication_section_list:
distribution = [x for x in current_publication_section_list if 'distribution' in x][0]
query = AndQuery(
Query(portal_type = ["Data Array"]),
Query(validation_state = "validated"),
AndQuery(Query(publication_section_relative_url = 'publication_section/file_system_image/process_state/processed'),
Query(publication_section_relative_url = 'publication_section/file_system_image/database_image'),
Query(publication_section_relative_url = 'publication_section/%s' % distribution)))
for old_version in context.portal_catalog(query=query):
old_version_publication_section_list = old_version.getPublicationSectionList()
old_version_publication_section_list.remove("file_system_image/process_state/processed")
old_version_publication_section_list.append("file_system_image/process_state/archived")
old_version.setPublicationSectionList(old_version_publication_section_list)
context.setPublicationSectionList(current_publication_section_list)
return
parent_array_list = context.getPredecessorValueList(portal_type='Data Array')
if parent_array_list:
next_database_distribution = None
parent_database_image = [x for x in parent_array_list if 'file_system_image/database_image' in x.getPublicationSectionList()]
if parent_database_image:
parent_database_image = parent_database_image[0]
for publication_section in parent_database_image.getPublicationSectionList():
if 'distribution' in publication_section:
distribution_property = context.portal_categories.restrictedTraverse('publication_section/%s' % publication_section)
#file_system_image/distribution/debian/debian10
parent_distribution = distribution_property.getParentValue()
distribution_list = [x for x in parent_distribution.objectValues(sort_on='int_index')]
index = distribution_list.index(distribution_property) + 1
# end, no more database found
if index >= len(distribution_list):
context.setPublicationSectionList(current_publication_section_list + [
'file_system_image/diff_end/different'])
return
next_database_distribution = distribution_list[index].getRelativeUrl()
break
else:
# parent are not database, so current array is result of self comparison
first_access_image = [x for x in parent_array_list if 'file_system_image/first_access' in x.getPublicationSectionList()][0]
for publication_section in first_access_image.getPublicationSectionList():
if 'distribution' in publication_section:
next_database_distribution = 'publication_section/%s' % publication_section
break
assert next_database_distribution is not None, "can't find next database distribution to compare"
query = AndQuery(
Query(portal_type = ["Data Array"]),
Query(validation_state = "validated"),
AndQuery(Query(publication_section_relative_url = 'publication_section/file_system_image/database_image'),
Query(publication_section_relative_url = 'publication_section/file_system_image/process_state/processed'),
Query(publication_section_relative_url = next_database_distribution)))
next_database_image = context.portal_catalog.getResultValue(query=query, sort_on=[('creation_date', 'descending')])
else:
# it's a first processing
# it's a first access, just change state
if 'file_system_image/first_access' in current_publication_section_list:
context.setPublicationSectionList(current_publication_section_list)
return
# not a first access, so get the first access data then compare
next_database_image = context.portal_catalog.getResultValue(
portal_type='Data Array',
validation_state='validated',
publication_section_relative_url ='publication_section/file_system_image/first_access',
causality_uid = context.getCausalityUid())
def getDiff(a1, a2):
return np.setdiff1d(a1[:], a2[:])
if next_database_image:
diff_array = getDiff(context.getArray(), next_database_image.getArray())
if len(diff_array) != 0:
new_data_array = context.data_array_module.newContent(portal_type='Data Array')
new_data_array.setArray(diff_array)
new_data_array.edit(
title='diff of %s and %s' %(context.getTitle(), next_database_image.getTitle()),
predecessor_value_list=[context, next_database_image],
publication_section=["file_system_image/node_image", "file_system_image/process_state/converted"],
causality = context.getCausality()
)
new_data_array.validate()
analysis_line = context.getAggregateRelatedValue(portal_type='Data Analysis Line')
if analysis_line:
analysis_line.setAggregateValueList(analysis_line.getAggregateValueList() + [new_data_array])
context.setPublicationSectionList(current_publication_section_list)
else:
context.setPublicationSectionList(current_publication_section_list + ['file_system_image/diff_end/identical'])
else:
context.setPublicationSectionList(current_publication_section_list + ['file_system_image/diff_end/different'])
master/bt5/slapos_abyss/PathTemplateItem/portal_callables/DataArray_processDataArray.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Python Script" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_local_properties</string> </key>
<value>
<tuple>
<dictionary>
<item>
<key> <string>id</string> </key>
<value> <string>reference</string> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>string</string> </value>
</item>
</dictionary>
</tuple>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>**kw</string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>DataArray_processDataArray</string> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Python Script</string> </value>
</item>
<item>
<key> <string>reference</string> </key>
<value> <string>DataArray_processDataArray</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>DataArray_processDataArray</string> </value>
</item>
<item>
<key> <string>workflow_history</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary>
<item>
<key> <string>edit_workflow</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle>
<global name="WorkflowHistoryList" module="Products.ERP5Type.Workflow"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_log</string> </key>
<value>
<list>
<dictionary>
<item>
<key> <string>action</string> </key>
<value> <string>edit</string> </value>
</item>
<item>
<key> <string>actor</string> </key>
<value> <string>zope</string> </value>
</item>
<item>
<key> <string>comment</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>error_message</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>serial</string> </key>
<value> <string>1005.7106.4846.16793</string> </value>
</item>
<item>
<key> <string>state</string> </key>
<value> <string>current</string> </value>
</item>
<item>
<key> <string>time</string> </key>
<value>
<object>
<klass>
<global name="DateTime" module="DateTime.DateTime"/>
</klass>
<tuple>
<none/>
</tuple>
<state>
<tuple>
<float>1671466273.93</float>
<string>UTC</string>
</tuple>
</state>
</object>
</value>
</item>
</dictionary>
</list>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/PathTemplateItem/portal_callables/DataOperation_convertDataStreamToArray.py 0 → 100644
View file @ 87a831cf
import numpy as np
import re
import json
import os.path
from DateTime import DateTime
def getEndAndJsonList(start, in_data_stream, chunk_size = 4 * 1024 * 1024):
end = min(start + chunk_size, in_data_stream.getSize())
unpacked = in_data_stream.readChunkList(start, end)
unpacked_string = "".join(unpacked)
if end < in_data_stream.getSize():
raw_data_string = '\n'.join(unpacked_string.splitlines()[:-1])
end = start + len(raw_data_string) + 1
else:
raw_data_string = unpacked_string
end_scan_regexp = re.compile('.*?fluentbit_end"}', re.DOTALL)
scan_end = end_scan_regexp.match(raw_data_string)
if not scan_end:
is_end_of_scan = False
else:
is_end_of_scan = True
end = start + len(scan_end.group()) + 1
raw_data_string = raw_data_string[:len(scan_end.group())]
return end, raw_data_string.splitlines(), is_end_of_scan
def removeFirstPath(data):
data['path'] = "/".join([''] + data['path'].split('/')[2:])
return data
def getTripletList(json_string_list, is_end_of_scan):
if is_end_of_scan:
# this lign deletes the "fluentbit_end" at the end of a scan
json_string_list = json_string_list[:-1]
tmp_data_list = []
for json_string in json_string_list:
tmp_data_list.append(json.loads(json_string))
tmp_data_list = [x for x in tmp_data_list if 'path' in x and 'hash' in x and 'sha256' in x['hash']]
# when server start with uefi, it is mounted at sysroot which makes path starts with /sysroot instead of /
if tmp_data_list and tmp_data_list[0]['path'].startswith('/sysroot'):
tmp_data_list = [removeFirstPath(x) for x in tmp_data_list]
data_list = []
for data in tmp_data_list:
in_list = False
for exclude_path in exclude_path_list:
if os.path.commonprefix([data['path'], exclude_path]) == exclude_path:
in_list = True
break
if not in_list:
data_list.append(data)
return [(data['path'], data['hash']['sha256']) for data in data_list if 'path' in data and 'hash' in data and 'sha256' in data['hash']]
def getUidList(triplet_list, data_stream):
uid_list = []
for triplet in triplet_list:
triplet_uid = data_mapping.addObject(triplet)
uid_list.append(triplet_uid)
return uid_list
progress_indicator = in_stream["Progress Indicator"]
in_data_stream = in_stream["Data Stream"]
data_mapping = in_stream["Data Mapping"]
out_data_array = out_array["Data Array"]
publication_section_list = out_data_array.getPublicationSectionList()
if "file_system_image/process_state/converted" in publication_section_list:
return
ingestion_line = in_data_stream.getAggregateRelatedValue(portal_type='Data Ingestion Line')
exclude_path_list = ingestion_line.getResourceValue().getProperty('exclude_path_list', [])
start = progress_indicator.getIntOffsetIndex()
end = in_data_stream.getSize()
if start >= end:
out_data_array.convertFile()
return
end, json_string_list, is_end_of_scan = getEndAndJsonList(start, in_data_stream)
if is_end_of_scan:
stop_date = json.loads(json_string_list[-1])["end_date"]
out_data_array.edit(stop_date= DateTime(stop_date))
triplet_list = getTripletList(json_string_list, is_end_of_scan)
uid_list = getUidList(triplet_list, in_data_stream)
uid_ndarray = np.ndarray((len(uid_list),), 'int64', np.array(uid_list))
zbigarray = out_data_array.getArray()
if zbigarray is None:
zbigarray = out_data_array.initArray(shape=(0,), dtype='int64')
start_date = json.loads(json_string_list[0])["beginning_date"]
out_data_array.edit(start_date = DateTime(start_date))
if len(uid_ndarray) > 0:
zbigarray.append(uid_ndarray)
progress_indicator.setIntOffsetIndex(end)
if is_end_of_scan:
out_data_array.setPublicationSectionList(publication_section_list + ["file_system_image/process_state/converted"])
if context.portal_workflow.isTransitionPossible(out_data_array, 'validate'):
out_data_array.validate()
return
if end < in_data_stream.getSize():
return 1
master/bt5/slapos_abyss/PathTemplateItem/portal_callables/DataOperation_convertDataStreamToArray.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Python Script" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_local_properties</string> </key>
<value>
<tuple>
<dictionary>
<item>
<key> <string>id</string> </key>
<value> <string>reference</string> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>string</string> </value>
</item>
</dictionary>
</tuple>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>in_stream=None, out_array=None</string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>DataOperation_convertDataStreamToArray</string> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Python Script</string> </value>
</item>
<item>
<key> <string>reference</string> </key>
<value> <string>DataOperation_convertDataStreamToArray</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>DataOperation_convertDataStreamToArray</string> </value>
</item>
<item>
<key> <string>workflow_history</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary>
<item>
<key> <string>edit_workflow</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle>
<global name="WorkflowHistoryList" module="Products.ERP5Type.Workflow"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_log</string> </key>
<value>
<list>
<dictionary>
<item>
<key> <string>action</string> </key>
<value> <string>edit</string> </value>
</item>
<item>
<key> <string>actor</string> </key>
<value> <string>zope</string> </value>
</item>
<item>
<key> <string>comment</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>error_message</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>serial</string> </key>
<value> <string>1004.65155.30706.36812</string> </value>
</item>
<item>
<key> <string>state</string> </key>
<value> <string>current</string> </value>
</item>
<item>
<key> <string>time</string> </key>
<value>
<object>
<klass>
<global name="DateTime" module="DateTime.DateTime"/>
</klass>
<tuple>
<none/>
</tuple>
<state>
<tuple>
<float>1671442810.27</float>
<string>UTC</string>
</tuple>
</state>
</object>
</value>
</item>
</dictionary>
</list>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/PathTemplateItem/portal_callables/IngestionPolicy_getSlaposIngestionOperationAndParameterDict.py 0 → 100644
View file @ 87a831cf
from DateTime import DateTime
now = DateTime()
today_string = now.strftime('%Y%m%d')
portal = context.getPortalObject()
portal_catalog = portal.portal_catalog
reference = movement_dict.get('reference', reference)
data_ingestion_id = "%s-%s" %(today_string, reference)
data_ingestion_query_kw = dict(
portal_type = 'Data Ingestion',
simulation_state = ['started', 'stopped'],
reference = reference)
# first search for applicable data ingestion
data_ingestion = portal_catalog.getResultValue(**data_ingestion_query_kw)
def init_input_line(input_line, operation_line):
input_line.setAggregateSet(
input_line.getAggregateList() + operation_line.getAggregateList())
data_stream = portal.portal_catalog.getResultValue(
portal_type='Data Stream',
validation_state="validated",
item_device_relative_url=operation_line.getAggregateDevice(),
item_project_relative_url=input_line.getDestinationProject(),
item_resource_uid=input_line.getResourceUid())
if data_stream is None:
compute_node = portal.portal_catalog.getResultValue(
portal_type = "Compute Node",
reference = reference
)
data_stream = portal.data_stream_module.newContent(
portal_type = 'Data Stream',
publication_section_list = compute_node.getPublicationSectionList(),
source=compute_node.getRelativeUrl(),
reference = reference)
data_stream.validate()
input_line.setAggregateValueList(
input_line.getAggregateValueList() + [data_stream])
input_line.setQuantity(1)
if data_ingestion is None:
document = portal.data_ingestion_module.get(data_ingestion_id)
if (document is not None) and document.getSimulationState() in ('started', 'stopped'):
data_ingestion = document
if data_ingestion is None:
specialise_query_kw = dict(portal_type = 'Data Supply',
reference = reference,
validation_state = 'validated')
specialise_list = [x.getRelativeUrl() for x in portal_catalog(**specialise_query_kw)]
# create a new data ingestion
data_ingestion = portal.ERP5Site_createDataIngestion(specialise_list,
reference,
data_ingestion_id)
for line in data_ingestion.objectValues(portal_type='Data Ingestion Line'):
# Mark so initialize later
if line.getResourceValue().getPortalType() == "Compute Node":
line.setQuantity(0)
break
for line in data_ingestion.objectValues(portal_type="Data Ingestion Line"):
if line.getResourceReference() == reference:
input_line = line
elif line.getResourceValue().getPortalType() == "Data Operation":
operation_line = line
if input_line.getQuantity() == 0:
init_input_line(input_line, operation_line)
data_operation = operation_line.getResourceValue()
parameter_dict = {
input_line.getReference(): \
{v.getPortalType(): v for v in input_line.getAggregateValueList()}}
return data_operation, parameter_dict
master/bt5/slapos_abyss/PathTemplateItem/portal_callables/IngestionPolicy_getSlaposIngestionOperationAndParameterDict.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Python Script" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_local_properties</string> </key>
<value>
<tuple>
<dictionary>
<item>
<key> <string>id</string> </key>
<value> <string>reference</string> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>string</string> </value>
</item>
</dictionary>
</tuple>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>movement_dict,reference</string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>IngestionPolicy_getSlaposIngestionOperationAndParameterDict</string> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Python Script</string> </value>
</item>
<item>
<key> <string>reference</string> </key>
<value> <string>IngestionPolicy_getSlaposIngestionOperationAndParameterDict</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>IngestionPolicy_getSlaposIngestionOperationAndParameterDict</string> </value>
</item>
<item>
<key> <string>workflow_history</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary>
<item>
<key> <string>edit_workflow</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle>
<global name="WorkflowHistoryList" module="Products.ERP5Type.Workflow"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_log</string> </key>
<value>
<list>
<dictionary>
<item>
<key> <string>action</string> </key>
<value> <string>edit</string> </value>
</item>
<item>
<key> <string>actor</string> </key>
<value> <string>zope</string> </value>
</item>
<item>
<key> <string>comment</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>error_message</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>serial</string> </key>
<value> <string>1005.27234.8266.51029</string> </value>
</item>
<item>
<key> <string>state</string> </key>
<value> <string>current</string> </value>
</item>
<item>
<key> <string>time</string> </key>
<value>
<object>
<klass>
<global name="DateTime" module="DateTime.DateTime"/>
</klass>
<tuple>
<none/>
</tuple>
<state>
<tuple>
<float>1672668071.19</float>
<string>UTC</string>
</tuple>
</state>
</object>
</value>
</item>
</dictionary>
</list>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/PathTemplateItem/portal_callables/IngestionPolicy_parseServerTag.py 0 → 100644
View file @ 87a831cf
if not reference:
raise ValueError('reference is None')
return {'reference': reference}
master/bt5/slapos_abyss/PathTemplateItem/portal_callables/IngestionPolicy_parseServerTag.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PyData Script" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_local_properties</string> </key>
<value>
<tuple>
<dictionary>
<item>
<key> <string>id</string> </key>
<value> <string>reference</string> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>string</string> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>id</string> </key>
<value> <string>comment</string> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>string</string> </value>
</item>
</dictionary>
</tuple>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>reference</string> </value>
</item>
<item>
<key> <string>comment</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>IngestionPolicy_parseServerTag</string> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>PyData Script</string> </value>
</item>
<item>
<key> <string>reference</string> </key>
<value> <string>IngestionPolicy_parseServerTag</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>IngestionPolicy_parseServerTag</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/PathTemplateItem/portal_ingestion_policies/metadata_upload.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Ingestion Policy" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_Access_contents_information_Permission</string> </key>
<value>
<tuple>
<string>Assignee</string>
<string>Assignor</string>
<string>Associate</string>
<string>Auditor</string>
<string>Manager</string>
</tuple>
</value>
</item>
<item>
<key> <string>_Add_portal_content_Permission</string> </key>
<value>
<tuple>
<string>Assignee</string>
<string>Assignor</string>
<string>Associate</string>
<string>Manager</string>
</tuple>
</value>
</item>
<item>
<key> <string>_Modify_portal_content_Permission</string> </key>
<value>
<tuple>
<string>Assignee</string>
<string>Assignor</string>
<string>Associate</string>
<string>Manager</string>
</tuple>
</value>
</item>
<item>
<key> <string>_View_Permission</string> </key>
<value>
<tuple>
<string>Assignee</string>
<string>Assignor</string>
<string>Associate</string>
<string>Auditor</string>
<string>Manager</string>
</tuple>
</value>
</item>
<item>
<key> <string>data_operation_script_id</string> </key>
<value> <string>IngestionPolicy_getSlaposIngestionOperationAndParameterDict</string> </value>
</item>
<item>
<key> <string>default_reference</string> </key>
<value> <string>metadata_upload</string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>metadata_upload</string> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Ingestion Policy</string> </value>
</item>
<item>
<key> <string>script_id</string> </key>
<value> <string>IngestionPolicy_parseServerTag</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Metadata Upload</string> </value>
</item>
<item>
<key> <string>version</string> </key>
<value> <string>001</string> </value>
</item>
<item>
<key> <string>workflow_history</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary>
<item>
<key> <string>edit_workflow</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value>
</item>
<item>
<key> <string>validation_workflow</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAQ=</string> </persistent>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle>
<global name="WorkflowHistoryList" module="Products.ERP5Type.Workflow"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_log</string> </key>
<value>
<list>
<dictionary>
<item>
<key> <string>action</string> </key>
<value> <string>edit</string> </value>
</item>
<item>
<key> <string>actor</string> </key>
<value> <string>zope</string> </value>
</item>
<item>
<key> <string>comment</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>error_message</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>serial</string> </key>
<value> <string>1004.20530.37821.4608</string> </value>
</item>
<item>
<key> <string>state</string> </key>
<value> <string>current</string> </value>
</item>
<item>
<key> <string>time</string> </key>
<value>
<object>
<klass>
<global name="DateTime" module="DateTime.DateTime"/>
</klass>
<tuple>
<none/>
</tuple>
<state>
<tuple>
<float>1669901322.82</float>
<string>UTC</string>
</tuple>
</state>
</object>
</value>
</item>
</dictionary>
</list>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="4" aka="AAAAAAAAAAQ=">
<pickle>
<global name="WorkflowHistoryList" module="Products.ERP5Type.Workflow"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_log</string> </key>
<value>
<list>
<dictionary>
<item>
<key> <string>action</string> </key>
<value> <string>validate</string> </value>
</item>
<item>
<key> <string>actor</string> </key>
<value> <string>zope</string> </value>
</item>
<item>
<key> <string>comment</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>error_message</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>time</string> </key>
<value>
<object>
<klass>
<global name="DateTime" module="DateTime.DateTime"/>
</klass>
<tuple>
<none/>
</tuple>
<state>
<tuple>
<float>1667569942.35</float>
<string>UTC</string>
</tuple>
</state>
</object>
</value>
</item>
<item>
<key> <string>validation_state</string> </key>
<value> <string>validated</string> </value>
</item>
</dictionary>
</list>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/PortalTypeBaseCategoryTemplateItem/base_category_list.xml 0 → 100644
View file @ 87a831cf
<base_category_list>
<portal_type id="Compute Node">
<item>publication_section</item>
</portal_type>
<portal_type id="Data Array">
<item>causality</item>
<item>predecessor</item>
</portal_type>
</base_category_list>
\ No newline at end of file
master/bt5/slapos_abyss/PortalTypePropertySheetTemplateItem/property_sheet_list.xml 0 → 100644
View file @ 87a831cf
<property_sheet_list>
<portal_type id="Compute Node">
<item>Aggregated</item>
</portal_type>
</property_sheet_list>
\ No newline at end of file
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Folder" module="OFS.Folder"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_local_properties</string> </key>
<value>
<tuple>
<dictionary>
<item>
<key> <string>id</string> </key>
<value> <string>business_template_skin_layer_priority</string> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>float</string> </value>
</item>
</dictionary>
</tuple>
</value>
</item>
<item>
<key> <string>_objects</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>business_template_skin_layer_priority</string> </key>
<value> <float>80.0</float> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>slapos_abyss</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/Alarm_processDataArray.py 0 → 100644
View file @ 87a831cf
portal = context.getPortalObject()
portal.portal_catalog.searchAndActivate(
portal_type="Data Array",
publication_section_relative_url = 'publication_section/file_system_image/process_state/converted',
validation_state='validated',
method_id='DataArray_processDataArray',
activate_kw={'tag': tag},
)
context.activate(after_tag=tag).getId()
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/Alarm_processDataArray.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>tag, fixit,**kw</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Alarm_processDataArray</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/Base_getDataMapping.py 0 → 100644
View file @ 87a831cf
mapping_for_path = getattr(context.data_mapping_module, 'mapping_for_path', None)
if not mapping_for_path:
mapping_for_path = context.data_mapping_module.newContent(
portal_type='Data Mapping',
reference='mapping_for_path',
id='mapping_for_path')
mapping_for_path.validate()
return mapping_for_path
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/Base_getDataMapping.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Base_getDataMapping</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/ComputeNode_createDataSupply.py 0 → 100644
View file @ 87a831cf
if not context.getReference():
return context.Base_redirect('view',keep_items={'portal_status_message': 'Reference is not defined'})
data_supply = context.portal_catalog.getResultValue(
validation_state='validated',
reference = context.getReference(),
portal_type='Data Supply')
if data_supply:
if batch:
return data_supply
return data_supply.Base_redirect('view',keep_items={'portal_status_message': 'Data Supply already created'})
data_supply = context.data_supply_module.newContent(
portal_type = 'Data Supply',
reference = context.getReference()
)
data_supply.newContent(
portal_type='Data Supply Line',
title='Data Stream',
reference='out_stream',
quantity=1,
int_index=2,
use='big_data/ingestion/stream',
resource_uid=context.getUid()
).validate()
data_supply.newContent(
portal_type='Data Supply Line',
title='Ingest Data',
reference='ingestion_operation',
quantity=1,
int_index=1,
resource='data_operation_module/wendelin_ingest_data'
).validate()
data_supply.validate()
if batch:
return data_supply
return data_supply.Base_redirect('view',keep_items={'portal_status_message': 'Data Supply is created'})
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/ComputeNode_createDataSupply.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>batch=0, **kw</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>ComputeNode_createDataSupply</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/ComputeNode_createDataTransformation.py 0 → 100644
View file @ 87a831cf
if not context.getReference():
return context.Base_redirect('view',keep_items={'portal_status_message': 'Reference is not defined'})
data_transformation = context.portal_catalog.getResultValue(
validation_state='validated',
resource_relative_url = context.getRelativeUrl(),
portal_type='Data Transformation')
if data_transformation:
if batch:
return data_transformation
return data_transformation.Base_redirect('view',keep_items={'portal_status_message': 'Data Transformation already created'})
data_transformation = context.data_transformation_module.newContent(
portal_type = 'Data Transformation',
reference = context.getReference(),
title=context.getTitle(),
resource_uid=context.getUid()
)
data_transformation.newContent(
portal_type='Data Transformation Resource Line',
title='In Stream',
reference='in_stream',
quantity=-1,
int_index=1,
aggregated_portal_type_list = ['Progress Indicator'],
trade_phase='data/convert',
resource_uid=context.getUid()
)
data_transformation.newContent(
portal_type='Data Transformation Resource Line',
title='Out Array',
reference='out_array',
quantity=1,
int_index=2,
aggregated_portal_type_list = ['Data Array'],
trade_phase='data/convert',
use='big_data/ingestion/stream',
resource='data_product_module/convert_data_stream_to_data_array'
)
data_transformation.newContent(
portal_type='Data Transformation Operation Line',
title='Convert Raw Data to Array',
reference='data_operation',
quantity=1,
int_index=3,
resource='data_operation_module/convert_data_stream_to_data_array'
)
data_transformation.validate()
if batch:
return data_transformation
return data_transformation.Base_redirect('view',keep_items={'portal_status_message': 'Data Transformation is created'})
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/ComputeNode_createDataTransformation.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>batch=0, **kw</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>ComputeNode_createDataTransformation</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/ComputeNode_hasModifiedFile.py 0 → 100644
View file @ 87a831cf
from Products.ZSQLCatalog.SQLCatalog import AndQuery, Query
query = AndQuery(
Query(portal_type = ["Data Array"]),
Query(causality_uid=context.getUid()),
AndQuery(Query(publication_section_relative_url = 'publication_section/file_system_image/diff_end'),
Query(publication_section_relative_url = 'publication_section/file_system_image/process_state/processed')))
data_array = context.portal_catalog.getResultValue(
query=query,
sort_on=[('creation_date', 'descending')])
if data_array:
if "file_system_image/diff_end/different" in data_array.getPublicationSectionList():
return data_array
return None
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/ComputeNode_hasModifiedFile.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>**kw</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>ComputeNode_hasModifiedFile</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/ComputeNode_viewOperatingSystemFileInfo.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ERP5 Form" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_objects</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>action</string> </key>
<value> <string>Base_edit</string> </value>
</item>
<item>
<key> <string>action_title</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>edit_order</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>encoding</string> </key>
<value> <string>UTF-8</string> </value>
</item>
<item>
<key> <string>enctype</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>group_list</string> </key>
<value>
<list>
<string>left</string>
<string>right</string>
<string>center</string>
<string>bottom</string>
<string>hidden</string>
</list>
</value>
</item>
<item>
<key> <string>groups</string> </key>
<value>
<dictionary>
<item>
<key> <string>bottom</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>center</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>hidden</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>left</string> </key>
<value>
<list>
<string>my_publication_section_list</string>
<string>my_exclude_path_list</string>
</list>
</value>
</item>
<item>
<key> <string>right</string> </key>
<value>
<list/>
</value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>ComputeNode_viewOperatingSystemFileInfo</string> </value>
</item>
<item>
<key> <string>method</string> </key>
<value> <string>POST</string> </value>
</item>
<item>
<key> <string>name</string> </key>
<value> <string>ComputeNode_viewOperatingSystemFileInfo</string> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>ERP5 Form</string> </value>
</item>
<item>
<key> <string>pt</string> </key>
<value> <string>form_view</string> </value>
</item>
<item>
<key> <string>row_length</string> </key>
<value> <int>4</int> </value>
</item>
<item>
<key> <string>stored_encoding</string> </key>
<value> <string>UTF-8</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>unicode_mode</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>update_action</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>update_action_title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/ComputeNode_viewOperatingSystemFileInfo/my_exclude_path_list.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ProxyField" module="Products.ERP5Form.ProxyField"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>delegated_list</string> </key>
<value>
<list>
<string>title</string>
</list>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>my_exclude_path_list</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string>my_lines_field</string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string>Base_viewFieldLibrary</string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string>Click to edit the target</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Exclude Path List</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/ComputeNode_viewOperatingSystemFileInfo/my_publication_section_list.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ProxyField" module="Products.ERP5Form.ProxyField"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>delegated_list</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>my_publication_section_list</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string>my_view_mode_publication_section_list</string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string>Base_viewDMSFieldLibrary</string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string>Click to edit the target</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataAnalysis_executeDataOperation.py 0 → 100644
View file @ 87a831cf
portal = context.getPortalObject()
operation = None
use_list = []
parameter_dict = {}
transient_output_item = None
context.checkConsistency(fixit=True)
initial_product = context.getSpecialiseValue(portal_type="Data Transformation").getResourceValue()
for analysis_line in context.objectValues(portal_type="Data Analysis Line"):
if analysis_line.getQuantity() < 0:
indicator = analysis_line.getAggregateProgressIndicatorValue()
stream = analysis_line.getAggregateDataStreamValue()
if indicator and stream and indicator.getIntOffsetIndex() >= stream.getSize():
return
for analysis_line in sorted(context.objectValues(portal_type="Data Analysis Line"),
key=lambda x: x.getIntIndex()):
resource = analysis_line.getResourceValue()
if resource == initial_product:
use_list = analysis_line.getUseList()
if resource is not None:
resource_portal_type = resource.getPortalType()
else:
resource_portal_type = ''
if resource_portal_type == 'Data Operation':
operation_analysis_line = analysis_line
operation = analysis_line.getResourceValue()
else:
parameter = {}
for portal_type in ["Data Array View", "Progress Indicator"] + \
list(portal.getPortalDataSinkTypeList()) + \
list(portal.getPortalDataDescriptorTypeList()):
value = analysis_line.getAggregateValue(portal_type=portal_type)
if value is not None:
parameter[portal_type] = value
# Input line, get correct data stream and mapping
if analysis_line.getQuantity() < 0:
if "Data Stream" not in parameter:
data_ingestion = analysis_line.getCausalityValue(portal_type='Data Ingestion')
data_stream_line = [x for x in data_ingestion.objectValues(portal_type='Data Ingestion Line') if x.getReference() =='out_stream'][0]
data_stream = data_stream_line.getAggregateDataStreamValue()
analysis_line.setAggregateValueList(analysis_line.getAggregateValueList() + [data_stream])
parameter['Data Stream'] = data_stream
parameter['Data Mapping'] = analysis_line.Base_getDataMapping()
# Get Correct Data array
data_array_list = analysis_line.getAggregateValueList(portal_type='Data Array')
if data_array_list:
# This is the first one, mark as database
if len(data_array_list) == 1 and 'file_system_image/process_state/processed' not in data_array_list[0].getPublicationSectionList():
data_array = data_array_list[0]
if 'file_system_image/first_access' not in data_array.getPublicationSectionList():
data_array.setPublicationSectionList(initial_product.getPublicationSectionList() + ['file_system_image/first_access'])
data_array.edit(causality_value=initial_product)
else:
all_data_array_is_processed = True
for data_array in data_array_list:
publication_section_list = data_array.getPublicationSectionList()
if ('file_system_image/process_state/processed' not in publication_section_list) and ('file_system_image/process_state/archived' not in publication_section_list):
all_data_array_is_processed = False
break
if all_data_array_is_processed:
module = portal.getDefaultModule('Data Array')
data_array = module.newContent(portal_type = 'Data Array',
title = initial_product.getTitle(),
reference = data_array.getReference()
)
analysis_line.setAggregateValueList(analysis_line.getAggregateValueList() + [data_array])
data_array.setPublicationSectionList(initial_product.getPublicationSectionList())
data_array.edit(causality_value=initial_product)
parameter['Data Array'] = data_array
if analysis_line.getQuantity() < 0 and "big_data/analysis/transient" in analysis_line.getUseList():
# at the moment we only support transient data arrays
parameter['Data Array'] = transient_input_item
if analysis_line.getQuantity() > 0 and "big_data/analysis/transient" in analysis_line.getUseList():
# at the moment we only support transient data arrays
transient_output_item = portal.data_array_module.newContent(portal_type='Data Array',
temp_object=True)
parameter['Data Array'] = transient_output_item
for base_category in analysis_line.getVariationRangeBaseCategoryList():
parameter[base_category] = analysis_line.getVariationCategoryItemList(
base_category_list=(base_category,))[0][0]
reference = analysis_line.getReference()
parameter["Start Date"] = analysis_line.getStartDate()
parameter["Stop Date"] = analysis_line.getStopDate()
parameter["causality_reference"] = analysis_line.getCausalityReference()
parameter["causality_relative_url"] = analysis_line.getCausality()
parameter["reference"] = analysis_line.getReference()
# several lines with same reference wil turn the parameter into a list
if reference in parameter_dict:
if not isinstance(parameter_dict[reference], list):
parameter_dict[reference] = [parameter_dict[reference]]
parameter_dict[reference].append(parameter)
else:
parameter_dict[reference] = parameter
if consuming_analysis_list is None:
consuming_analysis_list = []
if transient_output_item is not None and not consuming_analysis_list:
return
script_id = operation.getScriptId()
out = getattr(operation_analysis_line, script_id)(**parameter_dict)
for consuming_analysis in consuming_analysis_list:
portal.restrictedTraverse(consuming_analysis).DataAnalysis_executeDataOperation(transient_input_item = transient_output_item)
if out == 1:
context.activate(serialization_tag=str(context.getUid())).DataAnalysis_executeDataOperation(consuming_analysis_list)
else:
# only stop batch ingestions
if "big_data/ingestion/batch" in use_list:
context.stop()
# stop refresh
if context.getRefreshState() == "refresh_started":
context.stopRefresh()
return out
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataAnalysis_executeDataOperation.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>consuming_analysis_list=None, transient_input_item=None</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>DataAnalysis_executeDataOperation</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataArray_getArrayFileInfoList.py 0 → 100644
View file @ 87a831cf
from Products.ERP5Type.Document import newTempBase
if context.getArray() is None:
return []
class SequenceSliceMap():
def __init__(self, sequence_slice, usual_slice_length, total_length):
self.sequence_slice = sequence_slice
self.length = usual_slice_length
self.total_length = total_length
def __repr__(self):
return repr(list(self))
def __len__(self):
return self.total_length
def __getitem__(self, index):
return self.sequence_slice[index % self.length]
data_mapping = context.Base_getDataMapping()
def createTempBase(nr, row):
def getElementFromArray(array, index):
return data_mapping.getObjectFromValue(array[index])
def getElementFromScalar(scalar, index=None):
return data_mapping.getObjectFromValue(scalar)
column_list = [col for col in context.DataArray_getArrayColumnList() if col[0] != 'index']
column_iterator = enumerate(column_list)
if len(column_list) == 1:
getElement = getElementFromScalar
else:
getElement = getElementFromArray
return newTempBase(context.getPortalObject(),
str(id(row)),
index = nr,
**{col[0]: str(getElement(row, i)) for i, col in column_iterator})
length = context.getArrayShape()[0]
# never access more than 1000 lines at once
list_lines = min(list_lines, limit, 1000)
if context.REQUEST.has_key("limit"):
list_start = limit[0]
list_lines = limit[1] - limit[0]
orig_list_start = list_start
if orig_list_start + list_lines > length:
orig_list_start = length - length - (length % list_lines)
list_start = max(length - list_start - list_lines, 0)
if abs(orig_list_start) < list_lines and orig_list_start != 0:
list_end = abs(orig_list_start)
else:
list_end = max(list_start + list_lines, 0)
if list_start == list_end:
array_slice = [context.getArrayIndex(list_start)]
else:
array_slice = context.getArraySlice(list_start, list_end)
temp_base_list = list(reversed([createTempBase(nr + list_start, row) for nr, row in enumerate(array_slice)]))
# return lazy sequence of temp objects
return SequenceSliceMap(temp_base_list, list_lines, length)
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataArray_getArrayFileInfoList.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>list_start=0, list_lines=15, limit=1000, **kw</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>DataArray_getArrayFileInfoList</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataArray_view.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ERP5 Form" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_objects</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>action</string> </key>
<value> <string>Base_edit</string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>edit_order</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>encoding</string> </key>
<value> <string>UTF-8</string> </value>
</item>
<item>
<key> <string>enctype</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>group_list</string> </key>
<value>
<list>
<string>left</string>
<string>right</string>
<string>center</string>
<string>bottom</string>
<string>hidden</string>
</list>
</value>
</item>
<item>
<key> <string>groups</string> </key>
<value>
<dictionary>
<item>
<key> <string>bottom</string> </key>
<value>
<list>
<string>listbox</string>
</list>
</value>
</item>
<item>
<key> <string>center</string> </key>
<value>
<list>
<string>my_description</string>
</list>
</value>
</item>
<item>
<key> <string>hidden</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>left</string> </key>
<value>
<list>
<string>my_title</string>
<string>my_reference</string>
<string>my_version</string>
<string>my_publication_section_list</string>
<string>my_array_dtype</string>
<string>my_array_shape</string>
</list>
</value>
</item>
<item>
<key> <string>right</string> </key>
<value>
<list>
<string>my_start_date</string>
<string>my_predecessor_title_list</string>
<string>my_causality_title</string>
<string>my_translated_validation_state_title</string>
</list>
</value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>DataArray_view</string> </value>
</item>
<item>
<key> <string>method</string> </key>
<value> <string>POST</string> </value>
</item>
<item>
<key> <string>name</string> </key>
<value> <string>DataArray_view</string> </value>
</item>
<item>
<key> <string>pt</string> </key>
<value> <string>form_view</string> </value>
</item>
<item>
<key> <string>row_length</string> </key>
<value> <int>4</int> </value>
</item>
<item>
<key> <string>stored_encoding</string> </key>
<value> <string>UTF-8</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Data Array</string> </value>
</item>
<item>
<key> <string>unicode_mode</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>update_action</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>update_action_title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataArray_view/listbox.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ProxyField" module="Products.ERP5Form.ProxyField"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>delegated_list</string> </key>
<value>
<list>
<string>columns</string>
<string>list_method</string>
<string>portal_types</string>
<string>selection_name</string>
<string>title</string>
</list>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>listbox</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>columns</string> </key>
<value>
<list>
<tuple>
<string>reference</string>
<string>Reference</string>
</tuple>
<tuple>
<string>dtype</string>
<string>Dtype</string>
</tuple>
<tuple>
<string>index_expression</string>
<string>Index Expression</string>
</tuple>
</list>
</value>
</item>
<item>
<key> <string>field_id</string> </key>
<value> <string>my_view_mode_listbox</string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string>Base_viewFieldLibrary</string> </value>
</item>
<item>
<key> <string>list_method</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value>
<list>
<tuple>
<string>Data Acquisition Sensor</string>
<string>Data Acquisition Sensor</string>
</tuple>
</list>
</value>
</item>
<item>
<key> <string>portal_types</string> </key>
<value>
<list>
<tuple>
<string>Data Array Line</string>
<string>Data Array Line</string>
</tuple>
</list>
</value>
</item>
<item>
<key> <string>selection_name</string> </key>
<value> <string>data_array_line</string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string>Click to edit the target</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Data Array Lines</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="Method" module="Products.Formulator.MethodField"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>method_name</string> </key>
<value> <string>objectValues</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataArray_view/my_array_dtype.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="StringField" module="Products.Formulator.StandardFields"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>id</string> </key>
<value> <string>my_array_dtype</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
<item>
<key> <string>required_not_found</string> </key>
<value> <string>Input is required but no input given.</string> </value>
</item>
<item>
<key> <string>too_long</string> </key>
<value> <string>Too much input was given.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_maxwidth</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_width</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>extra</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>input_type</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>max_length</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>truncate</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>unicode</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>whitespace_preserve</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_maxwidth</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_width</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>extra</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>input_type</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>max_length</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>truncate</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>unicode</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>whitespace_preserve</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_maxwidth</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_width</string> </key>
<value> <int>20</int> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>extra</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>input_type</string> </key>
<value> <string>text</string> </value>
</item>
<item>
<key> <string>max_length</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Dtype</string> </value>
</item>
<item>
<key> <string>truncate</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>unicode</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>whitespace_preserve</string> </key>
<value> <int>0</int> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="TALESMethod" module="Products.Formulator.TALESField"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_text</string> </key>
<value> <string>python: str(here.getArrayDtype())</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataArray_view/my_array_shape.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="StringField" module="Products.Formulator.StandardFields"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>id</string> </key>
<value> <string>my_array_shape</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
<item>
<key> <string>required_not_found</string> </key>
<value> <string>Input is required but no input given.</string> </value>
</item>
<item>
<key> <string>too_long</string> </key>
<value> <string>Too much input was given.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_maxwidth</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_width</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>extra</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>input_type</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>max_length</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>truncate</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>unicode</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>whitespace_preserve</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_maxwidth</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_width</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>extra</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>input_type</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>max_length</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>truncate</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>unicode</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>whitespace_preserve</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_maxwidth</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_width</string> </key>
<value> <int>20</int> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>extra</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>input_type</string> </key>
<value> <string>text</string> </value>
</item>
<item>
<key> <string>max_length</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Shape</string> </value>
</item>
<item>
<key> <string>truncate</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>unicode</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>whitespace_preserve</string> </key>
<value> <int>0</int> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataArray_view/my_causality_title.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ProxyField" module="Products.ERP5Form.ProxyField"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>delegated_list</string> </key>
<value>
<list>
<string>portal_type</string>
<string>title</string>
</list>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>my_causality_title</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string>my_relation_field</string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string>Base_viewFieldLibrary</string> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value>
<list>
<tuple>
<string>Compute Node</string>
<string>Compute Node</string>
</tuple>
</list>
</value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Compute Node</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataArray_view/my_description.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ProxyField" module="Products.ERP5Form.ProxyField"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>delegated_list</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>my_description</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>extra_context</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>extra_context</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>extra_context</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>field_id</string> </key>
<value> <string>my_description</string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string>Base_viewPDMFieldLibrary</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataArray_view/my_predecessor_title_list.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ProxyField" module="Products.ERP5Form.ProxyField"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>delegated_list</string> </key>
<value>
<list>
<string>height</string>
<string>portal_type</string>
<string>title</string>
<string>view_separator</string>
</list>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>my_predecessor_title_list</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string>my_multi_relation_field</string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string>Base_viewFieldLibrary</string> </value>
</item>
<item>
<key> <string>height</string> </key>
<value> <int>5</int> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value>
<list>
<tuple>
<string>Data Array</string>
<string>Data Array</string>
</tuple>
</list>
</value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Predecessors</string> </value>
</item>
<item>
<key> <string>view_separator</string> </key>
<value> <string encoding="cdata"><![CDATA[
<br />
]]></string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataArray_view/my_publication_section_list.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ProxyField" module="Products.ERP5Form.ProxyField"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>delegated_list</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>my_publication_section_list</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string>my_view_mode_publication_section_list</string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string>Base_viewDMSFieldLibrary</string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string>Click to edit the target</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataArray_view/my_reference.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ProxyField" module="Products.ERP5Form.ProxyField"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>delegated_list</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>my_reference</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string>my_view_mode_reference</string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string>Base_viewDMSFieldLibrary</string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string>Click to edit the target</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataArray_view/my_start_date.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="DateTimeField" module="Products.Formulator.StandardFields"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>id</string> </key>
<value> <string>my_start_date</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>datetime_out_of_range</string> </key>
<value> <string>The date and time you entered were out of range.</string> </value>
</item>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
<item>
<key> <string>not_datetime</string> </key>
<value> <string>You did not enter a valid date and time.</string> </value>
</item>
<item>
<key> <string>required_not_found</string> </key>
<value> <string>Input is required but no input given.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>allow_empty_time</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>ampm_time_style</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>date_only</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>date_separator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default_now</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default_timezone</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>end_datetime</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden_day_is_last_day</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hide_day</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>input_order</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>input_style</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>start_datetime</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>time_separator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>timezone_style</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>allow_empty_time</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>ampm_time_style</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>date_only</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>date_separator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default_now</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default_timezone</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>end_datetime</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden_day_is_last_day</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hide_day</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>input_order</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>input_style</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>start_datetime</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>time_separator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>timezone_style</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>allow_empty_time</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>ampm_time_style</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>date_only</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>date_separator</string> </key>
<value> <string>/</string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>default_now</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>default_timezone</string> </key>
<value> <string>GMT</string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>end_datetime</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>hidden_day_is_last_day</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>hide_day</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>input_order</string> </key>
<value> <string>ymd</string> </value>
</item>
<item>
<key> <string>input_style</string> </key>
<value> <string>text</string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>start_datetime</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>time_separator</string> </key>
<value> <string>:</string> </value>
</item>
<item>
<key> <string>timezone_style</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Start Date</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataArray_view/my_title.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ProxyField" module="Products.ERP5Form.ProxyField"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>delegated_list</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>my_title</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string>my_view_mode_title</string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string>Base_viewFieldLibrary</string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string>Click to edit the target</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataArray_view/my_translated_validation_state_title.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ProxyField" module="Products.ERP5Form.ProxyField"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>delegated_list</string> </key>
<value>
<list>
<string>title</string>
</list>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>my_translated_validation_state_title</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string>my_view_mode_translated_workflow_state_title</string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string>Base_viewFieldLibrary</string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string>Click to edit the target</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>State</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataArray_view/my_version.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ProxyField" module="Products.ERP5Form.ProxyField"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>delegated_list</string> </key>
<value>
<list>
<string>description</string>
</list>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>my_version</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>description</string> </key>
<value> <string>Version of the web page</string> </value>
</item>
<item>
<key> <string>field_id</string> </key>
<value> <string>my_view_mode_version</string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string>Base_viewFieldLibrary</string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string>Click to edit the target</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataArray_viewMappedContent.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ERP5 Form" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_objects</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>action</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>action_title</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>edit_order</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>encoding</string> </key>
<value> <string>UTF-8</string> </value>
</item>
<item>
<key> <string>enctype</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>group_list</string> </key>
<value>
<list>
<string>left</string>
<string>right</string>
<string>center</string>
<string>bottom</string>
<string>hidden</string>
</list>
</value>
</item>
<item>
<key> <string>groups</string> </key>
<value>
<dictionary>
<item>
<key> <string>bottom</string> </key>
<value>
<list>
<string>listbox</string>
</list>
</value>
</item>
<item>
<key> <string>center</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>hidden</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>left</string> </key>
<value>
<list>
<string>my_title</string>
</list>
</value>
</item>
<item>
<key> <string>right</string> </key>
<value>
<list/>
</value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>DataArray_viewMappedContent</string> </value>
</item>
<item>
<key> <string>method</string> </key>
<value> <string>POST</string> </value>
</item>
<item>
<key> <string>name</string> </key>
<value> <string>DataArray_viewMappedContent</string> </value>
</item>
<item>
<key> <string>pt</string> </key>
<value> <string>form_view</string> </value>
</item>
<item>
<key> <string>row_length</string> </key>
<value> <int>4</int> </value>
</item>
<item>
<key> <string>stored_encoding</string> </key>
<value> <string>UTF-8</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Mapped Content</string> </value>
</item>
<item>
<key> <string>unicode_mode</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>update_action</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>update_action_title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataArray_viewMappedContent/listbox.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ProxyField" module="Products.ERP5Form.ProxyField"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>delegated_list</string> </key>
<value>
<list>
<string>columns</string>
<string>editable</string>
<string>hidden</string>
<string>list_method</string>
<string>title</string>
<string>url_columns</string>
</list>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>listbox</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>columns</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value>
</item>
<item>
<key> <string>list_method</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>url_columns</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>columns</string> </key>
<value>
<list>
<tuple>
<string>title</string>
<string>title</string>
</tuple>
</list>
</value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>field_id</string> </key>
<value> <string>my_view_mode_listbox</string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string>Base_viewFieldLibrary</string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>list_method</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAQ=</string> </persistent>
</value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string>Click to edit the target</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Content</string> </value>
</item>
<item>
<key> <string>url_columns</string> </key>
<value>
<list/>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="TALESMethod" module="Products.Formulator.TALESField"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_text</string> </key>
<value> <string>python: here.DataArray_getArrayColumnList()</string> </value>
</item>
</dictionary>
</pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle>
<global name="TALESMethod" module="Products.Formulator.TALESField"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_text</string> </key>
<value> <string>python: context.getArrayShape()[0] == 0</string> </value>
</item>
</dictionary>
</pickle>
</record>
<record id="4" aka="AAAAAAAAAAQ=">
<pickle>
<global name="Method" module="Products.Formulator.MethodField"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>method_name</string> </key>
<value> <string>DataArray_getArrayFileInfoList</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataArray_viewMappedContent/my_title.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ProxyField" module="Products.ERP5Form.ProxyField"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>delegated_list</string> </key>
<value>
<list>
<string>editable</string>
</list>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>my_title</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>editable</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>field_id</string> </key>
<value> <string>my_view_mode_title</string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string>Base_viewFieldLibrary</string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string>Click to edit the target</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataMappingModule_viewDataMappingList.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ERP5 Form" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_objects</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>action</string> </key>
<value> <string>Base_doSelect</string> </value>
</item>
<item>
<key> <string>action_title</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>edit_order</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>encoding</string> </key>
<value> <string>UTF-8</string> </value>
</item>
<item>
<key> <string>enctype</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>group_list</string> </key>
<value>
<list>
<string>left</string>
<string>right</string>
<string>center</string>
<string>bottom</string>
<string>hidden</string>
</list>
</value>
</item>
<item>
<key> <string>groups</string> </key>
<value>
<dictionary>
<item>
<key> <string>bottom</string> </key>
<value>
<list>
<string>listbox</string>
</list>
</value>
</item>
<item>
<key> <string>center</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>hidden</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>left</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>right</string> </key>
<value>
<list/>
</value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>DataMappingModule_viewDataMappingList</string> </value>
</item>
<item>
<key> <string>method</string> </key>
<value> <string>POST</string> </value>
</item>
<item>
<key> <string>name</string> </key>
<value> <string>DataMappingModule_viewDataMappingList</string> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>ERP5 Form</string> </value>
</item>
<item>
<key> <string>pt</string> </key>
<value> <string>form_list</string> </value>
</item>
<item>
<key> <string>row_length</string> </key>
<value> <int>4</int> </value>
</item>
<item>
<key> <string>stored_encoding</string> </key>
<value> <string>UTF-8</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Data Mappings</string> </value>
</item>
<item>
<key> <string>unicode_mode</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>update_action</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>update_action_title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataMappingModule_viewDataMappingList/listbox.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ProxyField" module="Products.ERP5Form.ProxyField"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>delegated_list</string> </key>
<value>
<list>
<string>columns</string>
<string>selection_name</string>
<string>title</string>
</list>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>listbox</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>columns</string> </key>
<value>
<list>
<tuple>
<string>title</string>
<string>Title</string>
</tuple>
<tuple>
<string>reference</string>
<string>Reference</string>
</tuple>
<tuple>
<string>translated_validation_state_title</string>
<string>State</string>
</tuple>
</list>
</value>
</item>
<item>
<key> <string>field_id</string> </key>
<value> <string>my_list_mode_listbox</string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string>Base_viewFieldLibrary</string> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value>
<list>
<tuple>
<string>Data Mapping</string>
<string>Data Mapping</string>
</tuple>
</list>
</value>
</item>
<item>
<key> <string>selection_name</string> </key>
<value> <string>data_mapping_module_selection</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Data Mappings</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataMapping_view.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ERP5 Form" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_objects</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>action</string> </key>
<value> <string>Base_edit</string> </value>
</item>
<item>
<key> <string>action_title</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>edit_order</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>encoding</string> </key>
<value> <string>UTF-8</string> </value>
</item>
<item>
<key> <string>enctype</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>group_list</string> </key>
<value>
<list>
<string>left</string>
<string>right</string>
<string>center</string>
<string>bottom</string>
<string>hidden</string>
</list>
</value>
</item>
<item>
<key> <string>groups</string> </key>
<value>
<dictionary>
<item>
<key> <string>bottom</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>center</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>hidden</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>left</string> </key>
<value>
<list>
<string>my_title</string>
<string>my_reference</string>
</list>
</value>
</item>
<item>
<key> <string>right</string> </key>
<value>
<list>
<string>my_translated_validation_state_title</string>
</list>
</value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>DataMapping_view</string> </value>
</item>
<item>
<key> <string>method</string> </key>
<value> <string>POST</string> </value>
</item>
<item>
<key> <string>name</string> </key>
<value> <string>DataMapping_view</string> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>ERP5 Form</string> </value>
</item>
<item>
<key> <string>pt</string> </key>
<value> <string>form_view</string> </value>
</item>
<item>
<key> <string>row_length</string> </key>
<value> <int>4</int> </value>
</item>
<item>
<key> <string>stored_encoding</string> </key>
<value> <string>UTF-8</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Data Mapping</string> </value>
</item>
<item>
<key> <string>unicode_mode</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>update_action</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>update_action_title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataMapping_view/my_reference.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="StringField" module="Products.Formulator.StandardFields"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>id</string> </key>
<value> <string>my_reference</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
<item>
<key> <string>required_not_found</string> </key>
<value> <string>Input is required but no input given.</string> </value>
</item>
<item>
<key> <string>too_long</string> </key>
<value> <string>Too much input was given.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_maxwidth</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_width</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>extra</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>input_type</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>max_length</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>truncate</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>unicode</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>whitespace_preserve</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_maxwidth</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_width</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>extra</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>input_type</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>max_length</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>truncate</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>unicode</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>whitespace_preserve</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_maxwidth</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_width</string> </key>
<value> <int>20</int> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>extra</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>input_type</string> </key>
<value> <string>text</string> </value>
</item>
<item>
<key> <string>max_length</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Reference</string> </value>
</item>
<item>
<key> <string>truncate</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>unicode</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>whitespace_preserve</string> </key>
<value> <int>0</int> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataMapping_view/my_title.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="StringField" module="Products.Formulator.StandardFields"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>id</string> </key>
<value> <string>my_title</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
<item>
<key> <string>required_not_found</string> </key>
<value> <string>Input is required but no input given.</string> </value>
</item>
<item>
<key> <string>too_long</string> </key>
<value> <string>Too much input was given.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_maxwidth</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_width</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>extra</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>input_type</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>max_length</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>truncate</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>unicode</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>whitespace_preserve</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_maxwidth</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_width</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>extra</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>input_type</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>max_length</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>truncate</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>unicode</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>whitespace_preserve</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_maxwidth</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_width</string> </key>
<value> <int>20</int> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>extra</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>input_type</string> </key>
<value> <string>text</string> </value>
</item>
<item>
<key> <string>max_length</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Title</string> </value>
</item>
<item>
<key> <string>truncate</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>unicode</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>whitespace_preserve</string> </key>
<value> <int>0</int> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/DataMapping_view/my_translated_validation_state_title.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ProxyField" module="Products.ERP5Form.ProxyField"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>delegated_list</string> </key>
<value>
<list/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>my_translated_validation_state_title</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>field_id</string> </key>
<value> <string>my_translated_workflow_state_title</string> </value>
</item>
<item>
<key> <string>form_id</string> </key>
<value> <string>Base_viewFieldLibrary</string> </value>
</item>
<item>
<key> <string>target</string> </key>
<value> <string>Click to edit the target</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/ERP5Site_createDataAnalysisList.py 0 → 100644
View file @ 87a831cf
from DateTime import DateTime
from Products.ZSQLCatalog.SQLCatalog import AndQuery, OrQuery, Query
from Products.ERP5Type.Errors import UnsupportedWorkflowMethod
portal = context.getPortalObject()
portal_catalog = portal.portal_catalog
today = DateTime().strftime('%d%m%Y')
if not include_delivered:
batch_simulation_state = "stopped"
stream_simulation_state = "started"
else:
batch_simulation_state = ["stopped", "delivered"]
stream_simulation_state = ["started", "stopped", "delivered"]
query = AndQuery(
Query(portal_type = ["Data Ingestion Line", "Data Analysis Line"]),
Query(**{"stock.quantity": "!=0"}),
Query(resource_portal_type = ("Data Product", "Compute Node")),
# Should be improved to support mor than one analysis per ingestion
#SimpleQuery(parent_causality_related_relative_url = None),
OrQuery(Query(simulation_state = batch_simulation_state,
use_relative_url = "use/big_data/ingestion/batch"),
Query(simulation_state = stream_simulation_state,
use_relative_url = "use/big_data/ingestion/stream")))
for movement in portal_catalog(query = query):
if movement.getQuantity() <= 0:
continue
if movement.DataIngestionLine_hasMissingRequiredItem():
raise ValueError("Transformation requires movement to have " +
"aggregated data ingestion batch")
delivery = movement.getParentValue()
data_supply = delivery.getSpecialiseValue(portal_type="Data Supply")
data_supply_list = delivery.getSpecialiseValueList(portal_type="Data Supply")
composed_data_supply = data_supply.asComposedDocument()
# Get applicable transformation
transformation_list = []
for transformation in composed_data_supply.getSpecialiseValueList(portal_type="Data Transformation"):
for line in transformation.objectValues():
if line.getResourceValue() == movement.getResourceValue() and line.getQuantity() < 0:
transformation_list.append(transformation)
break
for transformation in portal.portal_catalog(
portal_type = "Data Transformation",
validation_state = "validated",
resource_relative_url = movement.getResource()):
# Variation - only with resolution for now
if not transformation.getVariationCategoryList() or transformation.getVariationCategoryList() == movement.getVariationCategoryList():
transformation_list.append(transformation)
for transformation in transformation_list:
is_shared_data_analysis = False
data_analysis_id= "%s-%s-%s" % (today, delivery.getId(), transformation.getId())
# Check if analysis already exists
data_analysis = portal_catalog.getResultValue(
portal_type="Data Analysis",
specialise_relative_url = transformation.getRelativeUrl(),
causality_relative_url = delivery.getRelativeUrl())
# search again with ID in case data_analysis is not indexed yet
if data_analysis is None:
try:
data_analysis = portal.data_analysis_module[data_analysis_id]
except KeyError:
pass
if data_analysis is not None:
continue
# for first level analysis check if same kind of data analysis with same project and same source already exists
# If yes, then later add additional input lines to this shared data analysis
if delivery.getPortalType() == "Data Ingestion":
data_analysis = portal_catalog.getResultValue(
portal_type="Data Analysis",
specialise_relative_url = transformation.getRelativeUrl(),
source_relative_url = delivery.getSource(),
destination_project_relative_url = delivery.getDestinationProject())
if data_analysis is not None:
data_analysis.setDefaultCausalityValue(delivery)
data_analysis.setSpecialiseValueSet(data_analysis.getSpecialiseValueList() + data_supply_list)
is_shared_data_analysis = True
else:
# Create Analysis
data_analysis = portal.data_analysis_module.newContent(
portal_type = "Data Analysis",
title = transformation.getTitle(),
reference = delivery.getReference(),
start_date = delivery.getStartDate(),
stop_date = delivery.getStopDate(),
specialise_value_list = [transformation] + data_supply_list,
causality_value = delivery,
source = delivery.getSource(),
source_section = delivery.getSourceSection(),
source_project = delivery.getSourceProject(),
destination = delivery.getDestination(),
destination_section = delivery.getDestinationSection(),
destination_project = delivery.getDestinationProject(),
id=data_analysis_id)
data_analysis.checkConsistency(fixit=True)
# create input and output lines
for transformation_line in transformation.objectValues(
portal_type=["Data Transformation Resource Line",
"Data Transformation Operation Line"]):
resource = transformation_line.getResourceValue()
quantity = transformation_line.getQuantity()
if isinstance(quantity, tuple):
quantity = quantity[0]
# In case of shared data anylsis only add additional input lines
if is_shared_data_analysis and quantity > -1:
continue
aggregate_set = set()
# manually add device to every line
aggregate_set.add(movement.getAggregateDevice())
# If it is batch processing we additionally get items from the other
# batch movements and deliver the other batch movements
if transformation_line.getUse() == "big_data/ingestion/batch" and \
transformation_line.getPortalType() == \
"Data Transformation Resource Line" and quantity < 0:
batch_relative_url = movement.getAggregateDataIngestionBatch()
if batch_relative_url is not None:
related_movement_list = portal_catalog(
portal_type="Data Ingestion Line",
aggregate_relative_url=batch_relative_url,
resource_relative_url = resource.getRelativeUrl())
for related_movement in related_movement_list:
if "big_data/ingestion/batch" in related_movement.getUseList():
related_movement.getParentValue().deliver()
# create new item based on item_type if it is not already aggregated
aggregate_type_set = set(
[portal.restrictedTraverse(a).getPortalType() for a in aggregate_set])
for item_type in transformation_line.getAggregatedPortalTypeList():
# if item is not yet aggregated to this line, search it by related project
# and source If the item is a data configuration or a device configuration
# then we do not care the workflow state nor the related resource, nor
# the variation nor the related sensor. Data Array Lines are created
# by Data Operation.
if all(
[
# Do not create item if it is a Data Array Line, then it is created by data operation itself.
item_type not in aggregate_type_set,
# Do not create item if it is a transient Data Array.
not (item_type == "Data Array" and "big_data/analysis/transient" in transformation_line.getUseList()),
]
):
item = None
if any(
(
item_type in portal.getPortalDeviceConfigurationTypeList() + portal.getPortalDataConfigurationTypeList(),
item_type == "Progress Indicator"
)
):
if item_type not in ("Status Configuration", "Progress Indicator"):
item = portal.portal_catalog.getResultValue(
portal_type=item_type,
#validation_state="validated",
item_project_relative_url=delivery.getDestinationProject(),
item_source_relative_url=delivery.getSource())
elif item_type != "Data Array Line":
item_query_dict = dict(
portal_type=item_type,
validation_state="validated",
item_variation_text=transformation_line.getVariationText(),
item_device_relative_url=movement.getAggregateDevice(),
item_resource_uid=resource.getUid(),
item_source_relative_url=data_analysis.getSource())
if data_analysis.getDestinationProjectValue() is not None:
item_query_dict["item_project_relative_url"] = data_analysis.getDestinationProject()
item = portal.portal_catalog.getResultValue(**item_query_dict)
if item is None:
module = portal.getDefaultModule(item_type)
item = module.newContent(portal_type = item_type,
title = transformation.getTitle(),
reference = "%s-%s" %(transformation.getTitle(),
delivery.getReference()),
version = '001')
try:
item.validate()
except AttributeError:
pass
aggregate_set.add(item.getRelativeUrl())
tag = "%s-%s" %(data_analysis.getUid(), transformation_line.getUid())
data_analysis_line = data_analysis.newContent(
activate_kw={'tag': tag},
portal_type = "Data Analysis Line",
title = transformation_line.getTitle(),
reference = transformation_line.getReference(),
int_index = transformation_line.getIntIndex(),
resource_value = resource,
variation_category_list = transformation_line.getVariationCategoryList(),
quantity = quantity,
quantity_unit = transformation_line.getQuantityUnit(),
use_list = transformation_line.getUseList(),
aggregate_set = aggregate_set)
# for intput lines of first level analysis set causality and specialise
if quantity < 0 and delivery.getPortalType() == "Data Ingestion":
data_analysis_line.edit(
causality_value = delivery,
specialise_value_list = data_supply_list)
# fix consistency of line and all affected items. Do it after reindexing
# activities of newly created Data Analysis Line finished, because check
# consistency script might need to find the newly created Data Analysis
# Line in catalog.
data_analysis_line.checkConsistency(fixit=True)
for item in data_analysis_line.getAggregateValueList():
item.activate(after_tag=tag).checkConsistency(fixit=True)
try:
data_analysis.start()
except UnsupportedWorkflowMethod:
pass
master/bt5/slapos_abyss/SkinTemplateItem/portal_skins/slapos_abyss/ERP5Site_createDataAnalysisList.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>include_delivered=False</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>ERP5Site_createDataAnalysisList</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/TestTemplateItem/portal_components/test.erp5.testSlapOSAbyss.py 0 → 100644
View file @ 87a831cf
##############################################################################
#
# Copyright (c) 2002-2021 Nexedi SA and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
#
##############################################################################
from erp5.component.test.SlapOSTestCaseMixin import SlapOSTestCaseMixin
import json
class testSlapOSAbyss(SlapOSTestCaseMixin):
def afterSetUp(self):
super(testSlapOSAbyss, self).afterSetUp()
self.tic()
test_organisation = getattr(self.portal.organisation_module, 'test_organisation', None)
if not test_organisation:
test_organisation = self.portal.organisation_module.newContent(portal_type='Organisation', id='test_organisation')
self.compute_node_list = []
for compute_node_id in ('database_debian10', 'database_debian11', 'node_debian10', 'node_debian11'):
compute_node = getattr(self.portal.compute_node_module, compute_node_id, None)
if compute_node is None:
compute_node = self.portal.compute_node_module.newContent(
portal_type='Compute Node',
id=compute_node_id,
reference=compute_node_id)
self.tic()
data_supply = compute_node.ComputeNode_createDataSupply(batch=1)
self.compute_node_list.append(compute_node)
if not data_supply.getDestination():
data_supply.edit(
source_value = test_organisation,
source_section_value = test_organisation,
destination_value = test_organisation,
destination_section_value = test_organisation)
compute_node.ComputeNode_createDataTransformation(batch=1)
# last one is only used to check data inside data array test
self.compute_node_list.pop()
self.portal.compute_node_module['database_debian10'].edit(publication_section_list =['file_system_image/database_image', 'file_system_image/distribution/debian/debian10'])
self.portal.compute_node_module['database_debian11'].edit(publication_section_list =['file_system_image/database_image', 'file_system_image/distribution/debian/debian11'])
self.portal.compute_node_module['database_debian11'].edit(exclude_path_list=[])
self.portal.compute_node_module['node_debian10'].edit(publication_section_list =['file_system_image/node_image', 'file_system_image/distribution/debian/debian10'])
if not getattr(self.portal.portal_categories.publication_section.file_system_image.distribution, 'test_distribution', None):
test_distribution = self.portal.portal_categories.publication_section.file_system_image.distribution.newContent(portal_type='Category', id='test_distribution')
test_distribution.newContent(portal_category='Category', id='debian10', title='debian10', int_index=2)
test_distribution.newContent(portal_category='Category', id='debian11', title='debian11', int_index=1)
self.tic()
def beforeTearDown(self):
data_stream_id_list = []
data_ingestion_id_list = []
data_array_id_list = []
data_analysis_id_list = []
for compute_node in self.portal.portal_catalog(portal_type='Compute Node', reference=('database_debian10', 'database_debian11', 'node_debian10', 'node_debian11')):
data_analysis_line = compute_node.getResourceRelatedValue(portal_type='Data Analysis Line')
data_ingestion_line = compute_node.getResourceRelatedValue(portal_type='Data Ingestion Line')
data_array_list = compute_node.getCausalityRelatedValueList(portal_type='Data Array')
if data_ingestion_line:
data_stream = data_ingestion_line.getAggregateDataStreamValue()
data_ingestion_id_list.append(data_ingestion_line.getParentValue().getId())
if data_stream:
data_stream_id_list.append(data_stream.getId())
if data_analysis_line:
data_analysis_id_list.append(data_analysis_line.getParentValue().getId())
for data_array in data_array_list:
data_array_id_list.append(data_array.getId())
self.portal.data_stream_module.manage_delObjects(ids=data_stream_id_list)
self.portal.data_ingestion_module.manage_delObjects(ids=data_ingestion_id_list)
self.portal.data_array_module.manage_delObjects(ids=data_array_id_list)
self.portal.data_analysis_module.manage_delObjects(ids=data_analysis_id_list)
self.tic()
def _create_request_dict(self):
return {
'node_debian10': '{"beginning_date": "2022/11/15 17:07 CET"}\n\
{"mac_address": "fe:27:02:3d:26:26"}\n\
{"path": "/sysroot/home/test3/metadata-collect-agent/scan-filesystem/cython/command-line.main.pyx", "stat": {"st_dev": 65025, "st_ino": 150519, "st_mode": 33188, "st_nlink": 1, "st_uid": 1000, "st_gid": 1000, "st_rdev": 0, "st_size": 9153, "st_blksize": 4096, "st_blocks": 24, "st_atime": 1634293139, "st_mtime": 1632486702, "st_ctime": 1632486702, "st_atime_ns": 958685043, "st_mtime_ns": 839359789, "st_ctime_ns": 839359789}, "hash": {"md5": "8e29c0d260293bc592200a4ef37729e5", "sha1": "fed552c1e74f54275ba4f1106a51a3349e12bbda", "sha256": "776f9da4bc9ba9062c8ab9b8c0a2ab91ad204d6f1e1a7734be050c5d83db2a48", "sha512": "4c547b2c1b0cb76b6e960c21d34e758cd467158f7681042d80d2b7afdde697fdd33fa694e4075a2584bf18caa58a55ec3b15c4b3c13dc59288a216837f4a8d82"}}\n\
{}\n\
{"end_date": "2022/11/15 17:08 CET", "end_marker": "fluentbit_end"}\n',
'database_debian10': '{"beginning_date": "2022/11/15 17:07 CET"}\n\
{"mac_address": "fe:27:02:3d:26:26"}\n\
{}\n\
{"end_date": "2022/11/15 17:07 CET", "end_marker": "fluentbit_end"}\n',
'database_debian11': '{"beginning_date": "2022/11/15 17:07 CET"}\n\
{"mac_address": "fe:27:02:3d:26:26"}\n\
{"path": "/home/test3/metadata-collect-agent/scan-filesystem/cython", "stat": {"st_dev": 65025, "st_ino": 150513, "st_mode": 16877, "st_nlink": 6, "st_uid": 1000, "st_gid": 1000, "st_rdev": 0, "st_size": 4096, "st_blksize": 4096, "st_blocks": 8, "st_atime": 1634303447, "st_mtime": 1634303457, "st_ctime": 1634303457, "st_atime_ns": 659600793, "st_mtime_ns": 447628573, "st_ctime_ns": 447628573}}\n\
{"path": "/home/test3/metadata-collect-agent/scan-filesystem/cython/command-line.main.pyx", "stat": {"st_dev": 65025, "st_ino": 150519, "st_mode": 33188, "st_nlink": 1, "st_uid": 1000, "st_gid": 1000, "st_rdev": 0, "st_size": 9153, "st_blksize": 4096, "st_blocks": 24, "st_atime": 1634293139, "st_mtime": 1632486702, "st_ctime": 1632486702, "st_atime_ns": 958685043, "st_mtime_ns": 839359789, "st_ctime_ns": 839359789}, "hash": {"md5": "8e29c0d260293bc592200a4ef37729e5", "sha1": "fed552c1e74f54275ba4f1106a51a3349e12bbda", "sha256": "776f9da4bc9ba9062c8ab9b8c0a2ab91ad204d6f1e1a7734be050c5d83db2a48", "sha512": "4c547b2c1b0cb76b6e960c21d34e758cd467158f7681042d80d2b7afdde697fdd33fa694e4075a2584bf18caa58a55ec3b15c4b3c13dc59288a216837f4a8d82"}}\n\
{"path": "/home/test3/metadata-collect-agent/scan-filesystem/cython/test.main.pyx", "stat": {"st_dev": 65025, "st_ino": 150592, "st_mode": 33188, "st_nlink": 1, "st_uid": 1000, "st_gid": 1000, "st_rdev": 0, "st_size": 10359, "st_blksize": 4096, "st_blocks": 24, "st_atime": 1634286743, "st_mtime": 1632487276, "st_ctime": 1632487276, "st_atime_ns": 583684050, "st_mtime_ns": 164380968, "st_ctime_ns": 164380968}, "hash": {"md5": "a81f35167d92ba1bcafe643890a68d31", "sha1": "eb300bc4d66fc641237ffe43f990cda05431a73f", "sha256": "bd8a0403f0acf7fce29a8728e1efcbb26f8ca2ee663b12dffcd49ec729be692b", "sha512": "fe19cf16c194adc70bef79945f70904c6d76f12dbd684cb7415885addd923f76f57841c8a68ec350c26ab6abb3f1c7e74a69f4f650c6674702a638c62e29aa4f"}}\n\
{"path": "/home/test3/metadata-collect-agent/scan-filesystem/cython/parse_link_errors.py", "stat": {"st_dev": 65025, "st_ino": 148852, "st_mode": 33261, "st_nlink": 1, "st_uid": 1000, "st_gid": 1000, "st_rdev": 0, "st_size": 822, "st_blksize": 4096, "st_blocks": 8, "st_atime": 1634301617, "st_mtime": 1634301600, "st_ctime": 1634301600, "st_atime_ns": 70528398, "st_mtime_ns": 658499228, "st_ctime_ns": 658499228}, "hash": {"md5": "465ab32cdc7531623c1130211b30d20c", "sha1": "3439417bacc24c55db7f6b62256122cab5c4cdc7", "sha256": "64214702209370590c7976b599b5a4b1033e175460c9e8a8bb1c133614cd8dac", "sha512": "28b74c2b9e7b93d9e354c35d618cf94cb6285fc62bc1d94ce31fdc5a2d1fc42b8325bcf1292c71351422f66b2d121670b0dbaaffba2107de0ee5d4f19581c64d"}}\n\
{}\n\
{"end_date": "2022/11/15 17:08 CET", "end_marker": "fluentbit_end"}\n'
}
def _getRelatedDataStream(self, compute_node):
return self._getRelatedDataStreamList(compute_node)[0]
def _getRelatedDataStreamList(self, compute_node):
data_ingestion_line = compute_node.getResourceRelatedValue(portal_type='Data Ingestion Line')
if data_ingestion_line:
return data_ingestion_line.getAggregateValueList(portal_type='Data Stream')
def _getRelatedDataArrayList(self, compute_node):
data_array_list = compute_node.getCausalityRelatedValueList(portal_type='Data Array')
# ascending by creation date
return sorted(data_array_list, key=lambda x: x.getCreationDate())
def _ingestData(self, request_dict=None):
request = self.portal.REQUEST
if not request_dict:
request_dict = self._create_request_dict()
for reference in request_dict:
request.environ["REQUEST_METHOD"] = 'POST'
request.set('reference', reference)
request.set('data_chunk', request_dict[reference])
request.set('QUERY_STRING', 'ingestion_policy=metadata_upload')
self.portal.portal_slap.ingestData()
self.tic()
return request_dict
def test_data_ingestion(self):
request_dict = self._ingestData()
self.tic()
for compute_node in self.compute_node_list:
data_stream = self._getRelatedDataStream(compute_node)
self.assertEqual(request_dict[compute_node.getReference()], data_stream.getData())
def test_data_ingestion_create_only_one_data_stream_per_compute_node(self):
self._ingestData()
self.tic()
size_list = []
for compute_node in self.compute_node_list:
data_stream_list = self._getRelatedDataStreamList(compute_node)
self.assertEqual(len(data_stream_list), 1)
size_list.append(data_stream_list[0].getSize())
self._ingestData()
self.tic()
for i in range(3):
compute_node = self.compute_node_list[i]
data_stream_list = self._getRelatedDataStreamList(compute_node)
self.assertEqual(len(data_stream_list), 1)
# double size
self.assertEqual(data_stream_list[0].getSize(),size_list[i]*2)
def string_to_array(self, reference, string):
json_string_list = string.splitlines()[:-1]
data_list = [json.loads(json_string) for json_string in json_string_list]
if reference == 'node_debian10':
triplet_list = [("/".join([''] + data['path'].split('/')[2:]), data['hash']['sha256']) for data in data_list if 'path' in data and 'hash' in data and 'sha256' in data['hash']]
else:
triplet_list = [(data['path'], data['hash']['sha256']) for data in data_list if 'path' in data and 'hash' in data and 'sha256' in data['hash']]
data_mapping = self.portal.Base_getDataMapping()
uid_list = []
for triplet in triplet_list:
triplet_uid = data_mapping.addObject(triplet)
uid_list.append(triplet_uid)
return uid_list
def test_data_transformation(self):
request_dict = self._ingestData()
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
array_dict = {}
for reference in request_dict:
array_dict[reference] = self.string_to_array(reference, request_dict[reference])
for compute_node in self.compute_node_list:
data_array_list = self._getRelatedDataArrayList(compute_node)
self.assertEqual(len(data_array_list),1)
self.assertEqual(data_array_list[0].getArray()[:].tolist(), array_dict[compute_node.getReference()])
self.assertEqual(len(data_array_list[0].DataArray_getArrayFileInfoList()), len(array_dict[compute_node.getReference()]))
# no more data
for data_analysis in self.portal.portal_catalog(portal_type = "Data Analysis", simulation_state = "started"):
self.assertEqual(data_analysis.DataAnalysis_executeDataOperation(), None)
def test_data_transformation_with_exclude_path(self):
request = self.portal.REQUEST
request_dict = self._create_request_dict()
# only test compute node: database_debian11 is enough
request_dict.pop('node_debian10')
request_dict.pop('database_debian10')
self._ingestData(request_dict)
self.tic()
database_debian11 = self.portal.compute_node_module['database_debian11']
data_stream = self._getRelatedDataStream(database_debian11)
self.assertTrue(data_stream is not None)
database_debian11.edit(exclude_path_list=['/home/test3/metadata-collect-agent/scan-filesystem/cython/parse_link_errors.py'])
self.tic()
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
data_array = self._getRelatedDataArrayList(database_debian11)[0]
# /home/test3/metadata-collect-agent/scan-filesystem/cython/command-line.main.pyx, /home/test3/metadata-collect-agent/scan-filesystem/cython/test.main.pyx
self.assertEquals(len(data_array.getArray()), 2)
database_debian11.edit(exclude_path_list=['/home/test3/metadata-collect-agent/scan-filesystem/cython'])
self.tic()
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
request = self.portal.REQUEST
request.environ["REQUEST_METHOD"] = 'POST'
request.set('reference', 'database_debian11')
request.set('data_chunk', request_dict['database_debian11'])
request.set('QUERY_STRING', 'ingestion_policy=metadata_upload')
self.portal.portal_slap.ingestData()
self.tic()
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
new_array_list = self._getRelatedDataArrayList(database_debian11)
for new_array in new_array_list:
if 'file_system_image/process_state/converted' in new_array.getPublicationSectionList():
break
self.assertTrue('file_system_image/process_state/converted' in new_array.getPublicationSectionList())
self.assertEquals(len(new_array.getArray()), 0)
def test_data_processing_for_the_first_access(self):
self._ingestData()
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
for compute_node in self.compute_node_list:
# only one data array is created
data_array_list = self._getRelatedDataArrayList(compute_node)
self.assertEqual(len(data_array_list), 1)
self.assertEqual(data_array_list[0].getPublicationSectionList(), compute_node.getPublicationSectionList() + ['file_system_image/first_access', 'file_system_image/process_state/converted'])
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
for compute_node in self.compute_node_list:
# still one data array is created
data_array_list = self._getRelatedDataArrayList(compute_node)
self.assertEqual(len(data_array_list), 1)
self.assertEqual(data_array_list[0].getPublicationSectionList(), compute_node.getPublicationSectionList() + ['file_system_image/first_access', 'file_system_image/process_state/processed'])
def test_data_processing_no_data_array_is_created_if_previous_isnot_processed(self):
self._ingestData()
self.tic()
# now ingest different data for node_debian10
request_dict = {
'node_debian10': '{"beginning_date": "2022/11/15 17:07 CET"}\n\
{"mac_address": "fe:27:02:3d:26:26"}\n\
{"path": "/sysroot/home/test3/metadata-collect-agent/scan-filesystem/cython/command-line.main.pyx", "stat": {"st_dev": 65025, "st_ino": 150519, "st_mode": 33188, "st_nlink": 1, "st_uid": 1000, "st_gid": 1000, "st_rdev": 0, "st_size": 9153, "st_blksize": 4096, "st_blocks": 24, "st_atime": 1634293139, "st_mtime": 1632486702, "st_ctime": 1632486702, "st_atime_ns": 958685043, "st_mtime_ns": 839359789, "st_ctime_ns": 839359789}, "hash": {"md5": "8e29c0d260293bc592200a4ef37729e5", "sha1": "fed552c1e74f54275ba4f1106a51a3349e12bbda", "sha256": "VALUE_MODIFIED", "sha512": "4c547b2c1b0cb76b6e960c21d34e758cd467158f7681042d80d2b7afdde697fdd33fa694e4075a2584bf18caa58a55ec3b15c4b3c13dc59288a216837f4a8d82"}}\n\
{}\n\
{"end_date": "2022/11/15 17:08 CET", "end_marker": "fluentbit_end"}\n'
}
self._ingestData(request_dict)
self.tic()
# ingest data again
self._ingestData()
self.tic()
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
# now each compute node should have 1 data arrays
for compute_node in self.compute_node_list:
data_array_list = self._getRelatedDataArrayList(compute_node)
self.assertEqual(len(data_array_list), 1)
self.assertTrue("file_system_image/process_state/converted" in data_array_list[0].getPublicationSectionList())
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
# each compute node still should have 1 data arrays, no data array is created because previous one is not finished to process
for compute_node in self.compute_node_list:
data_array_list = self._getRelatedDataArrayList(compute_node)
self.assertEqual(len(data_array_list), 1)
self.assertTrue("file_system_image/process_state/converted" in data_array_list[0].getPublicationSectionList())
# process data array
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
# now each compute node should have 2 data arrays
for compute_node in self.compute_node_list:
data_array_list = self._getRelatedDataArrayList(compute_node)
self.assertEqual(len(data_array_list), 2)
self.assertTrue("file_system_image/process_state/converted" in data_array_list[-1].getPublicationSectionList())
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
# still the same
for compute_node in self.compute_node_list:
data_array_list = self._getRelatedDataArrayList(compute_node)
self.assertEqual(len(data_array_list), 2)
self.assertTrue("file_system_image/process_state/converted" in data_array_list[-1].getPublicationSectionList())
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
# 3 arrays, 2 previous and 1 diff with itself
data_array_list = self._getRelatedDataArrayList(self.portal.compute_node_module['node_debian10'])
self.assertEqual(len(data_array_list), 3)
self.assertTrue(data_array_list[-1].getPredecessorValue() is not None)
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
# 3 previous, 1 diff with database_debian10
data_array_list = self._getRelatedDataArrayList(self.portal.compute_node_module['node_debian10'])
self.assertEqual(len(data_array_list), 4)
self.assertTrue(data_array_list[-1].getPredecessorValue() is not None)
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
# 4 previous, 1 diff with database_debian11
data_array_list = self._getRelatedDataArrayList(self.portal.compute_node_module['node_debian10'])
self.assertTrue(data_array_list[-1].getPredecessorValue() is not None)
self.assertEqual(len(data_array_list), 5)
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
# still the same, previous not finished
data_array_list = self._getRelatedDataArrayList(self.portal.compute_node_module['node_debian10'])
self.assertEqual(len(data_array_list), 5)
# no more database to compare
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
data_array_list = self._getRelatedDataArrayList(self.portal.compute_node_module['node_debian10'])
self.assertEqual(len(data_array_list), 6)
self.assertTrue(data_array_list[-1].getPredecessorValue() is None)
def test_data_processing_for_sequence_data_ingestion(self):
# first access
self._ingestData()
self.tic()
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
# ingest data again
self._ingestData()
self.tic()
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
# now each compute node should have 2 data arrays
for compute_node in self.compute_node_list:
data_array_list = self._getRelatedDataArrayList(compute_node)
self.assertEqual(len(data_array_list), 2)
self.assertTrue("file_system_image/process_state/converted" in data_array_list[-1].getPublicationSectionList())
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
data_array_list = self._getRelatedDataArrayList(self.portal.compute_node_module['database_debian10'])
self.assertEqual(len(data_array_list), 2)
self.assertTrue("file_system_image/process_state/archived" in data_array_list[0].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[1].getPublicationSectionList())
data_array_list = self._getRelatedDataArrayList(self.portal.compute_node_module['database_debian11'])
self.assertEqual(len(data_array_list), 2)
self.assertTrue("file_system_image/process_state/archived" in data_array_list[0].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[1].getPublicationSectionList())
data_array_list = self._getRelatedDataArrayList(self.portal.compute_node_module['node_debian10'])
self.assertEqual(len(data_array_list), 2)
self.assertTrue("file_system_image/process_state/processed" in data_array_list[0].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[1].getPublicationSectionList())
self.assertTrue('file_system_image/diff_end/identical' in data_array_list[1].getPublicationSectionList())
# now ingest different data for node_debian10
request_dict = {
'node_debian10': '{"beginning_date": "2022/11/15 17:07 CET"}\n\
{"mac_address": "fe:27:02:3d:26:26"}\n\
{"path": "/sysroot/home/test3/metadata-collect-agent/scan-filesystem/cython/command-line.main.pyx", "stat": {"st_dev": 65025, "st_ino": 150519, "st_mode": 33188, "st_nlink": 1, "st_uid": 1000, "st_gid": 1000, "st_rdev": 0, "st_size": 9153, "st_blksize": 4096, "st_blocks": 24, "st_atime": 1634293139, "st_mtime": 1632486702, "st_ctime": 1632486702, "st_atime_ns": 958685043, "st_mtime_ns": 839359789, "st_ctime_ns": 839359789}, "hash": {"md5": "8e29c0d260293bc592200a4ef37729e5", "sha1": "fed552c1e74f54275ba4f1106a51a3349e12bbda", "sha256": "VALUE_MODIFIED", "sha512": "4c547b2c1b0cb76b6e960c21d34e758cd467158f7681042d80d2b7afdde697fdd33fa694e4075a2584bf18caa58a55ec3b15c4b3c13dc59288a216837f4a8d82"}}\n\
{}\n\
{"end_date": "2022/11/15 17:08 CET", "end_marker": "fluentbit_end"}\n'
}
self._ingestData(request_dict)
self.tic()
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
data_array_list = self._getRelatedDataArrayList(self.portal.compute_node_module['node_debian10'])
self.assertEqual(len(data_array_list), 3)
self.assertTrue("file_system_image/process_state/processed" in data_array_list[0].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[1].getPublicationSectionList())
# This is the new one with different value
self.assertTrue("file_system_image/process_state/converted" in data_array_list[2].getPublicationSectionList())
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
data_array_list = self._getRelatedDataArrayList(self.portal.compute_node_module['node_debian10'])
self.assertEqual(len(data_array_list), 4)
self.assertTrue("file_system_image/process_state/processed" in data_array_list[0].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[1].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[2].getPublicationSectionList())
# array 3 is the result of array 0 and array 2, Note: array 0 is the first access data
self.assertTrue("file_system_image/process_state/converted" in data_array_list[3].getPublicationSectionList())
predecessor_list = [x.getRelativeUrl() for x in data_array_list[3].getPredecessorValueList(portal_type='Data Array')]
self.assertTrue(data_array_list[0].getRelativeUrl() in predecessor_list)
self.assertTrue(data_array_list[2].getRelativeUrl() in predecessor_list)
# only have one diff
self.assertTrue(len(data_array_list[3].getArray()), 1)
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
# array 3 compare with database_debian10
data_array_list = self._getRelatedDataArrayList(self.portal.compute_node_module['node_debian10'])
self.assertEqual(len(data_array_list), 5)
self.assertTrue("file_system_image/process_state/processed" in data_array_list[0].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[1].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[2].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[3].getPublicationSectionList())
# array 4 is result of array 3 and database_debian10
self.assertTrue("file_system_image/process_state/converted" in data_array_list[4].getPublicationSectionList())
predecessor_list = [x.getRelativeUrl() for x in data_array_list[4].getPredecessorValueList(portal_type='Data Array')]
last_detabase_debian10 = self._getRelatedDataArrayList(self.portal.compute_node_module['database_debian10'])[-1]
self.assertTrue(last_detabase_debian10.getRelativeUrl() in predecessor_list)
self.assertTrue(data_array_list[3].getRelativeUrl() in predecessor_list)
self.assertTrue(len(data_array_list[4].getArray()), 1)
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
# array 4 compare with database_debian11
data_array_list = self._getRelatedDataArrayList(self.portal.compute_node_module['node_debian10'])
self.assertEqual(len(data_array_list), 6)
self.assertTrue("file_system_image/process_state/processed" in data_array_list[0].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[1].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[2].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[3].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[4].getPublicationSectionList())
# array 5 is result of array 4 and database_debian11
self.assertTrue("file_system_image/process_state/converted" in data_array_list[5].getPublicationSectionList())
predecessor_list = [x.getRelativeUrl() for x in data_array_list[5].getPredecessorValueList(portal_type='Data Array')]
last_detabase_debian11 = self._getRelatedDataArrayList(self.portal.compute_node_module['database_debian11'])[-1]
self.assertTrue(last_detabase_debian11.getRelativeUrl() in predecessor_list)
self.assertTrue(data_array_list[4].getRelativeUrl() in predecessor_list)
self.assertTrue(len(data_array_list[5].getArray()), 1)
# no more database to compare
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
self.assertTrue("file_system_image/process_state/processed" in data_array_list[5].getPublicationSectionList())
self.assertTrue('file_system_image/diff_end/different' in data_array_list[5].getPublicationSectionList())
def test_data_processing_for_multi_data_ingestion(self):
# almost same as test_data_processing_for_sequence_data_ingestion
# except ingest multi data instead of one by one
# 3 multi data ingestions for node debian10
# 2 multi data ingestions for database
self._ingestData()
self.tic()
# ingest data again
self._ingestData()
self.tic()
# now ingest different data for node_debian10
request_dict = {
'node_debian10': '{"beginning_date": "2022/11/15 17:07 CET"}\n\
{"mac_address": "fe:27:02:3d:26:26"}\n\
{"path": "/sysroot/home/test3/metadata-collect-agent/scan-filesystem/cython/command-line.main.pyx", "stat": {"st_dev": 65025, "st_ino": 150519, "st_mode": 33188, "st_nlink": 1, "st_uid": 1000, "st_gid": 1000, "st_rdev": 0, "st_size": 9153, "st_blksize": 4096, "st_blocks": 24, "st_atime": 1634293139, "st_mtime": 1632486702, "st_ctime": 1632486702, "st_atime_ns": 958685043, "st_mtime_ns": 839359789, "st_ctime_ns": 839359789}, "hash": {"md5": "8e29c0d260293bc592200a4ef37729e5", "sha1": "fed552c1e74f54275ba4f1106a51a3349e12bbda", "sha256": "VALUE_MODIFIED", "sha512": "4c547b2c1b0cb76b6e960c21d34e758cd467158f7681042d80d2b7afdde697fdd33fa694e4075a2584bf18caa58a55ec3b15c4b3c13dc59288a216837f4a8d82"}}\n\
{}\n\
{"end_date": "2022/11/15 17:08 CET", "end_marker": "fluentbit_end"}\n'
}
self._ingestData(request_dict)
self.tic()
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
# now each compute node should have 1 data arrays
for compute_node in self.compute_node_list:
data_array_list = self._getRelatedDataArrayList(compute_node)
self.assertEqual(len(data_array_list), 1)
self.assertTrue("file_system_image/process_state/converted" in data_array_list[0].getPublicationSectionList())
# process data array
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
data_array_list = self._getRelatedDataArrayList(self.portal.compute_node_module['database_debian10'])
self.assertEqual(len(data_array_list), 1)
self.assertTrue("file_system_image/process_state/processed" in data_array_list[0].getPublicationSectionList())
data_array_list = self._getRelatedDataArrayList(self.portal.compute_node_module['database_debian11'])
self.assertEqual(len(data_array_list), 1)
self.assertTrue("file_system_image/process_state/processed" in data_array_list[0].getPublicationSectionList())
data_array_list = self._getRelatedDataArrayList(self.portal.compute_node_module['node_debian10'])
self.assertEqual(len(data_array_list), 1)
self.assertTrue("file_system_image/process_state/processed" in data_array_list[0].getPublicationSectionList())
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
# now each compute node should have 2 data arrays
for compute_node in self.compute_node_list:
data_array_list = self._getRelatedDataArrayList(compute_node)
self.assertEqual(len(data_array_list), 2)
self.assertTrue("file_system_image/process_state/converted" in data_array_list[-1].getPublicationSectionList())
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
data_array_list = self._getRelatedDataArrayList(self.portal.compute_node_module['database_debian10'])
self.assertEqual(len(data_array_list), 2)
self.assertTrue("file_system_image/process_state/archived" in data_array_list[0].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[1].getPublicationSectionList())
data_array_list = self._getRelatedDataArrayList(self.portal.compute_node_module['database_debian11'])
self.assertEqual(len(data_array_list), 2)
self.assertTrue("file_system_image/process_state/archived" in data_array_list[0].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[1].getPublicationSectionList())
data_array_list = self._getRelatedDataArrayList(self.portal.compute_node_module['node_debian10'])
self.assertEqual(len(data_array_list), 2)
self.assertTrue("file_system_image/process_state/processed" in data_array_list[0].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[1].getPublicationSectionList())
self.assertTrue('file_system_image/diff_end/identical' in data_array_list[1].getPublicationSectionList())
self.portal.ERP5Site_createDataAnalysisList()
self.portal.ERP5Site_executeDataAnalysisList()
self.tic()
data_array_list = self._getRelatedDataArrayList(self.portal.compute_node_module['node_debian10'])
self.assertEqual(len(data_array_list), 3)
self.assertTrue("file_system_image/process_state/processed" in data_array_list[0].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[1].getPublicationSectionList())
# This is the new one with different value
self.assertTrue("file_system_image/process_state/converted" in data_array_list[2].getPublicationSectionList())
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
data_array_list = self._getRelatedDataArrayList(self.portal.compute_node_module['node_debian10'])
self.assertEqual(len(data_array_list), 4)
self.assertTrue("file_system_image/process_state/processed" in data_array_list[0].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[1].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[2].getPublicationSectionList())
# array 3 is the result of array 0 and array 2, Note: array 0 is the first access data
self.assertTrue("file_system_image/process_state/converted" in data_array_list[3].getPublicationSectionList())
predecessor_list = [x.getRelativeUrl() for x in data_array_list[3].getPredecessorValueList(portal_type='Data Array')]
self.assertTrue(data_array_list[0].getRelativeUrl() in predecessor_list)
self.assertTrue(data_array_list[2].getRelativeUrl() in predecessor_list)
# only have one diff
self.assertTrue(len(data_array_list[3].getArray()), 1)
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
# array 3 compare with database_debian10
data_array_list = self._getRelatedDataArrayList(self.portal.compute_node_module['node_debian10'])
self.assertEqual(len(data_array_list), 5)
self.assertTrue("file_system_image/process_state/processed" in data_array_list[0].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[1].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[2].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[3].getPublicationSectionList())
# array 4 is result of array 3 and database_debian10
self.assertTrue("file_system_image/process_state/converted" in data_array_list[4].getPublicationSectionList())
predecessor_list = [x.getRelativeUrl() for x in data_array_list[4].getPredecessorValueList(portal_type='Data Array')]
last_detabase_debian10 = self._getRelatedDataArrayList(self.portal.compute_node_module['database_debian10'])[-1]
self.assertTrue(last_detabase_debian10.getRelativeUrl() in predecessor_list)
self.assertTrue(data_array_list[3].getRelativeUrl() in predecessor_list)
self.assertTrue(len(data_array_list[4].getArray()), 1)
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
# array 4 compare with database_debian11
data_array_list = self._getRelatedDataArrayList(self.portal.compute_node_module['node_debian10'])
self.assertEqual(len(data_array_list), 6)
self.assertTrue("file_system_image/process_state/processed" in data_array_list[0].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[1].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[2].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[3].getPublicationSectionList())
self.assertTrue("file_system_image/process_state/processed" in data_array_list[4].getPublicationSectionList())
# array 5 is result of array 4 and database_debian11
self.assertTrue("file_system_image/process_state/converted" in data_array_list[5].getPublicationSectionList())
predecessor_list = [x.getRelativeUrl() for x in data_array_list[5].getPredecessorValueList(portal_type='Data Array')]
last_detabase_debian11 = self._getRelatedDataArrayList(self.portal.compute_node_module['database_debian11'])[-1]
self.assertTrue(last_detabase_debian11.getRelativeUrl() in predecessor_list)
self.assertTrue(data_array_list[4].getRelativeUrl() in predecessor_list)
self.assertTrue(len(data_array_list[5].getArray()), 1)
# no more database to compare
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
self.assertTrue("file_system_image/process_state/processed" in data_array_list[5].getPublicationSectionList())
self.assertTrue('file_system_image/diff_end/different' in data_array_list[5].getPublicationSectionList())
def test_data_processing_check_copmute_node_state(self):
self._ingestData()
self.tic()
self._ingestData()
self.tic()
# now ingest different data for node_debian10
request_dict = {
'node_debian10': '{"beginning_date": "2022/11/15 17:07 CET"}\n\
{"mac_address": "fe:27:02:3d:26:26"}\n\
{"path": "/sysroot/home/test3/metadata-collect-agent/scan-filesystem/cython/command-line.main.pyx", "stat": {"st_dev": 65025, "st_ino": 150519, "st_mode": 33188, "st_nlink": 1, "st_uid": 1000, "st_gid": 1000, "st_rdev": 0, "st_size": 9153, "st_blksize": 4096, "st_blocks": 24, "st_atime": 1634293139, "st_mtime": 1632486702, "st_ctime": 1632486702, "st_atime_ns": 958685043, "st_mtime_ns": 839359789, "st_ctime_ns": 839359789}, "hash": {"md5": "8e29c0d260293bc592200a4ef37729e5", "sha1": "fed552c1e74f54275ba4f1106a51a3349e12bbda", "sha256": "VALUE_MODIFIED", "sha512": "4c547b2c1b0cb76b6e960c21d34e758cd467158f7681042d80d2b7afdde697fdd33fa694e4075a2584bf18caa58a55ec3b15c4b3c13dc59288a216837f4a8d82"}}\n\
{}\n\
{"end_date": "2022/11/15 17:08 CET", "end_marker": "fluentbit_end"}\n'
}
self._ingestData(request_dict)
self.tic()
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
self.assertTrue(self.portal.compute_node_module['node_debian10'].ComputeNode_hasModifiedFile() is None)
self.portal.portal_alarms.wendelin_handle_analysis.activeSense()
self.tic()
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
# no more database to compare
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
self.assertTrue(self.portal.compute_node_module['node_debian10'].ComputeNode_hasModifiedFile() is not None)
def test_data_processing_check_value_in_data_array(self):
# create data array directly instead of tramsforming from data stream
# so we can easily check the value after each processing
server_uid_list_list = [
[7, 9, 10, 11, 15, 17, 18],
[2, 8, 14, 24],
[11, 17, 28, 31],
[2, 8, 14, 24],
[11, 17, 28, 31]
]
server_publication_list_list = [
['file_system_image/node_image', 'file_system_image/distribution/test_distribution/debian11'],
['file_system_image/node_image', 'file_system_image/distribution/test_distribution/debian11'],
['file_system_image/node_image', 'file_system_image/distribution/test_distribution/debian11'],
['file_system_image/node_image', 'file_system_image/distribution/test_distribution/debian10'],
['file_system_image/node_image', 'file_system_image/distribution/test_distribution/debian10']
]
server_uid_ndarray_list = []
for uid_list in server_uid_list_list:
server_uid_ndarray_list.append(uid_list)
reference_uid_list_list = [
[i for i in range(20, 30)],
[i for i in range(5, 15)]
]
reference_publication_section_list_list = [
['file_system_image/database_image', 'file_system_image/distribution/test_distribution/debian11'],
['file_system_image/database_image', 'file_system_image/distribution/test_distribution/debian10']
]
reference_uid_ndarray_list = []
for uid_list in reference_uid_list_list:
reference_uid_ndarray_list.append(uid_list)
# create empty first access data
tmp_data_array = self.portal.data_array_module.newContent(portal_type='Data Array', title='first_access_data')
tmp_data_array.setCausality('compute_node_module/node_debian10')
tmp_data_array.initArray(shape=(0,), dtype='int64')
tmp_data_array.setArray([])
tmp_data_array.setPublicationSectionList(['file_system_image/distribution/test_distribution/debian10', 'file_system_image/first_access', 'file_system_image/process_state/processed'])
tmp_data_array.validate()
tmp_data_array = self.portal.data_array_module.newContent(portal_type='Data Array', title='first_access_data')
tmp_data_array.setCausality('compute_node_module/node_debian11')
tmp_data_array.initArray(shape=(0,), dtype='int64')
tmp_data_array.setArray([])
tmp_data_array.setPublicationSectionList(['file_system_image/distribution/test_distribution/debian11', 'file_system_image/first_access', 'file_system_image/process_state/processed'])
tmp_data_array.validate()
self.tic()
server_data_array_list = []
for index, uid_ndarray in enumerate(server_uid_ndarray_list):
tmp_data_array = self.portal.data_array_module.newContent(portal_type='Data Array', title='%s-%s' % (server_publication_list_list[index][0],server_publication_list_list[index][1]))
if server_publication_list_list[index][1].endswith('debian10'):
tmp_data_array.setCausality('compute_node_module/node_debian10')
else:
tmp_data_array.setCausality('compute_node_module/node_debian11')
tmp_data_array.initArray(shape=(len(uid_ndarray),), dtype='int64')
tmp_data_array.setArray(uid_ndarray)
tmp_data_array.setPublicationSectionList(server_publication_list_list[index] + ['file_system_image/process_state/converted'])
tmp_data_array.validate()
server_data_array_list.append(tmp_data_array)
self.tic()
reference_data_array_list = []
for index, uid_ndarray in enumerate(reference_uid_ndarray_list):
tmp_data_array = self.getPortalObject().data_array_module.newContent(portal_type='Data Array', title='%s-%s' % (reference_publication_section_list_list[index][0], reference_publication_section_list_list[index][1]))
# mark it, so it can be deleted later
tmp_data_array.setCausality('compute_node_module/node_debian10')
tmp_data_array.initArray(shape=(len(uid_ndarray),), dtype='int64')
tmp_data_array.setArray(uid_ndarray)
tmp_data_array.setPublicationSectionList(reference_publication_section_list_list[index] + ['file_system_image/process_state/converted'])
reference_data_array_list.append(tmp_data_array)
tmp_data_array.validate()
self.tic()
# compare with itself
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
diff_server_data_array_list = []
for i in server_data_array_list:
self.assertTrue("file_system_image/process_state/processed" in i.getPublicationSectionList())
diff_server_data_array_list.append(i.getPredecessorRelatedValue(portal_type='Data Array'))
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
self.assertTrue(diff_server_data_array_list[0].getPredecessorRelatedValue(portal_type='Data Array') in reference_data_array_list[0].getPredecessorRelatedValueList(portal_type='Data Array'))
diff_1 = diff_server_data_array_list[0].getPredecessorRelatedValue(portal_type='Data Array')
self.assertEqual(len(diff_1.getPredecessorValueList(portal_type='Data Array')), 2)
self.assertTrue("file_system_image/process_state/converted" in diff_1.getPublicationSectionList())
self.assertTrue((diff_1.getArray() == [7, 9, 10, 11, 15, 17, 18]).all())
self.assertTrue(diff_server_data_array_list[1].getPredecessorRelatedValue(portal_type='Data Array') in reference_data_array_list[0].getPredecessorRelatedValueList(portal_type='Data Array'))
diff_2 = diff_server_data_array_list[1].getPredecessorRelatedValue(portal_type='Data Array')
self.assertEqual(len(diff_2.getPredecessorValueList(portal_type='Data Array')), 2)
self.assertTrue("file_system_image/process_state/converted" in diff_2.getPublicationSectionList())
self.assertTrue((diff_2.getArray() == [2, 8, 14]).all())
self.assertTrue(diff_server_data_array_list[2].getPredecessorRelatedValue(portal_type='Data Array') in reference_data_array_list[0].getPredecessorRelatedValueList(portal_type='Data Array'))
diff_3 = diff_server_data_array_list[2].getPredecessorRelatedValue(portal_type='Data Array')
self.assertEqual(len(diff_3.getPredecessorValueList(portal_type='Data Array')), 2)
self.assertTrue("file_system_image/process_state/converted" in diff_3.getPublicationSectionList())
self.assertTrue((diff_3.getArray() == [11, 17, 31]).all())
self.assertTrue(diff_server_data_array_list[3].getPredecessorRelatedValue(portal_type='Data Array') in reference_data_array_list[1].getPredecessorRelatedValueList(portal_type='Data Array'))
diff_4 = diff_server_data_array_list[3].getPredecessorRelatedValue(portal_type='Data Array')
self.assertEqual(len(diff_4.getPredecessorValueList(portal_type='Data Array')), 2)
self.assertTrue("file_system_image/process_state/converted" in diff_4.getPublicationSectionList())
self.assertTrue((diff_4.getArray() == [2, 24]).all())
self.assertTrue(diff_server_data_array_list[4].getPredecessorRelatedValue(portal_type='Data Array') in reference_data_array_list[1].getPredecessorRelatedValueList(portal_type='Data Array'))
diff_5 = diff_server_data_array_list[4].getPredecessorRelatedValue(portal_type='Data Array')
self.assertEqual(len(diff_5.getPredecessorValueList(portal_type='Data Array')), 2)
self.assertTrue("file_system_image/process_state/converted" in diff_5.getPublicationSectionList())
self.assertTrue((diff_5.getArray() == [17, 28, 31]).all(), diff_5.getRelativeUrl())
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
self.assertTrue('file_system_image/diff_end/different' in diff_4.getPublicationSectionList())
self.assertTrue('file_system_image/diff_end/different' in diff_5.getPublicationSectionList())
self.assertTrue(diff_1.getPredecessorRelatedValue(portal_type='Data Array') in reference_data_array_list[1].getPredecessorRelatedValueList(portal_type='Data Array'))
diff_6 = diff_1.getPredecessorRelatedValue(portal_type='Data Array')
self.assertEqual(len(diff_6.getPredecessorValueList(portal_type='Data Array')), 2)
self.assertTrue("file_system_image/process_state/converted" in diff_6.getPublicationSectionList())
self.assertTrue((diff_6.getArray() == [15, 17, 18]).all(), diff_6.getRelativeUrl())
self.assertTrue(diff_2.getPredecessorRelatedValue(portal_type='Data Array') in reference_data_array_list[1].getPredecessorRelatedValueList(portal_type='Data Array'))
diff_7 = diff_2.getPredecessorRelatedValue(portal_type='Data Array')
self.assertEqual(len(diff_7.getPredecessorValueList(portal_type='Data Array')), 2)
self.assertTrue("file_system_image/process_state/converted" in diff_7.getPublicationSectionList())
self.assertTrue((diff_7.getArray() == [2]).all(), diff_7.getRelativeUrl())
self.assertTrue(diff_3.getPredecessorRelatedValue(portal_type='Data Array') in reference_data_array_list[1].getPredecessorRelatedValueList(portal_type='Data Array'))
diff_8 = diff_3.getPredecessorRelatedValue(portal_type='Data Array')
self.assertEqual(len(diff_8.getPredecessorValueList(portal_type='Data Array')), 2)
self.assertTrue("file_system_image/process_state/converted" in diff_8.getPublicationSectionList())
self.assertTrue((diff_8.getArray() == [17, 31]).all(), diff_8.getRelativeUrl())
self.portal.portal_alarms.slapos_process_data_array.activeSense()
self.tic()
for i in [diff_6, diff_7, diff_8]:
self.assertTrue('file_system_image/diff_end/different' in i.getPublicationSectionList())
master/bt5/slapos_abyss/TestTemplateItem/portal_components/test.erp5.testSlapOSAbyss.xml 0 → 100644
View file @ 87a831cf
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Test Component" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>default_reference</string> </key>
<value> <string>testSlapOSAbyss</string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>test.erp5.testSlapOSAbyss</string> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Test Component</string> </value>
</item>
<item>
<key> <string>sid</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>text_content_error_message</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>text_content_warning_message</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>version</string> </key>
<value> <string>erp5</string> </value>
</item>
<item>
<key> <string>workflow_history</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary>
<item>
<key> <string>component_validation_workflow</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle>
<global name="WorkflowHistoryList" module="Products.ERP5Type.Workflow"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_log</string> </key>
<value>
<list>
<dictionary>
<item>
<key> <string>action</string> </key>
<value> <string>validate</string> </value>
</item>
<item>
<key> <string>validation_state</string> </key>
<value> <string>validated</string> </value>
</item>
</dictionary>
</list>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
master/bt5/slapos_abyss/bt/dependency_list 0 → 100644
View file @ 87a831cf
erp5_wendelin
slapos_cloud
\ No newline at end of file
master/bt5/slapos_abyss/bt/template_action_path_list 0 → 100644
View file @ 87a831cf
Compute Node | create_data_supply
Compute Node | create_data_transformation
Compute Node | jump_to_data_array
Compute Node | jump_to_data_stream
Compute Node | operating_system_file_info_view
Data Array | mapped_content
\ No newline at end of file
master/bt5/slapos_abyss/bt/template_keep_last_workflow_history_only_path_list 0 → 100644
View file @ 87a831cf
portal_callables/IngestionPolicy_parseServerTag
portal_callables/DataOperation_convertDataStreamToArray
portal_callables/DataArray_processDataArray
portal_callables/IngestionPolicy_getSlaposIngestionOperationAndParameterDict
portal_ingestion_policies/metadata_upload
data_product_module/convert_data_stream_to_data_array
data_operation_module/convert_data_stream_to_data_array
\ No newline at end of file
master/bt5/slapos_abyss/bt/template_path_list 0 → 100644
View file @ 87a831cf
data_operation_module/convert_data_stream_to_data_array
data_product_module/convert_data_stream_to_data_array
portal_alarms/slapos_process_data_array
portal_callables/DataArray_processDataArray
portal_callables/DataOperation_convertDataStreamToArray
portal_callables/IngestionPolicy_getSlaposIngestionOperationAndParameterDict
portal_callables/IngestionPolicy_parseServerTag
portal_ingestion_policies/metadata_upload
\ No newline at end of file
master/bt5/slapos_abyss/bt/template_portal_type_base_category_list 0 → 100644
View file @ 87a831cf
Compute Node | publication_section
Data Array | causality
Data Array | predecessor
\ No newline at end of file
master/bt5/slapos_abyss/bt/template_portal_type_property_sheet_list 0 → 100644
View file @ 87a831cf
Compute Node | Aggregated
\ No newline at end of file
master/bt5/slapos_abyss/bt/template_skin_id_list 0 → 100644
View file @ 87a831cf
slapos_abyss
\ No newline at end of file
master/bt5/slapos_abyss/bt/template_test_id_list 0 → 100644
View file @ 87a831cf
test.erp5.testSlapOSAbyss
\ No newline at end of file
master/bt5/slapos_abyss/bt/title 0 → 100644
View file @ 87a831cf
slapos_abyss
\ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7