Skip to content

W
wendelin
Commit fd728553 authored by Ivan Tyagov's avatar Ivan Tyagov
Browse files
Options

Add a new interaction workflow which will publish / invalidate all Data Streams per Data Set.

parent 562b39c3
Show whitespace changes
Inline Side-by-side
Showing with 560 additions and 1 deletion
bt5/erp5_wendelin_data_lake_ingestion/PortalTypeWorkflowChainTemplateItem/workflow_chain_type.xml
View file @ fd728553
<workflow_chain> <workflow_chain>
<chain> <chain>
<type>Data Set</type> <type>Data Set</type>
<workflow>-validation_workflow, data_validation_workflow</workflow> <workflow>-validation_workflow, data_set_interaction_workflow, data_validation_workflow</workflow>
</chain> </chain>
<chain> <chain>
<type>Data Stream</type> <type>Data Stream</type>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/DataSet_getDataStreamList.py 0 → 100644
View file @ fd728553
"""
Return list of Data Streams belonging to a Date Set.
Data Ingestion line aggregates both Data Set and Data Stream.
Note: This code is quite computationally costly (for Data Streams having thousands of iles) as it needs to:
1. Query MariaDB to find ingestion lines
2. Read from ZODB both Data Ingestion Lines and Data Streams (whoch itself can be big too)
"""
data_ingestion_line_list = context.portal_catalog(
portal_type = "Data Ingestion Line",
aggregate_uid = context.getUid())
return [x.getAggregateValue(portal_type = "Data Stream") \
for x in data_ingestion_line_list]
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/DataSet_getDataStreamList.xml 0 → 100644
View file @ fd728553
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>DataSet_getDataStreamList</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow.xml 0 → 100644
View file @ fd728553
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="InteractionWorkflowDefinition" module="Products.ERP5.InteractionWorkflow"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_objects</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>creation_guard</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>groups</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>data_set_interaction_workflow</string> </value>
</item>
<item>
<key> <string>manager_bypass</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Data Set Interaction Workflow</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/interactions.xml 0 → 100644
View file @ fd728553
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Interaction" module="Products.ERP5.Interaction"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_mapping</string> </key>
<value>
<dictionary/>
</value>
</item>
<item>
<key> <string>_objects</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>interactions</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/interactions/invalidate.xml 0 → 100644
View file @ fd728553
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="InteractionDefinition" module="Products.ERP5.Interaction"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>actbox_category</string> </key>
<value> <string>workflow</string> </value>
</item>
<item>
<key> <string>actbox_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>actbox_url</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>activate_script_name</string> </key>
<value>
<list>
<string>DataSet_invalidateDataStreamList</string>
</list>
</value>
</item>
<item>
<key> <string>after_script_name</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>before_commit_script_name</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>guard</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>invalidate</string> </value>
</item>
<item>
<key> <string>method_id</string> </key>
<value>
<list>
<string>invalidate</string>
</list>
</value>
</item>
<item>
<key> <string>once_per_transaction</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>portal_type_filter</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>portal_type_group_filter</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>script_name</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>temporary_document_disallowed</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>trigger_type</string> </key>
<value> <int>2</int> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/interactions/publish.xml 0 → 100644
View file @ fd728553
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="InteractionDefinition" module="Products.ERP5.Interaction"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>actbox_category</string> </key>
<value> <string>workflow</string> </value>
</item>
<item>
<key> <string>actbox_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>actbox_url</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>activate_script_name</string> </key>
<value>
<list>
<string>DataSet_publishDataStreamList</string>
</list>
</value>
</item>
<item>
<key> <string>after_script_name</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>before_commit_script_name</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>guard</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>publish</string> </value>
</item>
<item>
<key> <string>method_id</string> </key>
<value>
<list>
<string>publish</string>
</list>
</value>
</item>
<item>
<key> <string>once_per_transaction</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>portal_type_filter</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>portal_type_group_filter</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>script_name</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>temporary_document_disallowed</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>trigger_type</string> </key>
<value> <int>2</int> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts.xml 0 → 100644
View file @ fd728553
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Scripts" module="Products.DCWorkflow.Scripts"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_mapping</string> </key>
<value>
<dictionary/>
</value>
</item>
<item>
<key> <string>_objects</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>scripts</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts/DataSet_invalidateDataStreamList.py 0 → 100644
View file @ fd728553
"""
Publish all Data Streams for a Data Set.
"""
data_set = state_change['object']
for data_stream in data_set.DataSet_getDataStreamList():
data_stream.activate().invalidate()
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts/DataSet_invalidateDataStreamList.xml 0 → 100644
View file @ fd728553
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>state_change,**kw</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>DataSet_invalidateDataStreamList</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts/DataSet_publishDataStreamList.py 0 → 100644
View file @ fd728553
"""
Publish all Data Streams for a Data Set.
"""
data_set = state_change['object']
for data_stream in data_set.DataSet_getDataStreamList():
data_stream.activate().publish()
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts/DataSet_publishDataStreamList.xml 0 → 100644
View file @ fd728553
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>state_change,**kw</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>DataSet_publishDataStreamList</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/variables.xml 0 → 100644
View file @ fd728553
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Variables" module="Products.DCWorkflow.Variables"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_mapping</string> </key>
<value>
<dictionary/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>variables</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/worklists.xml 0 → 100644
View file @ fd728553
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Worklists" module="Products.DCWorkflow.Worklists"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_mapping</string> </key>
<value>
<dictionary/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>worklists</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/bt/template_portal_type_workflow_chain_list
View file @ fd728553
Data Set | -validation_workflow Data Set | -validation_workflow
Data Set | data_set_interaction_workflow
Data Set | data_validation_workflow Data Set | data_validation_workflow
Data Stream | -validation_workflow Data Stream | -validation_workflow
Data Stream | data_validation_workflow Data Stream | data_validation_workflow
\ No newline at end of file
bt5/erp5_wendelin_data_lake_ingestion/bt/template_workflow_id_list
View file @ fd728553
data_set_interaction_workflow
data_validation_workflow data_validation_workflow
\ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7