Skip to content

W
wendelin
Commit 004be34a authored by Martin Manchev's avatar Martin Manchev
1 Browse files
Options

Changes in 'erp5_wendelin_data_lake' ...

parent 40ee4d19
Hide whitespace changes
Inline Side-by-side
Showing with 405 additions and 285 deletions
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/data_operation_module/ingestion_operation.xml
View file @ 004be34a
...@@ -167,9 +167,11 @@ ...@@ -167,9 +167,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global name="DateTime" module="DateTime.DateTime"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global name="DateTime" module="DateTime.DateTime"/>
<global name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
...@@ -220,9 +222,11 @@ ...@@ -220,9 +222,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global name="DateTime" module="DateTime.DateTime"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global name="DateTime" module="DateTime.DateTime"/>
<global name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
......
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/data_operation_module/transform_fif.xml
View file @ 004be34a
...@@ -167,9 +167,11 @@ ...@@ -167,9 +167,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global name="DateTime" module="DateTime.DateTime"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global name="DateTime" module="DateTime.DateTime"/>
<global name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
...@@ -220,9 +222,11 @@ ...@@ -220,9 +222,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global name="DateTime" module="DateTime.DateTime"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global name="DateTime" module="DateTime.DateTime"/>
<global name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
......
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/data_product_module/fif_array.xml
View file @ 004be34a
...@@ -175,9 +175,11 @@ ...@@ -175,9 +175,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global name="DateTime" module="DateTime.DateTime"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global name="DateTime" module="DateTime.DateTime"/>
<global name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
...@@ -228,9 +230,11 @@ ...@@ -228,9 +230,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global name="DateTime" module="DateTime.DateTime"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global name="DateTime" module="DateTime.DateTime"/>
<global name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
......
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/data_product_module/fif_data.xml
View file @ 004be34a
...@@ -176,9 +176,11 @@ ...@@ -176,9 +176,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global name="DateTime" module="DateTime.DateTime"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global name="DateTime" module="DateTime.DateTime"/>
<global name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
...@@ -229,9 +231,11 @@ ...@@ -229,9 +231,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global name="DateTime" module="DateTime.DateTime"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global name="DateTime" module="DateTime.DateTime"/>
<global name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
......
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/data_product_module/fif_descriptor.xml
View file @ 004be34a
...@@ -167,9 +167,11 @@ ...@@ -167,9 +167,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global name="DateTime" module="DateTime.DateTime"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global name="DateTime" module="DateTime.DateTime"/>
<global name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
...@@ -220,9 +222,11 @@ ...@@ -220,9 +222,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global name="DateTime" module="DateTime.DateTime"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global name="DateTime" module="DateTime.DateTime"/>
<global name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
......
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/data_supply_module/default_ebulk.xml
View file @ 004be34a
...@@ -99,9 +99,11 @@ ...@@ -99,9 +99,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global id="1.1" name="DateTime" module="DateTime.DateTime"/> <global id="1.1" name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global id="1.2" name="DateTime" module="DateTime.DateTime"/>
<global id="1.3" name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
...@@ -119,6 +121,8 @@ ...@@ -119,6 +121,8 @@
<object> <object>
<klass> <reference id="1.1"/> </klass> <klass> <reference id="1.1"/> </klass>
<tuple> <tuple>
<reference id="1.2"/>
<reference id="1.3"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
...@@ -273,9 +277,11 @@ ...@@ -273,9 +277,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global name="DateTime" module="DateTime.DateTime"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global name="DateTime" module="DateTime.DateTime"/>
<global name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
...@@ -340,9 +346,11 @@ ...@@ -340,9 +346,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global name="DateTime" module="DateTime.DateTime"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global name="DateTime" module="DateTime.DateTime"/>
<global name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
......
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/data_supply_module/default_ebulk/1.xml
View file @ 004be34a
...@@ -225,9 +225,11 @@ ...@@ -225,9 +225,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global name="DateTime" module="DateTime.DateTime"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global name="DateTime" module="DateTime.DateTime"/>
<global name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
...@@ -280,9 +282,11 @@ ...@@ -280,9 +282,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global name="DateTime" module="DateTime.DateTime"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global name="DateTime" module="DateTime.DateTime"/>
<global name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
......
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/data_supply_module/default_ebulk/2.xml
View file @ 004be34a
...@@ -231,9 +231,11 @@ ...@@ -231,9 +231,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global name="DateTime" module="DateTime.DateTime"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global name="DateTime" module="DateTime.DateTime"/>
<global name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
...@@ -286,9 +288,11 @@ ...@@ -286,9 +288,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global name="DateTime" module="DateTime.DateTime"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global name="DateTime" module="DateTime.DateTime"/>
<global name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
......
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/portal_alarms/wendelin_check_datastream_consistency.xml
View file @ 004be34a
...@@ -61,9 +61,11 @@ ...@@ -61,9 +61,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global id="1.1" name="DateTime" module="DateTime.DateTime"/> <global id="1.1" name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global id="1.2" name="DateTime" module="DateTime.DateTime"/>
<global id="1.3" name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
...@@ -81,6 +83,8 @@ ...@@ -81,6 +83,8 @@
<object> <object>
<klass> <reference id="1.1"/> </klass> <klass> <reference id="1.1"/> </klass>
<tuple> <tuple>
<reference id="1.2"/>
<reference id="1.3"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
......
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/portal_alarms/wendelin_data_lake_handle_analysis.xml
View file @ 004be34a
...@@ -61,9 +61,11 @@ ...@@ -61,9 +61,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global id="1.1" name="DateTime" module="DateTime.DateTime"/> <global id="1.1" name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global id="1.2" name="DateTime" module="DateTime.DateTime"/>
<global id="1.3" name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
...@@ -81,6 +83,8 @@ ...@@ -81,6 +83,8 @@
<object> <object>
<klass> <reference id="1.1"/> </klass> <klass> <reference id="1.1"/> </klass>
<tuple> <tuple>
<reference id="1.2"/>
<reference id="1.3"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
...@@ -179,9 +183,11 @@ ...@@ -179,9 +183,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global name="DateTime" module="DateTime.DateTime"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global name="DateTime" module="DateTime.DateTime"/>
<global name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
......
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/portal_callables/DataIngestionLine_writeEbulkIngestionToDataStream.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/portal_callables/IngestionPolicy_parseEbulkIngestionTag.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/portal_categories/use/big_data/ingestion/batch_ingestion.xml
View file @ 004be34a
...@@ -156,9 +156,11 @@ ...@@ -156,9 +156,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global name="DateTime" module="DateTime.DateTime"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global name="DateTime" module="DateTime.DateTime"/>
<global name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
...@@ -223,9 +225,11 @@ ...@@ -223,9 +225,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global name="DateTime" module="DateTime.DateTime"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global name="DateTime" module="DateTime.DateTime"/>
<global name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
......
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/portal_categories/use/big_data/ingestion/stream.xml 0 → 100644
View file @ 004be34a
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Category" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_Add_portal_content_Permission</string> </key>
<value>
<tuple>
<string>Assignor</string>
<string>Manager</string>
</tuple>
</value>
</item>
<item>
<key> <string>_Add_portal_folders_Permission</string> </key>
<value>
<tuple>
<string>Assignor</string>
<string>Manager</string>
</tuple>
</value>
</item>
<item>
<key> <string>_Copy_or_Move_Permission</string> </key>
<value>
<tuple>
<string>Assignor</string>
<string>Manager</string>
</tuple>
</value>
</item>
<item>
<key> <string>_Delete_objects_Permission</string> </key>
<value>
<tuple>
<string>Assignor</string>
<string>Manager</string>
</tuple>
</value>
</item>
<item>
<key> <string>_Modify_portal_content_Permission</string> </key>
<value>
<tuple>
<string>Assignee</string>
<string>Assignor</string>
<string>Manager</string>
<string>Owner</string>
</tuple>
</value>
</item>
<item>
<key> <string>categories</string> </key>
<value>
<tuple>
<string>use/big_data/ingestion/1</string>
</tuple>
</value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>stream</string> </value>
</item>
<item>
<key> <string>language</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Category</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Stream Ingestion</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/portal_ingestion_policies/default_ebulk.xml
View file @ 004be34a
...@@ -165,9 +165,11 @@ ...@@ -165,9 +165,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global name="DateTime" module="DateTime.DateTime"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global name="DateTime" module="DateTime.DateTime"/>
<global name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
...@@ -218,9 +220,11 @@ ...@@ -218,9 +220,11 @@
<value> <value>
<object> <object>
<klass> <klass>
<global name="DateTime" module="DateTime.DateTime"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple> <tuple>
<global name="DateTime" module="DateTime.DateTime"/>
<global name="object" module="__builtin__"/>
<none/> <none/>
</tuple> </tuple>
<state> <state>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/Alarm_checkDataStreamStateConsistency.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/Alarm_dataLakeHandleAnalysis.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/DataLake_stopIngestionList.py
View file @ 004be34a
from erp5.component.module.Log import log from erp5.component.module.Log import log
from Products.ZSQLCatalog.SQLCatalog import Query, SimpleQuery from Products.ZSQLCatalog.SQLCatalog import Query
import hashlib import hashlib
CHUNK_SIZE = 200000 CHUNK_SIZE = 200000
...@@ -14,7 +14,7 @@ def getHash(data_stream): ...@@ -14,7 +14,7 @@ def getHash(data_stream):
end_offset = n_chunk*chunk_size+chunk_size end_offset = n_chunk*chunk_size+chunk_size
try: try:
data_stream_chunk = ''.join(data_stream.readChunkList(start_offset, end_offset)) data_stream_chunk = ''.join(data_stream.readChunkList(start_offset, end_offset))
except: except e:
# data stream is empty # data stream is empty
data_stream_chunk = "" data_stream_chunk = ""
hash_md5.update(data_stream_chunk) hash_md5.update(data_stream_chunk)
...@@ -24,7 +24,7 @@ def getHash(data_stream): ...@@ -24,7 +24,7 @@ def getHash(data_stream):
def isInterruptedAbandonedSplitIngestion(reference): def isInterruptedAbandonedSplitIngestion(reference):
from DateTime import DateTime from DateTime import DateTime
now = DateTime() # now = DateTime()
day_hours = 1.0/24/60*60*24 day_hours = 1.0/24/60*60*24
# started split data ingestions for reference # started split data ingestions for reference
catalog_kw = {'portal_type': 'Data Ingestion', catalog_kw = {'portal_type': 'Data Ingestion',
...@@ -90,8 +90,8 @@ for data_ingestion in portal_catalog(portal_type = "Data Ingestion", ...@@ -90,8 +90,8 @@ for data_ingestion in portal_catalog(portal_type = "Data Ingestion",
portal.data_stream_module.deleteContent(data_stream.getId()) portal.data_stream_module.deleteContent(data_stream.getId())
if last_data_stream_id.endswith(reference_end_split): if last_data_stream_id.endswith(reference_end_split):
portal.ERP5Site_invalidateSplitIngestions(data_ingestion.getReference(), success=True) portal.ERP5Site_invalidateSplitIngestions(data_ingestion.getReference(), success=True)
hash = getHash(full_data_stream) hash1 = getHash(full_data_stream)
full_data_stream.setVersion(hash) full_data_stream.setVersion(hash1)
if full_data_stream.getValidationState() != "validated": if full_data_stream.getValidationState() != "validated":
full_data_stream.validate() full_data_stream.validate()
related_split_ingestions = portal_catalog(portal_type = "Data Ingestion", related_split_ingestions = portal_catalog(portal_type = "Data Ingestion",
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/DataLake_stopIngestionList.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/DataSet_checkMd5DataStreamList.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_activeLogin.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_checkIngestionReferenceExists.py
View file @ 004be34a
...@@ -44,7 +44,7 @@ if data_ingestion != None: ...@@ -44,7 +44,7 @@ if data_ingestion != None:
# previous ingestion was interrumped # previous ingestion was interrumped
log(''.join(["[WARNING] User has restarted an interrumpted ingestion for reference ", data_ingestion.getReference(), ". Previous split ingestions will be discarted and full ingestion restarted."])) log(''.join(["[WARNING] User has restarted an interrumpted ingestion for reference ", data_ingestion.getReference(), ". Previous split ingestions will be discarted and full ingestion restarted."]))
portal.ERP5Site_invalidateSplitIngestions(data_ingestion.getReference(), success=False) portal.ERP5Site_invalidateSplitIngestions(data_ingestion.getReference(), success=False)
except: except Exception:
pass pass
# the ingestion attemp corresponds to a split ingestion in course, accept # the ingestion attemp corresponds to a split ingestion in course, accept
return FALSE return FALSE
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_checkIngestionReferenceExists.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_checkReferenceInvalidated.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/DataLake_createDataAnalysisList.py → bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_createDataAnalysisList.py
View file @ 004be34a
from DateTime import DateTime from DateTime import DateTime
portal = context.getPortalObject() portal = context.getPortalObject()
portal_catalog = portal.portal_catalog portal_catalog = portal.portal_catalog
from erp5.component.module.Log import log # from erp5.component.module.Log import log
now = DateTime() now = DateTime()
query_dict = { query_dict = {
"portal_type": "Data Ingestion Line", "portal_type": "Data Ingestion Line",
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/DataLake_createDataAnalysisList.xml → bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_createDataAnalysisList.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
...@@ -54,7 +54,7 @@ ...@@ -54,7 +54,7 @@
</item> </item>
<item> <item>
<key> <string>id</string> </key> <key> <string>id</string> </key>
<value> <string>DataLake_createDataAnalysisList</string> </value> <value> <string>ERP5Site_createDataAnalysisList</string> </value>
</item> </item>
</dictionary> </dictionary>
</pickle> </pickle>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/DataLake_executeDataAnalysisList.xml → bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_executeDataAnalysisList.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
...@@ -54,7 +54,7 @@ ...@@ -54,7 +54,7 @@
</item> </item>
<item> <item>
<key> <string>id</string> </key> <key> <string>id</string> </key>
<value> <string>DataLake_executeDataAnalysisList</string> </value> <value> <string>ERP5Site_executeDataAnalysisList</string> </value>
</item> </item>
</dictionary> </dictionary>
</pickle> </pickle>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/DataLake_executeDataOperation.xml → bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_executeDataOperation.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
...@@ -54,7 +54,7 @@ ...@@ -54,7 +54,7 @@
</item> </item>
<item> <item>
<key> <string>id</string> </key> <key> <string>id</string> </key>
<value> <string>DataLake_executeDataOperation</string> </value> <value> <string>ERP5Site_executeDataOperation</string> </value>
</item> </item>
</dictionary> </dictionary>
</pickle> </pickle>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_getDataStreamChunk.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_getDataStreamCount.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_getDataStreamList.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5_getDescriptorHTMLContent.py → bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_getDescriptorHTMLContent.py
View file @ 004be34a
from Products.ZSQLCatalog.SQLCatalog import Query, SimpleQuery, ComplexQuery from Products.ZSQLCatalog.SQLCatalog import Query, ComplexQuery
portal = context.getPortalObject() portal = context.getPortalObject()
portal_catalog = portal.portal_catalog portal_catalog = portal.portal_catalog
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5_getDescriptorHTMLContent.xml → bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_getDescriptorHTMLContent.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
...@@ -54,7 +54,7 @@ ...@@ -54,7 +54,7 @@
</item> </item>
<item> <item>
<key> <string>id</string> </key> <key> <string>id</string> </key>
<value> <string>ERP5_getDescriptorHTMLContent</string> </value> <value> <string>ERP5Site_getDescriptorHTMLContent</string> </value>
</item> </item>
</dictionary> </dictionary>
</pickle> </pickle>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_getIngestionConstantsJson.py
View file @ 004be34a
import json import json
portal = context.getPortalObject() portal = context.getPortalObject()
dict = {'invalid_suffix':portal.ERP5Site_getIngestionReferenceDictionary()['invalid_suffix'], dictionary = {'invalid_suffix':portal.ERP5Site_getIngestionReferenceDictionary()['invalid_suffix'],
'split_end_suffix':portal.ERP5Site_getIngestionReferenceDictionary()['split_end_suffix'], 'split_end_suffix':portal.ERP5Site_getIngestionReferenceDictionary()['split_end_suffix'],
'single_end_suffix':portal.ERP5Site_getIngestionReferenceDictionary()['single_end_suffix'], 'single_end_suffix':portal.ERP5Site_getIngestionReferenceDictionary()['single_end_suffix'],
'split_first_suffix':portal.ERP5Site_getIngestionReferenceDictionary()['split_first_suffix'], 'split_first_suffix':portal.ERP5Site_getIngestionReferenceDictionary()['split_first_suffix'],
...@@ -10,4 +10,4 @@ dict = {'invalid_suffix':portal.ERP5Site_getIngestionReferenceDictionary()['inva ...@@ -10,4 +10,4 @@ dict = {'invalid_suffix':portal.ERP5Site_getIngestionReferenceDictionary()['inva
'reference_length':portal.ERP5Site_getIngestionReferenceDictionary()['reference_length'], 'reference_length':portal.ERP5Site_getIngestionReferenceDictionary()['reference_length'],
'invalid_chars':portal.ERP5Site_getIngestionReferenceDictionary()['invalid_chars'], 'invalid_chars':portal.ERP5Site_getIngestionReferenceDictionary()['invalid_chars'],
} }
return json.dumps(dict) return json.dumps(dictionary)
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_getIngestionConstantsJson.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_getIngestionReferenceDictionary.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_increaseDatasetVersion.py
View file @ 004be34a
portal = context.getPortalObject() portal = context.getPortalObject()
portal_catalog = portal.portal_catalog # portal_catalog = portal.portal_catalog
data_set = portal.data_set_module.get(reference) data_set = portal.data_set_module.get(reference)
if data_set is not None: if data_set is not None:
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_increaseDatasetVersion.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_invalidateIngestionObjects.py
View file @ 004be34a
from Products.ZSQLCatalog.SQLCatalog import Query, SimpleQuery, ComplexQuery from Products.ZSQLCatalog.SQLCatalog import Query, ComplexQuery
portal = context.getPortalObject() portal = context.getPortalObject()
portal_catalog = portal.portal_catalog portal_catalog = portal.portal_catalog
...@@ -16,5 +16,5 @@ for document in portal_catalog(**kw_dict): ...@@ -16,5 +16,5 @@ for document in portal_catalog(**kw_dict):
portal.ERP5Site_invalidateReference(document) portal.ERP5Site_invalidateReference(document)
try: try:
document.invalidate() document.invalidate()
except: except Exception:
pass # fails if it's already invalidated, draft or if it doens't allow invalidation (e.g. DI) pass # fails if it's already invalidated, draft or if it doens't allow invalidation (e.g. DI)
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_invalidateIngestionObjects.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_invalidateOldDatasets.py
View file @ 004be34a
...@@ -20,11 +20,11 @@ for data_set in portal_catalog(portal_type="Data Set", **catalog_kw): ...@@ -20,11 +20,11 @@ for data_set in portal_catalog(portal_type="Data Set", **catalog_kw):
portal.ERP5Site_invalidateIngestionObjects(data_stream.getReference()) portal.ERP5Site_invalidateIngestionObjects(data_stream.getReference())
try: try:
data_stream.invalidate() data_stream.invalidate()
except: except Exception:
pass # fails if it's already invalidated, draft or if it doens't allow invalidation (e.g. DI) pass # fails if it's already invalidated, draft or if it doens't allow invalidation (e.g. DI)
portal.ERP5Site_invalidateReference(data_set) portal.ERP5Site_invalidateReference(data_set)
try: try:
data_set.invalidate() data_set.invalidate()
except: except Exception:
pass # fails if it's already invalidated, draft or if it doens't allow invalidation (e.g. DI) pass # fails if it's already invalidated, draft or if it doens't allow invalidation (e.g. DI)
return printed return printed
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_invalidateOldDatasets.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_invalidateReference.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_invalidateSplitIngestions.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_newCredentialRequest.py
View file @ 004be34a
...@@ -79,5 +79,5 @@ if not batch_mode: ...@@ -79,5 +79,5 @@ if not batch_mode:
return portal.Base_redirect(form_id='login_form', return portal.Base_redirect(form_id='login_form',
keep_items = dict(portal_status_message=context.Base_translateString(message_str))) keep_items = dict(portal_status_message=context.Base_translateString(message_str)))
else: else:
return json.dumps({'msg': message_str, return json.dumps({'msg': message_str,
'code':0}) 'code':0})
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_newCredentialRequest.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_renameIngestion.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_revalidateReference.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_setDatasetDescription.py
View file @ 004be34a
...@@ -2,7 +2,7 @@ import base64 ...@@ -2,7 +2,7 @@ import base64
import json import json
portal = context.getPortalObject() portal = context.getPortalObject()
portal_catalog = portal.portal_catalog # portal_catalog = portal.portal_catalog
request = context.REQUEST request = context.REQUEST
data_chunk = request.get('data_chunk') data_chunk = request.get('data_chunk')
...@@ -11,10 +11,10 @@ data_set = portal.data_set_module.get(dataset) ...@@ -11,10 +11,10 @@ data_set = portal.data_set_module.get(dataset)
if data_set is not None: if data_set is not None:
decoded = base64.b64decode(data_chunk) decoded = base64.b64decode(data_chunk)
data_set.setDescription(decoded) data_set.setDescription(decoded)
dict = { 'status_code': 0, 'message': 'Dataset description successfully set.' } # dictionary = { 'status_code': 0, 'message': 'Dataset description successfully set.' }
else: else:
message = "No remote dataset found for reference '%s'" % (dataset) message = "No remote dataset found for reference '%s'" % (dataset)
dict = { 'status_code': 1, 'message': message } # dictionary = { 'status_code': 1, 'message': message }
context.logEntry(message) context.logEntry(message)
return json.dumps(dict) return json.dumps(dict)
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_setDatasetDescription.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_setIdGenerator.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_stopIngestionList.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/CredentialRequest_updatePersonAssignment.xml → bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_updatePersonAssignment.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
...@@ -63,7 +63,7 @@ ...@@ -63,7 +63,7 @@
</item> </item>
<item> <item>
<key> <string>id</string> </key> <key> <string>id</string> </key>
<value> <string>CredentialRequest_updatePersonAssignment</string> </value> <value> <string>ERP5Site_updatePersonAssignment</string> </value>
</item> </item>
</dictionary> </dictionary>
</pickle> </pickle>
......
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/IngestionPolicy_getIngestionOperationAndParameterDictEbulk.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/TestTemplateItem/portal_components/test.erp5.testDataLakeIngestion.py
View file @ 004be34a
...@@ -103,7 +103,7 @@ class TestDataIngestion(SecurityTestCase): ...@@ -103,7 +103,7 @@ class TestDataIngestion(SecurityTestCase):
def stepIngest(self, extension, delimiter, randomize_ingestion_reference=False, data_set_reference=False): def stepIngest(self, extension, delimiter, randomize_ingestion_reference=False, data_set_reference=False):
file_name = "file_name.csv" file_name = "file_name.csv"
reference = self.getRandomReference() reference = self.getRandomReference()
array = [[random.random() for i in range(self.CHUNK_SIZE_CSV + 10)] for j in range(self.CHUNK_SIZE_CSV + 10)] array = [[random.random() for _ in range(self.CHUNK_SIZE_CSV + 10)] for _ in range(self.CHUNK_SIZE_CSV + 10)]
np.savetxt(file_name, array, delimiter=delimiter) np.savetxt(file_name, array, delimiter=delimiter)
chunk = [] chunk = []
with open(file_name, 'r') as csv_file: with open(file_name, 'r') as csv_file:
......
bt5/erp5_wendelin_data_lake_ingestion/TestTemplateItem/portal_components/test.erp5.testDataLakeIngestion.xml
View file @ 004be34a
...@@ -6,12 +6,6 @@ ...@@ -6,12 +6,6 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>_recorded_property_dict</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
<item> <item>
<key> <string>default_reference</string> </key> <key> <string>default_reference</string> </key>
<value> <string>testDataLakeIngestion</string> </value> <value> <string>testDataLakeIngestion</string> </value>
...@@ -45,10 +39,7 @@ ...@@ -45,10 +39,7 @@
<item> <item>
<key> <string>text_content_warning_message</string> </key> <key> <string>text_content_warning_message</string> </key>
<value> <value>
<tuple> <tuple/>
<string>W:106, 34: Unused variable \'i\' (unused-variable)</string>
<string>W:106, 76: Unused variable \'j\' (unused-variable)</string>
</tuple>
</value> </value>
</item> </item>
<item> <item>
...@@ -58,28 +49,13 @@ ...@@ -58,28 +49,13 @@
<item> <item>
<key> <string>workflow_history</string> </key> <key> <string>workflow_history</string> </key>
<value> <value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent> <persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value> </value>
</item> </item>
</dictionary> </dictionary>
</pickle> </pickle>
</record> </record>
<record id="2" aka="AAAAAAAAAAI="> <record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary/>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle> <pickle>
<global name="PersistentMapping" module="Persistence.mapping"/> <global name="PersistentMapping" module="Persistence.mapping"/>
</pickle> </pickle>
...@@ -92,7 +68,7 @@ ...@@ -92,7 +68,7 @@
<item> <item>
<key> <string>component_validation_workflow</string> </key> <key> <string>component_validation_workflow</string> </key>
<value> <value>
<persistent> <string encoding="base64">AAAAAAAAAAQ=</string> </persistent> <persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value> </value>
</item> </item>
</dictionary> </dictionary>
...@@ -101,7 +77,7 @@ ...@@ -101,7 +77,7 @@
</dictionary> </dictionary>
</pickle> </pickle>
</record> </record>
<record id="4" aka="AAAAAAAAAAQ="> <record id="3" aka="AAAAAAAAAAM=">
<pickle> <pickle>
<global name="WorkflowHistoryList" module="Products.ERP5Type.Workflow"/> <global name="WorkflowHistoryList" module="Products.ERP5Type.Workflow"/>
</pickle> </pickle>
......
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts/DataSet_invalidateDataStreamList.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts/DataSet_publishDataStreamList.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_validation_workflow/scripts/checkConsistency.py
View file @ 004be34a
object = state_change['object'] state_change_object = state_change["object"]
object.Base_checkConsistency() state_change_object.Base_checkConsistency()
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_validation_workflow/scripts/checkConsistency.xml
View file @ 004be34a
...@@ -6,18 +6,18 @@ ...@@ -6,18 +6,18 @@
</pickle> </pickle>
<pickle> <pickle>
<dictionary> <dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item> <item>
<key> <string>_bind_names</string> </key> <key> <string>_bind_names</string> </key>
<value> <value>
<object> <object>
<klass> <klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/> <global name="_reconstructor" module="copy_reg"/>
</klass> </klass>
<tuple/> <tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state> <state>
<dictionary> <dictionary>
<item> <item>
......
bt5/erp5_wendelin_data_lake_ingestion/bt/dependency_list
View file @ 004be34a
erp5_wendelin erp5_wendelin
erp5_credential \ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7