Commit 503c9033 authored by Ivan Tyagov's avatar Ivan Tyagov

Call pure transformation in an activity rather than execute in current...

Call pure transformation in an activity rather than execute in current transaction. This way we split ingestion part from transformation part.
Note: this commit will serialize argument_list to activity's mysql table in case of big packets this can be slow, in case of small appends it can be acceptable.
Instead we should call with start and end offset only and data should be read from Data Stream itself (WIP).
parent d40a41cf
......@@ -58,7 +58,10 @@ argument_list = state_change[\'kwargs\'][\'workflow_method_args\']\n
\n
# call you own script to handle newly appended data which \n
# is not processed yet\n
data_stream.DataStream_transformTail(*argument_list)\n
# Note: this will serialize argument_list to activity\'s mysql table\n
# in case of big packets this can be slow, in =case of small appends it can be \n
# acceptable.\n
data_stream.activate().DataStream_transformTail(*argument_list)\n
  • For me, it would ok to remove the argument_list already.

  • Probalby we have to make sure that only one DataStream_transformTail is called at a time:

    if not data_stream.hasActivity():

    Of course this check can also be done in the project specific DataStream_transformTail method, but it could be a trap for many projects

Please register or sign in to reply
</string> </value>
</item>
<item>
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment