Commit 226c6458 authored by Ivan Tyagov's avatar Ivan Tyagov

This script should only append data to stream and NOT do any transformations...

This script should only append data to stream and NOT do any transformations on it keeping a simple rule of ERP5: save exactly what was entered by user or passed to us by remote agent (fluentd).
parent 984f0d71
...@@ -56,7 +56,7 @@ ...@@ -56,7 +56,7 @@
Example of ingesting data in ERP5 coming from fluentd.\n Example of ingesting data in ERP5 coming from fluentd.\n
Fluentd sends to us a JSON dictionary using msgpack protocol.\n Fluentd sends to us a JSON dictionary using msgpack protocol.\n
In this implementation we find respective Data Stream and simply\n In this implementation we find respective Data Stream and simply\n
append data there. We save raw JSON dictionary.\n append data there. We save raw JSON dictionary as string.\n
Ingestion Policy -> Data Supply -> Data Supply Line -> Sensor\n Ingestion Policy -> Data Supply -> Data Supply Line -> Sensor\n
-> Data Stream\n -> Data Stream\n
"""\n """\n
...@@ -101,19 +101,11 @@ if data_chunk is not None and reference is not None:\n ...@@ -101,19 +101,11 @@ if data_chunk is not None and reference is not None:\n
if data_stream is not None:\n if data_stream is not None:\n
pretty_data_chunk_list = []\n pretty_data_chunk_list = []\n
data_chunk_list = context.unpack(data_chunk)\n data_chunk_list = context.unpack(data_chunk)\n
\n \n
# in some case we can format data based on content type\n for data_chunk in data_chunk_list:\n
content_type = data_stream.getContentType()\n pretty_data = str(data_chunk[1])\n
if content_type in (\'application/csv\',):\n pretty_data_chunk_list.append(pretty_data)\n
for data_chunk in data_chunk_list:\n data = \'\'.join(pretty_data_chunk_list)\n
pretty_data = \',\'.join([\'"%s"\' %x for x in data_chunk[1].values()])\n
pretty_data_chunk_list.append(pretty_data)\n
data = \'\\n\'.join(pretty_data_chunk_list)\n
else:\n
for data_chunk in data_chunk_list:\n
pretty_data = str(data_chunk[1])\n
pretty_data_chunk_list.append(pretty_data)\n
data = \'\'.join(pretty_data_chunk_list)\n
\n \n
# append data\n # append data\n
data_stream.appendData(data)\n data_stream.appendData(data)\n
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment