Add an example of MSGPack transmission to Wendelin from a file.

example/to_wendelin.conf

@@ -8,42 +8,44 @@
   port 8888
 </source>
 
-# output tag=sensor.** to Wendelin, this will match all HTTP request like so
+# This will match all HTTP request like so
 # ivan@debian: ~$ curl -X POST -d 'json={"foo1":"bar1"}' http://localhost:8888/default_http_json
+# then send to Wendelin and append as strings at Wendelin side 
 <match default_http_json.**>
-  @type            wendelin
+  @type             wendelin
   @id wendelin_out
   streamtool_uri    <Wendelin_URL>/erp5/portal_ingestion_policies/default_http_json   
   # TODO ^^^ do not check peer's certificate   
-  user      <your_wendelin_user>   
-  password  <your_wendelin_password>
+  user              <your_wendelin_user>
+  password          <your_wendelin_password>
 
-  use_keep_alive   true  
-  buffer_type      memory
-  flush_interval   5s
+  use_keep_alive    true  
+  buffer_type       memory
+  flush_interval    5s
 </match>
 
 # tail existing file and send its contents to Wendelin
 <source>
-  @type            tail
-  path             test.log
-  pos_file         test.log.pos
-  tag              fluentd-1
+  @type             tail
+  path              test.log
+  pos_file          test.log.pos
+  tag               test_sensor.test_product
   <parse>
     @type none
   </parse>
-
 </source>
 
-<match fluentd-1>
-  @type            wendelin
-  @id wendelin_out
-  streamtool_uri    <Wendelin_URL>/erp5/portal_ingestion_policies/default_http_json   
+
+# This will send and streo at Wendelin side in a MsgPack format
+<match test_sensor.test_product>
+  @type             wendelin
+  @id fluentd_out
+  streamtool_uri    <Wendelin_URL>/erp5/portal_ingestion_policies/default   
   # TODO ^^^ do not check peer's certificate   
-  user      <your_wendelin_user>   
-  password  <your_wendelin_password>
+  user              <your_wendelin_user>
+  password          <your_wendelin_password>
 
-  use_keep_alive   true  
-  buffer_type      memory
-  flush_interval   5s
+  use_keep_alive    true  
+  buffer_type       memory
+  flush_interval    5s
 </match>

[Klaus Wölfel]

@Tyagov Please note that it is not the ingestion policy which decides how to store incoming data in wendelin (e.g. when sending to portal_ingestion_policies/default then data can be stored in msgpack or in unpacked string).

The script defined in ingestion policy only decides how to parse the tag to find a data ingestion or data supply

It is the Data Operation Script which is defined in data ingestion / data supply which decides how it wants to store data.

Therefore the current comment is a bit misleading.

[Ivan Tyagov]

@klaus , yes sorry for this.

Here's what I plan to fix as current state (pure Wendelin side) is too confusing:

• Unification between "default" ingestion policy and "default_http_json"
  • add to erp5_wendelin the needed Data Supplies and Data Products so "default" can work without any changes and fix tests (not create Data Supply)
  • fix tutorial here https://wendelin.nexedi.com/NXD-Tutorial.HowTo.Ingest.Data.With.Fluentd to use "default"

Overall goal: 1 "default" ingestion policy which works without any changes needed Wendelin side right now. Based on outcome of storage tests it will either be MSgPack or "append as string" - but let us see (doing it now)

[Klaus Wölfel]

@Tyagov in this case, I suggest that this work is done using merge request so it can be discussed before being comitted to master.

My current thinking is that there can be one "default" data ingestion policy, but there must be several data supplies preinstalled and in the case of fluentd there must not be a single default data supply, because the choice depends how you acquire data (log file, execute command, ...), the type of data (time series or not) and the original data format (json, binary, ...)

[Ivan Tyagov]

@klaus , I completely agree. This is as state the overall goal.