Data transformation

Data transformation components and concepts

You can send data to Live Objects using two sources, a device can send data using a supported protocol (LoRa®, SMS, MQTT…​) or a business application by using HTTPS or MQTTS. Each source is processed by one or many components. Data transformation is then relies upon:

  • the data enrichment process which makes the link between Live Objects Data Message and Device Management. Data enrichment is a merge of third-party data model from device source with a Live Objects common data model.

  • the decoding service which process Live Objects message for handling encoded data collected from various networks.

  • the custom pipeline service which process Live Objects Data Messages to perform various enrichment steps

EnrichmentprocessDecodingserviceAPIPipelinesserviceAPIstore & searchserviceAPIdata provided byDeviceover MQTT, LORA & SMSdata provided by aBusiness applicationoverHTTPS or MQTTusingApplication modeEncodeddatamessage
Figure 1. Data journey.

Data Message Enrichment process

Overview

The Live Objects Data Message pipeline goes through an enrichment process that applies the following logic. This pipeline concerns messages coming from LoRa® and MQTT protocols.

Enrichment logic

The enrichment logic uses Device Management repository to enrich a DataMessage with Live Objects device information .

  • when using LoRa® protocol a device is retrieved by using devEUI.

  • when using MQTT protocol a device is retrieved by using clientId.

Table 1. Data Message fields modified by Enricher Logic

streamId

if not provided, will be set to Live Objects urn

tags

will be enriched by merging this fields with the device tags

extra

will be enriched by merging with the device properties

metadata.group

will be set with the device group

metadata.encoding

will be set with the device’s interface definition.encoding value. It will override the value from the original data message.

The enrichment process is executed before the decoding process

Decoding service

Overview

The data messages sent to the Live Objects platform can be encoded in a customer specific format. For instance, the payload may be a string containing an hexadecimal value or a csv value.

The data decoding feature enables you to provision your own decoding grammar. On receiving the encoded message, the Live Objects platform will use the grammar to decode the payload into plain text JSON fields and record the JSON in the Store service. The stored message will then be searchable with the Advanced Search service.

A "template" option allows to perform mathematical operations on the decoded fields or to define an output format.

A "model" option allows to override original data 'model' field.

The decoding feature is not activated by default.

Binary decoding

Decoder provisioning

The custom decoder describes the grammar to be used to decode the message payload. The Live Objects API to manage the decoders are described in the swagger documentation: https://liveobjects.orange-business.com/swagger-ui/index.html.

The binary decoding module uses the Java Binary Block Parser (JBBP) library in version 2.0.6.
You must use the JBBP DSL language to describe the binary payload format for your decoder.

Available types

The following JBBP types are not available: floatj, doublej and stringj. Please use respectively float, double and utf8 or asciichar instead (see decoding examples).
Field names

Each field can have case insensitive name which must not contain '.' (reserved for links to structure field values) and '#'(reserved for internal library use). A field name must not be started with either number or chars '$' and ''. Keep in mind that field names are case insensitive!_

Naming fields to avoid :

myString =/= mystring

Prohibited field names examples :

Not authorized : my.string, my#string, $mystring,_my.string,
Basic decoder

Example: create a binary decoder with the REST API:

POST /api/v0/decoders/binary
X-API-Key: <your API key>
Accept: application/json
{
"encoding":"twointegers",  (1)
"enabled":true,  (2)
"format":"int pressure;int temperature;", (3)
"template":"{\"pressure\":{{pressure}},
  \"temperature\" : \"{{#math}}{{temperature}}/10{{/math}} celsius\"}", (4)
"model":"model_twointegers"  (5)
}
1 identifies the decoder. This name will be associated to the devices during the provisioning and will be present in the data message.
2 activation/deactivation of the decoder.
3 describes the payload frame (cf. JBBP DSL language). The name of the fields will be found in the resulting decoded payload JSON.
4 optional parameter describing a post-decoding template format. In this example, the output temperature will be divided by 10 and stored in a string format including its unit. More information on templates.
5 optional parameter that will override the 'model' field in decoded data. If empty, the original value of 'model' field of the encoded data will be used. More information on model field
When to change the model in the decoder ?

  • The "model" field is set in order to enable elasticsearch (ES) indexing and queries on the decoded value.

  • If the "model" field is not set when provisioning the decoder, the data message will still be stored but it will not be searchable with ES queries.

  • The "model" must be changed when the type of a field already indexed in ES is modified. Example :

    • Suppose you have created a binary decoder with the following payload description "format":"utf8[4] field1; int field2;"

    • You have defined a model named "my_model_v0" associated with this decoder.

    • You start using this decoder with your devices. Data messages are stored and indexed in Live Objects.

    • Then, you can perform searches in your data messages using the model. See search examples.

    • If, after checking your decoded data, you find out that field1 is a float, not a UTF8 string, you may modify your decoder format. "format":"float field1; int field2;"

    • In this case, if you want your new messages to be stored and indexed properly in elasticsearch, you must change the model name in your decoder (for instance "my_model_v1"). Otherwise, "field1" will still be mapped as String and the message will be rejected by elasticsearch.

Endianness

The decoding service uses the big-endian order (the high bytes come first). If your device uses little-endian architecture, you can use the < character to prefix a type in your format description.

Example: create a binary decoder for a device sending data in little-endian format:

POST /api/v0/decoders/binary
X-API-Key: <your API key>
Accept: application/json
{
"encoding":"my_little_endian_encoding",
"enabled":true,
"format":"<float temperature;" (1)
}
1 : float means 32-bit float sent in little-endian.
How to test the binary decoder format?

The Live Objects API provides a "test" endpoint which takes a payload format and a payload value as input and provides the decoded value in the response body, if the decoding is successful. Optionally, you can provide a post-decoding template which will describe the output format.

In the following example, the decoded value for the pressure will remain unchanged, while the decoded value for temperature will be divided by 10.
The test endpoint is described in swagger.

Request:

POST /api/v0/decoders/binary/test
X-API-Key: <your API key>
Accept: application/json
{
"binaryPayloadStructure":"int  pressure; int temperature;",
"binaryPayloadHexString":"000003F5000000DD",
"template":"{\"pressure\":{{pressure}}, \"temperature\" : \"{{temperature}}/10\"}"
}

Response:

{
   "parsingOk": true,
   "decodingResult":    {
      "temperature": 22.1,
      "pressure": 1013
   },
   "descriptionValid": true
}
How to customize the fields once the payload has been decoded?

The fields resulting of a decoded payload might need to be processed using a template description, in order to change their output format. More information on templates.

Message decoding

The data message is decoded using the decoder previously provisioned and the decoded fields are added to the value. The encoded raw payload is kept in the decoded message. Once the message has been decoded and stored, "Advanced Search" requests can be performed using the newly decoded fields.

Table 2. Examples :
Frame format Payload example Decoded payload (JSON) 
int temperature;

000000DD

 
{
  "value": {
    "payload" : "000000DD",
    "temperature": 221
  }
}
 
int signed;
uint unsigned;

80000000
80000000

 
{
  "value": {
    "payload" : "8000000080000000",
    "signed": -2147483648,
    "unsigned": 2147483648
  }
}
 
float max_value;

7F7FFFFF

 
{
  "value": {
    "payload" : "7F7FFFFF",
    "max_value": 3.4028235e+38
  }
}
 
double max_value;

7FEFFFFFFFFFFFFF

 
{
  "value": {
    "payload" : "7FEFFFFFFFFFFFFF",
    "max_value": 1.7976931348623157e+308
  }
}
 
ubyte temperature;

DD

 
{
  "value": {
    "payload" : "DD",
    "temperature": 221
  }
}
 
utf8 [17] mystring;

2855332e3632542b323144323235503029

 
{
  "value": {
    "payload" : "2855332e3632542b323144323235503029",
    "mystring": "(U3.62T+21D225P0)"
  }
}
 
asciichar[13] name;

616E4153434949537472696E67

 
{
  "value": {
    "payload" : "616E4153434949537472696E67",
    "myasciistring": "anASCIIString"
  }
}
 
byte is_led_on;
float pressure;
float temperature;
float altitude;
ubyte battery_lvl;
byte[6] raw_gps;
ushort altitude_gps;

00
447CE000
41CEF5C3
45CAB8CD
38
00000000000
0FFFF

 
{
  "value": {
    "payload" : "00447CE00041CEF5C345CAB8CD38000000000000FFFF",
    "is_led_on": 0,
    "pressure": 1011.5,
    "temperature": 25.87,
    "altitude": 6487.1,
    "battery_lvl": 56,
    "raw_gps_list": [0,0,0,0,0,0],
    "altitude_gps": 65535
  }
}
 
float pi;
measure[2] {
  int length;
  utf8 [length] name;
  float value;
}

4048F5C3
0000000B
C2A955544638537472696E67
41480000
00000012
C2A9616E6F7468657255544638537472696E67
447D4000

 
{
  "value": {
    "payload" : "4048F5C3
      0000000BC2A955544638537472696E6741480000
      00000012C2A9616E6F7468657255544638537472696E67447D4000",
    "pi": 3.14,
    "measure_list":[
      {
        "length":11,
        "name":"©UTF8String",
        "value":12.5
      },
      {
        "length":18,
        "name":"©anotherUTF8String",
        "value":1013.0
      }
    ]
  }
}

The binary decoding feature can handle long type (i.e 64-bit integer). Nevertheless, when displayed in a web portal, the long number might be rounded unaccurately due to javascript limitations (max safe integer = 2^53-1 in javascript).
Table 3. JSON fields

value.payload

a string containing the encoded payload in hexadecimal (raw value)

metadata.encoding

contains the decoder name

model

remains unchanged after decoding if model field of decoder is empty; else it will be set with the value of model field in the decoder

additional LoRa® fields (lora port, snr…​) in the value

remain unchanged after decoding.

Csv decoding

More information on csv format

Decoder provisioning

The custom decoder describes the columns format and options to be used to decode the message csv payload. The Live Objects API to manage the decoders are described in the swagger documentation : https://liveobjects.orange-business.com/swagger-ui/index.html.

When provisioning a csv decoder, you must specify an ordered list of column names and their associated type. Three column types are available : STRING, NUMERIC or BOOLEAN.
Several options (column separator char, quote char, escape char…​) may be set to customize the csv decoding.

A template option enables you to provide a post-decoding output format including mathematical evaluation. More information on templates.

Column types

  • STRING column may contain UTF-8 characters

  • NUMERIC column may contain integer (32 bits), long (64 bits), float or double values. The values may be signed.

  • BOOLEAN column must contain true or false.

Table 4. Available options
name default definition example

quoteChar

double-quote "\""

character used for quoting values that contain column separator characters or linefeed.

"pierre, dupont",25,true will be decoded as 3 fields.

columnSeparator

comma ","

character used to separate values.

lineFeedSeparator

"\n"

character used to separate data rows. If the message payload contains several rows, only the first one will be decoded.

the decoding result for pierre,35,true\nmarie,25,false will be 3 fields containing pierre, 35 and true.

useEscapeChar

false

set to true if you want to use an escape char.

escapeChar

backslash "\\"

character used to escape values.

skipWhiteSpace

false

if set to true, will trim the decoded values (white spaces before and after will be removed).

Example 1: create a simple csv decoder with the REST API:

POST /api/v0/decoders/csv
X-API-Key: <your API key>
Accept: application/json
{
    "encoding":"my csv encoding", (1)
    "enabled":true, (2)
    "columns": [ (3)
        {"name":"column1","jsonType":"STRING"},
        {"name":"column2","jsonType":"NUMERIC"},
        {"name":"column3","jsonType":"BOOLEAN"}
    ],
    "model":"model_csv_decoded"  (4)
}
1 identifies the decoder. This name will be associated to the devices during the provisioning and will be present in the data message.
2 activation/deactivation of the decoder.
3 an ordered list of column descriptions.
4 optional parameter that will override the 'model' field of decoded data. If empty, the original value of 'model' field of the encoded data will be used. More information on model field.

Example 2: create a csv decoder with options, using the REST API:

POST /api/v0/decoders/csv
X-API-Key: <your API key>
Accept: application/json
{
    "encoding":"my csv encoding with options",
    "enabled":true,
    "columns": [
        {"name":"unit","jsonType":"STRING"},
        {"name":"temperature","jsonType":"NUMERIC"},
        {"name":"normal","jsonType":"BOOLEAN"}
    ],
    "options" : {
        "columnSeparator": "|",
        "quoteChar": "\"",
        "lineFeedSeparator": "/r/n"
    }
}
In the POST request, you can provide only the options you wish to modify. The other options will keep the default values.
How to customize the fields once the payload has been decoded?

The fields resulting of a decoded payload might need to be processed using a template description, in order to change their output format. More information on templates.

How to test the csv decoder ?

The Live Objects API provides a "test" endpoint which takes a csv format description and a payload value as input and provides the decoded value in the response body, if the decoding is successful. The test endpoint is described in swagger.

Request:

POST /api/v0/decoders/csv/test
X-API-Key: <your API key>
Accept: application/json
{
    "columns": [
        {"name":"unit","jsonType":"STRING"},
        {"name":"temperature","jsonType":"NUMERIC"},
        {"name":"threasholdReached","jsonType":"BOOLEAN"}
    ] ,
    "options":{
        "columnSeparator": ","
    },
    "csvPayload":"celsius,250,true",
    "template":"{\"temperature\" : \"{{temperature}}/10\",
      \"unit\":\"{{unit}}\", \"thresholdReached\":\"{{thresholdReached}}\"} "
}

Response:

{
   "parsingOk": true,
   "decodingResult":    {
      "unit": "celsius",
      "thresholdReached": "true",
      "temperature": 25
   },
   "descriptionValid": true
}

  • The csv decoding feature can handle long type (i.e 64-bit integer) but numbers above 2^31-1 are displayed in exponential notation (2147483648 will be displayed as 2.147483648E9). The decoding of very large numbers might lead to inaccurate rounding when the precision is above 15 digits.

Example : 9007199254740997 is displayed as 9.007199254740996E15.

  • For large numbers above 16 digits, you should use the STRING type when decoding your payload to avoid any rounding issue. Please note also that using string type for a number has impact on indexing/searching in Elasticsearch (data search feature). Since the data will be indexed and stored as string, it will not be possible to do aggregations or calculations.

Referencing a decoder in a LoRa® device

When provisioning a LoRa® device, you may reference the decoder to be used for the device so that Live Objects will automatically decode all the payloads received from this device, using the referenced decoder.

Message decoding

The data message is decoded using the decoder previously provisioned and the decoded fields are added to the value. The csv encoded raw payload is kept in the decoded message. Once the message has been decoded and stored, "Advanced Search" requests can be performed using the newly decoded fields.

Example in https:

Request
POST /api/v0/data/streams/{streamId}
X-API-Key: <your API key>
Accept: application/json
{
  "value": {"payload":"celsius,25,true"},
  "model": "temperature_v0",
  "metadata" : {"encoding" : "my csv encoding"}
 }

The data message will be stored as:

{
      "id": "585aa47de4b019917e342edd",
      "streamId": "stream0",
      "timestamp": "2016-12-21T15:49:17.693Z",
      "model": "temperature_v0",
      "value":       {
         "payload": "celsius,25,true",
         "normal": true,
         "unit": "celsius",
         "temperature": 25
      },
      "metadata": {"encoding": "my csv encoding"},
      "created": "2016-12-21T15:49:17.750Z"
}
Table 5. JSON fields

value.payload

a string containing the csv encoded payload (raw value)

metadata.encoding

contains the decoder name

model

remains unchanged after decoding if model field of decoder is empty; else it will be set with the value of model field in the decoder

Templating

The Live Objects provides, for the decoder creation and the decoder test APIs, an optional parameter named "template". This parameter is a string field describing the target output fields in a mustache-like format.

Table 6. Available functions :

{{#math}}{{/math}}

performs mathematical operations on a field

{{#toUpperCase}}{{/toUpperCase}}

converts a string to upper case

{{#toLowerCase}}{{/toLowerCase}}

converts a string to lower case

The following examples shows, for the same raw binary payload, the output if you are not using any template, or if you define a custom template.

Request (WITHOUT the template parameter)
POST /api/v0/decoders/binary/test
X-API-Key: <your API key>
Accept: application/json
{
"binaryPayloadStructure": "byte:1 led; ushort pressure; ushort temperature; ushort altitude; ubyte battery; byte[6] raw_gps; ushort altitude_gps;",
"binaryPayloadHexString":"0027830a1bfd6738000000000000ffff"
}
Response
{
   "parsingOk": true,
   "decodingResult":{
      "led": 0,
      "pressure": 10115,
      "temperature": 2587,
      "altitude": 64871,
      "battery": 56,
      "raw_gps": [0, 0, 0, 0, 0, 0],
      "altitude_gps": 65535
   },
   "descriptionValid": true
}
Request (WITH the template parameter)
POST /api/v0/decoders/binary/test
X-API-Key: <your API key>
Accept: application/json
{
  "binaryPayloadStructure":"byte:1 led; ushort pressure; ushort temperature;ushort altitude; ubyte battery; byte[6] raw_gps; ushort altitude_gps;",
  "binaryPayloadHexString":"0027830a1bfd6738000000000000ffff",
  "template":"{\"pressure\": \"{{pressure}} / 10\",\"temperature\": \"{{temperature}} / 100\",\"altitude\": \"{{altitude}} / 10\",\"view\": {\"Pressure\": \"{{#math}}{{pressure}}/10{{/math}} hPa\",\"Temperature\": \"{{#math}}{{temperature}}/100{{/math}} C\",\"Altitude\": \"{{#math}}{{altitude}}/100{{/math}} m\",\"GPSAltitude\": \"{{altitude_gps}} m\",\"Battery\": \"{{battery}} %\"}}"
}
Response
{
   "parsingOk": true,
   "decodingResult":    {
      "altitude": 6487.1,
      "view":       {
         "Pressure": "1011.5 hPa",
         "Temperature": "25.87 C",
         "Altitude": "648.71 m",
         "GPSAltitude": "65535 m",
         "Battery": "56 %"
      },
      "temperature": 25.87,
      "pressure": 1011.5,
      "led": 0,
      "battery": 56,
      "raw_gps": [0, 0, 0, 0, 0, 0],
      "altitude_gps": 65535
   },
   "descriptionValid": true
}

Please do not use dot-separated fields in the template.

Example:

`"template":"{\"field.with.dot\": {{temperature}}/10}"` will be rejected.

The {{#math}}{{/math}} template is needed only if you wish to evaluate a mathematical expression within a string.

Example for a template containing:

\"Temperature\": \"{{temperature}}/100 celsius\" (1)
\"Temperature\": \"{{#math}}{{temperature}}/100{{/math}} celsius\" (2)
\"Temperature\": \"{{#math}}{{temperature}}/100{{/math}}\" (3)
\"Temperature\": \"{{temperature}}/100\" (4)
1 the output will be like Temperature": "2587/100 celsius" (the division is not evaluated).
2 the output will be like Temperature": "25.87 celsius" (a string output. the division is evaluated).
3 the output will be like Temperature": 25.87 (a numeric). In this case, the {{#math}} function is not needed.
4 the output will be like Temperature": 25.87 (a numeric).

You need to specify in the template, all the fields you wish to get in the output, even if they are not modified by the template.

Example:

 `"template":"{\"pressure\":{{pressure}}, \"temperature\" : {{temperature}}/10}"`
If you omit the *pressure* field in the template, it will simply not appear in the output.
location

If the decoded value contains a location field with latitude and longitude, it will override the location field provided in Live Objects at the same JSON level as the value field.

Example:

A LoRa® message contains an encoded payload. The default location in the message is the location provided by the LoRa® network.

  "streamId": "urn:lora:xxxxxxxxxxxx1009!uplink",
  "created": "2019-04-05T14:37:12.047Z",
  "location": {
      "provider": "lora",
      "alt": 0,
      "accuracy": 7500,
      "lon": -3.469017,
      "lat": 48.762119
   },
   "model": "model_yyyy",
   "value": {
      "payload": "01005ca767ee00400810",
   },
   "timestamp": "2019-04-05T14:37:08.489Z",

If the device is a tracker, after decoding the payload, the decoded message may have the following format :

  "streamId": "urn:lora:xxxxxxxxxxxx1009!uplink",
  "created": "2019-04-05T14:37:12.047Z",
  "location": {(2)
        "provider": "device", (3)
        "lon": -3.469020,
        "lat": 48.762120
   },
   "model": "model_yyyy",
   "value": {
      "payload": "01005ca767ee00400810",
      "location": {  (1)
            "lon": -3.469020,
            "lat": 48.762120
        },
      "temperature": {
         "unit": "°C",
         "currentTemperature": {
            "values": [
               20.64
            ]
         }
      }
   },
   "timestamp": "2019-04-05T14:37:08.489Z",
1 if the decoded location has the expected "location" format,
2 the decoded location replaces the location provided by the LoRa® network,
3 and the provider is now "device" instead of "lora"

Javascript decoding

If the payloads are more complex (several payloads, specific calculations/transformations…​), you can develop a scriptable (javascript) decoder. The javascript version must be compliant with ECMA Script 5. We also provide a development framework that can be used with Eclipse or IntelliJ IDE that can ease the development of the decoder. More info in the links below :

Split decoding

If your device sends several measures in a single data message, you can use a « split decoder ». It is a special type of javascript decoder that enables you to « split » the original data message into several data messages (typically one per measure). Each resulting data message will then follow the usual data processing path described here.

You will find an example in the scriptable decoders documentation.

  • metadata.transformation.decoder.split.rank is added in the messages produced by a split decoder (it is not present in the other messages). It can be used to filter out the « split » messages from a search query. This is for instance necessary if you want to count the messages sent by your devices (in that case you only want to count one of the « split » messages).

  • The metadata.network section of the message is only preserved in one of the data messages produced by a split decoder (in the one with a « split rank » equal to 0).

Custom Pipelines service for External enrichment

Overview

The Data Messages sent to Live Objects can be enriched by the Custom Pipelines service.

You can create your own pipeline to perform one or several enrichment steps on specific Data Messages.

Interactive

As described below, a pipeline is composed by a filter section and at least one step section.

Each pipeline targets Data Messages based on its filter configuration, and has a priorityLevel : at most one pipeline can handle a Data Message.

When a Data Message is handled by a pipeline, pipeline id and the status of the execution will be added in the metadata.transformation.pipeline field.

In case of pipeline failure, the original Data Message is stored and an AuditLog message will be sent with failure details.

Pipeline description

pipeline description
{
    "name": "base64 decoding",
    "description": "pipeline to decode base64 encoded message with external transformation",
    "priorityLevel": 10,
    "enabled": true,
    "filter": {
        "connectors": [ "mqtt" ],
        "encodings": [ "base64" ],
        "groupPaths": [ { "path": "/europe", "includeSubPath": true } ],
        "tags": [["PROD"]]
    },
    "steps" : [ {
        "type": "externalTransformation",
        "url": "http://lo-data-transformation.appspot.com:80/base64_decode"
    } ]
}
Field Type Description

name

String

(Mandatory). Name of the pipeline. Max 1000 characters

description

String

(Optional). Description of the pipeline. Max 2000 characters

priorityLevel

Integer

(Mandatory). Used to prioritize pipelines when a Data Message matches with the filter of several pipelines. The pipeline with the lowest priorityLevel value will be selected. In case of equal priorityLevel value, the older pipeline (based on its creation date) will be picked.

enabled

Boolean

(Mandatory). Indicates if the pipeline can apply or not.

filter

Object

(Optional). Define which Data Message can be processed by this pipeline. A null filter means all Data Message can be processed by the pipeline. Criteria in a filter are combined with a AND boolean logic. OR operator is applied between each elements inside filter’s list.

filter.connectors

List<String>

(Optional). If set, then only Data Message sent through one of these connectors (based on its metadata.connector field) will be selected and enter into the pipeline. Possible connectors are: "http", "lora", "mqtt", "sms", "x-connector"

filter.encodings

List<String>

(Optional). If set, then only Data Message with one of these encodings (based on its metadata.encoding field) will be selected and enter into the pipeline.

filter.groupPaths

List<GroupPath>

(Optional). If set, then only Data Message originated from one of these groupPaths (based on its metadata.group field) will be selected and enter into the pipeline.

filter.tags

List<List<String>>

(Optional). If set, then only Data Message with these tags description (based on its tags field) will be selected. There is a match if at least one group of tags is a match. A group of tags is a match if the tags of the message contains all elements of this group. For instance [["HIGH", "ALERT"],["PROD"]] will match any message containing 'PROD' tag; and also any message containing both 'HIGH' and 'ALERT' tags.

steps

List<Steps>

(Mandatory). Define the processing steps of the pipeline (see steps description below).
A dataMessage can only go through one pipeline. (see priorityLevel field for more information). This information is useful when there is overlap between pipeline filters.
A pipeline can have multiple steps. In this case, the steps are executed sequentially

Pipeline Steps description

For now, one type of pipeline steps is available: externalTransformation.

"externalTransformation" step

Interactive

This step will POST a http request with a DataMessage as body toward an external webhook url. Additional headers can be set for each request.

The Data Message format is described in the messages data model section, without the id field as it is not yet stored.

The response to this request must be the transformed (decoded, enriched…​) Data Message using the same format.

Live Objects will then manage this transformed Data Message response:

Behavior Fields

The value of these fields is kept from the original dataMessage. i.e. not overriden

  • streamId

  • metadata

  • extra

  • value.payload

  • created

The value of these fields is replaced by the value contained in the response dataMessage. i.e. overriden Even if 'null'

  • model

  • location

  • tags

  • value (except value.payload)

If not null in the response dataMessage, the value overrides the original field

  • timestamp

Ignored by the pipeline

Any other fields
The important fields that the response can override even if null are: model, location, tags and value.
The http status of the response must be of the 2xx success family in order to be taken into account
In order to ensure that all Data Messages are quickly delivered into other services (Data Store, Alarming, Routing) there is no retry mechanism. Also, only a limited number of Data Messages can wait in a pipeline queue. You need to ensure that your webhook server is available and can hold the Data Messages traffic.
externalTransformation step
{
    "type": "externalTransformation",
    "name": "additional identifier enrichment",
    "url": "http://lo-data-transformation.appspot.com:80/enrich",
    "headers" : [ "x-transform-header" : [ "account-1234" ] ]
}
Field Type Description

type

String

(Mandatory). Must be set to 'externalTransformation'.

name

String

(Optional). Name of the step. It will be pushed in the 'x-orange-lo-pipeline-step-name' http header. Max 1000 characters

url

String

(Mandatory). URL to POST the Data Message. Authorized ports are: 80, 443, 8080, 8443 and 9243.

headers

Map<String, List<String>>

(Optional). If present, these headers will be added in the http POST request.

If the external transformation fails (for ex. because the JSON is badly formatted, server responds http 503…​), Live Objects will store the original Data Message, and an AuditLog message will be logged with failure details.

Example

Implementing a base64 decoder for all dataMessages with 'base64' metadata.decoding:

First, you must deploy your server that will accept POST request and return decoded Data Message.

Then create the following pipeline:

pipeline description
{
    "name": "base64 decoding",
    "priorityLevel": 10,
    "enabled": true,
    "filter": {
        "encodings": [ "base64" ]
    },
    "steps" : [ {
        "type": "externalTransformation",
        "name": "base64 external decoder",
        "url": "http://lo-data-transformation.appspot.com:80/base64_decode"
    } ]
}

The following Data Message is sent by a device:

Data Message sent by a device
{
   "streamId":"device-001-alarm",
   "timestamp":"2019-12-10T13:57:03Z",
   "model":"v1",
   "value":{
      "payload":"bG93IGJhdHRlcnkgYWxhcm0gOiA5JSByZW1haW5pbmc="
   },
   "metadata":{
      "encoding":"base64"
   }
}

This Data Message will arrive into the custom pipeline service and match with the filter of the provisioned pipeline. So, this message will be POST to the url:

POST http request triggered by the DataMessage
POST http://lo-data-transformation.appspot.com:80/base64_decode
'Content-Type': 'application/json'
'x-orange-lo-pipeline-execution-id': '5e9f80ca-6a72-4252-9251-f014b61cf682'
'x-orange-lo-pipeline-step-name': 'base64 external decoder'
{
   "type":"dataMessage",
   "version":1,
   "streamId":"device-001-alarm",
   "timestamp":"2019-12-10T13:57:03Z",
   "model":"v1",
   "value":{
      "payload":"bG93IGJhdHRlcnkgYWxhcm0gOiA5JSByZW1haW5pbmc="
   },
   "metadata":{
      "encoding":"base64"
   }
}

Here is the body response from the remote server (that has performed the payload decoding and has added a tag):

POST http response 200 OK
{
   "model":"v1_base64_decoded",
   "value":{
      "alarm":"low battery alarm : 9% remaining",
      "battery_level":9
   },
   "tags":[
      "ALARM",
      "BASE64_DECODED"
   ]
}

Then the stored Data Message will be:

Stored Data Message
{
   "streamId":"device-001-alarm",
   "timestamp":"2019-12-10T13:57:03Z",
   "model":"v1_base64_decoded",
   "value":{
      "payload":"bG93IGJhdHRlcnkgYWxhcm0gOiA5JSByZW1haW5pbmc=",
      "alarm":"low battery alarm : 9% remaining",
      "battery_level":9
   },
   "tags":[
      "ALARM",
      "BASE64_DECODED"
   ],
   "metadata":{
      "encoding":"base64",
      "transformation":{
         "pipeline":{
            "id":"a07767f6-809e-4943-8b79-5efb96bd1535",
            "success":true
         }
      }
   }
}
Version 2026.7.0 • Updated date 2026-08-11 12:37:29 UTC