Data Transformation

Data Transformation#

In many industrial scenarios, raw data coming from sensors and assets requires processing before it can be effectively used in Digital Twins. This is where data transformation becomes crucial. By applying various transformation techniques such as format conversion, data validation, calculations, and filtering, raw data can be converted into meaningful information suitable for specific applications or analyses.

Node-RED, a powerful flow-based development tool, serves as an excellent middleware solution for data transformation in the BaSyx ecosystem. It provides a visual programming environment that allows users to create sophisticated data processing pipelines with ease.

This section demonstrates how Node-RED can be integrated with BaSyx components to create a robust data transformation pipeline that processes MQTT sensor data before storing it in an Asset Administration Shell.

Note

The example can be found on in the Examples on GitHub. Feel free to try it out yourself!

Data Transformation Architecture#

The following diagram illustrates the data flow from an MQTT client through Node-RED for transformation, before being stored in the AAS Environment:

@startuml rectangle "MQTT Client/Sensor" as client rectangle "Mosquitto Broker" as broker rectangle "Node-RED" as nodered rectangle "AAS Environment" as aas client -down-> broker : Publishes raw data broker -down-> nodered : MQTT topic note right of nodered Data Transformation: • Format conversion • Data validation • Calculations • Filtering end note nodered -down-> aas : REST API @enduml

Detailed Transformation Process#

The sequence diagram below shows the step-by-step process of how Node-RED receives, transforms, and forwards data:

@startuml participant "MQTT Client" as Client participant "Mosquitto Broker" as Broker participant "Node-RED" as NodeRED participant "Function Node" as Function participant "AAS Environment" as AAS Client -> Broker : Publish raw sensor data note over Broker : Topic: /sensors/raw_temperature\nPayload: {"temp": "23.5°C", "unit": "celsius"} NodeRED -> Broker : Subscribe to MQTT topic Broker -> NodeRED : Forward raw message NodeRED -> Function : Process raw data note over Function : Transform data:\n• Parse temperature value\n• Convert to numeric\n• Apply calibration\n• Add metadata Function -> NodeRED : Return transformed data note over NodeRED : Transformed payload:\n{"value": 23.5, "unit": "°C", "timestamp": "2025-08-27T10:00:00Z"} NodeRED -> AAS : HTTP PUT request with transformed data note over AAS : Update AAS property\nwith processed value AAS -> NodeRED : Confirmation response @enduml

Key Benefits of Node-RED for Data Transformation#

  • Visual Programming: Create data flows using a drag-and-drop interface

  • Extensive Library: Access to thousands of pre-built nodes for various protocols and services

  • Real-time Processing: Handle streaming data with low latency

  • Flexible Transformations: Apply complex logic, calculations, and data manipulations

  • Protocol Bridge: Convert between different communication protocols (MQTT, HTTP, OPC-UA, etc.)

  • Error Handling: Built-in mechanisms for handling data validation and error scenarios

BaSyx Node-RED Example#

This example showcases the integration of BaSyx with Node-RED for seamless data flow and management in industrial IoT applications. Here, an example MQTT client for an environmental sensor is used to send data via Node-RED to the BaSyx AAS Environment.

How to run the BaSyx + Node-RED Example#

  1. Open a terminal in this folder

  2. Run the following command to start the BaSyx containers:

docker-compose up -d

To run the example containers, you need to have Docker installed on your device.

View the working Example#

To see the working example, open the BaSyx AAS Web UI and navigate to the SensorExampleAAS. You can see the data coming from the MQTT client in the SensorData submodel. To see updates in real-time, active the Auto-Sync feature in the AAS Web UI (see top right corner of the UI).

BaSyx AAS Web UI

Where to find the configuration#

Node-RED#

You can visit the Node-RED Editor to view and edit the flows.

The preconfigured Node-RED flow can be found in the nodered folder. The flow is defined in the flows.json file.

Node-RED Flow

MQTT Client#

The configuration for the MQTT client can be found in the mqtt-client folder. It is a small Python script that publishes data to the MQTT broker.

MQTT Broker#

The MQTT brokers configuration can be found in the mosquitto folder. The configuration is defined in the config/mosquitto.conf file.

BaSyx Components#

The configuration for the BaSyx components can be found in the basyx folder. The AAS used in this example is located in the aas folder.

Additional Resources#

For more information about data transformation with BaSyx: