Introduction
Apache StreamPipes 0.97.0 brings exciting new features, enhanced security, and significant UX improvements to streamline your IIoT data processing workflows. This release introduces Adapters & Pipelines as Code, enabling users to define and deploy StreamPipes resources programmatically using YAML or JSON. Security enhancements, such as OPC UA certificate support and Kafka SSL/SASL authentication, improve connectivity and data integrity. Additionally, we have made various UX improvements, including an enhanced Data Explorer and asset-based filtering.
In total, the new release closes 159 issues including new features, bug fixes and improvements.
Read on to discover what’s new in StreamPipes 0.97.0!
Adapters & Pipeline Elements
Adapter & Pipeline as Code
StreamPipes now supports managing adapters and pipelines programmatically using compact Json or YAML definitions. These features are ideal for automating deployments, integrating with CI/CD pipelines, and applying infrastructure-as-code practices.
Adapters as Code
With Adapters as Code, you can define and deploy adapters via a compact description, streamlining the creation of multiple similar adapters and simplifying automation. For example, here's a sample YAML snippet to define an OPC-UA adapter:
name: My OPC Adapter
id: testadapter
description: Test
appId: org.apache.streampipes.connect.iiot.adapters.opcua
configuration:
- opc_host_or_url: OPC_URL
opc_server_url: opc.tcp://localhost:62541/milo
- adapter_type: PULL_MODE
pulling_interval: 1000
- access_mode: UNAUTHENTICATED
- available_nodes:
- "ns=2;s=Leakage Test Station/temperature"
- "ns=2;s=Leakage Test Station/pressure"
schema:
temperature:
propertyScope: measurement
label: Temp
description: Temperature value
semanticType: http://schema.org/temperature
enrich:
timestamp: timestamp
transform:
rename:
pressure: pressure3
createOptions:
persist: true
start: true
This approach considerably simplifies the programmatic creation of adapters Learn more about Adapters as Code
Pipelines as Code
Pipelines as Code lets you define end-to-end data flows—from data sources through processors to sinks. This function is ideal for the standardization and automation of pipeline deployments. For example, here’s a sample YAML snippet for a simple pipeline:
id: my-pipeline
name: ""
description: ""
pipelineElements: # Define pipeline components here
- type: stream # Data source
ref: <reference> # Unique reference ID
id: <data-stream-id> # ID of the stream
- type: processor # Data transformation
ref: <reference> # Unique reference ID
id: <processor-id> # ID of the processor
connectedTo: # Previous pipeline element reference(s)
- <reference>
configuration: # Processor-specific configurations
- <configuration-option>
- type: sink # Data sink (output)
ref: <reference> # Unique reference ID
id: <sink-id> # ID of the sink
connectedTo: # Previous pipeline element reference(s)
- <reference>
configuration: # Sink-specific configurations
- <configuration-option>
createOptions:
start: <true|false> # Whether to start the pipeline immediately
The code preview of a pipeline has also been added to the Start Pipeline dialog, which has been further improved to make the execution status of pipelines more clear.
This method provides a programmatic interface to deploy and manage pipelines quickly and easily. Learn more about Pipelines as Code
OPC UA Certificates
The OPC UA adapter now supports both signed/encrypted and unencrypted communication. Certificate provisioning is handled externally via environment variables, following a dedicated security directory layout. Refer to the OPC UA adapter documentation for further details.
https://github.com/apache/streampipes/pull/3354
Kafka Avro & Certificates
The Kafka adapter and sink now support SSL and SASL for enhanced security. Additionally, users can provide custom producer/consumer configurations. A new Avro parser has been added along with an input field for specifying the consumer group in the Kafka adapter.
New Features & Deprecations
Asset Browser
StreamPipes now allows users to model real-world assets and associate them with StreamPipes resources such as adapters, pipelines, and dashboards. These asset mappings enable intuitive filtering across different modules, making it easier to navigate and manage related resources.
The screenshot below illustrates an example where a StreamPipes instance is configured with two machines and various stations. Clicking on a machine or station dynamically filters the associated resources in the list on the right. This asset-based filtering is available across all main modules of StreamPipes.
The asset overview panel has been redesigned to provide a more intuitive and user-friendly experience. It is possible to assign labels, locations, and ISA95 types to assets. For each asset, resources such as pipelines, adapters and dashboards can be assigned. The asset overview contains an intuitive overview of all linked resources.
To manage locations & sites, we extended the configuration menu with additional options to add a custom map layer provider and to configure the map view.
UX improvements
File Relocation
The file upload configuration has been moved from the top-level navigation to the settings menu, improving the overall structure and clarity of the main navigation.
This change does not affect file uploads required by adapters or pipeline elements—these configurations remain unchanged. Further, users can still upload files as before via the files configuration section.
Adapter Data Preview
Viewing live data streams in the adapter configuration has been improved. By clicking on "details", a full-screen view of the data preview is displayed. Changing data values are highlighted, so that it is ease to track changes. The Code panel shows the configuration in JSON or YAML format.
Data Explorer Configuration Improvements
Configuring data views in the Data Explorer is now more intuitive and user-friendly. We have enhanced validation, making it easier to select data sources and specify queries while reducing potential misconfigurations.
Key improvements include:
- Default Table Visualization: New data views now default to a table view, minimizing setup effort.
- Improved Time Range Picker: More flexibility in selecting time ranges, including support for different time ranges per widget in Data Lake dashboards.
- Auto-Refresh for Dashboards: Live updates are now possible with the new auto-refresh functionality.
- Better Error Messages: Enhanced feedback when configurations are incorrect to improve usability.
- Custom color coding for pie chart: In the pie chart, custom colors can be mapped to specific data values.
- Traffic light visualization: A new traffic light visualization has been added to the Data Explorer.
Here is a screenshot of the new traffic light visualization:
Pipeline Details
The pipeline details view has been completely remodeled. It now shows the pipeline in full-screen mode, making it easier to view and edit complex pipelines. A sidebar contains further information on the pipeline, such as logs, pipeline elements, and execution status. In addition, execution metrics are now intuitively accessible right in the pipeline details view.
Deprecation Notice: Dashboard
The current live dashboard feature in StreamPipes is being deprecated and will be removed in a future release. As a replacement, the Data Explorer dashboards has been enhanced with auto-refresh functionality, ensuring a seamless transition for users who rely on real-time data visualization.
The harmonization of data explorer and live dashboard functionalities will provide a more consistent user experience and will ease the addition of new charts in future version, since all visualizations now rely on a single visualization library.
We encourage all users to migrate their existing dashboards to the Data Explorer to continue leveraging powerful visualization and analysis capabilities.
If you think any feature is missing in the data explorer, please let us know!
Conclusion
We appreciate the ongoing support and contributions from our community in making StreamPipes better with each release. Version 0.97.0 brings improvements in usability, security, and automation, helping users to analyze their IIoT data sources.
As always, we welcome your feedback and suggestions to continue refining and enhancing StreamPipes. Feel free to share your thoughts or get involved through GitHub or mailing list.