Introduction​

Apache StreamPipes 0.97.0 brings exciting new features, enhanced security, and significant UX improvements to streamline your IIoT data processing workflows. This release introduces Adapters & Pipelines as Code, enabling users to define and deploy StreamPipes resources programmatically using YAML or JSON. Security enhancements, such as OPC UA certificate support and Kafka SSL/SASL authentication, improve connectivity and data integrity. Additionally, we have made various UX improvements, including an enhanced Data Explorer and asset-based filtering.

In total, the new release closes 159 issues including new features, bug fixes and improvements.

Read on to discover what’s new in StreamPipes 0.97.0!

Adapters & Pipeline Elements​

Adapter & Pipeline as Code​

StreamPipes now supports managing adapters and pipelines programmatically using compact Json or YAML definitions. These features are ideal for automating deployments, integrating with CI/CD pipelines, and applying infrastructure-as-code practices.

Adapters as Code​

With Adapters as Code, you can define and deploy adapters via a compact description, streamlining the creation of multiple similar adapters and simplifying automation. For example, here's a sample YAML snippet to define an OPC-UA adapter:

name: My OPC Adapter
id: testadapter
description: Test
appId: org.apache.streampipes.connect.iiot.adapters.opcua
configuration:
  - opc_host_or_url: OPC_URL
    opc_server_url: opc.tcp://localhost:62541/milo
  - adapter_type: PULL_MODE
    pulling_interval: 1000
  - access_mode: UNAUTHENTICATED
  - available_nodes:
    - "ns=2;s=Leakage Test Station/temperature"
    - "ns=2;s=Leakage Test Station/pressure"

schema:
  temperature:
    propertyScope: measurement
    label: Temp
    description: Temperature value
    semanticType: http://schema.org/temperature

enrich:
  timestamp: timestamp

transform:
  rename:
    pressure: pressure3

createOptions:
  persist: true
  start: true

This approach considerably simplifies the programmatic creation of adapters Learn more about Adapters as Code

Pipelines as Code​

Pipelines as Code lets you define end-to-end data flows—from data sources through processors to sinks. This function is ideal for the standardization and automation of pipeline deployments. For example, here’s a sample YAML snippet for a simple pipeline:

id: my-pipeline
name: ""
description: ""
pipelineElements:  # Define pipeline components here
  - type: stream     # Data source
    ref: <reference> # Unique reference ID
    id: <data-stream-id> # ID of the stream

  - type: processor  # Data transformation
    ref: <reference> # Unique reference ID
    id: <processor-id> # ID of the processor
    connectedTo:     # Previous pipeline element reference(s)
      - <reference>
    configuration:   # Processor-specific configurations
      - <configuration-option>

  - type: sink       # Data sink (output)
    ref: <reference> # Unique reference ID
    id: <sink-id>    # ID of the sink
    connectedTo:     # Previous pipeline element reference(s)
      - <reference>
    configuration:   # Sink-specific configurations
      - <configuration-option>

createOptions:
  start: <true|false>  # Whether to start the pipeline immediately

The code preview of a pipeline has also been added to the Start Pipeline dialog, which has been further improved to make the execution status of pipelines more clear.

This method provides a programmatic interface to deploy and manage pipelines quickly and easily. Learn more about Pipelines as Code

OPC UA Certificates​

The OPC UA adapter now supports both signed/encrypted and unencrypted communication. Certificate provisioning is handled externally via environment variables, following a dedicated security directory layout. Refer to the OPC UA adapter documentation for further details.

https://github.com/apache/streampipes/pull/3354

Kafka Avro & Certificates​

The Kafka adapter and sink now support SSL and SASL for enhanced security. Additionally, users can provide custom producer/consumer configurations. A new Avro parser has been added along with an input field for specifying the consumer group in the Kafka adapter.

New Features & Deprecations​

Asset Browser​

StreamPipes now allows users to model real-world assets and associate them with StreamPipes resources such as adapters, pipelines, and dashboards. These asset mappings enable intuitive filtering across different modules, making it easier to navigate and manage related resources.

The screenshot below illustrates an example where a StreamPipes instance is configured with two machines and various stations. Clicking on a machine or station dynamically filters the associated resources in the list on the right. This asset-based filtering is available across all main modules of StreamPipes.

The asset overview panel has been redesigned to provide a more intuitive and user-friendly experience. It is possible to assign labels, locations, and ISA95 types to assets. For each asset, resources such as pipelines, adapters and dashboards can be assigned. The asset overview contains an intuitive overview of all linked resources.

To manage locations & sites, we extended the configuration menu with additional options to add a custom map layer provider and to configure the map view.

UX improvements​

File Relocation​

The file upload configuration has been moved from the top-level navigation to the settings menu, improving the overall structure and clarity of the main navigation.

This change does not affect file uploads required by adapters or pipeline elements—these configurations remain unchanged. Further, users can still upload files as before via the files configuration section.

Adapter Data Preview​

Viewing live data streams in the adapter configuration has been improved. By clicking on "details", a full-screen view of the data preview is displayed. Changing data values are highlighted, so that it is ease to track changes. The Code panel shows the configuration in JSON or YAML format.

Data Explorer Configuration Improvements​

Configuring data views in the Data Explorer is now more intuitive and user-friendly. We have enhanced validation, making it easier to select data sources and specify queries while reducing potential misconfigurations.

Key improvements include:

• Default Table Visualization: New data views now default to a table view, minimizing setup effort.
• Improved Time Range Picker: More flexibility in selecting time ranges, including support for different time ranges per widget in Data Lake dashboards.
• Auto-Refresh for Dashboards: Live updates are now possible with the new auto-refresh functionality.
• Better Error Messages: Enhanced feedback when configurations are incorrect to improve usability.
• Custom color coding for pie chart: In the pie chart, custom colors can be mapped to specific data values.
• Traffic light visualization: A new traffic light visualization has been added to the Data Explorer.

Here is a screenshot of the new traffic light visualization:

Pipeline Details​

The pipeline details view has been completely remodeled. It now shows the pipeline in full-screen mode, making it easier to view and edit complex pipelines. A sidebar contains further information on the pipeline, such as logs, pipeline elements, and execution status. In addition, execution metrics are now intuitively accessible right in the pipeline details view.

Deprecation Notice: Dashboard​

The current live dashboard feature in StreamPipes is being deprecated and will be removed in a future release. As a replacement, the Data Explorer dashboards has been enhanced with auto-refresh functionality, ensuring a seamless transition for users who rely on real-time data visualization.

The harmonization of data explorer and live dashboard functionalities will provide a more consistent user experience and will ease the addition of new charts in future version, since all visualizations now rely on a single visualization library.

We encourage all users to migrate their existing dashboards to the Data Explorer to continue leveraging powerful visualization and analysis capabilities.

If you think any feature is missing in the data explorer, please let us know!

Conclusion​

We appreciate the ongoing support and contributions from our community in making StreamPipes better with each release. Version 0.97.0 brings improvements in usability, security, and automation, helping users to analyze their IIoT data sources.

As always, we welcome your feedback and suggestions to continue refining and enhancing StreamPipes. Feel free to share your thoughts or get involved through GitHub or mailing list.

We're excited to share the release of StreamPipes 0.95.0, bringing with it a range of bug fixes and new features. In total, the newest release closes over 200 issues. This release doesn't include many large feature drops, we have worked a lot on making small things smoother and improving StreamPipes under the hood. As we continue to work on the 1.0 release of Apache StreamPipes probably later this year, this release features many improvements to the end-user experience and bug fixes.

Let's delve into the key updates introduced in this release.

Apache StreamPipes saves the day when it comes to connecting to data sources in the IIoT world. Want to do more with your IIoT data than just analyze it in a dashboard? If so, this blog post is for you! We'll show you how to extract historical data from StreamPipes, use it to train a machine learning model, bring the model back to StreamPipes using ONNX, and apply the model to live data.

We're excited to share the release of StreamPipes 0.93.0, bringing with it a range of bug fixes and new features. This release places a strong emphasis on refining the technical foundations of StreamPipes, featuring significant changes that set the stage for our major 1.0 release. Nevertheless, we haven't overlooked the end-user experience and have implemented several enhancements. Let's delve into the key updates introduced in this release.

We are pleased to announce the release of StreamPipes 0.92.0, which includes several bug fixes, performance improvements, and new features. This version focuses on enhancing the technical aspects of StreamPipes. Let's explore the key updates in this release:

The Apache StreamPipes community is delighted to announce the release of Apache StreamPipes version 0.91.0! The biggest highlight of this release is the birth of our official StreamPipes Python library. Apart from that, we have made a lot of improvements under the hood, especially in terms of improving the development experience (e.g., by introducing and enforcing code style rules). Nevertheless, our new version includes more than 90 improvements and bug fixes in total.

The Apache StreamPipes community is pleased to announce the immediate availability of Apache StreamPipes version 0.90.0! In total, this release was focused on stabilization and closes 57 issues with improvements and bug fixes as well as some new features such as adapters and data sinks.

The Apache StreamPipes (incubating) community is delighted to announce the release of Apache StreamPipes version 0.70.0! Key highlights of this release include many enhancements to the data explorer for fast exploration of IoT data and a brand new asset management that allows importing and exporting of StreamPipes resources. In addition, our new version includes more than 50 improvements and bug fixes.

This tutorial shows how you can stream sensor data into StreamPipes from a simulation environment (Factory I/O). The simulation environment is controlled by a "Siemens S7-1200" PLC and shows a conveyor belt which sorts packages by their height. The tutorial is based on the upcoming version 0.70.0 and teaches you how to load the sensor data, build a pipeline, preprocess the data and create visualizations.

The Apache StreamPipes (incubating) community is pleased to announce Apache StreamPipes version 0.69.0! The most notable highlights of this release include a completely reworked data explorer for quick exploration of IoT data and advanced user and access rights management. In addition, our new release includes more than 80 improvements and bug fixes.