So why not use Podman Desktop, it’s Docker compatible? Well yes, almost, would be the answer.
For Podman Desktop to handle Apache StreamPipes we need to do a few tricks.

This guide walks you through installing and running Apache StreamPipes using Podman inside a WSL environment and will try to explain to you the how’s and why’s.

Prerequisits​

After installing Podman Desktop you need to run the setup to enable Podman core.

Alter this you can setup a virtual machine in which to host Apache Streampipes on.

Now we have to embark on a voyage into the (dark) Linux terminal country, so that we can later enjoy the use of a familiar Windows UI in Podman Desktop.

Connect to Your Podman WSL Machine​

Open PowerShell to get a list of available machines.

wsl -l

Or you can see the Podman machine in the tray menu

In PowerShell connect to the Podman machine.

wsl -d podman-machine-default

Install Required Tools​

Podman uses Fedora Linux and the dnf package manager. Install the following packages to make your life easier.

• podman-compose

sudo dnf install podman-compose.noarch 

• unzip

sudo dnf install unzip.x86_64 

• nano

sudo dnf install nano.x86_64 

• mc

sudo dnf install mc.x86_64 

Get Apache StreamPipes​

The current release can be downloaded here.

For ease save it in a subfolder on your C-drive, e.g. c:\Download
Lets make a folder in WSL to hold our files, and go into it.

mkdir StreamPipes
cd StreamPipes

We can now directly copy the downloaded *.zip file to our WSL, note the version number on the zip file.

cp /mnt/c/Download/apache-streampipes-0.98.0-source-release.zip /home/user/StreamPipes/

If the terminal is still a bit outside you'r comfort zone Midnight Commander can help you.

mc

It is possible to reach your WSL from Windows filemanager, but unzipping files will give issues due to long filenames. Therefore the approch of handling the files from WSL is preferable.

Unpack and Navigate to the Compose Directory​

Unzip the package by using the unzip command and navigate to the extracted folder and go to 'installer/compose' directory.

unzip apache-streampipes-0.98.0-source-release.zip

cd apache-streampipes-0.98.0/installer/compose

Update docker-compose.yml​

Modify the docker-compose-yml with nano

nano docker-compose.yml

And add the following.

- ./nginx_config/default.conf:/etc/nginx/conf.d/default.conf:ro

Looking like this.

Remove this line since it is not relevant for this setup.

environment:
      - SP_HTTP_SERVER_ADAPTER_ENDPOINT=extensions-all-iiot:8090

And yes you can copy paste from windows into nano. Use CRTL + x to quit, and answer ‘y’ to save the file. Here is how the docker-compose.yml should look loke

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# global logging
x-logging:
  &default-logging
  options:
    max-size: '12m'
    max-file: '5'
  driver: json-file

services:

  # StreamPipes services
  backend:
    image: "${SP_DOCKER_REGISTRY}/backend:${SP_VERSION}"
    depends_on:
      - couchdb
    environment:
      - SP_PRIORITIZED_PROTOCOL=nats
    volumes:
      - backend:/root/.streampipes
    logging: *default-logging
    restart: unless-stopped
    networks:
      spnet:

  ui:
    image: "${SP_DOCKER_REGISTRY}/ui:${SP_VERSION}"
    ports:
      - "80:8088"
    depends_on:
      - couchdb
      - backend
    volumes:
      - nginx:/etc/nginx/
      - ./nginx_config/default.conf:/etc/nginx/conf.d/default.conf:ro
    logging: *default-logging
    restart: unless-stopped
    networks:
      spnet:

  couchdb:
    image: couchdb:3.3.1
    environment:
      - COUCHDB_USER=admin
      - COUCHDB_PASSWORD=admin
    volumes:
      - couchdb:/opt/couchdb/data
    logging: *default-logging
    restart: unless-stopped
    networks:
      spnet:

  influxdb:
    image: influxdb:2.6
    environment:
      - INFLUXDB_DATA_ENGINE=tsm1
      - INFLUXDB_REPORTING_DISABLED=false
      - INFLUXDB_ADMIN_ENABLED=true
      - DOCKER_INFLUXDB_INIT_USERNAME=admin
      - DOCKER_INFLUXDB_INIT_PASSWORD=sp-admin
      - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=sp-admin
      - DOCKER_INFLUXDB_INIT_ORG=sp
      - DOCKER_INFLUXDB_INIT_BUCKET=sp
      - DOCKER_INFLUXDB_INIT_MODE=${SP_INFLUX_INIT_MODE}
    volumes:
      - influxdb:/var/lib/influxdb
      - influxdb2:/var/lib/influxdb2
    logging: *default-logging
    restart: unless-stopped
    networks:
      spnet:

  nats:
    image: nats
    restart: unless-stopped
    networks:
      spnet:

  # Lite version pipeline elements
  extensions-all-iiot:
    image: "${SP_DOCKER_REGISTRY}/extensions-all-iiot:${SP_VERSION}"
    logging: *default-logging
    restart: unless-stopped
    networks:
      spnet:

volumes:
  backend:
  couchdb:
  influxdb:
  influxdb2:
  nginx:


networks:
  spnet:
    driver: bridge

Add Custom Nginx Configuration​

Create a 'nginx_config' directory

mkdir nginx_config

In that directory you add a 'default.conf' with the help of Nano.

cd nginx_config
nano default.conf

The following nginx.conf is almost identical to the default Docker-based setup, with the only change being that the resolve parameter is removed as it doesn't work reliably with Podman's networking.

server {
  listen       8088 default_server;
  root /usr/share/nginx/html;
 
  # Enable gzip compression
  gzip on;
  gzip_static on;
  gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
  gzip_proxied  any;
  gzip_vary on;
  gzip_comp_level 6;
  gzip_buffers 16 8k;
  gzip_http_version 1.1;
 
  # StreamPipes Backend Proxy

  location /streampipes-backend {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-NginX-Proxy true;
    proxy_pass http://backend:8030;
    proxy_ssl_session_reuse off;
    proxy_set_header Host $http_host;
    proxy_redirect off;
  }
}

Now we are almost there, and you are getting good at this terminal thing, so is there a use for Podman Desktop, you ask yourself?

Start Up StreamPipes Using Podman Compose​

Do

sudo podman-compose -f docker-compose.yml up -d

to import and run all containers.

Congratulations, now we can start to use the strongpoints of Podman which enable us in a fast and terminal free way to make backups of our projects or scale them further with Kubernetes.

Happy StreamPiping on Windows with Podman Desktop

Functions are ideal when:

• A full processor would be overkill
• You want logic that runs with the extensions service lifecycle
• You need direct access to incoming events

Previously, functions could maintain in-memory state during runtime, but any restart of the extensions service resulted in a complete state reset.

We now provide a new functionality in developer preview that introduces typed state persistence for Functions, allowing state to survive service restarts.

This allows functions to survive restarts and resume exactly where they stopped.

Why Stateful Functions?​

Many real-world use cases require minimal but persistent runtime state:

• Counting processed events
• Tracking the last processed timestamp
• Storing threshold configurations
• Keeping lightweight usage statistics

Previously, developers had to implement external persistence manually.
Now, state management is built directly into the Function lifecycle.

The New State Store API​

The FunctionContext now provides access to a typed state store:

StateStore<T> getStateStore(Class<T> stateClass)

This API allows you to:

• Load persisted state during startup
• Persist state on shutdown

State handling is fully optional and backward compatible. If you don’t request a state store, your function behaves exactly as before.

Lifecycle Integration​

Functions still follow the same lifecycle:

• onServiceStarted(FunctionContext context)
• onEvent(Event event, String streamId)
• onServiceStopped()

The only difference is that you can now initialize a StateStore during startup.

Example: Threshold Monitoring Function​

Let’s implement a simple function that:

• Counts processed events
• Tracks the last processed timestamp
• Logs a message when a threshold is reached

1. Define the State Object​

State is defined as a typed POJO.

public class ProcessingState {

    private int schemaVersion = 1;
    private long processedEventCount = 0;
    private long lastProcessedEpochMs = 0;
    private long alertThreshold = 1000;
    
   public ProcessingState() {
   }

    // getters/setters omitted for brevity
}

2. Implement the Stateful Function​

public class ThresholdMonitoringFunction implements StreamPipesFunction {

    private StateStore<ProcessingState> stateStore;
    private ProcessingState state;

    @Override
    public String getFunctionId() {
        return "threshold-monitoring-function";
    }

    @Override
    public List<String> requiredStreamIds() {
        return List.of("your-stream-resource-id");
    }

    @Override
    public void onServiceStarted(FunctionContext context) {

        // Request typed state store
        this.stateStore = context.getStateStore(ProcessingState.class);

        // Load existing state or use default
        this.state = stateStore.load(new ProcessingState());
    }

    @Override
    public void onEvent(Event event, String streamId) {

        state.setProcessedEventCount(
            state.getProcessedEventCount() + 1
        );

        state.setLastProcessedEpochMs(
            System.currentTimeMillis()
        );

        if (state.getProcessedEventCount() 
                >= state.getAlertThreshold()) {

            // Handle alert once when threshold is reached
        }
    }

    @Override
    public void onServiceStopped() {

        // Persist state before shutdown
        stateStore.persist(state);
    }
}

State Persistence Lifecycle​

The persistence mechanism follows a simple and predictable pattern:

Startup​

state = stateStore.load(new ProcessingState());

• If persisted state exists → it is restored
• If not → the provided default object is used

Runtime​

• State is modified in memory during onEvent

Shutdown​

stateStore.persist(state);

• The current state is stored before the service stops

After a restart, the function continues with the restored values.

No external database setup. No manual serialization logic. No additional infrastructure required.

Backward Compatibility​

Stateful behavior is opt-in.

Existing functions:

• Continue to work unchanged
• Require no migration
• Remain fully stateless unless getStateStore(...) is used

Conclusion & Feedback​

Functions and state persistence are currently available in developer preview, and the API may evolve.

We are open to feedback from developers, if you try out stateful functions, let us know your thoughts and suggestions.

This release focuses on consolidating visualization and dashboard functionality, improving operational transparency, and modernizing the underlying architecture. Highlights include dashboard cloning and kiosk mode, new aggregation functions, enhanced OPC UA authentication and certificate handling, file logging support, and a modernized UI.

In total, this release closes around 100 issues across enhancements, bug fixes, and maintenance tasks.

Read on to discover what’s new in StreamPipes 0.98.0!

Industrial Connectivity & OPC-UA​

Redesign of the adapter creation page​

StreamPipes Connect is the central interface to connect a wide variety of industrial protocols with Apache StreamPipes. In this release, a new overview page simplifies selection of appropriate adapters, taking less space and allowing users to get a better overview over available connectors.

OPC UA X.509 Authentication & Certificate Management​

Support for OPC-UA security is greatly enhanced in 0.98.0. We added a new authentication option that supports X.509 user authentication. Users can now directly upload their X.509 certificates which are securely stored within StreamPipes.

For encrypted communication, we also added enterprise features to handle OPC-UA certificates. Until now, certificates had to be managed through environment variables. In the new release, server certificates can be centrally managed from the user interface, simplifying secure deployments. In case a non-trusted certificate is provided by the OPC-UA server, the certificate is stored in the StreamPipes core for manual approval. In the settings menu, admins can explicitly trust or reject a given certificate. The UI view also contains extended certificate information, making it easy to get an overview of all OPC-UA certificates that are currently used.

Version 0.98.0 also supports more OPC-UA data types such as better support of ExtensionObjects, allowing more complex OPC-UA data structures to be processed.

These improvements strengthen StreamPipes as a secure and enterprise-ready IIoT integration platform.

Other improvements​

Minor improvements for other adapters include:

• Configurable wait and lease times for PLC connectors.
• Improved adapter state handling and recovery options.
• Better schema guessing and duplicate field handling in OPC-UA adapters.

Dashboards & Data Explorer​

Harmonized Data Explorer & Dashboard Experience​

The consolidation of visualization tools is finished in 0.98.0. The Chart and Dashboard views have been further harmonized, ensuring consistent configuration and behavior. The deprecated dashboard has been removed and the new dashboard tool is fully in place. Users can now add any chart directly to the live dashboard.

This work lays the foundation for faster innovation and a more consistent visualization experience going forward.

Dashboard Cloning & Kiosk Mode​

Dashboards have received further several long-awaited improvements:

• Dashboard cloning allows users to quickly duplicate existing dashboards and reuse configurations.
• Kiosk mode enables distraction-free, full-screen dashboards—ideal for shop floor displays and control rooms.
• Improved configuration panel now clearly indicate missing or incomplete dashboard and chart configurations.

These enhancements significantly simplify dashboard reuse and deployment in operational environments.

New Visualizations & Chart Improvements​

Several new visualization and charting capabilities were added: The status heatmap visualization was enhanced with a dedicated color-mapping component, in the chart appearance settings, additional chart-specific configurations were added, including improved heatmap configuration with customizable min/max values. To improve usability, icons and descriptions were added to the chart type dropdown, making chart selection more intuitive.

New Aggregation Functions​

The chart view now supports additional aggregation functions:

• MEDIAN
• SPREAD
• STDDEV

These functions enable more advanced statistical analysis directly within pipelines and the Chart view.

Excel export​

Besides CSV and JSON, datasets can now be directly exported to a Microsoft Excel format (xlsx). In addition, Excel templates can be uploaded that can be used to add any company-specific header data to an Excel export.

Security & Authentication​

Since we aim for enterprise-grade security, we added additional advanced configuration options to integrate Apache StreamPipes with 3rd-party authentication providers.

OAuth & External Authentication Improvements​

Security and identity management received several upgrades.

External role assignment is now supported in the OAuth configurations. Besides default roles, users can now be assigned their roles directly from an external authentication service. To further increase flexibility, StreamPipes-internal roles and groups can be assigned an alternate ID, which can be used to map external role names to StreamPipes roles and groups.

OAuth and OpenID configurations are now loaded at startup time, so that invalid settings are detected earlier upon startup.

User Management Enhancements​

User transparency was improved with:

• Tracking and displaying user creation date and last login date in the user configuration.
• Optional terms acknowledgment, allowing organizations to enforce acceptance of usage terms.

UX & UI Improvements​

Modernized UI with Material 3​

The StreamPipes UI has been upgraded to Material Design 3, delivering a cleaner, more modern look and feel. Additional UI improvements include:

• Broader use of design tokens so that theming becomes easier for external developers.
• Panel resize components for flexible layouts.
• Improved breadcrumb navigation in dashboards.

StreamPipes becomes multilingual​

Release 0.98.0 starts to add internationalization features to Apache StreamPipes. While this will be a greater effort spanning multiple releases, this new release lays the groundwork for a completely localized Apache StreamPipes experience.

As a first feature, we've localized most parts of the UI to the german language. More languages will follow in the future - if you are willing to help us translate StreamPipes to more languages, let us know!

Actions menu​

To ensure a consistent design and a clean, streamlined look and feel, all data tables across StreamPipes now use a unified actions menu. Instead of displaying multiple action buttons inline, available actions are grouped under a three-dot menu. Simply hover over the icon to reveal the actions for a given resource, making tables easier to scan while keeping all functionality readily accessible.

Developer experience​

Migration to the New Data Processor API​

The migration to the new Data Processor API continues, improving consistency and performance across pipeline elements. As part of this effort:

• The legacy StreamPipesDataProcessor and StreamPipesDataSink APIs are now deprecated.
• The Flink-based wrapper has also been deprecated and will be removed in a future release.

Developers are encouraged to migrate to the new APIs to ensure long-term compatibility.

Custom theming​

With the upgrade of the design system, we changed the way how the StreamPipes UI can be customized with company-specific color settings. Settings that can be used for custom theming have been isolated in two files (_custom-variables.scss and _theme-colors.scss). We encourage developers to investigate the modified prebuild script and the two scss files to generate individual theming files.

Breaking Change​

Default Messaging System Switched to NATS 💣​

The default docker-compose setup in the installer directory now uses NATS instead of Apache Kafka for internal messaging.

⚠️ Important:
This change only affects users who are updating from a previous installation.
If your existing StreamPipes instance uses Kafka, continue using the provided docker-compose.kafka.yml file.

Deprecations ⚠️​

The following components and features are deprecated and scheduled for removal in future versions:

• Deprecated domain property APIs (use semantic type API)
• Nested rule creation functionality
• Legacy configurable data format APIs
• Flink wrapper
• IO-Link IFM adapter
• NetIO adapter

Users are encouraged to migrate early to avoid disruption in upcoming releases.

Other improvements​

As usual, StreamPipes 0.98.0 includes a variety of minor improvements, bug fixes and dependency updates. Users are encouraged to update as soon as possible.

Conclusion​

Apache StreamPipes 0.98.0 delivers major improvements in dashboarding, OPC UA integration, security, and user experience, while continuing the architectural cleanup and modernization started in previous releases.

We would like to thank all contributors and community members who made this release possible.
As always, we welcome your feedback—join the discussion on GitHub or the mailing list, and help shape the future of StreamPipes!

This blog post introduces an easy, scalable approach to bridging this IT/OT divide using IO-Link, a standardized protocol for industrial sensors, and Apache StreamPipes. We'll walk you through a complete setup—from connecting IO-Link sensors to visualizing actionable insights—with a focus on a vibration sensor and an IO-Link Master.

Whether you're building condition monitoring systems, predictive maintenance pipelines, or just starting with industrial data analytics, this guide will show how StreamPipes makes it simple to move from raw sensor data to intelligent decisions.

Why Choose IO-Link Sensors?​

IO-Link is an increasingly popular choice in modern industrial environments—and for good reason. It enables seamless sensor integration without the headaches of proprietary protocols or specialized wiring.

Benefits of IO-Link for IIoT​

IO-Link sensors support a wide range of measurements including vibration, temperature, flow, and pressure. Thanks to IODD (IO Device Description) files, these sensors are interoperable across manufacturers, making configuration fast and reliable. With simple three-wire cabling and support for up to eight sensors via a single IO-Link master, IO-Link dramatically reduces the complexity of sensor installations.

What is IO-Link?​

IO-Link is a globally standardized communication protocol (IEC 61131-9) used to connect industrial sensors and actuators. Unlike traditional analog or proprietary fieldbus protocols, IO-Link communicates digitally and supports detailed sensor diagnostics, parameterization, and status monitoring. It also integrates easily into IT environments, helping to close the gap between OT data generation and IT-driven analytics.

OI4: The Key to MQTT-Based IO-Link Integration​

To simplify integration with modern IoT platforms, the Open Industry 4.0 Alliance (OI4) provides a standardized interface specification for IO-Link over MQTT. This includes well-defined JSON structures, topic hierarchies, and security mechanisms such as TLS encryption and authentication. OI4-compliant devices ensure that sensor data can be consumed and interpreted reliably—no matter the vendor.

Use Case: Multi-Sensor Condition Monitoring​

Let’s consider a typical industrial use case: monitoring the condition of electric motors using vibration, temperature, and pressure sensors. These sensors are connected to a Hilscher sensorEDGE FIELD gateway, which publishes all sensor values over MQTT using the OI4 format.

Real-Time Insights with StreamPipes​

In Apache StreamPipes, this data can be ingested using the OI4 adapter and routed through various processing pipelines. For instance, you can configure alerts for high vibration (e.g., aPeak > 8.0 m/s²) and high temperature (> 80°C) to detect early signs of mechanical stress or overheating. Additionally, a rolling average calculation on pressure data helps identify leaks or irregularities.

This setup not only enables instant notifications via email but also stores the processed data in a timeseries database for long-term trend analysis—reducing the need for manual inspections and improving asset reliability.

Expanding Beyond Vibration​

IO-Link sensors can be used across many other domains in industrial environments. Here are a few examples:

Integration Overview: From Hardware to StreamPipes​

Setting up the hardware and data flow is straightforward—even for teams without OT experience.

After connecting the VVB001 vibration sensor to an IO-Link Master, it is automatically detected and begins sending data. This gateway then publishes sensor readings to a local or remote MQTT broker in a structured JSON format, following the OI4 specification.

Apache StreamPipes, with its OI4 adapter, subscribes to these MQTT topics, automatically infers the data schema, and prepares the events for downstream processing—completely eliminating the need for manual parsing or custom integration scripts.

StreamPipes Configuration Guide​

Setting up the data stream in StreamPipes is quick and intuitive. Here's how it works:

Adding the OI4 Adapter​

Start by opening the Connect section in the StreamPipes UI. From the list of available adapters, choose the OI4 MQTT Adapter. This adapter is purpose-built for consuming OI4-compliant MQTT streams, so it comes with built-in support for schema detection and mapping.

Configuring the Adapter​

In the configuration screen, specify the connection to your MQTT broker. For this use case, the broker runs at tcp://192.168.188.73:1883, with unauthenticated access. Select the desired sensor type (VVB001) and confirm that all available sensors should be included.

You’ll also need to enter the App ID that identifies the publishing application—in this case, hilscher.com/.../netfield-app-opc-ua-io-link-adapter-1. Once configured, click Next, and StreamPipes will automatically detect the event schema based on the live payloads from the broker.

Deep Dive: VVB001 Vibration Sensor Data​

The VVB001 sensor is an IO-Link-enabled device capable of measuring several key vibration and temperature metrics. These metrics are essential for detecting imbalances, misalignments, and early-stage mechanical wear in rotating equipment.

Key Metrics Explained​

• vRMS (Velocity RMS): Indicates the effective vibration velocity over time. Higher values suggest increasing mechanical stress or imbalance.
• aPeak (Acceleration Peak): Captures the maximum acceleration value in a sampling window. Sudden spikes may point to impacts or misalignment.
• aRMS (Acceleration RMS): Reflects consistent vibration intensity. Useful for trend analysis and baseline monitoring.
• Crest Factor: Calculated as the ratio between peak and RMS acceleration. A rising Crest Factor often signals early-stage bearing damage.
• Temperature: Measures the internal temperature of the sensor or its surroundings—crucial for thermal condition monitoring.

Sample JSON Payload​

{
  "Temperature": 37.9,
  "sensorId": "000008740649",
  "Crest": 4,
  "vRms": 0,
  "aRms": 0,
  "aPeak": 0.2,
  "timestamp": 1600599689805
}

Preprocessing in StreamPipes​

Before this data can be used for analytics or alerting, it must be scaled and cleaned. For example:

• vRMS values need to be multiplied by 0.0001.
• aPeak, aRMS, Temperature, and Crest values are scaled by 0.1.

The StreamPipes connect functionality van handle this effortlessly. You can edit the event schema to apply scaling and remove unused fields.

Visual Data Exploration​

Once the data is flowing, you can use the Data Explorer in StreamPipes to monitor sensor metrics in real-time. Line charts help you visualize trends, spikes, and anomalies in vibration and temperature levels.

Whether you’re fine-tuning processing logic or monitoring asset health, StreamPipes visualizations make it easy to detect anomalies and trends in real time.

Alerting with Pipelines​

With pipelines, you can define rule-based logic to trigger alerts based on sensor values. For example, if the Crest Factor exceeds a defined threshold, you can send an email notification to maintenance staff.

Pipelines are extensible—you can integrate third party tools, store data in databases, or implement predictive analytics using machine learning models.

Conclusion​

Apache StreamPipes and IO-Link together offer a highly accessible yet powerful IIoT solution. From a simple dashboard setup to complex, multi-sensor condition monitoring pipelines, this integration enables IT teams to drive real-time insights from machine data—without needing a background in industrial automation.

Introduction

While an unencrypted security policy is the way most users try when they start testing StreamPipes for their first time, production-grade scenarios typically require encrypted communication.

In case you are already using Apache StreamPipes, you might have noticed that the OPC UA adapter now supports encrypted security policies:

In this guide, I'll show how to set up a secure connection between your OPC UA server and Apache StreamPipes. This guide is intended to be a step-by-step-guide starting from certificate generation to the configuration of the OPC UA adapter in StreamPipes.

I'll show a working example on a local installation using Eclipse Milo as the OPC UA server. The same steps can be applied to production systems by integrating certificates into the volume of the extension service. There are other ways to configure OPC-UA secure connections, but this guide will focus on a complete approach using self-signed certificates for beginners.

Prerequisites

• Apache StreamPipes 0.97.0 or newer
• An OPC-UA server supporting security policies Sign and Sign & Encrypt
• OpenSSL installed on your system

Generate client certificates

When creating a new OPC-UA adapter in Apache StreamPipes, the extension service where the adapter is running in serves as the client. First, we need to create a client certificate for the client. In this example, we'll create a self-signed certificate.

[req]
default_bits       = 2048
prompt             = no
default_md         = sha256
req_extensions     = req_ext
x509_extensions    = v3_ca
distinguished_name = dn

[dn]
CN = stream-pipes-client

[req_ext]
subjectAltName = @alt_names
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment, keyCertSign
extendedKeyUsage = serverAuth, clientAuth

[v3_ca]
subjectKeyIdentifier = hash
basicConstraints = CA:false
subjectAltName = @alt_names
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment, keyCertSign
extendedKeyUsage = serverAuth, clientAuth

[alt_names]
URI.1 = urn:org:apache:streampipes:opcua:client

When using a self-signed certificate, it is requried to to provide the CA:false flag under basicConstraints. In addition, note the subject alternative name that later needs to be matched by the application URI of the OPC-UA client.

The next step is to generate the certificate and import it into a PKCS12 keystore. The keystore is used to store the key pair we are generating in the first step.

openssl req -x509 -nodes -days 365 \
  -newkey rsa:2048 \
  -keyout client.key \
  -out client.crt \
  -config cert.conf \
  && openssl pkcs12 -export \
  -in client.crt \
  -inkey client.key \
  -out client.p12 \
  -name streampipes-client

When I was asked for a password, I chose streampipes. You can choose any password you like, but make sure to remember it, as we will need it in the next step.

Check that the keystore lists the correct key by executing:

keytool -list -v -storetype PKCS12 -keystore client.p12

You'll see the key you've just generated.

Configure StreamPipes to support encrypted connections

To enable the OPC UA adapter to use the generated client certificate, we need to provide a few environment variables to StreamPipes.

Here is a list of all environment variables that are available:

• SP_OPCUA_SECURITY_DIR (default: /streampipes-security/opcua): This is the directory where the keystore and trusted/rejected server certificates are stored.
• SP_OPCUA_KEYSTORE_FILE (default: keystore.pfx): This is the keystore file that contains the client certificate.
• SP_OPCUA_KEYSTORE_PASSWORD (default: password): The password for the keystore file.
• SP_OPCUA_KEYSTORE_TYPE (default: PKCS12): The type of the keystore file.
• SP_OPCUA_KEYSTORE_ALIAS (default: apache-streampipes): The alias of the client certificate in the keystore.
• SP_OPCUA_APPLICATION_URI (default: urn:org:apache:streampipes:opcua:client): The application URI of the client certificate.

We could have made the previous configuration easier by reusing some of the defaults. For learning purposes, we used some other names for the alias and keystore filename.

In my setup, I've started a local StreamPipes instance. When using Docker, you can add the environment variables to the extensions service (e.g., extensions-all-iiot). In my local instance, I just add the emnvironment variables to my IDE settings.

I add the following variables:

SP_OPCUA_SECURITY_DIR=/home/user/streampipes-security/opcua
SP_OPCUA_KEYSTORE_FILE=client.p12
SP_OPCUA_KEYSTORE_PASSWORD=streampipes
SP_OPCUA_KEYSTORE_ALIAS=streampipes-client

Connecting to an OPC-UA server

For this demo, I'm using the Eclipse Milo OPC UA server. You can download it from here. Once the server is running, you can connect to it using the following URL:

opc.tcp://localhost:62541/milo

Now it's time to switch to StreamPipes!

Go to Connect -> New adapter -> OPC UA and enter the following configuration:

Pull interval 1000
Incomplete Events Ignore
Security Mode Sign & Encrypt
Security Policy Basic256Sha256
URL opc.tcp://localhost:62541/milo

Defocus the URL input field and wait for the OPC-UA server to connect. After a few seconds, you should see an error message appearing:

This is expected, as we haven't added the client certificate to the trusted certificates of the OPC-UA server yet. So we have to let Milo know that the client certificate can be trusted. In my local setup, I need to move the client certificate from the rejected to the trusted/certs folder as seen below:

So let's try again by clicking Reload in the Available Nodes section of the adapter configuration page. We'll get another error saying that the server certificate is not trusted. This is again expected, as we haven't added the server certificate to the trusted certificates of the OPC-UA client yet.

To do that, we can have a look at the folder structure under SP_OPC_UA_SECURITY_DIR:

Several new folders have been created by the Apache StreamPipes OPC-UA client. The trusted folder contains the trusted server certificates, while the rejected folder contains the rejected server certificates. We'll move the server certificate from the rejected folder to the trusted/certs folder.

No we'll try again - and this time, we should see the available nodes in the adapter configuration page:

You'll see the beautiful tree view where you can navigate through all OPC-UA nodes.

Now you can select the nodes you want to use in your adapter.

That's it! You've successfully configured the OPC-UA adapter to use encrypted communication with self-signed certificates.

However, with Apache StreamPipes, even limited data — such as a simple boolean signal indicating activity — can be leveraged to implement usage-based maintenance. In this blog post, we demonstrate how this is achieved using a real-world example: a packaging machine that processes goods at high throughput. You'll learn how to count machine operations, store event data, visualize usage over time, and trigger notifications when maintenance is due — all with just a few clicks in StreamPipes.

The Scenario: Packaging Machine Monitoring​

Let’s consider a packaging machine that wraps goods into individual packages. The wear and tear on this machine is directly related to the number of items processed. However, due to the machine’s age, only minimal data is available.

Currently:

• Maintenance occurs at fixed intervals.
• These intervals were determined on the basis of the technicians' experience.
• There is suspicion that maintenance is done too frequently, leading to unnecessary costs.

With Apache StreamPipes, we’ll build a pipeline that:

• Counts packaging operations in real time.
• Stores data for analysis and visualization.
• Sends notifications once a maintenance threshold is reached.

The Data: A Simple Boolean Signal​

The machine provides a boolean flag — active — which is true when a packaging operation occurs. This flag is streamed using an Apache StreamPipes S7 adapter. The stream also includes a few additional process parameters:

Part 1: Absolute Counter with Notification​

We’ll begin by building a pipeline that counts how often the machine packages a product and sends an email alert when a predefined threshold is reached.

Pipeline Overview​

A simple StreamPipes pipeline for monitoring the parts produced at the machine

Step-by-Step Guide​

1. Add the Data Source

   • Drag the data stream of the packaging machine into the pipeline editor.

2. Insert the Boolean Counter Processor

   • Monitors a boolean field and counts transitions (e.g., false → true).
   • Configuration:
     • Boolean Field: active
     • Flank Parameter: FALSE → TRUE

   The processor emits a new event each time the machine packages an item, incrementing a counter value.

3. Persist the Count

   • Add the Data Lake sink to store these events for visualization.

4. Add a Threshold Alert

   • Insert the Numerical Filter processor.
     • Field: counter
     • Filter Operation: >
     • Threshold: 100000
   • Connect this to the Email Notification sink.
     • Configure recipient, subject, and set the Silent Period to avoid spamming the team.

5. Visualize in Dashboard

   • Start the pipeline.
   • In the dashboard, add an Indicator Widget displaying the counter value.

The live dashboard with an indicator chart showing the current number of goods produced

This setup gives you a live counter of how many products were packaged and notifies the maintenance team when attention is needed.

Part 2: Aggregated Counting Over Time​

While a cumulative counter is useful, often it's more insightful to understand how usage varies over time — for example, how many items are processed each day.

A Time-Aggregated Approach​

Instead of a total counter, we’ll now generate a single event for each packaging operation and then use the Data Explorer to visualize activity per hour or per day.

Alternative Pipeline Using Signal Edge Filter​

Pipeline to recognize when a part is being processed

Step-by-Step:

1. Replace the Boolean Counter with a Signal Edge Filter processor.

   • Filters events based on signal transitions.
   • Configuration:
     • Boolean Field: active
     • Signal Edge: FALSE → TRUE
     • Delay: 0
     • Output Event Selection: First

2. Store the filtered stream in the Data Lake sink.

This gives one clean event per packaging operation — ideal for aggregation.

Visualization in the Data Explorer​

Now that we have one event per packaging action, we can analyze the data over time in the Data Explorer.

1. Open the Data Explorer
2. Table View Configuration:
   • Data Source: Select the stream used in the pipeline.
   • Aggregation: Group by Hour or Day.
   • Field Aggregation: Use count on any field to count the number of events.

3. Bar Chart Visualization:
   • Switch to a Time-Series Chart.
   • Select the counted field.
   • Choose bar from the chart type dropdown.

You now have a clear, time-based view of machine usage. This helps spot trends, detect unusual downtimes, and better plan maintenance.

Conclusion​

In this post, we’ve shown how even limited sensor data — like a simple boolean signal — can be used to build powerful maintenance logic using Apache StreamPipes. With just a few clicks, you can:

• Count operations and trigger maintenance alerts
• Store and visualize historical data
• Understand machine usage over time with time-based aggregations

By replacing static maintenance intervals with actual usage insights, you can reduce unnecessary work and improve efficiency — all without needing to retrofit expensive new sensors.

Need a quick sentiment check, a rolling summary, or a safety rule that’s just a bit too fuzzy for SQL?
With the Multi-Model Prompt Processor you can drop the full might of OpenAI, Anthropic, or your own Llama into any Apache StreamPipes pipeline—no extra code required.

Below you’ll find:

• A quick tour of the options
• Five every day use-cases
• A hands-on GIF tutorial
• Tips on writing prompts that return only what you need
• A full list of the models that work out of the box

What does the processor do?​

The processor sends your prompt + current event data to the model, grabs the reply, and writes it to a new field llmResponse; ideal when the answer must flow straight into the next processor.

Five ways to use it​

(Replace the $placeholders$ with the actual stream attributes in the mapping dialog.)

Tutorial – from stream to JSON warning​

Below is a short clip showing the processor in action in a temperature pipeline.
The prompt forces the model to output a two-field JSON—perfect for machine parsing.

Model: claude-3-5-sonnet-20241022
Prompt Used:

You are an assistant that ALWAYS replies with a single JSON object
containing two string fields: "status" and "message".

Rules
1. If the temperature is strictly greater than 45 °C:
     • "status" must be "WARNING"
     • "message" must explain that the temperature is above the safe limit
2. Otherwise:
     • "status" must be "OK"
     • "message" must note that the temperature is within the safe range
3. Do not output anything except the JSON object.

### Examples

Temperature: 50
{
  "status": "WARNING",
  "message": "Temperature exceeds the 45 °C threshold"
}

Temperature: 30
{
  "status": "OK",
  "message": "Temperature is within the safe range"
}

Temperature: 45
{
  "status": "OK",
  "message": "Temperature is within the safe range"
}

Temperature: 60
{
  "status": "WARNING",
  "message": "Temperature exceeds the 45 °C threshold"
}

The model now returns either:

{
  "status": "WARNING",
  "message": "Temperature exceeds the 45 °C threshold"
}

or

{
  "status": "OK",
  "message": "Temperature is within the safe range"
}

Nothing else—exactly what we want for downstream processors.

Prompt writing tips for event streams​

1. State the output format first: “Reply with OK or WARNING only.”
2. Give one or two concrete examples – LLMs copy patterns.
3. End with a clear cue: “Answer:” or “Output:”.
4. Keep temperature ≤ 0.3 when strict structure matters.
5. Short replies reduce context size in Windowed / Full modes.

Supported models​

claude-3-7-sonnet-20250219
claude-3-5-sonnet-20241022
claude-3-5-haiku-20241022
claude-3-5-sonnet-20240620
claude-3-opus-20240229
claude-3-sonnet-20240229
claude-3-haiku-20240307
claude-2.1
claude-2.0

gpt-3.5-turbo              gpt-3.5-turbo-1106     gpt-3.5-turbo-0125
gpt-3.5-turbo-16k          gpt-4                  gpt-4-0613
gpt-4-turbo-preview        gpt-4-1106-preview     gpt-4-0125-preview
gpt-4-turbo                gpt-4-turbo-2024-04-09 gpt-4-32k
gpt-4-32k-0613             gpt-4o                 gpt-4o-2024-05-13
gpt-4o-2024-08-06          gpt-4o-2024-11-20      gpt-4o-mini
gpt-4o-mini-2024-07-18     o1                     o1-mini
o1-2024-12-17              o1-mini-2024-09-12     o1-preview
o1-preview-2024-09-12      o3-mini-2025-01-31

Final thoughts​

Whether you need a lightweight rule engine, live summaries, or just a smarter filter, the Multi-Model Prompt Processor drops straight into your pipeline and starts working within minutes.

Give it a try, and let us know what you build!

The JavaScript Evaluator Processor in Apache StreamPipes allows you to define custom event logic with just a few lines of JavaScript. Whether you want to modify values, filter events, or even add stateful logic, this processor empowers you to do just that—without writing a full processing element in Java.

Let’s take a closer look at how it works, what the configuration options are, and how you can use it in three different ways.

What is the JavaScript Evaluator?​

Sometimes you need to apply custom logic to your data stream. Maybe it's a quick transformation or a complex filter that isn’t supported out of the box. The JavaScript Evaluator Processor is designed exactly for these cases.

With a simple JavaScript function, you can:

• Modify existing fields
• Add new fields
• Filter out unwanted events
• Maintain state across events

All you need is a little coding and a bit of creativity.

Configuring the JavaScript Processor​

When adding the JavaScript Evaluator to your pipeline, you’ll notice two key configuration sections:

1. The JavaScript Function
   This function is triggered once for each incoming event.

2. The Output Schema
   This defines the structure of your transformed events. It’s important to ensure that the output schema matches what your function returns—otherwise, downstream processors might not be able to validate the events correctly.

Let’s explore some examples that show different ways to apply custom logic to your event streams using JavaScript.

Example 1: Stateless Event Transformation​

The simplest use case is when you want to modify values in an event.

Use case: You want to halve the temperature value of each incoming event.

function process(event) {
    event.temperature = event.temperature / 2;
    return event;
}

That’s it! Each event passes through this function, gets modified, and is sent on its way. No extra setup needed.

Example 2: Custom Event Filtering​

You don’t always want to forward every event. The JavaScript Evaluator lets you return null (or just omit return) to drop an event.

Use case: Only forward events where the temperature is between 50 and 100.

function process(event) {
    if (event.temperature > 50 && event.temperature < 100) {
        return event;
    }
}

With this setup, only "valid" events get through. All others are silently ignored. Perfect for quick, custom filtering logic.

Example 3: Stateful Logic​

Need to track information across multiple events? You can wrap your function in an Immediately Invoked Function Expression (IIFE) to create persistent variables.

Use case: Count how many events have passed through the processor and append that number to each event.

(() => {
    let count = 0;

    return function process(event) {
        count += 1;
        event.count = count;
        return event;
    };
})()

This small trick makes it easy to introduce state without needing any external dependencies or full-blown processor development. You can track counts, store intermediate values, or even implement basic alerting. Just make sure that any new properties you add to the event, like count in the example, are also included in the output event schema during processor configuration, otherwise the pipeline may fail validation.

Wrap-Up​

The JavaScript Evaluator is one of the most flexible processors in StreamPipes. Whether you're a beginner or a power user, it lets you:

✅ Write custom transformations
✅ Implement simple filters
✅ Track state between events
✅ Prototype complex logic—without deploying a new microservice

So, next time you think "I wish there was a processor that could...", just try the JavaScript Evaluator. Chances are, it can.

Got a cool use case using this processor? Let us know or share it with the community—we’d love to see what you’re building with Apache StreamPipes!

Are you excited to explore Apache StreamPipes but find yourself without any sensors to play with? Or maybe you're looking to validate StreamPipes for your use case and kick off a Proof of Concept (PoC) but the hardware is not yet available? No worries—the File Stream Adapter is here to save the day!

With the File Stream Adapter, you can upload a file containing sensor data in various formats (JSON, CSV, XML) and replay this data, giving you a hands-on way to dive into StreamPipes. It's not meant for production use, but it's perfect for experimenting with StreamPipes using your own data.

This blog post will take you on a exciting journey through configuring the File Stream Adapter and using it to simulate sensor data streams.

Adapter Configuration​

Let's break down the configuration options for the File Stream Adapter:

Configuration Options Explained​

1. File: Choose an existing file or upload a new one.
2. Overwrite file time: If checked, the current time at which the event is triggered is used for the timestamp of the event, regardless of the timestamp value in the file.
3. Replay Once: Decide if the file should be replayed once or continuously until stopped.
   • Options: yes (file is replayed once), no (file is replayed continuously)
4. Replay Speed: Set the frequency of events.
   • Options: Keep original time, Fastest (Ignore original time), Speed Up Factor
5. Format: Choose the format to parse the events.
   • Options: JSON, CSV, XML, Image (experimental feature)

Use Case: Create a Sensor Simulator​

Let's create a simulation data stream using your own sensor data. First, store the sample CSV data in a local file.

timestamp,density,mass_flow,sensorId,sensor_fault_flags,temperature,volume_flow
1719897720887,41.82,6.306,flowrate01,false,41.9,3.365
1719897721952,45.09,9.659,flowrate01,false,43.2,4.103
1719897722958,42.84,9.532,flowrate01,true,42.6,3.110
1719897723964,49.58,1.929,flowrate01,true,40.3,3.4684
1719897724970,41.83,4.173,flowrate01,false,40.7,0.246
1719897725975,48.11,10.90,flowrate01,false,46.4,5.275

To set up the adapter, follow these steps:

1. Open StreamPipes Connect.
2. Click on + New Adapter.
3. Select the File Stream Adapter.
4. Enter the following configuration:
   • File: Upload your sample data file.
   • Overwrite file time: Check this box to use the current timestamp instead of the original time of the event in the file.
   • Replay Once: Choose no to create an ongoing data stream.
   • Replay Speed: Select Keep original time to maintain the original event frequency.
   • Format: Pick CSV, set the delimiter as , and check the Header box.
5. Click Next once you've added the correct configuration.
6. Configure the event schema by editing the timestamp field and selecting Mark as timestamp. Save your changes.
7. You can add additional meta information to the event schema, but it's not required for this tutorial. Click Next when done.
8. Enter an Adapter Name.
9. Check the Persist events box to store the data in the StreamPipes data store.
10. Click Start Adapter.

Once configured, you should see the following preview:

Use Case: Persist CSV File to Data Lake​

Now, let's store the entire content of the file in the data lake. Once stored, you can analyze it with the Data Explorer. Here’s how to configure the adapter:

• Overwrite file time: Leave this unchecked to retain the original timestamps.
• Replay Once: Choose yes to read the file only once.
• Replay Speed: Select Fastest (Ignore original time) to write data as fast as possible.

Conclusion​

In this tutorial, we've explored how to use the File Stream Adapter in Apache StreamPipes to create a simulation data stream and persist CSV data into a data lake. This adapter lets you experiment with StreamPipes' capabilities and explore its functionality using your own data. Happy streaming!

Apache StreamPipes 0.97.0 brings exciting new features, enhanced security, and significant UX improvements to streamline your IIoT data processing workflows. This release introduces Adapters & Pipelines as Code, enabling users to define and deploy StreamPipes resources programmatically using YAML or JSON. Security enhancements, such as OPC UA certificate support and Kafka SSL/SASL authentication, improve connectivity and data integrity. Additionally, we have made various UX improvements, including an enhanced Data Explorer and asset-based filtering.

In total, the new release closes 159 issues including new features, bug fixes and improvements.

Read on to discover what’s new in StreamPipes 0.97.0!

Adapters & Pipeline Elements​

Adapter & Pipeline as Code​

StreamPipes now supports managing adapters and pipelines programmatically using compact Json or YAML definitions. These features are ideal for automating deployments, integrating with CI/CD pipelines, and applying infrastructure-as-code practices.

Adapters as Code​

With Adapters as Code, you can define and deploy adapters via a compact description, streamlining the creation of multiple similar adapters and simplifying automation. For example, here's a sample YAML snippet to define an OPC-UA adapter:

name: My OPC Adapter
id: testadapter
description: Test
appId: org.apache.streampipes.connect.iiot.adapters.opcua
configuration:
  - opc_host_or_url: OPC_URL
    opc_server_url: opc.tcp://localhost:62541/milo
  - adapter_type: PULL_MODE
    pulling_interval: 1000
  - access_mode: UNAUTHENTICATED
  - available_nodes:
    - "ns=2;s=Leakage Test Station/temperature"
    - "ns=2;s=Leakage Test Station/pressure"

schema:
  temperature:
    propertyScope: measurement
    label: Temp
    description: Temperature value
    semanticType: http://schema.org/temperature

enrich:
  timestamp: timestamp

transform:
  rename:
    pressure: pressure3

createOptions:
  persist: true
  start: true

This approach considerably simplifies the programmatic creation of adapters Learn more about Adapters as Code

Pipelines as Code​

Pipelines as Code lets you define end-to-end data flows—from data sources through processors to sinks. This function is ideal for the standardization and automation of pipeline deployments. For example, here’s a sample YAML snippet for a simple pipeline:

id: my-pipeline
name: ""
description: ""
pipelineElements:  # Define pipeline components here
  - type: stream     # Data source
    ref: <reference> # Unique reference ID
    id: <data-stream-id> # ID of the stream

  - type: processor  # Data transformation
    ref: <reference> # Unique reference ID
    id: <processor-id> # ID of the processor
    connectedTo:     # Previous pipeline element reference(s)
      - <reference>
    configuration:   # Processor-specific configurations
      - <configuration-option>

  - type: sink       # Data sink (output)
    ref: <reference> # Unique reference ID
    id: <sink-id>    # ID of the sink
    connectedTo:     # Previous pipeline element reference(s)
      - <reference>
    configuration:   # Sink-specific configurations
      - <configuration-option>

createOptions:
  start: <true|false>  # Whether to start the pipeline immediately

The code preview of a pipeline has also been added to the Start Pipeline dialog, which has been further improved to make the execution status of pipelines more clear.

This method provides a programmatic interface to deploy and manage pipelines quickly and easily. Learn more about Pipelines as Code

OPC UA Certificates​

The OPC UA adapter now supports both signed/encrypted and unencrypted communication. Certificate provisioning is handled externally via environment variables, following a dedicated security directory layout. Refer to the OPC UA adapter documentation for further details.

https://github.com/apache/streampipes/pull/3354

Kafka Avro & Certificates​

The Kafka adapter and sink now support SSL and SASL for enhanced security. Additionally, users can provide custom producer/consumer configurations. A new Avro parser has been added along with an input field for specifying the consumer group in the Kafka adapter.

New Features & Deprecations​

Asset Browser​

StreamPipes now allows users to model real-world assets and associate them with StreamPipes resources such as adapters, pipelines, and dashboards. These asset mappings enable intuitive filtering across different modules, making it easier to navigate and manage related resources.

The screenshot below illustrates an example where a StreamPipes instance is configured with two machines and various stations. Clicking on a machine or station dynamically filters the associated resources in the list on the right. This asset-based filtering is available across all main modules of StreamPipes.

The asset overview panel has been redesigned to provide a more intuitive and user-friendly experience. It is possible to assign labels, locations, and ISA95 types to assets. For each asset, resources such as pipelines, adapters and dashboards can be assigned. The asset overview contains an intuitive overview of all linked resources.

To manage locations & sites, we extended the configuration menu with additional options to add a custom map layer provider and to configure the map view.

UX improvements​

File Relocation​

The file upload configuration has been moved from the top-level navigation to the settings menu, improving the overall structure and clarity of the main navigation.

This change does not affect file uploads required by adapters or pipeline elements—these configurations remain unchanged. Further, users can still upload files as before via the files configuration section.

Adapter Data Preview​

Viewing live data streams in the adapter configuration has been improved. By clicking on "details", a full-screen view of the data preview is displayed. Changing data values are highlighted, so that it is ease to track changes. The Code panel shows the configuration in JSON or YAML format.

Data Explorer Configuration Improvements​

Configuring data views in the Data Explorer is now more intuitive and user-friendly. We have enhanced validation, making it easier to select data sources and specify queries while reducing potential misconfigurations.

Key improvements include:

• Default Table Visualization: New data views now default to a table view, minimizing setup effort.
• Improved Time Range Picker: More flexibility in selecting time ranges, including support for different time ranges per widget in Data Lake dashboards.
• Auto-Refresh for Dashboards: Live updates are now possible with the new auto-refresh functionality.
• Better Error Messages: Enhanced feedback when configurations are incorrect to improve usability.
• Custom color coding for pie chart: In the pie chart, custom colors can be mapped to specific data values.
• Traffic light visualization: A new traffic light visualization has been added to the Data Explorer.

Here is a screenshot of the new traffic light visualization:

Pipeline Details​

The pipeline details view has been completely remodeled. It now shows the pipeline in full-screen mode, making it easier to view and edit complex pipelines. A sidebar contains further information on the pipeline, such as logs, pipeline elements, and execution status. In addition, execution metrics are now intuitively accessible right in the pipeline details view.

Deprecation Notice: Dashboard​

The current live dashboard feature in StreamPipes is being deprecated and will be removed in a future release. As a replacement, the Data Explorer dashboards has been enhanced with auto-refresh functionality, ensuring a seamless transition for users who rely on real-time data visualization.

The harmonization of data explorer and live dashboard functionalities will provide a more consistent user experience and will ease the addition of new charts in future version, since all visualizations now rely on a single visualization library.

We encourage all users to migrate their existing dashboards to the Data Explorer to continue leveraging powerful visualization and analysis capabilities.

If you think any feature is missing in the data explorer, please let us know!

Conclusion​

We appreciate the ongoing support and contributions from our community in making StreamPipes better with each release. Version 0.97.0 brings improvements in usability, security, and automation, helping users to analyze their IIoT data sources.

As always, we welcome your feedback and suggestions to continue refining and enhancing StreamPipes. Feel free to share your thoughts or get involved through GitHub or mailing list.

Let's delve into the key updates introduced in this release.

The current release can be downloaded here.

Adapters & Pipeline Elements​

Improve deletion workflow of adapters​

Within StreamPipes 0.93.0 we introduced the possibility to edit existing adapters. This release improves adapter management by conveniently allowing to delete existing adapters even if they are used in a pipeline. When deleting a new adapter, a dialog appears which allows users to delete all pipelines which are using the data stream produced by the adapter.

https://github.com/apache/streampipes/issues/1936

Enhance data lake schema management​

We've added a new option to the data lake sink which provides users more flexibility when writing data to the internal time-series storage. Users can choose between the Update Schema and Extend existing schema options. By choosing Extend schema, it is possible to also see "old" data in the data explorer after the schema has changed.

https://github.com/apache/streampipes/issues/2252

New Adapter: Open Industry 4.0 devices​

A new adapter in 0.95.0 is support for the Open Industry 4.0 specification over MQTT. The adapter can be used to connect to a variety of devices which support the data format defined in the Development Guidelines for Open Edge Computing.

For instance, the adapter makes it easy to use IO-Link sensors together with an OI4-compatible IO-Link master.

https://github.com/apache/streampipes/pull/2511

New Adapter(s): All PLC4X protocols supported as adapters​

Since our "sister project" Apache PLC4X has introduced a new metadata API for gathering available drivers, StreamPipes 0.95.0 now supports a much larger collection of PLC connectors. The list of newly supported protocols include:

• Allen Bradley ETH
• BACnet/IP
• Beckhoff TwinCat ADS
• EthernetIP
• Modbus ASCII/RTU/TCP

The already existing S7 adapter is now also available with additional configuration options.

Configurations on protocol and transport level are dynamically resolved from the PLC4X API. This ensures that users now have much greater flexibility when connecting to industrial protocols.

The screenshot below shows the advanced options panel for Modbus TCP:

The new adapters should be considered a beta feature and we still recommend to use the existing Modbus and S7 adapters until the new adapters have proven their stability. More information on the usage of these new adapters and supported features can also be found on the PLC4X website.

https://github.com/apache/streampipes/issues/2632

New Processor: Static Metadata enrichment​

We added a new processor to add static metadata (such as sensor ID or location IDs) to any data stream within a pipeline.

https://github.com/apache/streampipes/issues/2350

New Processor: Datetime from String​

The new Datetime from String processor allows to convert a datetime string to a timestamp value and lets users conveniently choose the desired target time zone.

https://github.com/apache/streampipes/issues/1865

New Sink: MS Teams​

StreamPipes 0.95.0 comes with a new data sink for Microsoft Teams, which allows to publish notifications from a pipeline to a teams channel using Webhooks.

https://github.com/apache/streampipes/pull/2248

Remove deprecated adapters​

We removed several outdated adapters that were deprecated in release 0.93.0:

• CoindeskBitcoinAdapter
• GdeltAdapter
• FlicMQTTAdapter
• TISensorTag
• IexCloudAdapter
• OpenSenseMapAdapter
• WikipediaAdapter

https://github.com/apache/streampipes/issues/2128

UX improvements​

Display file types in different colors​

Different file types in the Files overview are now better highlighted by a dedicated color.

https://github.com/apache/streampipes/issues/2107

Download button for asset & files​

In the asset and file overview, resources can be now directly downloaded by using a new download button.

https://github.com/apache/streampipes/issues/2192 https://github.com/apache/streampipes/issues/2074

StreamPipes Python​

As already mentioned in our blog post, we added an example to use an existing ONNX model as part of a StreamPipes function.

https://github.com/apache/streampipes/pull/2687

Administration & Development​

Remove Consul entirely​

Following the replacement of Consul with an internal service registration and discovery mechanism, StreamPipes 0.95.0 also removes Consul from all installation files.

https://github.com/apache/streampipes/issues/2133

New Apache StreamPipes Go-Client​

Apache StreamPipes now includes a new client library for Golang. Our first version of the Go client interacts with the StreamPipes API and provides various ways to gather data from StreamPipes.

Here is an example on the usage of the Go client:

package main

import (
    "github.com/apache/streampipes/streampipes-client-go/streampipes"
    "github.com/apache/streampipes/streampipes-client-go/streampipes/config"
    "log"
)

/*
    Here are some examples of using go client, including outputting the data returned by streams.
    Only supports outputting model data
*/

func main() {

    clientConfig := config.StreamPipesClientConfig{
        Url: "http://localhost:8030",
        Credential: config.StreamPipesApiKeyCredentials{
            UserName: "<Your-User-Name>",
            ApiKey:   "<Your-API-Key>",
        },
    }

    streamPipesClient, err := streampipes.NewStreamPipesClient(clientConfig)
    if err != nil {
        log.Fatal(err)
    }

    dataSeries, err := streamPipesClient.DataLakeMeasures().GetSingleDataSeries("measureName")
    if err != nil {
        log.Fatal(err)
    }
    dataSeries.Print()

    /*
            output format:

            There are 2 pieces of DataSerie in the Dataseries
            The 1 DataSeries
            time                   msg                   test
            2024-02-23T13:37:09.052Z   go-client_test   2f4556
            2024-02-23T13:37:26.044Z   go-client_test   2f4556
            2024-02-23T13:37:29.007Z   go-client_test   2f4556
            The 2 DataSeries
            time                   msg                   test
            2024-02-23T13:38:06.052Z   go-client_test   2f4556
            2024-02-23T13:38:35.044Z   go-client_test   2f4556
            2024-02-23T13:38:38.007Z   go-client_test   2f4556

    */

}

https://github.com/apache/streampipes/pull/2437

First release, are working on more examples and better docs - stay tuned

Final Remarks​

We highly recommend updating to StreamPipes 0.95.0 to take advantage of these new features, bug fixes, and improvements. For more detailed information, please refer to the release notes.

We appreciate your continued support and valuable feedback. StreamPipes is continually evolving, and we are dedicated to providing you with a powerful and reliable platform for your IIoT streaming data needs.

Motivation​

With this blogpost we want to illustrate how one can easily extract historical IIoT data collected with StreamPipes, train a machine learning model on this data and bringing the model back to StreamPipes with the interoperability standard ONNX to make inference in live data.

A very common use case in the area of IIoT is the detection of anomalies, so we want to tackle this challenge in this article as well. We will use data generated by the Machine Data Simulator adapter. More specifically, we will focus on the flowrate data, which consists of various sensor values coming from a water pipe system. Our goal is to keep an eye on the parameter volume_flow, which represents the current volume flow in cubic meters/second. For this parameter, we want to detect anomalies that could indicate problems such as leaks, blockages, etc.

To get the concerned data, we simply need to create an instance of the machine data simulator and persist the data in the data lake:

Set Up & Prepare Python Client​

As a prerequisite, we need to install the StreamPipes Python client and all other dependencies,

pip install git+https://github.com/apache/streampipes.git#subdirectory=streampipes-client-python
pip install scikit-learn==1.4.0 skl2onnx==1.16.0 onnxruntime==1.17.1

The next step is to configure and initialize an instance of the client.

import os
from streampipes.client import StreamPipesClient
from streampipes.client.config import StreamPipesClientConfig
from streampipes.client.credential_provider import StreamPipesApiKeyCredentials

os.environ["BROKER-HOST"] = "localhost"
os.environ["KAFKA-PORT"] = "9094"  # When using Kafka as message broker

config = StreamPipesClientConfig(
    credential_provider=StreamPipesApiKeyCredentials(
        username="admin@streampipes.apache.org",
        api_key="TOKEN",
    ),
    host_address="localhost",
    https_disabled=True,
    port=80
)

client = StreamPipesClient(client_config=config)

In case you have never worked with the Python client before and have problems to get started, please have a look at our tutorial.

If you already have an ONNX model and are only interested in applying it with StreamPipes on a data stream, you can skip the following section.

Model Training with Historic Data​

As said above, the aim of our model is to detect anomalies of the volume_flow parameter. For this task, we will use Isolation Forests. Please note that the focus of the tutorial is not on training the model, so please be patient even though the training is very simplified and lacks important preparation steps such as standardization.

As a first step, lets query the flowrate data from the StreamPipes data lake and extract the values of volume_flow as a feature:

flowrate_df = client.dataLakeMeasureApi.get("flow-rate").to_pandas()
X = flowrate_df["volume_flow"].values.reshape(-1, 1).astype("float32")

As a next step, we can already train our model with the historic data:

from sklearn.ensemble import IsolationForest

model = IsolationForest(contamination=0.01)
model.fit(X)

The contamination parameter models the proportion of outliers in the data. See the scikit-learn documentation for more information.

Here you can see how this simple model performs:

This doesn't look too bad, right? Let's continue by converting our model to the ONNX representation.

from onnxconverter_common import FloatTensorType
from skl2onnx import to_onnx

model_onnx = to_onnx(
    model,
    initial_types=[('input', FloatTensorType([None, X.shape[1]]))],
    target_opset={'ai.onnx.ml': 3, 'ai.onnx': 15, '': 15}
)

with open("isolation_forest.onnx", "wb") as f:
    f.write(model_onnx.SerializeToString())

Model Inference with Live Data​

Utilizing a pre-trained model within StreamPipes becomes seamless with the ONNX interoperability standard, enabling effortless application of your existing model on live data streams.

Interacting with live data from StreamPipes is facilitated through StreamPipes functions. Below, we'll create a Python StreamPipes function that leverages an ONNX model to generate predictions for each incoming event, making the results accessible as a data stream within StreamPipes for subsequent steps.

So let's create an ONNXFunction that is capable of applying a model in ONNX representation to a StreamPipes data stream. If you'd like to read more details about how functions are defined, refer to our tutorial.

import numpy as np
import onnxruntime as rt

from streampipes.functions.broker.broker_handler import get_broker_description
from streampipes.functions.streampipes_function import StreamPipesFunction
from streampipes.functions.utils.data_stream_generator import create_data_stream, RuntimeType
from streampipes.functions.utils.function_context import FunctionContext
from streampipes.model.resource import FunctionDefinition, DataStream

from typing import Dict, Any, List


class ONNXFunction(StreamPipesFunction):

    def __init__(self, feature_names: list[str], input_stream: DataStream):
        output_stream = create_data_stream(
            name="flowrate-prediction",
            attributes={
                "is_anomaly": RuntimeType.BOOLEAN.value
            },
            broker=get_broker_description(input_stream)
        )

        function_definition = FunctionDefinition(
            consumed_streams=[input_stream.element_id]
        ).add_output_data_stream(output_stream)

        self.feature_names = feature_names
        self.input_name = None
        self.output_name = None
        self.session = None

        super().__init__(function_definition=function_definition)
    
    ...

First, we need to take care about the data stream that is required to send the predictions from our function to StreamPipes. Thus, we create a dedicated output data stream which we need to provide with the attributes our event will consist of (a timestamp attribute is always added automatically). This output data stream needs to be registered at the function definition which is to be passed to the parent class. Lastly, we need to define some instance variables that are mainly required for the ONNX runtime.

Next, we need to ensure that ONNX runtime session is created on start up. Thus, we need to invoke an InferenceSession and retrieving the corresponding configuration parameters:

class ONNXFunction(StreamPipesFunction):
  
    ...
    
    def onServiceStarted(self, context: FunctionContext) -> None:
        self.session = rt.InferenceSession(
            path_or_bytes="isolation_forest.onnx",
            providers=rt.get_available_providers(),
        )
        self.input_name = self.session.get_inputs()[0].name
        self.output_name = self.session.get_outputs()[0].name
        
    ...

Lastly, we need to implement the inference logic that is applied to every event. If you have brought up your own model, you need to adapt line 10-13:

class ONNXFunction(StreamPipesFunction):

    ...

    def onEvent(self, event: Dict[str, Any], streamId: str) -> None:
        feature_vector = []
        for feature in self.feature_names:
            feature_vector.append(event[feature])

        prediction = self.session.run(
            [self.output_name],
            {self.input_name: np.expand_dims(np.array(feature_vector), axis=0).astype("float32")}
        )[0]

        output = {
            "is_anomaly": int(prediction[0]) == -1
        }

        self.add_output(
            stream_id=self.function_definition.get_output_stream_ids()[0],
            event=output
        )

    def onServiceStopped(self) -> None:
        pass

Having the function code in place, we can start the function with the following:

from streampipes.functions.registration import Registration
from streampipes.functions.function_handler import FunctionHandler

stream = [
    stream
    for stream
    in client.dataStreamApi.all()
    if stream.name == "flow-rate"
][0]

function = ONNXFunction(
    feature_names=["volume_flow"],
    input_stream=stream
)

registration = Registration()
registration.register(function)
function_handler = FunctionHandler(registration, client)
function_handler.initializeFunctions()

We can now access the live values of the prediction in the StreamPipes UI, e.g., in the pipeline editor.

From here on you can further work with the prediction events in StreamPipes, e.g., by sending notifications to MS Teams.

We're excited to share the release of StreamPipes 0.93.0, bringing with it a range of bug fixes and new features. This release places a strong emphasis on refining the technical foundations of StreamPipes, featuring significant changes that set the stage for our major 1.0 release. Nevertheless, we haven't overlooked the end-user experience and have implemented several enhancements. Let's delve into the key updates introduced in this release.

The current release can be downloaded here.

Adapters & Pipeline Elements​

Editing existing adapters​

StreamPipes has streamlined the process of modifying existing adapter instances, eliminating the inconvenience users previously faced. In prior versions, altering an adapter required a cumbersome sequence of stopping, deleting, and then recreating the adapter. However, with the introduction of StreamPipes 0.93.0, this arduous process is now a thing of the past.

Users can now seamlessly edit running adapters, and the changes will be applied automatically, obviating the need for the aforementioned manual steps. To guide you through this newfound capability, refer to the below animation, which demonstrates how to modify an existing adapter.

While the automatic migration seamlessly handles the majority of cases, there are instances where it may not be feasible. One such scenario arises when an event property, used by a pipeline element, is removed. In such cases, StreamPipes proactively alerts users before initiating the adapter update. Following the alert, the system takes precautionary measures by halting the adapter and marking the associated pipeline for review.

OPC-UA allows detailed & asynchronous browsing​

We have fully reworked the browsing interface of our OPC-UA adapter.

Now, all you need to do is specify the URL where the OPC UA server is accessible, eliminating the need to concern yourself with namespaces or node ids. Once the URL is provided, the adapter effortlessly presents all top-level nodes within the user interface. You can then navigate through the node tree interactively by clicking on the desired node.

For every data node, accessing node information is a breeze – simply click on the eye icon. If you wish to subscribe to a particular node, a straightforward click on the plus icon is all it takes.

All the nodes you select become integral to the event structure of this adapter, accessible by clicking on Next. The ensuing view not only displays the OPC UA status for each node but also allows for the removal of a node with ease, especially if its status is not Good.

Adapter for S7 PLCs supports array-based data​

Until now, our S7 PLC adapter lacked the capability to read values structured in arrays. We're pleased to announce that this limitation has been addressed through the introduction of an Advanced register input mode.

This new mode empowers you to configure the registers to be queried in a manner specific to the requirements of our underlying dependency, PLC4X. For detailed insights and guidance on the configuration process, refer to the PLC4X S7 documentation.

When dealing with array definitions, the configuration would resemble the following:

# propertyName=%{Memory-Area}{start-address}:{Data-Type}[{array-size}]
temperature=%DB3.DB0.0[BOOL]

New Adapter: ifm IOLink​

The IO-Link Master (AL1350) from ifm comes equipped with an IoT interface designed for transmitting sensor data via MQTT. With the introduction of this adapter, users gain the capability to effortlessly incorporate IO-Link sensor data into StreamPipes. This experimental integration empowers users to effortlessly connect to data from the ifm sensor VVB001.

For more details refer to the adapter documentation.

For the integration of additional sensors we are thinking about supporting IODD. Your feedback on these considerations is invaluable to us. If you find the integration of IODD for supporting more sensors within StreamPipes appealing or have insights to share, we welcome your input. Please let us know your thoughts on this potential feature enhancement.

New Processor: Numeric Rounding​

Introducing our latest data processor, Numeric Rounding , designed to perform rounding operations on numeric values within an event stream. This processor enables users to select one or more numeric values from their event data, specifying the desired rounding precision and the rounding mode.

For more details refer to the processor documentation.

UX improvements​

Ordered measurement units​

In the adapter configuration step, when setting up a transformation rule for an event value, the available units of measurement are now conveniently organized in alphabetical order for improved accessibility and ease of selection.

Extended information for existing adapters​

When accessing the Connect module, a comprehensive list of all existing adapters is presented. Within this view, each adapter now includes details about the timestamp of the last event creation and the total count of elements published since startup. This enhancement provides a quick and efficient means to evaluate the health of each adapter.

Interactive tutorial is back​

Upon initiating a new StreamPipes instance, a dialog prompts users to embark on an interactive tutorial showcasing the platform's key features. We're pleased to announce that this tutorial is now operational again. Enjoy exploring and discovering the most important functionalities in StreamPipes. Happy learning!

Deployment​

Kubernetes Deployment​

Add probes to the Helm chart​

Integrating readiness, liveness, and startup probes into StreamPipes' Kubernetes containers enhances the system's health monitoring. These probes automatically detect and handle anomalies, significantly improving application availability and stability. The readiness probe ensures preparedness, the liveness probe validates ongoing functionality, and the startup probe assesses initialization status, collectively fortifying StreamPipes against potential disruptions.

Rework the values file​

We've refined and simplified the values file for our Helm chart, optimizing its structure. Furthermore, we've invested in comprehensive documentation for all available options. Check out the README file of our Helm chart for detailed information on configuration and deployment.

Add Apache Pulsar to Helm file​

In our last release, we've expanded StreamPipes' capabilities to include Apache Pulsar as a supported messaging layer. Alongside this enhancement, our Helm chart has now been updated to seamlessly deploy StreamPipes on Kubernetes with Pulsar as the underlying messaging infrastructure.

Monitoring Integration (Grafana & Prometheus)​

We've now integrated monitoring support for StreamPipes through Prometheus and Grafana, making it seamlessly available in both our Kubernetes and Docker deployments. If you're using Kubernetes, we recommend checking out our Helm chart for straightforward setup instructions. For Docker users, stay tuned for an upcoming blog post where we'll provide a step-by-step guide on configuring Prometheus and Grafana to monitor StreamPipes effectively.

New Deployment mode for first time users​

To simplify onboarding with Apache StreamPipes and demonstrate its capabilities, we've added a quickstart deployment option. This docker-compose variant provides users with a pre-configured StreamPipes instance and rich examples. Explore it in our user guide.

StreamPipes Python​

Our official Python support for Apache StreamPipes got some small improvements as well. Notable among other improvements are especially the following two:

• Support for Python 3.11
• StreamPipes Python does now also support other protocols than Apache Kafka to interact with live data

Administration & Development​

Email Templates are configurable​

StreamPipes already supported the sending of notifications, such as those for password recovery and account registration, when properly configured. However, in previous versions, the email content was predefined and couldn't be adjusted. In this latest release, administrators now have the flexibility to modify the email templates, offering greater customization options for notifications.

Remove legacy resources​

In this release, we've eliminated legacy resources that were initially employed for demonstration purposes. As these resources are no longer actively supported, we have decided to completely remove them. The affected components include the vehicle simulator and watertank simulator. This removal has led to the discontinuation of the StreamPipes package streampipes-sources.

Furthermore, we are pleased to announce the forthcoming removal of Consul as an additional dependency required to run StreamPipes in the upcoming release. Within this release, we have already implemented the fundamental steps to eliminate the need for Consul. This marks a significant stride towards making StreamPipes a more lightweight platform. It's crucial to highlight that this modification is purely a technical enhancement and will have no impact on the end user experience.

Enable migrations for pipeline elements​

We have established a crucial technical foundation to simplify the update process for upcoming versions of StreamPipes. It is now possible to define migrations for adapters, data processors, and data sinks. For a more in-depth understanding of how migrations function and how to define them, please refer to our documentation.

New Homepage​

We've undergone a comprehensive redesign of our website, streampipes.apache.org. Take a moment to explore the new look, and we welcome your feedback on your experience. Let us know what you think!

Security​

We've addressed a vulnerability in the default password generation for new users during the self-registration process. Previously, a cryptographically weak pseudo-random number generator (PRNG) was utilized, but we've now rectified this by implementing Apache Commons RNG. We extend our sincere thanks to Alessandro Albani from the Digital Security Division Var Group for bringing this to our attention.

Final Remarks​

We highly recommend updating to StreamPipes 0.93.0 to take advantage of these new features, bug fixes, and improvements. For more detailed information, please refer to the release notes.

We appreciate your continued support and valuable feedback. StreamPipes is continually evolving, and we are dedicated to providing you with a powerful and reliable platform for your IIoT streaming data needs.

We are pleased to announce the release of StreamPipes 0.92.0, which includes several bug fixes, performance improvements, and new features. This version focuses on enhancing the technical aspects of StreamPipes. Let's explore the key updates in this release:

The current release can be downloaded here.

Improved Python Support:​

In response to valuable user feedback, we have improved Python support in StreamPipes. Users can now seamlessly integrate Python-based scripts, libraries, and modules into their functions. This enhancement enables leveraging the power of machine learning and advanced analytics algorithms for IIoT data sources. Please refer to the Python documentation for further details.

New Processing Elements: Introducing the Swinging Door Trending (SDT) Filter Processor:​

To expand the functionality of StreamPipes, we have introduced several new processing elements. Among them is the Swinging Door Trending (SDT) Filter Processor. This processor provides a linear trend compression algorithm. Essentially, it replaces a series of continuous points (timestamp, value) with a straight line determined by the start and end points.

Bug Fixes and Improvements:​

We have addressed various bugs and issues reported by our users, resulting in a more stable and reliable StreamPipes experience. We have also made several performance and usability improvements to enhance overall system efficiency.

Deprecation of Set Adapters​

In order to streamline the StreamPipes ecosystem, we have deprecated the functionality of Set Adapters. Going forward, our focus will solely be on streaming event sources. However, we understand the importance of working with files of sensor data, especially during development. To accommodate this need, we have extended the File Stream Adapter, which now offers the following functionalities:

1. Create a live simulation of sensor data based on the file.
2. Replay the data in the file once, for example, to store it in the Data Lake.

IMPORTANT: The set adapters will be removed in the next release.

From this release onwards, we encourage you to utilize the File Stream Adapter instead of the File Set Adapter. To simulate a file set using the File Stream Adapter, you can configure it with Replay Once: Yes and Replay Speed: Fastest (Ignore original time). See the example below:

Final Remarks​

We highly recommend updating to StreamPipes 0.92.0 to take advantage of these new features, bug fixes, and improvements. For more detailed information, please refer to the release notes.

We appreciate your continued support and valuable feedback. StreamPipes is continually evolving, and we are dedicated to providing you with a powerful and reliable platform for your IIoT streaming data needs.

The Apache StreamPipes community is delighted to announce the release of Apache StreamPipes version 0.91.0! The biggest highlight of this release is the birth of our official StreamPipes Python library. Apart from that, we have made a lot of improvements under the hood, especially in terms of improving the development experience (e.g., by introducing and enforcing code style rules). Nevertheless, our new version includes more than 90 improvements and bug fixes in total.

The current release can be downloaded here.

StreamPipes Python​

Apache StreamPipes aims to enable non-technical users to connect and analyze IoT data streams. To this end, it provides an easy-to-use and convenient user interface that allows one to connect to an IoT data source and create some visual graphs within a few minutes.
Although this is the main use case of Apache StreamPipes, it can also provide great value for people who are eager to work on data analysis or data science with IoT data, but don't want to get in touch with all the hassle associated with extracting data from devices in a suitable format. In this scenario, StreamPipes helps you connect to your data source and extract the data for you. You then can make the data available outside StreamPipes by writing it into an external source, such as a database, Kafka, etc. While this requires another component, you can also extract your data directly from StreamPipes programmatically using the StreamPipes API. For convenience, we also provide you with a StreamPipes client both available for Java and Python. Specifically with StreamPipes Python, we want to address the amazing data analytics and data science community in Python and benefit from the great universe of Python libraries out there.

Getting started​

Besides our official version, which you can access here, we provide Apache StreamPipes as a convenient release via PyPI. Therefore, you can easily install it via the following command

pip install streampipes

All documentation related to StreamPipes Python can be found here.

But now enough words are written, let's just start with a few examples. As a first step, we want to establish a connection to a running StreamPipes instance.

from streampipes.client import StreamPipesClient
from streampipes.client.config import StreamPipesClientConfig
from streampipes.client.credential_provider import StreamPipesApiKeyCredentials

config = StreamPipesClientConfig(
    credential_provider = StreamPipesApiKeyCredentials(
        username = "test@streampipes.apache.org",
        api_key = "DEMO-KEY",
        ),
    host_address = "localhost",
    https_disabled = True,
    port = 80
)

client = StreamPipesClient(client_config=config)

client.describe() gives you an overview about what can be discovered via the Python client:

Hi there!
You are connected to a StreamPipes instance running at http://localhost:80.
The following StreamPipes resources are available with this client:
1x DataLakeMeasures
1x DataStreams

The required API key can be retrieved from StreamPipes via the UI:

Query Data from StreamPipes data lake​

As a first step, we can see which data sets are available in the data lake:

client.dataLakeMeasureApi.all().to_pandas()

To query data for a specific data lake measure, the following command takes you there:

client.dataLakeMeasureApi.get(identifier="flow-rate").to_pandas()

There are some parameters that allow you to query data more specifically, visit the docs for more detailed information.

If you are curious which features are available for StreamPipes Python, have a look at our tutorial section.

We plan to further extend the Python library in the next releases - we welcome your ideas on any features you'd like to see!

Create Adapter without starting​

Until now, when creating an adapter in StreamPipes, the adapter was started immediately after completion. From now on you can choose via the adapter overview in the last dialog whether the adapter should be started immediately or later (this is then possible in the adapter overview).

Redirect after Login​

This is another small user interface improvement: when a user opens a StreamPipes link even though they are not logged in, e.g., http://streampipes.demo/#/connect, StreamPipes displays the login dialog and now redirects the user directly to the target view (in this case the connection overview).

Support Output Streams for Standalone Functions​

Within the last release, 0.90.0, we introduced StreamPipes Functions as a lightweight alternative to processing elements.
StreamPipes Functions do now support having output streams, which means that they allow not only to consume but to send data to StreamPipes as well.

Below you can see a simple example function that demonstrates how events can be pushed back to StreamPipes again.

public class FunctionPublishExample extends StreamPipesFunction {

  private static final Logger LOG = LoggerFactory.getLogger(FunctionPublishExample.class);

  private static final String FUNCTION_ID = "org.apache.streampipes.example.function.publish";
  private static final String STREAM_APP_ID = "example-output-stream-1";

  private SpOutputCollector outputCollector;

  @Override
  public List<String> requiredStreamIds() {
    return List.of("urn:streampipes.apache.org:eventstream:EtMUkN");
  }

  @Override
  public void onServiceStarted(FunctionContext context) {
    LOG.info("Service started");
    this.outputCollector = context.getOutputCollectors().get(STREAM_APP_ID);
  }

  @Override
  public void onEvent(Event event, String streamId) {
    LOG.info("on event");
    var ev = new Event();
    ev.addField("timestamp", System.currentTimeMillis());
    ev.addField("example-property", "abc");
    this.outputCollector.collect(ev);
  }

  @Override
  public void onServiceStopped() {
    LOG.info("service stopped");
  }

  @Override
  public FunctionConfig getFunctionConfig() {

    return FunctionConfigBuilder
        .create(FunctionId.from(FUNCTION_ID, 1))
        .withOutputStream(DataStreamBuilder.create(STREAM_APP_ID, "My Function Stream", "")
            .property(EpProperties.timestampProperty("timestamp"))
            .property(EpProperties.stringEp(
                Labels.from("my-example-property", "test", "test"),
                "example-property",
                SO.TEXT,
                PropertyScope.MEASUREMENT_PROPERTY))
            .format(Formats.jsonFormat())
            .protocol(Protocols.kafka("localhost", 9094, STREAM_APP_ID))
            .build())
        .build();
  }
}

This functionality is also already supported in StreamPipes Python! 🚀

This means that you can write a StreamPipes function in Python that allows you to consume streams of data from StreamPipes, perform any data manipulation in Python, and send the data back to StreamPipes, where it can be further processed.

The Apache StreamPipes community is pleased to announce the immediate availability of Apache StreamPipes version 0.90.0! In total, this release was focused on stabilization and closes 57 issues with improvements and bug fixes as well as some new features such as adapters and data sinks.

The current release can be downloaded here.

Improvements​

The focus of this release cycle was on general usability improvements and bug fixes. We continued to harmonize the StreamPipes web interface, so that all views follow a similar navigation. Many improvements were added to the data explorer:

• Incomplete events with missing values are now displayed in the data explorer
• Improvements to the time selector and edit mode
• More configuration options for data download
• Improved overall data explorer performance

In addition, we improved features to monitor pipelines and extensions services. The pipeline monitoring component has been restructured and now works independent from the underlying message broker. In addition, we extended the SDK so that runtime errors can be forwarded to the StreamPipes UI. Further improvements to the monitoring stack will be added in the next versions.

New Adapters and Data Sinks​

This version improves support for existing message brokers in form of adapters and sinks:

• Improvements of the Apache Pulsar (which is also available as a data sink)
• A new adapter and data sink for Apache RocketMQ
• A new adapter for NATS (also available as a data sink)

GitHub Community Features​

Apache StreamPipes is a community effort - everyone is welcome not only to contribute code, but also to provide feedback and other input which helps improving the product. To improve community interaction, we switched our issue management from Jira to GitHub issues - making it much easier to create issues and to discuss future features. In addition, we've enabled Github Discussions - feel free to participate in the existing discussions and ask questions if you have problems!

• GitHub Issues
• GitHub Discussions

And if you prefer the mailing list, subscribe to dev@streampipes.apache.org or users@streampipes.apache.org. Tech-related and general project discussions as well as release votes will continue to happen on the dev list.

Preview of Standalone Functions​

An exciting new feature is available in preview mode: StreamPipes Functions are a new way to interact with connected data streams in StreamPipes. As an open system, there are already several ways to extend the functionality of StreamPipes: New adapter can be added to connect to non-supported systems. In addition, the SDK supports building custom data processors and sinks, so that StreamPipes can be easily extended at runtime in case new analytics operators are needed.

But providing extensions in form of pipeline elements are not always the best choice: E.g., in cases where non-generic algorithms should be executed on a specific data stream, providing a reusable component for pipeline development is unsuitable.

This is where Functions come into play! StreamPipes Functions allow to define standalone components, which consume one or more connected data streams from StreamPipes and can be used to add custom processing logic. E.g., it becomes easy to apply an ML model on continuous data without the need to build a specific data processing pipeline.

Functions can be registered in an extensions service similar to data processors and sinks by extending a single class:

public class MyFunction extends StreamPipesFunction {
  @Override
  public FunctionId getFunctionId() {
    return FunctionId.from("my-app-id", 1);
  }

  @Override
  public List<String> requiredStreamIds() {
    return List.of("my-stream-id");
  }

  @Override
  public void onServiceStarted(FunctionContext context) {
    // called on service startup
  }

  @Override
  public void onEvent(Event event, String streamId) {
    // called for each incoming event
  }

  @Override
  public void onServiceStopped() {
    // called on service shutdown
  }
}

This concept, which will also come for more programming languages besides Java offer new ways for developers to interact with available data in StreamPipes and build custom extensions for IIoT analytics.

Functions are available in preview mode and their API should therefore be considered unstable.

Future Roadmap​

After this stabilization release, the next version will most probably bring a major new feature - Python support! The community is currently working hard to bring the StreamPipes client and functions to the Python world. Stay tuned!

The Apache StreamPipes (incubating) community is delighted to announce the release of Apache StreamPipes version 0.70.0! Key highlights of this release include many enhancements to the data explorer for fast exploration of IoT data and a brand new asset management that allows importing and exporting of StreamPipes resources. In addition, our new version includes more than 50 improvements and bug fixes.

The current release can be downloaded here.

Preview of Event Schema​

A very important and inevitable step when connecting to a data source with Apache StreamPipes is the definition of the event schema. By collecting some sample events, Apache StreamPipes is able to suggest an event schema that describes the data structure of the incoming events. Although this works quite well in the vast majority of cases, there is usually a need to modify the proposed event schema to some degree or add additional fields. So far, a user has gotten little idea of what the revised data source event schema looks like until the adapter has been created and some data was consumed. With this release, we have included a before/after comparison of the events while defining the event schema. This allows to compare the proposed events of Apache StreamPipes (before) with the events after all the user's changes have been applied (after). The after view adapts to each interaction of the user.

Export and Import StreamPipes Resource​

With this release we introduce a completely new module called Asset Management. This can be selected in the menu bar of the modules on the left side as the third last icon and is marked by a construction icon. This module allows to combine several StreamPipes resources to a so called Asset, which can be exported and re-imported later. To do this, a new asset must first be created. Afterwards the asset can be edited and the following detail view allows to link all resources available in StreamPipes to the asset. After the asset has been finally saved, the asset (inlcuding all resources) can be downloaded within the configuration module by switching to the Export/Import tab. All existing assets are listed there and can be selected for download. The downloaded file can then be re-imported later in the same or another StreamPipes instance at the same place.

Download Data of Configured Query from Data Lake​

Another new feature is the ability to download or export data from the StreamPipes data lake defined by a user-defined query. Previously, it was only possible to download all events belonging to a data lake index from the Data Lake tab of the configuration menu. In addition to this, each widget in the Data Explorer offers the possibility to download the corresponding data. The underlying query is created from the configuration of the widget and is also used to create the widget's figure. The download supports both CSV and JSON format.

Slide View in Data Explorer​

Traditionally, the graphs created in the Data Explorer are displayed in a grid view, where the user can adjust the arrangement of the figures as well as their height and width. With this release, another option for displaying the dashboard widgets is available: the slide view. This view allows to display a single graph in full-screen mode and switch between views using a horizontal menu bar. When a new Data View is created in the Data Explorer, a choice can be made between grid view and slide view as the default display mode. Once a Data View is created, the display mode can be changed by clicking the icon in the upper left corner right next to the Back button.

Templates for Adapter Configuration​

Similar to the processing elements, it is now possible to create templates for adapter configurations. This allows you to start from a previously defined set of configuration settings when creating similar adapters. To create a template for an adapter configuration, start by creating a new adapter and configuring all the settings that will be part of the template. Before proceeding to the field configuration (the second step of adapter creation), save the current configuration as a template. Whenever a new instance of the same adapter is created, the template can be selected from the drop-down menu that appears. Templates are only available for the same type of adapter (obviously).

Upload Multiple Files at Once​

A rather small but very handy improvement is the possibility to upload multiple files at once to StreamPipes. When you click the UPLOAD NEW FILE button in the File Management module, it is now possible to select as many files as you want in the file upload dialog.

To get a comprehensive overview of all changes released within Apache StreamPipes 0.70.0 please visit our release notes.

This tutorial shows how you can stream sensor data into StreamPipes from a simulation environment (Factory I/O). The simulation environment is controlled by a "Siemens S7-1200" PLC and shows a conveyor belt which sorts packages by their height. The tutorial is based on the upcoming version 0.70.0 and teaches you how to load the sensor data, build a pipeline, preprocess the data and create visualizations.

1. Factory I/O​

• Set up Factory I/O and make sure that the Sorting by Height (Basic) simulation works.

2. Adapter​

The next step is to connect the PLC to StreamPipes. To achieve this we need to create an adapter.

• Start StreamPipes, go to the * *CONNECT menu and create a NEW ADAPTER**.
• Select the PLC4X S7 adapter, insert the IP-address of your PLC and import the file with the PLC tags (see below). Then click NEXT on the bottom right.

• If the warning "MISSING TIMESTAMP" appears, press ADD TIMESTAMP and then NEXT. This warning occurs because some data sources don't provide a timestamp.

• You can name the adapter "S7_sorting" and select the option PERSIST EVENTS to make sure the data is persisted. The adapter is now ready to START. (You can check if the adapter is working by pressing the ? symbol in the connect menu and clicking on VALUES to see a live preview).

3. Dashboard for raw data​

A dashboard is used to monitor live data from a data stream. We will use it to make sure that the sensor values are updated when the simulation runs.

• Go to the DASHBOARD menu and create a NEW DASHBOARD "Dashboard_sorting". Press the pen symbol to edit the dashboard.
• Click ADD VISUALIZATION and select the automatically created pipeline "Persist S7_sorting". Choose the TABLE widget, configure it by pressing SELECT ALL and CREATE the widget.
• Change the size of the table as you like and SAVE the dashboard.
• The values should now update every 5 seconds. Test it by starting the simulation in Factory I/O.

4. Creating a Data View​

The Data View is used to create visualization for persisted data in a Data Lake. You can choose various visualization to explore your data. We will create a table to see the stored data and a bar chart to take a closer look at a single sensor.

• Create a NEW DATA VIEW "Sorting" in the DATA EXPLORER menu and edit the view.
• Select the "Persist S7_sorting" pipeline at the NEW WIDGET menu, SELECT ALL fields and press NEXT.
• Select the TABLE visualization. Click SELECT ALL and CREATE.

• Now we want to look closer at a single sensor value. Therefore click on the + symbol on the top left to add a new widget.
• Choose the "Persist S7_sorting" pipeline again, set the LIMIT to 1000 and only select the I_At_right_entry field. Now select DISTRIBUTION as visualization and press CREATE. This chart shows how often the sensor returned true or false (package in front of it or not).

5. Modifying the Pipeline​

The next step is to change the pipeline to add some preproccessing steps. We want to count the number of large packages which get transported to the right side and we want to measure the time between two arriving packages.

• Therefore STOP the running "Persist S7_sorting" pipeline in the PIPELINE menu and MODIFY the pipeline.
• Add the BOOLEAN COUNTER as new pipeline element. Connect the data source to the counter and configure the element. Select I_At_right_entry as boolean field and change the flank parameter to FALSE -> TRUE. Then add a DATA LAKE and connect the BOOLEAN COUNTER to it. Use "Boolean_Counter" as identifier.
• Add the BOOLEAN TIMER to the pipeline and connect the data source to it. Select I_At_right_entry as boolean field too and select TRUE and SECONDS in the configuration window. Add another DATA LAKE and connect the * *BOOLEAN TIMER** to it. Use "Boolean_Timer" as identifier.
• SAVE the pipeline and click the checkbox START PIPELINE IMMEDIATELY.

6. Dashboard for preprocessed data​

After we created two new data field in the last step, we will monitor them in the dashboard.

• Go to the DASHBOARD menu and EDIT the dashboard.
• Click ADD VISUALIZATION and select "Persist S7_sorting: Boolean_Counter". Create a TABLE widget as before and SELECT ALL fields.
• Now add one last table. Choose "Persist S7_sorting: Boolean_Timer" and only select the measured_time field
• Test your pipeline by starting the simulation again. The counter should now count up when a package passes the sensor on the right side and the "measured_time" should show the time since the last package arrived.

7. Final Data View​

Finally we want to improve the Data Explorer and create visualizations with the preprocessed data. We will create a time series to verify when a new package arrived and a heat map to check the time between two packages.

• Go to the DATA EXPLORER menu and edit the data view.
• Click ADD WIDGET and choose the pipeline "Persist S7_sorting: Boolean_Counter". Select the counter field and choose the TIME SERIES visualization.
• Create another widget. Choose the "Persist S7_sorting: BooleanCounter" pipeline and select the measured_time field. Then select the HEATMAP and use "measured_time" as HEAT VALUE

Summary​

In this tutorial we learned how to connect a PLC to StreamPipes, monitor the data stream with a dashboard, create a pipeline with different preprocessing steps and visualize the data in the Data View. But we just focused on the basics, so feel free to try out your own ideas and use different preprocessing steps, create new visualizations or try out another simulation environment.

CSV file for PLC tags​

Runtime Name,Path,Data Type,Node Name,Comment,Hmi Visible,Hmi Accessible,Hmi Writeable,Typeobject ID,Version ID
I_High_sensor,Tag table_1,Bool,%I0.0,,True,True,True,,
I_Low_sensor,Tag table_1,Bool,%I0.1,,True,True,True,,
I_Pallet_sensor,Tag table_1,Bool,%I0.2,,True,True,True,,
I_Loaded,Tag table_1,Bool,%I0.3,,True,True,True,,
I_At_left_entry,Tag table_1,Bool,%I0.4,,True,True,True,,
I_At_left_exit,Tag table_1,Bool,%I0.5,,True,True,True,,
I_At_right_entry,Tag table_1,Bool,%I0.6,,True,True,True,,
I_At_right_exit,Tag table_1,Bool,%I0.7,,True,True,True,,
O_Conveyor_entry,Tag table_1,Bool,%Q0.0,,True,True,True,,
O_Load,Tag table_1,Bool,%Q0.1,,True,True,True,,
O_Unload,Tag table_1,Bool,%Q0.2,,True,True,True,,
O_Transf_left,Tag table_1,Bool,%Q0.3,,True,True,True,,
O_Tranf_right,Tag table_1,Bool,%Q0.4,,True,True,True,,
O_Conveyor_left,Tag table_1,Bool,%Q0.5,,True,True,True,,

The Apache StreamPipes (incubating) community is pleased to announce Apache StreamPipes version 0.69.0! The most notable highlights of this release include a completely reworked data explorer for quick exploration of IoT data and advanced user and access rights management. In addition, our new release includes more than 80 improvements and bug fixes.

The current release can be downloaded here.

Data Explorer​

Apache StreamPipes 0.69.0 includes a heavily improved data explorer. The new user interface allows to visually explore data streams within minutes. Any pipeline which uses the Data Lake sink can be visualized in the data explorer. The completely new query interface allows to easily select date and time ranges and users can choose from either displaying raw data or aggregated time series. The widget library includes charts to visualize time-series data (e.g., line chart, scatter plots or heatmaps), value distributions (e.g., density or histograms) or single values. The widget interface is completely customized and users can build rich dashboards.

User and access rights management​

StreamPipes now supports multiple users and roles. The new User management section in the settings page allows to create individual users and groups. Various roles can be assigned to users and groups to manage permissions and visible views. For instance, you can now create users which only see dashboards, but cannot create pipelines or connect new data streams. In addition, individual permissions can be assigned to various resource objects such as pipelines, data streams and dashboards.

Communication between extensions services and the core are now authenticated by default. Service accounts can be created to allow extensions services to access StreamPipes resources.

Improved pipeline element modeling​

We've removed one of the major drawbacks of previous StreamPipes versions: Pipelines can now be flexibly reconfigured. In previous versions, due to the validation of pipelines and their configuration, only root elements could be modified. The new version now gives users full flexibility when modeling pipelines (e.g., by removing or modifying elements from the middle of a pipeline), while pipelines are still continuously validated and modeling errors are shown. Updating pipelines should now be much less distracting and faster.

Other new features​

Mail configuration​

StreamPipes now includes a configuration interface for mail server settings. These settings are used by the email data sink used to trigger email notifications. In addition, several system-wide mail services exist, e.g., to restore forgotten passwords.

Login configuration​

The login mechanism can now be configured directly in the user interface. A self-registration can be activated to allow users to create new accounts. In addition, passwords can be restored by using a two-stage email confirmation process. The profile view allows users to change their passwords individually.

Redesign of pipeline editor​

The pipeline editor has a new look: The pipeline element selection bar has been moved from the top to the left side. The icons are smaller and for each pipeline element, the name and description are shown by default. This makes it easier to find the elements you're looking for.

Improved service discovery​

Under the hood, we've improved the discovery of extensions services. This is part of a larger effort which will be further improved in the next versions. Extensions services now automatically register within the core depending on the IP address of the service. Pipeline element descriptions are now free of the hostname and can be easily updated. Several environment variables exist to manually assign the hostname to an extensions service.

Performance​

We've improved the overall system performance by replacing the triple store with a standard NoSQL database. This leads to huge performance improvements in terms of system start time, pipeline element installation and overall loading time.

Dark mode ;-)​

We know that you wanted this: An experimental dark mode has been added to StreamPipes. The color scheme can be flexibly changed in the profile view and is stored for each user. The dark mode should be considered experimental, so please expect some layout flaws.

Migration​

While we are not yet ready for automatic migration, a migration guide explains several new concepts introduced in this StreamPipes version. We aim at providing a backwards compatible version with release 1.0, planned for later this year.

The Apache StreamPipes (incubating) community is pleased to announce Apache StreamPipes version 0.68.0! The latest release brings many improvements related to health management of pipelines and pipeline monitoring, improved user guidance such as a live preview in the pipeline editor and better handling of pipeline updates as well as several new data processors and sinks such as support for NATS. In addition, the release contains several bug fixes related to K8s deployment, PLC adapters and pipeline management.

The current release can be downloaded here.

User interface​

The user interface brings several improvements for users when working with pipelines. For this release, we focused on providing a better user experience when pipelines and associated dashboard widgets are updated.

Pipeline preview​

A new pipeline preview allows to quickly inspect the output of data streams and processors during pipeline development. The preview can be enabled in the pipeline canvas and displays the latest value of each pipeline element every second.

Editable dashboard widgets​

Apache StreamPipes 0.68.0 improves the modification of existing dashboard widgets. In the live dashboard, the visualization type and the underlying pipeline can now easily be changed without recreating a widget. In addition, the dashboard shows additional error messages and hints in case pipelines are stopped or have been deleted. In the first case, pipelines can be conveniently started directly from the dashboard.

Besides that, the dashboard now provides more visualization types (e.g., bar race chart, status light and word cloud) based on Apache ECharts.

Monitoring & Resilience​

The latest version comes with several improvements related to better monitoring of running pipelines and internal health checking. To allow users to inspect currently running pipelines, a new pipeline monitoring module has been added to the user interface which shows the current throughput of each pipeline element. Individual pipeline elements can be selected in the preview panel. For each pipeline element, the number of processed input events, produced output events and the lag is displayed. The displayed numbers may be slightly inaccurate for higher throughputs, but give an estimate whether the pipeline is running smoothly or lagging behind.

Besides monitoring, StreamPipes 0.68.0 brings many improvements in terms of system resilience. Pipelines are now gracefully stopped once the StreamPipes installation is stopped. Running pipelines are flagged and restarted at startup time. In addition, StreamPipes checks at every startup for any pipelines that were not properly stopped and restarts these pipelines accordingly. For all running pipelines, StreamPipes now regularly checks the state of all involved pipeline elements and detects any failures. Problems are displayed in the UI and pipeline elements are automatically re-invoked once they become available after a failure.

Altogether, Apache StreamPipes is now able to recover and resume its state even when the power plug is pulled! Resiliency is part of an ongoing larger community effort and will be further improved with a new service discovery approach that will be part of the upcoming version 0.69.0.

The Docker Compose-based installation now restarts StreamPipes automatically after a system restart by default.

StreamPipes Client​

To allow third-party applications to access concepts managed by Apache StreamPipes, version 0.68.0 brings a preview of the StreamPipes client and REST API. The public REST API and the client expose several features that were previously only internally available to developers, such as reading pipelines, performing operations on pipelines and subscribing to data sources managed by StreamPipes. An API token can be easily generated in the profile section of the user interface.

Currently provided REST interfaces can be seen in the Swagger documentation, which is linked on the login page. The client is provided as a Java library, an example of its usage is provided in our Examples Github repository

StreamPipes Extensions​

We also added several new adapters to connect data sources and pipeline elements:

• A new NATS sink allows to forward events to a NATS broker
• The PLC adapter now makes use of the PooledPLCDriverManager
• Improvements to the MQTT sink
• The IoTDB adapter now uses the latest IoTDB version
• All SQL-based sinks now share a common and more stable codebase

Other features and improvements​

• The connect-master module has been removed and integrated into the core
• Library upgrades to Angular (11), Spring Boot, and others
• Improvements to the helm chart for K8s deployment

See the release notes for a complete list of new features and improvements.

Upgrading

StreamPipes 0.68.0 is an intermediate release towards a stable version 1.0. Under the hood, we are performing several code migrations to further improve the overall stability and performance of StreamPipes. Therefore, we currently recommend to start with a clean installation when upgrading. We are working towards an auto-update feature which will be available later this year.

Feedback​

We are absolutely open to your suggestions for further improvements! Let us know (by mail or twitter) and we'll consider your feature request in the next release!

And do not forget to subscribe to our new mailing lists: https://streampipes.apache.org/mailinglists.html

This blog post is the second part of a series on how we want to bring StreamPipes closer to the edge. The first part describes how we managed to build and deploy StreamPipes on one Raspberry Pi (Model 4), while the second part further covers how to deploy StreamPipes over a Kubernetes cluster of two Raspberry Pi's via k3s using our StreamPipes helm chart.

<TL;DR>

With new advancements that came in the last StreamPipes release, we managed to reduce the required resources (mainly memory) drastically. To this extent, we cross-compiled StreamPipes Docker images for ARM support and deployed our lite version on a Raspberry Pi (Model 4). We showcase, that it is feasible to run analytical pipelines with ease and operate StreamPipes at moderate load. Additionally, we demonstrate how to deploy StreamPipes over a two-node k3s cluster using our StreamPipes helm chart.

Introduction​

While it is easy to get started with StreamPipes in no time using our single-node deployment instructions using Docker and Docker-Compose, StreamPipes naturally is designed to run in a distributed cluster setup through its microservice architecture.

This architectural design paired with the advancements of multi-architecture StreamPipes images as described in the previous blog post allows us to run certain analytical pipeline element containers with ease on various target nodes as well as target hardware architecture wether it is X86, ARM32v7, or ARM64v8. This becomes more valuable, as we shift the focus away from centralized cloud or data center only deployments where we generally are provided with a homogenous compute landscape in terms of resources towards edge deployments that oftentimes envolve resource-limited, constrained devices that not only differ in terms of compute but also in terms of their chip architecture.

Thereby, having multi-architecture images in your repertoire is key to deal with the challenge of heterogeneity in compute resources along the edge-cloud continuum. However, there are other challenges that occur when dealing with distributed container deployments such as automating deployments itself, scaling them, managing the containerized applications, taking care of networking between services, or realizing fault-tolerance by comparing the desired state against the current one. If there wasn't some solution we could leverage that takes care of all of this.

Well, which K to use? Kubernetes, K3s​

Fortunately, there exists a few solutions in the container ecosystem with the open source container orchestrator Kubernetes (K8s) being the defacto standard, thereby incorporating the best ideas of Google's internal large-scale container cluster management system Borg (see academic paper about the details published at EuroSys 2015). Since we are not going to write "yet another K8s basics blog post", we refer to their docs or other awesome blog posts such as the Kubernetes 101 to get familiarized with the concepts and terminology. With all the power, concepts and terminology K8s provides, it is oftentimes hard for newcomers to get started. Additionally, not all use cases necessarily need these extra features that K8s adds.

On the other hand, there exists K3s, a lightweight Kubernetes distro by Rancher that was introduced early 2019 and has since then gained huge popularity. As it is stated on their website:

• Perfect for Edge: K3s is a highly available, certified Kubernetes distribution designed for production workloads in unattended, resource-constrained, remote locations or inside IoT appliances
• Simplified & Secure: K3s is packaged as a single <40MB binary that reduces the dependencies and steps needed to install, run and auto-update a production Kubernetes cluster.
• Optimized for ARM: Both ARM64 and ARMv7 are supported with binaries and multiarch images available for both. K3s works great from something as small as a Raspberry Pi to an AWS a1.4xlarge 32GiB server.

Thus, this lead us to the point to say, hey why not use K3s to manage StreamPipes in all its glory. The containerized microservice design in addition to the newly available multiarch images are a great fit.

StreamPipes Helm Chart​

While you can easily write an deploy K8s YAML-based manifests for your deployments and services and so on the standard way, it is way more convenient to leverage Helm, the package manager for K8s. Helm describes complex, distributed deployments using charts in order to provide repeatable application installation, easy updates, rollbacks etc.

To this extent, we wrote a Helm chart for StreamPipes that can be found on our Github incubator-streampipes-installer repository. This allows to deploy StreamPipes as one liner as shown later. Helm charts follow a certain directory structure as depicted in the following:

helm-chart/
  Chart.yaml
  values.yaml
  templates/
  ...

The Chart.yaml file contains a description of the chart, e.g., metadata such as chart name, version, maintainers etc.

The values.yaml file contains default values for all charts and in our case serves the purpose to provide a central place to store all variable/values, e.g., streampipes.registry: "apachestreampipes", that replace chart templates such as the registry and version of the StreamPipes backend image:

...
image: {{ .Values.streampipes.registry }}/backend:{{ .Values.streampipes.version }}
...

Lastly, the template/ directory is for template files. When Helm evaluates a chart, it sends all of the files in this directory through the template rendering engine before collecting the results and finally forwarding them to K8s. If interested, you can read more on this in the official Helm documentation.

The almighty StreamPi K3s Cluster​

As pointed out in the first part of the series, we have gained valueable insights on how to run and deploy StreamPipes on a Raspberry Pi. This worked out so greatly, that we thought, why not step up the game, use another Pi and built the almighty StreamPi cluster by leveraging K3s for orchestration and management.

Note: We are aware that this is far off from a production-grade setup, especially running StreamPipes core services such as the central pipeline management backend, or the UI in a two-node cluster. However, it provides a solid starting point for more elaborate deployments along the edge-cloud continuum.

As shown in the following figure, starting from the multiarch images hosted on Dockerhub, we first provisioned the two Pi's with Docker, install and setup K3s and finally deploy our StreamPipes helm chart in order for users to connect, analyze and exploit their IIoT data with StreamPipes.

In the remainder, we cover how we installed Helm and K3s to form our almighty StreamPi K3s cluster. At the time of writing, we tested our deployment using the latest version of the Docker, Helm as well as K3s in the following setup

• 2x Raspberry Pi Model 4 (4GiB RAM): streampi-01, streampi-02
• Raspbian Buster 10
• Docker 19.03.8
• Helm 3.1.2
• K3s v1.17.4+k3s1

1. Install Docker​

SSH into your Raspberry Pi's and run the following command:

curl -sSL https://get.docker.com | sh

2. Install K3s​

Now that we got Helm installed, we can proceed and install K3s. Thus, we must check wether we fulfill all the requirements. These are mainly network related, such that specific ports (server on 6443) are accessible. You should carefully read about this in their official documentation.

Other than that, lean back and enjoy - the installation happens so fast, you might miss it if you get a coffee :)

SSH into K3s server (streampi-01)​

On the server node, run the following command:

sudo curl -sfL https://get.k3s.io | sh -

A kubeconfig file is written to /etc/rancher/k3s/k3s.yaml and the service is automatically started or restarted. The install script will install k3s and additional utilities, such as kubectl, crictl, k3s-killall.sh, and k3s-uninstall.sh.

The K3s installer creates a systemd service which can be used to stop, start, restart and check the status of the K3s server.

sudo systemctl status k3s.service

● k3s.service - Lightweight Kubernetes
   Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2020-04-11 15:54:07 BST; 3 days ago
     Docs: https://k3s.io
  Process: 10193 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
  Process: 10194 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
 Main PID: 10195 (k3s-server)
 (...)

Next, copy the server node token in order to connect additional agent nodes to the cluster.

sudo cat /var/lib/rancher/k3s/server/node-token

SSH into K3s agent (streampi-02)​

Now, open a new terminal session and SSH into your agent and run the following command, thereby adding the copied K3S_TOKEN to connect to the server, that is specified using K3S_URL.

sudo curl -sfL https://get.k3s.io | K3S_URL=https://streampi-01:6443 \
K3S_TOKEN=<NODE_TOKEN> sh -

The K3s installer creates a systemd service which can be used to stop, start, restart and check the status of the K3s agent.

sudo systemctl status k3s-agent.service

● k3s-agent.service - Lightweight Kubernetes
   Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2020-04-11 16:05:37 BST; 3 days ago
     Docs: https://k3s.io
  Process: 24469 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
  Process: 24470 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
 Main PID: 24471 (k3s-agent)
 (...)

That's it ... that's all!

When you go back to the terminal session on the K3s server you can list all nodes in the cluster using standard kubectl commands.

kubectl get nodes
NAME              STATUS   ROLES    AGE   VERSION
streampi-02   Ready    <none>   3d    v1.17.4+k3s1
streampi-01   Ready    master   3d    v1.17.4+k3s1

Hmmm - but wait. We did only add one agent node to the cluster right? Well, that's true, however when installing K3s using the script the server automatically installs and runs an agent on the same node.

3. Install Helm​

SSH into the RPi that should act as the K3s server later on - in our case this is streampi-01 and install Helm. Since we are using Helm 3 there is no need to install tiller. Thus, installing helm is as simple as running the following command:

curl -sSL https://get.helm.sh/helm-v3.1.2-linux-arm.tar.gz | sudo tar -xzf - -C /usr/local/bin/ linux-arm/helm --strip-components=1

This command downloads the Helm binary release from Github, extracts only the binary and puts it in /usr/local/bin. Try go check helm version to see if it worked.

version.BuildInfo{Version:"v3.1.2", GitCommit:"d878d4d45863e42fd5cff6743294a11d28a9abce", GitTreeState:"clean", GoVersion:"go1.13.8"}

To use Helm, we can set and export KUBECONFIG environment variable to point to the kubeconfig under /etc/rancher/k3s/k3s.yaml.

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

Alternatively, if you want this permanently you can either add it to your .bashrc/.zshrc, or create a .kube directory in your home directory and symlink the k3s.yaml.

mkdir $HOME/.kube
ln -s /etc/rancher/k3s/k3s.yaml $HOME/.kube/config

Deploy StreamPipes on K3s using Helm​

Now, that we have all set up, we can go ahead and deploy StreamPipes using our Helm chart. Therefore, we go to the server node (streampi-01), where we installed Helm. Next, we clone our StreamPipes installer repository (dev branch) and navigate to the helm-chart directory.

Note: At the point of writing, the helm chart is only available in the dev branch. This will change in the future as we update the repository.

git clone -b dev --single-branch https://github.com/apache/incubator-streampipes-installer.git
cd incubator-streampipes-installer

At this point, we can leverage Helm's install command in order to install the StreamPipes Helm chart. Since Helm reads the K3s kubeconfig, it forwards the deployment request to K3s. As per default, we deploy StreamPipes lite version, where we only start standalone Java-based pipeline elements and disregard the ones building on top of our Flink wrapper.

helm install streampipes helm-chart

NAME: streampipes
LAST DEPLOYED: Tue Apr 14 18:13:19 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

Next, K3s receives this requests and goes ahead and downloads the corresponding Docker images and starts the deployments in the right order according to the specified dependencies, creates services as well as persistent volume claims when needed. This step takes some time depending on your internet connection and the hardware when starting the containers. You can check the status of the StreamPipes Helm installation using kubectl get pods -o wide (when adding -w, you can automatically watch for changes). You can also use the custom command below to only display the pod name, status as well as the node where it was scheduled and deployed to.

kubectl get pods -o=custom-columns=NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName

NAME                                           STATUS    NODE
activemq-66d58f47cf-b88nb                      Running   streampi-01
zookeeper-5d9947686f-7vpjb                     Running   streampi-02
couchdb-77db98cf7b-6nn99                       Running   streampi-02
consul-55965f966b-tjcfb                        Running   streampi-02
influxdb-b95b6479-mvgbg                        Running   streampi-02
backend-6cdfb7bbf9-b4vgh                       Running   streampi-02
pipeline-elements-all-jvm-64795974-g2xhv       Running   streampi-02
kafka-657b5fb77-xtw8p                          Running   streampi-01
connect-master-56f5f947d-n2t5b                 Running   streampi-01
sources-watertank-simulator-6958f7844c-2jv78   Running   streampi-01
connect-worker-fdbd6f464-m6vm8                 Running   streampi-01
nginx-5d4498d4f5-fcccq                         Running   streampi-02