Skip to main content

Apache StreamPipes release 0.95.0

· 5 min read
Dominik Riemer

We're excited to share the release of StreamPipes 0.95.0, bringing with it a range of bug fixes and new features. In total, the newest release closes over 200 issues. This release doesn't include many large feature drops, we have worked a lot on making small things smoother and improving StreamPipes under the hood. As we continue to work on the 1.0 release of Apache StreamPipes probably later this year, this release features many improvements to the end-user experience and bug fixes.

Let's delve into the key updates introduced in this release.

The current release can be downloaded here.

Adapters & Pipeline Elements​

Improve deletion workflow of adapters​

Within StreamPipes 0.93.0 we introduced the possibility to edit existing adapters. This release improves adapter management by conveniently allowing to delete existing adapters even if they are used in a pipeline. When deleting a new adapter, a dialog appears which allows users to delete all pipelines which are using the data stream produced by the adapter.

https://github.com/apache/streampipes/issues/1936

Enhance data lake schema management​

We've added a new option to the data lake sink which provides users more flexibility when writing data to the internal time-series storage. Users can choose between the Update Schema and Extend existing schema options. By choosing Extend schema, it is possible to also see "old" data in the data explorer after the schema has changed.

https://github.com/apache/streampipes/issues/2252

New Adapter: Open Industry 4.0 devices​

A new adapter in 0.95.0 is support for the Open Industry 4.0 specification over MQTT. The adapter can be used to connect to a variety of devices which support the data format defined in the Development Guidelines for Open Edge Computing.

For instance, the adapter makes it easy to use IO-Link sensors together with an OI4-compatible IO-Link master.

https://github.com/apache/streampipes/pull/2511

New Adapter(s): All PLC4X protocols supported as adapters​

Since our "sister project" Apache PLC4X has introduced a new metadata API for gathering available drivers, StreamPipes 0.95.0 now supports a much larger collection of PLC connectors. The list of newly supported protocols include:

  • Allen Bradley ETH
  • BACnet/IP
  • Beckhoff TwinCat ADS
  • EthernetIP
  • Modbus ASCII/RTU/TCP

The already existing S7 adapter is now also available with additional configuration options.

Configurations on protocol and transport level are dynamically resolved from the PLC4X API. This ensures that users now have much greater flexibility when connecting to industrial protocols.

The screenshot below shows the advanced options panel for Modbus TCP:

The new adapters should be considered a beta feature and we still recommend to use the existing Modbus and S7 adapters until the new adapters have proven their stability. More information on the usage of these new adapters and supported features can also be found on the PLC4X website.

https://github.com/apache/streampipes/issues/2632

New Processor: Static Metadata enrichment​

We added a new processor to add static metadata (such as sensor ID or location IDs) to any data stream within a pipeline.

https://github.com/apache/streampipes/issues/2350

New Processor: Datetime from String​

The new Datetime from String processor allows to convert a datetime string to a timestamp value and lets users conveniently choose the desired target time zone.

https://github.com/apache/streampipes/issues/1865

New Sink: MS Teams​

StreamPipes 0.95.0 comes with a new data sink for Microsoft Teams, which allows to publish notifications from a pipeline to a teams channel using Webhooks.

https://github.com/apache/streampipes/pull/2248

Remove deprecated adapters​

We removed several outdated adapters that were deprecated in release 0.93.0:

  • CoindeskBitcoinAdapter
  • GdeltAdapter
  • FlicMQTTAdapter
  • TISensorTag
  • IexCloudAdapter
  • OpenSenseMapAdapter
  • WikipediaAdapter

https://github.com/apache/streampipes/issues/2128

UX improvements​

Display file types in different colors​

Different file types in the Files overview are now better highlighted by a dedicated color.

https://github.com/apache/streampipes/issues/2107

Download button for asset & files​

In the asset and file overview, resources can be now directly downloaded by using a new download button.

https://github.com/apache/streampipes/issues/2192 https://github.com/apache/streampipes/issues/2074

StreamPipes Python​

As already mentioned in our blog post, we added an example to use an existing ONNX model as part of a StreamPipes function.

https://github.com/apache/streampipes/pull/2687

Administration & Development​

Remove Consul entirely​

Following the replacement of Consul with an internal service registration and discovery mechanism, StreamPipes 0.95.0 also removes Consul from all installation files.

https://github.com/apache/streampipes/issues/2133

New Apache StreamPipes Go-Client​

Apache StreamPipes now includes a new client library for Golang. Our first version of the Go client interacts with the StreamPipes API and provides various ways to gather data from StreamPipes.

Here is an example on the usage of the Go client:



package main

import (
    "github.com/apache/streampipes/streampipes-client-go/streampipes"
    "github.com/apache/streampipes/streampipes-client-go/streampipes/config"
    "log"
)

/*
    Here are some examples of using go client, including outputting the data returned by streams.
    Only supports outputting model data
*/

func main() {

    clientConfig := config.StreamPipesClientConfig{
        Url: "http://localhost:8030",
        Credential: config.StreamPipesApiKeyCredentials{
            UserName: "<Your-User-Name>",
            ApiKey:   "<Your-API-Key>",
        },
    }

    streamPipesClient, err := streampipes.NewStreamPipesClient(clientConfig)
    if err != nil {
        log.Fatal(err)
    }

    dataSeries, err := streamPipesClient.DataLakeMeasures().GetSingleDataSeries("measureName")
    if err != nil {
        log.Fatal(err)
    }
    dataSeries.Print()

    /*
            output format:

            There are 2 pieces of DataSerie in the Dataseries
            The 1 DataSeries
            time                   msg                   test
            2024-02-23T13:37:09.052Z   go-client_test   2f4556
            2024-02-23T13:37:26.044Z   go-client_test   2f4556
            2024-02-23T13:37:29.007Z   go-client_test   2f4556
            The 2 DataSeries
            time                   msg                   test
            2024-02-23T13:38:06.052Z   go-client_test   2f4556
            2024-02-23T13:38:35.044Z   go-client_test   2f4556
            2024-02-23T13:38:38.007Z   go-client_test   2f4556

    */

}

https://github.com/apache/streampipes/pull/2437

First release, are working on more examples and better docs - stay tuned

Final Remarks​

We highly recommend updating to StreamPipes 0.95.0 to take advantage of these new features, bug fixes, and improvements. For more detailed information, please refer to the release notes.

We appreciate your continued support and valuable feedback. StreamPipes is continually evolving, and we are dedicated to providing you with a powerful and reliable platform for your IIoT streaming data needs.