Skip to main content

Apache StreamPipes release 0.90.0

· 4 min read
Dominik Riemer

5 minutes to read


The Apache StreamPipes community is pleased to announce the immediate availability of Apache StreamPipes version 0.90.0! In total, this release was focused on stabilization and closes 57 issues with improvements and bug fixes as well as some new features such as adapters and data sinks.

The current release can be downloaded here.

Improvements​

The focus of this release cycle was on general usability improvements and bug fixes. We continued to harmonize the StreamPipes web interface, so that all views follow a similar navigation. Many improvements were added to the data explorer:

  • Incomplete events with missing values are now displayed in the data explorer
  • Improvements to the time selector and edit mode
  • More configuration options for data download
  • Improved overall data explorer performance

In addition, we improved features to monitor pipelines and extensions services. The pipeline monitoring component has been restructured and now works independent from the underlying message broker. In addition, we extended the SDK so that runtime errors can be forwarded to the StreamPipes UI. Further improvements to the monitoring stack will be added in the next versions.

New Adapters and Data Sinks​

This version improves support for existing message brokers in form of adapters and sinks:

  • Improvements of the Apache Pulsar (which is also available as a data sink)
  • A new adapter and data sink for Apache RocketMQ
  • A new adapter for NATS (also available as a data sink)

GitHub Community Features​

Apache StreamPipes is a community effort - everyone is welcome not only to contribute code, but also to provide feedback and other input which helps improving the product. To improve community interaction, we switched our issue management from Jira to GitHub issues - making it much easier to create issues and to discuss future features. In addition, we've enabled Github Discussions - feel free to participate in the existing discussions and ask questions if you have problems!

And if you prefer the mailing list, subscribe to dev@streampipes.apache.org or users@streampipes.apache.org. Tech-related and general project discussions as well as release votes will continue to happen on the dev list.

Github Discussions overview

Preview of Standalone Functions​

An exciting new feature is available in preview mode: StreamPipes Functions are a new way to interact with connected data streams in StreamPipes. As an open system, there are already several ways to extend the functionality of StreamPipes: New adapter can be added to connect to non-supported systems. In addition, the SDK supports building custom data processors and sinks, so that StreamPipes can be easily extended at runtime in case new analytics operators are needed.

But providing extensions in form of pipeline elements are not always the best choice: E.g., in cases where non-generic algorithms should be executed on a specific data stream, providing a reusable component for pipeline development is unsuitable.

This is where Functions come into play! StreamPipes Functions allow to define standalone components, which consume one or more connected data streams from StreamPipes and can be used to add custom processing logic. E.g., it becomes easy to apply an ML model on continuous data without the need to build a specific data processing pipeline.

Functions can be registered in an extensions service similar to data processors and sinks by extending a single class:

public class MyFunction extends StreamPipesFunction {
  @Override
  public FunctionId getFunctionId() {
    return FunctionId.from("my-app-id", 1);
  }

  @Override
  public List<String> requiredStreamIds() {
    return List.of("my-stream-id");
  }

  @Override
  public void onServiceStarted(FunctionContext context) {
    // called on service startup
  }

  @Override
  public void onEvent(Event event, String streamId) {
    // called for each incoming event
  }

  @Override
  public void onServiceStopped() {
    // called on service shutdown
  }
}

This concept, which will also come for more programming languages besides Java offer new ways for developers to interact with available data in StreamPipes and build custom extensions for IIoT analytics.

Functions are available in preview mode and their API should therefore be considered unstable.

Future Roadmap​

After this stabilization release, the next version will most probably bring a major new feature - Python support! The community is currently working hard to bring the StreamPipes client and functions to the Python world. Stay tuned!


To get a comprehensive overview of all changes released within Apache StreamPipes 0.90.0 please visit our [release notes](https://github.com/apache/incubator-streampipes/blob/release/0.90.0/RELEASE_NOTES.md#0900).