The Apache StreamPipes (incubating) community is pleased to announce Apache StreamPipes version 0.68.0! The latest release brings many improvements related to health management of pipelines and pipeline monitoring, improved user guidance such as a live preview in the pipeline editor and better handling of pipeline updates as well as several new data processors and sinks such as support for NATS. In addition, the release contains several bug fixes related to K8s deployment, PLC adapters and pipeline management.
The current release can be downloaded here.
User interface
The user interface brings several improvements for users when working with pipelines. For this release, we focused on providing a better user experience when pipelines and associated dashboard widgets are updated.
Pipeline preview
A new pipeline preview allows to quickly inspect the output of data streams and processors during pipeline development. The preview can be enabled in the pipeline canvas and displays the latest value of each pipeline element every second.
Editable dashboard widgets
Apache StreamPipes 0.68.0 improves the modification of existing dashboard widgets. In the live dashboard, the visualization type and the underlying pipeline can now easily be changed without recreating a widget. In addition, the dashboard shows additional error messages and hints in case pipelines are stopped or have been deleted. In the first case, pipelines can be conveniently started directly from the dashboard.
Besides that, the dashboard now provides more visualization types (e.g., bar race chart, status light and word cloud) based on Apache ECharts.
Monitoring & Resilience
The latest version comes with several improvements related to better monitoring of running pipelines and internal health checking. To allow users to inspect currently running pipelines, a new pipeline monitoring module has been added to the user interface which shows the current throughput of each pipeline element. Individual pipeline elements can be selected in the preview panel. For each pipeline element, the number of processed input events, produced output events and the lag is displayed. The displayed numbers may be slightly inaccurate for higher throughputs, but give an estimate whether the pipeline is running smoothly or lagging behind.
Besides monitoring, StreamPipes 0.68.0 brings many improvements in terms of system resilience. Pipelines are now gracefully stopped once the StreamPipes installation is stopped. Running pipelines are flagged and restarted at startup time. In addition, StreamPipes checks at every startup for any pipelines that were not properly stopped and restarts these pipelines accordingly. For all running pipelines, StreamPipes now regularly checks the state of all involved pipeline elements and detects any failures. Problems are displayed in the UI and pipeline elements are automatically re-invoked once they become available after a failure.
Altogether, Apache StreamPipes is now able to recover and resume its state even when the power plug is pulled! Resiliency is part of an ongoing larger community effort and will be further improved with a new service discovery approach that will be part of the upcoming version 0.69.0.
The Docker Compose-based installation now restarts StreamPipes automatically after a system restart by default.
StreamPipes Client
To allow third-party applications to access concepts managed by Apache StreamPipes, version 0.68.0 brings a preview of the StreamPipes client and REST API. The public REST API and the client expose several features that were previously only internally available to developers, such as reading pipelines, performing operations on pipelines and subscribing to data sources managed by StreamPipes. An API token can be easily generated in the profile section of the user interface.
Currently provided REST interfaces can be seen in the Swagger documentation, which is linked on the login page. The client is provided as a Java library, an example of its usage is provided in our Examples Github repository
StreamPipes Extensions
We also added several new adapters to connect data sources and pipeline elements:
- A new NATS sink allows to forward events to a NATS broker
- The PLC adapter now makes use of the PooledPLCDriverManager
- Improvements to the MQTT sink
- The IoTDB adapter now uses the latest IoTDB version
- All SQL-based sinks now share a common and more stable codebase
Other features and improvements
- The connect-master module has been removed and integrated into the core
- Library upgrades to Angular (11), Spring Boot, and others
- Improvements to the helm chart for K8s deployment
See the release notes for a complete list of new features and improvements.
Upgrading
StreamPipes 0.68.0 is an intermediate release towards a stable version 1.0. Under the hood, we are performing several code migrations to further improve the overall stability and performance of StreamPipes. Therefore, we currently recommend to start with a clean installation when upgrading. We are working towards an auto-update feature which will be available later this year.
Feedback
We are absolutely open to your suggestions for further improvements! Let us know (by mail or twitter) and we'll consider your feature request in the next release!
And do not forget to subscribe to our new mailing lists: https://streampipes.apache.org/mailinglists.html