We're excited to share the release of StreamPipes 0.93.0, bringing with it a range of bug fixes and new features. This release places a strong emphasis on refining the technical foundations of StreamPipes, featuring significant changes that set the stage for our major 1.0 release. Nevertheless, we haven't overlooked the end-user experience and have implemented several enhancements. Let's delve into the key updates introduced in this release.
The current release can be downloaded here.
Adapters & Pipeline Elements
Editing existing adapters
StreamPipes has streamlined the process of modifying existing adapter instances, eliminating the inconvenience users previously faced. In prior versions, altering an adapter required a cumbersome sequence of stopping, deleting, and then recreating the adapter. However, with the introduction of StreamPipes 0.93.0, this arduous process is now a thing of the past.
Users can now seamlessly edit running adapters, and the changes will be applied automatically, obviating the need for the aforementioned manual steps. To guide you through this newfound capability, refer to the below animation, which demonstrates how to modify an existing adapter.
While the automatic migration seamlessly handles the majority of cases, there are instances where it may not be feasible. One such scenario arises when an event property, used by a pipeline element, is removed. In such cases, StreamPipes proactively alerts users before initiating the adapter update. Following the alert, the system takes precautionary measures by halting the adapter and marking the associated pipeline for review.
OPC-UA allows detailed & asynchronous browsing
We have fully reworked the browsing interface of our OPC-UA adapter.
Now, all you need to do is specify the URL where the OPC UA server is accessible, eliminating the need to concern yourself with namespaces or node ids. Once the URL is provided, the adapter effortlessly presents all top-level nodes within the user interface. You can then navigate through the node tree interactively by clicking on the desired node.
For every data node, accessing node information is a breeze – simply click on the eye icon. If you wish to subscribe to a particular node, a straightforward click on the plus icon is all it takes.
All the nodes you select become integral to the event structure of this adapter, accessible by clicking on Next. The
ensuing view not only displays the OPC UA status for each node but also allows for the removal of a node with ease,
especially if its status is not Good.
Adapter for S7 PLCs supports array-based data
Until now, our S7 PLC adapter lacked the capability to read values structured in arrays. We're pleased to announce that
this limitation has been addressed through the introduction of an Advanced register input mode.
This new mode empowers you to configure the registers to be queried in a manner specific to the requirements of our underlying dependency, PLC4X. For detailed insights and guidance on the configuration process, refer to the PLC4X S7 documentation.
When dealing with array definitions, the configuration would resemble the following:
# propertyName=%{Memory-Area}{start-address}:{Data-Type}[{array-size}]
temperature=%DB3.DB0.0[BOOL]
New Adapter: ifm IOLink
The IO-Link Master (AL1350) from ifm comes equipped with an IoT interface designed for transmitting sensor data via MQTT. With the introduction of this adapter, users gain the capability to effortlessly incorporate IO-Link sensor data into StreamPipes. This experimental integration empowers users to effortlessly connect to data from the ifm sensor VVB001.
For more details refer to the adapter documentation.
For the integration of additional sensors we are thinking about supporting IODD. Your feedback on these considerations is invaluable to us. If you find the integration of IODD for supporting more sensors within StreamPipes appealing or have insights to share, we welcome your input. Please let us know your thoughts on this potential feature enhancement.
New Processor: Numeric Rounding
Introducing our latest data processor, Numeric Rounding , designed to perform rounding operations on numeric values within an event stream. This processor enables users to select one or more numeric values from their event data, specifying the desired rounding precision and the rounding mode.
For more details refer to the processor documentation.
UX improvements
Ordered measurement units
In the adapter configuration step, when setting up a transformation rule for an event value, the available units of measurement are now conveniently organized in alphabetical order for improved accessibility and ease of selection.
Extended information for existing adapters
When accessing the Connect module, a comprehensive list of all existing adapters is presented. Within this view, each adapter now includes details about the timestamp of the last event creation and the total count of elements published since startup. This enhancement provides a quick and efficient means to evaluate the health of each adapter.
Interactive tutorial is back
Upon initiating a new StreamPipes instance, a dialog prompts users to embark on an interactive tutorial showcasing the platform's key features. We're pleased to announce that this tutorial is now operational again. Enjoy exploring and discovering the most important functionalities in StreamPipes. Happy learning!
Deployment
Kubernetes Deployment
Add probes to the Helm chart
Integrating readiness, liveness, and startup probes into StreamPipes' Kubernetes containers enhances the system's health monitoring. These probes automatically detect and handle anomalies, significantly improving application availability and stability. The readiness probe ensures preparedness, the liveness probe validates ongoing functionality, and the startup probe assesses initialization status, collectively fortifying StreamPipes against potential disruptions.
Rework the values file
We've refined and simplified the values file for our Helm chart, optimizing its structure. Furthermore, we've invested in comprehensive documentation for all available options. Check out the README file of our Helm chart for detailed information on configuration and deployment.
Add Apache Pulsar to Helm file
In our last release, we've expanded StreamPipes' capabilities to include Apache Pulsar as a supported messaging layer. Alongside this enhancement, our Helm chart has now been updated to seamlessly deploy StreamPipes on Kubernetes with Pulsar as the underlying messaging infrastructure.
Monitoring Integration (Grafana & Prometheus)
We've now integrated monitoring support for StreamPipes through Prometheus and Grafana, making it seamlessly available in both our Kubernetes and Docker deployments. If you're using Kubernetes, we recommend checking out our Helm chart for straightforward setup instructions. For Docker users, stay tuned for an upcoming blog post where we'll provide a step-by-step guide on configuring Prometheus and Grafana to monitor StreamPipes effectively.
New Deployment mode for first time users
To simplify onboarding with Apache StreamPipes and demonstrate its capabilities, we've added a quickstart deployment option. This docker-compose variant provides users with a pre-configured StreamPipes instance and rich examples. Explore it in our user guide.
StreamPipes Python
Our official Python support for Apache StreamPipes got some small improvements as well. Notable among other improvements are especially the following two:
- Support for Python 3.11
- StreamPipes Python does now also support other protocols than Apache Kafka to interact with live data
Administration & Development
Email Templates are configurable
StreamPipes already supported the sending of notifications, such as those for password recovery and account registration, when properly configured. However, in previous versions, the email content was predefined and couldn't be adjusted. In this latest release, administrators now have the flexibility to modify the email templates, offering greater customization options for notifications.
Remove legacy resources
In this release, we've eliminated legacy resources that were initially employed for demonstration purposes. As these
resources are no longer actively supported, we have decided to completely remove them. The affected components include
the vehicle simulator and watertank simulator.
This removal has led to the discontinuation of the StreamPipes package streampipes-sources.
Furthermore, we are pleased to announce the forthcoming removal of Consul as an additional dependency required to run StreamPipes in the upcoming release. Within this release, we have already implemented the fundamental steps to eliminate the need for Consul. This marks a significant stride towards making StreamPipes a more lightweight platform. It's crucial to highlight that this modification is purely a technical enhancement and will have no impact on the end user experience.
Enable migrations for pipeline elements
We have established a crucial technical foundation to simplify the update process for upcoming versions of StreamPipes. It is now possible to define migrations for adapters, data processors, and data sinks. For a more in-depth understanding of how migrations function and how to define them, please refer to our documentation.
New Homepage
We've undergone a comprehensive redesign of our website, streampipes.apache.org. Take a moment to explore the new look, and we welcome your feedback on your experience. Let us know what you think!
Security
We've addressed a vulnerability in the default password generation for new users during the self-registration process. Previously, a cryptographically weak pseudo-random number generator (PRNG) was utilized, but we've now rectified this by implementing Apache Commons RNG. We extend our sincere thanks to Alessandro Albani from the Digital Security Division Var Group for bringing this to our attention.
Final Remarks
We highly recommend updating to StreamPipes 0.93.0 to take advantage of these new features, bug fixes, and improvements. For more detailed information, please refer to the release notes.
We appreciate your continued support and valuable feedback. StreamPipes is continually evolving, and we are dedicated to providing you with a powerful and reliable platform for your IIoT streaming data needs.