2019-10-25
The world's leading open source IoT software provider EMQ (Github Project ) officially open-sourced a lightweight streaming data processing project EMQ X Kuiper (hereafter referred to as Kuiper) on October 23, 2019 . With the release of Kuiper, EMQ will accelerate the arrangement of the IoT middleware domain, so as to form a full coverage of IoT middleware software from messaging to data processing, from edge to cloud.
In 2019, as the first year of 5G commercial use, with the deepening of 5G deployment, edge computing has become increasingly popular. Edge computing increases computing power with the computer room close to the access network, so as to:
A large part of the IoT edge computing refers to the processing of streaming data, which is a set of sequential, massive, fast and continuous data series. Generally, streaming data can be regarded as a dynamic data set that grows infinitely over time. Streaming data processing helps users understand the status of system devices in real time and respond quickly to abnormal situations. The computing resources (CPU, memory, etc.) at the edge are not as rich as in the cloud. Therefore, for the traditional streaming data processing framework such as Apache Storm or Apache Flink, because of its large installation package, complex deployment structure and process, and high runtime consumption, it is not suitable for the resource constrained edge devices (IPCs, gateways, or devices with low-profile X86 servers) with limited resources. Kuiper open-sourced by EMQ is to solve these problems on the IoT edge devices.
Kuiper can be applied all kinds of edge computing user scenarios, the typical sample scenarios including,
The previous scenarios describe the value of edge computing: local data processing improves user experience; local data processing also improves data security, and protects personal privacy; Save the costs of bandwidth and cloud storage. Kuiper can be run at those resource constrained devices, and realized the streaming processing with low cost.
As shown in the following figure, Kuiper Server consists of three parts:
Kuiper's usage process is similar to traditional streaming tools, mainly including three steps: streaming definition, rule definition and submit rule.
Streaming definition: This step is mainly used to define the data format of the data source, similar to the process of defining the table structure in the traditional relational database. A data streaming named demo is defined as follows:
# bin/cli create stream demo '(temperature float, humidity bigint) WITH (FORMAT="JSON", DATASOURCE="demo")'
Rule definition: this step is mainly used to define the following three parts
{
"sql": "SELECT * from demo where temperature > 30",
"actions": [{
"log": {}
}]
}
As shown above, a rule is defined to extract data with a temperature greater than 30 from the streaming named demo, and output the data satisfying the condition to the log file. Rules can be saved in a text file, such as myRule
Submit rule
# bin/cli create rule ruleDemo -f myRule
Once the rule is completed, it can be committed by creating a rule command. The data that satisfies the conditions will be output to the log file.
...
[{"humidity":17,"temperature":60}]
[{"humidity":59,"temperature":68}]
[{"humidity":67,"temperature":31}]
[{"humidity":5,"temperature":45}]
[{"humidity":34,"temperature":97}]
[{"humidity":90,"temperature":73}]
[{"humidity":72,"temperature":59}]
[{"humidity":39,"temperature":72}]
[{"humidity":76,"temperature":92}]
[{"humidity":23,"temperature":40}]
[{"humidity":74,"temperature":91}]
[{"humidity":50,"temperature":76}]
[{"humidity":77,"temperature":44}]
...
Readers can complete Kuiper's first experience by following the Tutorial Documentation on Github. The Kuiper project is based on the Apache 2.0 open source license and any problems found in the use process can be directly submitted in the project.
Welcome to our open source project github.com/emqx/emqx. Please visit the official documentation for details.