# JQ Functions

JQ (opens new window) is a powerful command line tool and programming language designed primarily for transforming and querying data encoded as JSON (opens new window).

Typical JQ programs describe simple transformations or filters for JSON data, but one can also use JQ to perform complex computations when needed.

# How to use

The rule's SQL language integrates JQ through two functions:

FunctionParametersReturned value
jq1. string containing valid jq program
2. JSON encoded string or object
3. Optional, integer timeout value (milliseconds, default is 10 seconds)
list of objects corresponding to the JSON objects generated by the given JQ program (parameter 1) when given the input provided by parameter 2. The function throws an exception if the execution did not finish before the timeout or if the jq program throws an exception.

The default timeout value can be configured via rule_engine.jq_function_default_timeout.

The JQ documentation (opens new window) is a good resource for learning how to write JQ programs. You can also try out JQ programs in this online environment (opens new window), install it on your computer, or watch EMQX's JQ introduction video (opens new window). JQ is a Turing-complete (opens new window) high-level functional programming language.

The following are some examples of simple jq function calls and their results:

# Example 1

Simple jq function calls example:

jq('.', '{"temprature": 10}') = [json_decode('{"temprature": 10}')]
jq('.', json_decode('{"temprature": 10}')) = [json_decode('{"temprature": 10}')]
jq('.temprature', '{"temprature": 10}') = [10]
jq('{temprature_C:.temprature,temprature_F: (.temprature * 1.8 + 32)}', '{"temprature": 10}') = [json_decode('{"temprature_C": 10, "temprature_F": 50}')]
jq('.temprature,(.temprature * 1.8 + 32)', '{"temprature": 10}') = [10, 50]
1
2
3
4
5

# Example 2

The above examples only scratch the surface of what can be done with JQ. The below example illustrates what a more complex JQ program might look like and how one can combine the jq function with the FOREACH statement to divide JQ's output objects into multiple messages.

FOREACH   jq('def rem_first: ' +
             '    if length > 2 then del(.[0]) else . end;' +
             'def rem_last:' +
             '    if length > 1 then del(.[-1]) else . end;' +
             '.date as $date |' +
             '.sensors[] |' +
             '  (.data | sort | rem_first | rem_last | add / length) as $average |' +
             '  {$average, $date}',
             payload)
FROM    "jq_demo/complex_rule/jq/#"
1
2
3
4
5
6
7
8
9
10

Below is an example of input payload for the code above:

{
  "date": "2020-04-24",
  "sensors": [
    {
      "name": "a",
      "data": [3, 1, 2, 4, 5, 5]
    },
    {
      "name": "b",
      "data": [1, -100, 2, 3, 4, 5, 2000]
    },
    {
      "name": "c",
      "data": [3, 7, 9]
    }
  ]
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

The rule SQL snippet above will create one output message for each sensor in the input data. Each message will be a JSON object containing one field for the date and one field for the average of the sensor's data field after the smallest and largest values have been removed (as they might be outliers). Thus given the payload above, the three output messages will have the following payloads:

Message 1:

{
  "average": 3.5,
  "date": "2020-04-24"
}
1
2
3
4

Message 2:

{
  "average": 3,
  "date": "2020-04-24"
}
1
2
3
4

Message 3:

{
  "average": 7,
  "date": "2020-04-24"
}
1
2
3
4

# Notice

JQ functions can be convenient for transformations that are difficult or impossible to do with only the rule SQL language and its simple functions.

It is not recommended to do long-running computations in the rule as this can significantly slow down the rate at which EMQX can process new messages. The JQ functions have timeouts to prevent possibly buggy JQ programs (that may have gotten stuck in an infinite loop) from taking too much processing time from the rest of the EMQX system.