# Migrate from V4
This document provides guidelines for migrating from EMQX 4.x to 5.0. EMQX 5.0 is not backward compatible with 4.x in management APIs and clustering APIs, however most of the functionalities are not significantly changed.
To learn about the new features we have added in EMQX 5.0, checkout 5.0.0 Release (opens new window).
# Log
Comparing with 4.x, the biggest change in log is the format. In 4.x, the log text is mostly free text aiming for good human readability. Starting from 5.0, we moved to structured logging without losing readability. For instance, most of the log fields uses underscore as word separators, which makes it in general more search-friendly, also helps log indexing tools to index the logs more effectively.
2022-06-29T16:58:53.235042+02:00 [info] foo: bar, msg: msg_for_human_to_read_but_also_easy_to_index
Find more details in Log & Trace.
# Default listeners
By default, the following listeners will no longer be provided:
Listeners | Description |
---|---|
MQTT-TCP 11883 | Listeners for backend applications |
Management-HTTP 8081 | For REST API, merged to port 18083 |
# Plugins
The previous official plug-ins have been migrated to EMQX as built-in functions. Plug-ins developed by users on version 4.x need to be adapted before they can be used in EMQX 5.0. The following is a comparison table between official plug-ins and existing functions:
4.x | 5.0 |
---|---|
emqx_auth_http | AuthN/AuthZ - HTTP data source |
emqx_auth_jwt | AuthN - JWT |
emqx_auth_mnesia | AuthN/AuthZ - Built-in database |
emqx_auth_mongo | AuthN/AuthZ - MongoDB data source |
emqx_auth_mysql | AuthN/AuthZ - MySQL data source |
emqx_auth_pgsql | AuthN/AuthZ - PostgreSQL data source |
emqx_auth_redis | AuthN/AuthZ - Redis data source |
emqx_sasl | AuthN/AuthZ - MQTT 5 Enhanced Authentication |
emqx_auth_ldap | - |
emqx_rule_engine | Data Integration |
emqx_bridge_mqtt | Data Bridge - MQTT Sink/MQTT Source |
emqx_web_hook | Data Bridge - WebHook |
emqx_coap | CoAP Gateway |
emqx_dashboard | Dasboard |
emqx_exhook | ExHook |
emqx_exproto | ExProto Gateway |
emqx_lwm2m | LwM2M Gateway |
emqx_sn | MQTT-SN Gateway |
emqx_stomp | STOMP Gateway |
emqx_lua_hook | - |
emqx_management | - |
emqx_prometheus | Prometheus |
emqx_psk_file | AuthN - PSK (psk_authentication.enable = true ) |
emqx_recon | Old features still available from CLI emqx_ctl observer |
emqx_retainer | Retain |
emqx_telemetry | Telemetry |
# HTTP API
Previously, "Appication" was used to manage API access credentials, which has now been renamed to "API Key". And Secret Key is only returned once when it is successfully created, but cannot be obtained again later.
Port 8081 has been merged into port 18083, and the basic path of API access has been switched from /api/v4
to /api/v5
, please call the API through this port and path.
String timestamps are now in RFC3339 (opens new window) format.
# Data Format Changes
When the response is successful, the business status code code
will no longer be returned with the data, and the corresponding 4xx/5xx HTTP status code and error prompt will be returned when an error occurs. Users can access GET /error_codes
to get all possible error codes, the following is a comparative example of the response format:
# 4.x successful response
## HTTP StatusCode = 200
GET /rules/my_rule
{ "code": 0, "data": { ... } }
# 4.x error response
## HTTP StatusCode = 200
GET /rules/my_rule
{ "code": 404, "message": "Not Found" }
# 5.0 successful response
## HTTP StatusCode = 200
GET /rules/my_rule
{ ... }
# 5.0 error response
## HTTP StatusCode = 404
GET /rules/my_rule
{ "code": "NOT_FOUND", "message": "Rule Id Not Found" }
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# Major API Changes
The API has undergone major changes, and some APIs have been made compatible. The following is a comparison table of commonly used API changes.
TIP
Compatibility Notes:
- Compatibility: Use the old API path and parameters, or keep the old API
- Partially compatible: API path remains unchanged, but some API fields have changed
- Incompatible: API paths and fields have changed
4.x | 5.0 | Compatibility | Notes |
---|---|---|---|
Publish/Subscribe | |||
POST /mqtt/publish | POST /publish | Compatibility | |
POST /mqtt/publish_batch | POST /publish/bulk | Compatibility | |
POST /mqtt/subscribe | POST /clients/{clientid}/subscribe | Compatibility | |
POST /mqtt/subscribe_batch | POST /clients/{clientid}/subscribe/bulk | Compatibility | |
POST /mqtt/unsubscribe | POST /clients/{clientid}/unsubscribe | Compatibility | |
POST /mqtt/unsubscribe_batch | POST /clients/{clientid}/unsubscribe/bulk | Compatibility | |
Clients/Topics/Subscriptions | |||
GET /clients | GET /clients | Partially compatible | |
GET /routes{/topic} | GET /topics{/topic} | Incompatible | routes rename to topics |
GET /subscriptions | GET /subscriptions | Partially compatible | |
GET /subscriptions/{clientid} | GET /clients/{clientid}/subscriptions | Incompatible | |
Node/Stats/Metrics | |||
GET /nodes | GET /nodes | Partially compatible | |
GET /brokers | - | Incompatible | merged to GET /nodes |
GET /stats | GET /stats | Partially compatible | |
GET /metrics | GET /metrics | Partially compatible | |
Users/Alarms | |||
GET /users | GET /users | Partially compatible | |
GET /alarms{/activated} | GET /alarms?activated={true,false} | Incompatible | |
GET /alarms{/deactivated} | GET /alarms?activated={true,false} | Incompatible |
# Auth/ACL
The Auth and ACL plugins (emqx_auth_*) have been removed, and the corresponding functions are built into EMQX, and users can configure them through Dashboard or configuration files.
Auth was renamed Authenticate and ACL was renamed Authorization. Now they are independent of each other.
Version 5.0 supports the authentication methods and most data sources in 4.x, but there are certain adjustments in the usage.
# Fixed order of execution
When multiple authenticators or authorization checkers are enabled at the same time, the checks are no longer performed according to the startup order but a fixed configuration order, which can be adjusted in the configuration file and Dashboard.
# Replace variable extraction syntax
Previously, the Auth plugin could use %u
similar syntax to construct placeholders to extract variables, dynamically splicing client information into SQL statements, Redis query commands and HTTP requests. Now EMQX use a new syntax ${}
, such as ${username}
, ${clientid}
, which is consistent with the rule SQL.
Usage example:
# 4.x
# etc/emqx_auth_mysql.conf
auth.mysql.auth_query = select password from mqtt_user where username = '%u' limit 1
# 5.0
# emqx.conf
authentication = [
{
...
mechanism = "password_based"
backend = "mysql"
query = "SELECT password_hash, salt FROM mqtt_user where username = ${username} LIMIT 1"
}
]
2
3
4
5
6
7
8
9
10
11
12
13
14
# Auth
# Remove anonymous mechanism
Remove the alow_anonymous
configuration item, all client connections are allowed by default. If add and enable any authenticator, EMQX will perform authentication check on all new connections.
Remove the bypass_auth_plugins
configuration item. When a listener needs to disable authentication, it can be set through the listeners.{type}.{name}.enable_authn = true | false
configuration item.
# Built-in database (Mnesia)
- Mnesia renamed to built-in database;
- Only one search method can be selected: based on user name or based on client ID;
- The data format and REST API have changed. For more information, please refer to
POST /authentication/{id}/users
.
Users can use the data import API to import data from older versions into EMQX 5.0, see POST /authentication/{id}/import_users
for details.
# HTTP
- Use the JSON fields inside the response body instead of the HTTP response status codes to decide on the authentication result;
- Remove the super user query configuration. If you need the super user function, please set it through JSON in the response body of successful authentication.
Success response status code:
200 or 204
The authenticator will be ignored if the request fails or returns another status code.
Success Response Body (JSON):
Name | Type | Required | Description |
---|---|---|---|
result | Enum | true | allow | deny | ignore |
is_supseruser | Boolean | false |
{
"result": "allow",
"is_supseruser": true
}
2
3
4
# MySQL/PostgreSQL
- The password field required in the query result is changed from
password
topassword_hash
, if you do not want to change the database column name, you can useas
to complete the migration; - Remove the superuser query SQL. If you need superuser, please make sure that the authentication SQL result contains the
is_superuser
field.
SELECT password as password_hash, salt, is_superuser FROM mqtt_user where username = ${username} LIMIT 1
# MongoDB
- Remove the superuser query. If you need a superuser, please specify the
is_superuser_field
field.
authentication = [
{
...
mechanism = "password_based"
backend = "mongodb"
# is_superuser_field = "is_superuser"
}
]
2
3
4
5
6
7
8
# Redis
- Only supports Redis Hashes (opens new window) data structure and
HGET
,HMGET
query commands, must usepassword_hash
orpassword
(compatible with 4.x version) as password field name; - Remove the superuser query, if you need the superuser, please add the
is_superuser
field to the Redis query command.
# bad
GET emqx_user:${username}
# bad
HMGET emqx_user:${username} passwd
# good
HMGET emqx_user:${username} password_hash
# good
HMGET emqx_user:${username} password_hash is_superuser
2
3
4
5
6
7
8
9
10
# LDAP
Not supported for now.
# JWT
No changes.
# ACL
# ACL File
- Remove the
acl_file
configuration, the file-based ACL (acl.conf) will be used as one of the authorization checkers and added to EMQX by default; - The
acl.conf
file is moved from theetc
directory to thedata/authz
directory, please ensure that EMQX has read and write permissions for this file; acl.conf
data file syntax has changed
4.x | 5.0.0 | Compatibility |
---|---|---|
user | username | Yes |
client | clientid | Yes |
pubsub | all | No |
# Built-in database (Mnesia)
- Mnesia renamed to built-in database;
- The data format and REST API have changed. For more information, please refer to
POST /authorization/built_in_database/clientid
.
The data export command ./bin/emqx_ctl data export
in 4.x contains ACL data, and users can process the data into the format required by 5.0 and import it through the corresponding REST API.
# MySQL/PostgreSQL
- The
ipaddr/username/clientid
field is no longer required in the query result; - The
access
field name is changed toaction
, the data type is changed from integer to character or character enumeration, and the numerical correspondence is shown in the following table; - The
allow
field name is changed topermission
, the data type is changed from integer to character or character enumeration, and the numerical correspondence is shown in the following table.
access-action field mapping
4.x (int) | 5.0.0 (varchar/enum) | action |
---|---|---|
1 | subscribe | subscribe |
2 | publish | publish |
3 | all | subscribe & publish |
allow-permission field mapping
4.x (int) | 5.0.0 (varchar/enum) | permission |
---|---|---|
0 | deny | deny |
1 | allow | allow |
# MongoDB
- MongoDB data source can be used in black/white list mode. Previously, only white list mode was supported. It is required to set
acl_nomatch = deny
; - You need to query the data list containing the
action
permission
topics
fields from MongoDB. For details, see AuthZ-MongoDB.
If you want to continue using the data in 4.x, please migrate the adaptation manually.
Data example in 5.0
[
{
"username": "emqx_u",
"clientid": "emqx_c",
"ipaddress": "127.0.0.1",
"permission": "allow",
"action": "all",
"topics": ["#"]
}
]
2
3
4
5
6
7
8
9
10
# Redis
- Redis data source still only supports whitelist mode, which requires setting
acl_nomatch = deny
; - The
access
field name changes toaction
, and the data changes from numbers to action strings. The corresponding relationship is shown in the table below.
4.x | 5.0.0 | action |
---|---|---|
1 | subscribe | subscribe |
2 | publish | publish |
3 | all | subscribe & publish |
If you want to continue using the data in 4.x, please migrate the adaptation manually.
Data example in 5.0
HSET mqtt_acl:emqx_u t/# subscribe
HSET mqtt_acl:emqx_u # all
HSET mqtt_acl:emqx_u a/1 publish
2
3
# HTTP
- Use the JSON fields inside the response body instead of the HTTP response status codes to decide on the authentication result.
Success response status code:
200 or 204
Other status codes or this authorizer will be ignore
if the request fails.
Success Response Body (JSON):
Name | Type | Required | Description |
---|---|---|---|
result | Enum | true | allow | deny | ignore |
{
"result": "deny"
}
2
3
# Rule Engine
The rules engine has been renamed Data Integration, which includes rules and data bridge.
Rule SQL is fully compatible with 4.x syntax, but the actions under the rule are split into built-in actions (republish, console) and data bridges (WebHook, MQTT Sink, MQTT Source), so as to realize the reuse of actions.
# WebHook
The WebHook plugin (emqx_web_hook) has been removed, please use the WebHook data bridge in Data Integration instead.
# MQTT Bridge
The MQTT bridge plugin (emqx_bridge_mqtt) has been removed, please use the MQTT Sink and MQTT Source data bridge in Data Integration instead.
# Prometheus
Prometheus scraping endpoint is by default enabled and there is no authentication required to scrap the metrics.
You can use curl
command to inspect the metrics: curl -f "127.0.0.1:18083/api/v5/prometheus/stats"
If you wish to enable push-gateway, it is configurable from prometheus {...}
config block. Or you could as well configure and enable it from the Dashboard.
Changes:
4.4.4 | 5.0 | Description |
---|---|---|
emqx_client_auth_success_anonymous | emqx_client_auth_anonymous | Renamed |
emqx_client_check_acl | emqx_client_authorize counter | Renamed |
- | emqx_mria_last_intercepted_trans | New |
- | emqx_mria_replicants | New |
- | emqx_mria_server_mql | New |
- | emqx_mria_weight | New |
emqx_routes_count | emqx_topics_count | Renamed |
emqx_routes_max | emqx_topics_max | Renamed |
emqx_session_takeovered | emqx_session_takenover | Renamed |
erlang_vm_ets_tables | - | Removed |
- | erlang_vm_memory_dets_tables | New |
- | erlang_vm_memory_ets_tables | New |
- | erlang_vm_msacc_alloc_seconds_total | New |
- | erlang_vm_msacc_aux_seconds_total | New |
- | erlang_vm_msacc_bif_seconds_total | New |
- | erlang_vm_msacc_busy_wait_seconds_total | New |
- | erlang_vm_msacc_check_io_seconds_total | New |
- | erlang_vm_msacc_emulator_seconds_total | New |
- | erlang_vm_msacc_ets_seconds_total | New |
- | erlang_vm_msacc_gc_full_seconds_total | New |
- | erlang_vm_msacc_gc_seconds_total | New |
- | erlang_vm_msacc_nif_seconds_total | New |
- | erlang_vm_msacc_other_seconds_total | New |
- | erlang_vm_msacc_port_seconds_total | New |
- | erlang_vm_msacc_send_seconds_total | New |
- | erlang_vm_msacc_sleep_seconds_total | New |
- | erlang_vm_msacc_timers_seconds_total | New |
- | erlang_vm_statistics_dirty_cpu_run_queue_length | New |
- | erlang_vm_statistics_dirty_io_run_queue_length | New |
- | erlang_vm_wordsize_bytes | New |
# Gateway/Multi-Protocol Supports
Clients of other protocols (LwM2M, CoAP, STOMP, MQTT-SN) will no longer be mapped as MQTT clients and cannot be obtained through the Dashboard client page and the GET /clients
API. Users can go to the gateway page details page or get it through the GET /gateway/{name}/clients
API.
# Telemetry
The telemetry plugin (emqx_telemetry) has been removed, please configure it through the telemetry {}
configuration item or the Dashboard System Settings -> Settings page.