# User Guide

# Authentication

The EMQ broker supports to authenticate MQTT clients with ClientID, Username/Password, IpAddress and even HTTP Cookies.

The authentication is provided by a list of plugins such as MySQL, PostgreSQL and Redis...

If we enable several authentication plugins at the same time, the authentication process:

           ----------------           ----------------           -------------
Client --> |   Username   | -ignore-> |   ClientID   | -ignore-> | Anonymous |
           ----------------           ----------------           -------------
                  |                         |                         |
                 \|/                       \|/                       \|/
            allow | deny              allow | deny              allow | deny

The authentication plugins implemented by default:

PluginDescription
emq_auth_clientid (opens new window)ClientId Auth Plugin
emq_auth_username (opens new window)Username Auth Plugin
emq_auth_ldap (opens new window)LDAP Auth Plugin
emq_auth_http (opens new window)HTTP Auth/ACL Plugin
emq_auth_mysql (opens new window)MySQL Auth/ACL Plugin
emq_auth_pgsql (opens new window)Postgre Auth/ACL Plugin
emq_auth_redis (opens new window)Redis Auth/ACL Plugin
emq_auth_mongo (opens new window)MongoDB Auth/ACL Plugin

# Allow Anonymous

Configure etc/emq.conf to allow anonymous authentication:

## Allow Anonymous authentication
mqtt.allow_anonymous = true

# Username/Password

Authenticate MQTT client with Username/Password:

Configure default users in etc/plugins/emq_auth_username.conf:

auth.user.$N.username = admin
auth.user.$N.password = public

Enable emq_auth_username (opens new window) plugin:

./bin/emqttd_ctl plugins load emq_auth_username

Add user by './bin/emqttd_ctl users' command:

$ ./bin/emqttd_ctl users add \<Username> \<Password>

# ClientId

Authentication with MQTT ClientId.

Configure Client Ids in etc/plugins/emq_auth_clientid.conf:

auth.client.$N.clientid = clientid
auth.client.$N.password = passwd

Enable emq_auth_clientid (opens new window) plugin:

./bin/emqttd_ctl plugins load emq_auth_clientid

# LDAP

etc/plugins/emq_auth_ldap.conf:

auth.ldap.servers = 127.0.0.1

auth.ldap.port = 389

auth.ldap.timeout = 30

auth.ldap.user_dn = uid=%u,ou=People,dc=example,dc=com

auth.ldap.ssl = false

Enable LDAP plugin:

./bin/emqttd_ctl plugins load emq_auth_ldap

# HTTP

etc/plugins/emq_auth_http.conf:

## Variables: %u = username, %c = clientid, %a = ipaddress, %P = password, %t = topic

auth.http.auth_req = http://127.0.0.1:8080/mqtt/auth
auth.http.auth_req.method = post
auth.http.auth_req.params = clientid=%c,username=%u,password=%P

auth.http.super_req = http://127.0.0.1:8080/mqtt/superuser
auth.http.super_req.method = post
auth.http.super_req.params = clientid=%c,username=%u

Enable HTTP Plugin:

./bin/emqttd_ctl plugins load emq_auth_http

# MySQL

Authenticate with MySQL database. Suppose that we create a mqtt_user table:

CREATE TABLE `mqtt_user` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `username` varchar(100) DEFAULT NULL,
  `password` varchar(100) DEFAULT NULL,
  `salt` varchar(20) DEFAULT NULL,
  `created` datetime DEFAULT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `mqtt_username` (`username`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

Configure the 'auth_query' and 'password_hash' in etc/plugins/emq_auth_mysql.conf:

## Mysql Server
auth.mysql.server = 127.0.0.1:3306

## Mysql Pool Size
auth.mysql.pool = 8

## Mysql Username
## auth.mysql.username =

## Mysql Password
## auth.mysql.password =

## Mysql Database
auth.mysql.database = mqtt

## Variables: %u = username, %c = clientid

## Authentication Query: select password only
auth.mysql.auth_query = select password from mqtt_user where username = '%u' limit 1

## Password hash: plain, md5, sha, sha256, pbkdf2
auth.mysql.password_hash = sha256

## %% Superuser Query
auth.mysql.super_query = select is_superuser from mqtt_user where username = '%u' limit 1

Enable MySQL plugin:

./bin/emqttd_ctl plugins load emq_auth_mysql

# PostgreSQL

Authenticate with PostgreSQL database. Create a mqtt_user table:

CREATE TABLE mqtt_user (
  id SERIAL primary key,
  username character varying(100),
  password character varying(100),
  salt character varying(40)
);

Configure the 'auth_query' and 'password_hash' in etc/plugins/emq_auth_pgsql.conf:

## Postgre Server
auth.pgsql.server = 127.0.0.1:5432

auth.pgsql.pool = 8

auth.pgsql.username = root

#auth.pgsql.password =

auth.pgsql.database = mqtt

auth.pgsql.encoding = utf8

auth.pgsql.ssl = false

## Variables: %u = username, %c = clientid, %a = ipaddress

## Authentication Query: select password only
auth.pgsql.auth_query = select password from mqtt_user where username = '%u' limit 1

## Password hash: plain, md5, sha, sha256, pbkdf2
auth.pgsql.password_hash = sha256

## sha256 with salt prefix
## auth.pgsql.password_hash = salt sha256

## sha256 with salt suffix
## auth.pgsql.password_hash = sha256 salt

## Superuser Query
auth.pgsql.super_query = select is_superuser from mqtt_user where username = '%u' limit 1

Enable the plugin:

./bin/emqttd_ctl plugins load emq_auth_pgsql

# Redis

Authenticate with Redis. MQTT users could be stored in redis HASH, the key is "mqtt_user:<Username>".

Configure 'auth_cmd' and 'password_hash' in etc/plugins/emq_auth_redis.conf:

## Redis Server
auth.redis.server = 127.0.0.1:6379

## Redis Pool Size
auth.redis.pool = 8

## Redis Database
auth.redis.database = 0

## Redis Password
## auth.redis.password =

## Variables: %u = username, %c = clientid

## Authentication Query Command
auth.redis.auth_cmd = HGET mqtt_user:%u password

## Password hash: plain, md5, sha, sha256, pbkdf2
auth.redis.password_hash = sha256

## Superuser Query Command
auth.redis.super_cmd = HGET mqtt_user:%u is_superuser

Enable the plugin:

./bin/emqttd_ctl plugins load emq_auth_redis

# MongoDB

Create a mqtt_user collection:

{
    username: "user",
    password: "password hash",
    is_superuser: boolean (true, false),
    created: "datetime"
}

Configure super_query , auth_query in etc/plugins/emq_auth_mongo.conf:

## Mongo Server
auth.mongo.server = 127.0.0.1:27017

## Mongo Pool Size
auth.mongo.pool = 8

## Mongo User
## auth.mongo.user =

## Mongo Password
## auth.mongo.password =

## Mongo Database
auth.mongo.database = mqtt

## auth_query
auth.mongo.auth_query.collection = mqtt_user

auth.mongo.auth_query.password_field = password

auth.mongo.auth_query.password_hash = sha256

auth.mongo.auth_query.selector = username=%u

## super_query
auth.mongo.super_query.collection = mqtt_user

auth.mongo.super_query.super_field = is_superuser

auth.mongo.super_query.selector = username=%u

Enable the plugin:

./bin/emqttd_ctl plugins load emq_auth_mongo

# ACL

The ACL of EMQ broker is responsible for authorizing MQTT clients to publish/subscribe topics.

The ACL rules define:

Allow|Deny Who Publish|Subscribe Topics

Access Control Module of EMQ broker will match the rules one by one:

          ---------              ---------              ---------
Client -> | Rule1 | --nomatch--> | Rule2 | --nomatch--> | Rule3 | --> Default
          ---------              ---------              ---------
              |                      |                      |
            match                  match                  match
             \|/                    \|/                    \|/
        allow | deny           allow | deny

# Internal

The default ACL of EMQ broker is implemented by an 'internal' module.

Enable the 'internal' ACL module in etc/emq.conf:

## ACL nomatch
mqtt.acl_nomatch = allow

## Default ACL File
mqtt.acl_file = etc/acl.conf

The ACL rules of 'internal' module are defined in 'etc/acl.conf' file:

%% Allow 'dashboard' to subscribe '$SYS/#'
{allow, {user, "dashboard"}, subscribe, ["$SYS/#"]}.

%% Allow clients from localhost to subscribe any topics
{allow, {ipaddr, "127.0.0.1"}, pubsub, ["$SYS/#", "#"]}.

%% Deny clients to subscribe '$SYS#' and '#'
{deny, all, subscribe, ["$SYS/#", {eq, "#"}]}.

%% Allow all by default
{allow, all}.

# HTTP API

ACL by HTTP API: https://github.com/emqtt/emq_auth_http (opens new window)

Configure etc/plugins/emq_auth_http.conf and enable the plugin:

## 'access' parameter: sub = 1, pub = 2
auth.http.acl_req = http://127.0.0.1:8080/mqtt/acl
auth.http.acl_req.method = get
auth.http.acl_req.params = access=%A,username=%u,clientid=%c,ipaddr=%a,topic=%t

# MySQL

ACL with MySQL database. The mqtt_acl table and default data:

CREATE TABLE `mqtt_acl` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `allow` int(1) DEFAULT NULL COMMENT '0: deny, 1: allow',
  `ipaddr` varchar(60) DEFAULT NULL COMMENT 'IpAddress',
  `username` varchar(100) DEFAULT NULL COMMENT 'Username',
  `clientid` varchar(100) DEFAULT NULL COMMENT 'ClientId',
  `access` int(2) NOT NULL COMMENT '1: subscribe, 2: publish, 3: pubsub',
  `topic` varchar(100) NOT NULL DEFAULT '' COMMENT 'Topic Filter',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

INSERT INTO mqtt_acl (id, allow, ipaddr, username, clientid, access, topic)
VALUES
    (1,1,NULL,'$all',NULL,2,'#'),
    (2,0,NULL,'$all',NULL,1,'$SYS/#'),
    (3,0,NULL,'$all',NULL,1,'eq #'),
    (5,1,'127.0.0.1',NULL,NULL,2,'$SYS/#'),
    (6,1,'127.0.0.1',NULL,NULL,2,'#'),
    (7,1,NULL,'dashboard',NULL,1,'$SYS/#');

Configure 'acl-query' and 'acl_nomatch' in etc/plugins/emq_auth_mysql.conf:

## ACL Query Command
auth.mysql.acl_query = select allow, ipaddr, username, clientid, access, topic from mqtt_acl where ipaddr = '%a' or username = '%u' or username = '$all' or clientid = '%c'

# PostgreSQL

ACL with PostgreSQL database. The mqtt_acl table and default data:

CREATE TABLE mqtt_acl (
  id SERIAL primary key,
  allow integer,
  ipaddr character varying(60),
  username character varying(100),
  clientid character varying(100),
  access  integer,
  topic character varying(100)
);

INSERT INTO mqtt_acl (id, allow, ipaddr, username, clientid, access, topic)
VALUES
    (1,1,NULL,'$all',NULL,2,'#'),
    (2,0,NULL,'$all',NULL,1,'$SYS/#'),
    (3,0,NULL,'$all',NULL,1,'eq #'),
    (5,1,'127.0.0.1',NULL,NULL,2,'$SYS/#'),
    (6,1,'127.0.0.1',NULL,NULL,2,'#'),
    (7,1,NULL,'dashboard',NULL,1,'$SYS/#');

Configure 'acl_query' and 'acl_nomatch' in etc/plugins/emq_auth_pgsql.conf:

## ACL Query. Comment this query, the acl will be disabled.
auth.pgsql.acl_query = select allow, ipaddr, username, clientid, access, topic from mqtt_acl where ipaddr = '%a' or username = '%u' or username = '$all' or clientid = '%c'

# Redis

ACL with Redis. The ACL rules are stored in a Redis HashSet:

HSET mqtt_acl:<username> topic1 1
HSET mqtt_acl:<username> topic2 2
HSET mqtt_acl:<username> topic3 3

Configure acl_cmd and acl_nomatch in etc/plugins/emq_auth_redis.conf:

## ACL Query Command
auth.redis.acl_cmd = HGETALL mqtt_acl:%u

# MongoDB

Store ACL Rules in a mqtt_acl collection:

{
    username: "username",
    clientid: "clientid",
    publish: ["topic1", "topic2", ...],
    subscribe: ["subtop1", "subtop2", ...],
    pubsub: ["topic/#", "topic1", ...]
}

For example, insert rules into mqtt_acl collection:

db.mqtt_acl.insert({username: "test", publish: ["t/1", "t/2"], subscribe: ["user/%u", "client/%c"]})
db.mqtt_acl.insert({username: "admin", pubsub: ["#"]})

Configure acl_query and acl_nomatch in etc/plugins/emq_auth_mongo.conf:

## acl_query
auth.mongo.acl_query.collection = mqtt_user

auth.mongo.acl_query.selector = username=%u

# MQTT Publish/Subscribe

MQTT is a an extremely lightweight publish/subscribe messaging protocol desgined for IoT, M2M and Mobile applications.

image

Install and start the EMQ broker, and then any MQTT client could connect to the broker, subscribe topics and publish messages.

MQTT Client Libraries: https://github.com/mqtt/mqtt.github.io/wiki/libraries (opens new window)

For example, we use mosquitto_sub/pub commands:

mosquitto_sub -t topic -q 2
mosquitto_pub -t topic -q 1 -m "Hello, MQTT!"

MQTT V3.1.1 Protocol Specification: http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqtt-v3.1.1.html (opens new window)

MQTT Listener of the EMQ broker is configured in etc/emq.conf:

## TCP Listener: 1883, 127.0.0.1:1883, ::1:1883
listener.tcp.external = 1883

## Size of acceptor pool
listener.tcp.external.acceptors = 8

## Maximum number of concurrent clients
listener.tcp.external.max_clients = 1024

MQTT(SSL) Listener, Default Port is 8883:

## SSL Listener: 8883, 127.0.0.1:8883, ::1:8883
listener.ssl.external = 8883

## Size of acceptor pool
listener.ssl.external.acceptors = 4

## Maximum number of concurrent clients
listener.ssl.external.max_clients = 512

# HTTP Publish API

The EMQ broker provides a HTTP API to help application servers publish messages to MQTT clients.

HTTP API: POST http://host:8080/mqtt/publish (opens new window)

Web servers such as PHP, Java, Python, NodeJS and Ruby on Rails could use HTTP POST to publish MQTT messages to the broker:

curl -v --basic -u user:passwd -d "qos=1&retain=0&topic=/a/b/c&message=hello from http..." -k http://localhost:8080/mqtt/publish

Parameters of the HTTP API:

NameDescription
clientclientid
qosQoS(0, 1, 2)
retainRetain(0, 1)
topicTopic
messagePayload

Tip

The API uses HTTP Basic Authentication.

The url of this API has been changed to 'api/v2/mqtt/publish' in v2.3-beta.2 release. Read the doc in /rest .

# MQTT Over WebSocket

Web browsers could connect to the emqttd broker directly by MQTT Over WebSocket.

WebSocket URI:ws(s)://host:8083/mqtt
Sec-WebSocket-Protocol:'mqttv3.1' or 'mqttv3.1.1'

The Dashboard plugin provides a test page for WebSocket:

http://127.0.0.1:18083/websocket.html

Listener of WebSocket and HTTP Publish API is configured in etc/emq.config:

## MQTT/WebSocket Listener
listener.ws.external = 8083
listener.ws.external.acceptors = 4
listener.ws.external.max_clients = 64

# $SYS Topics

The EMQ broker periodically publishes internal status, MQTT statistics, metrics and client online/offline status to $SYS/# topics.

For the EMQ broker could be clustered, the $SYS topic path is started with:

$SYS/brokers/${node}/

'${node}' is the erlang node name of emqttd broker. For example:

$SYS/brokers/emqttd@127.0.0.1/version

$SYS/brokers/emqttd@host2/uptime

Tip

The broker only allows clients from localhost to subscribe $SYS topics by default.

Sys Interval of publishing $SYS messages, could be configured in etc/emqttd.config:

## System Interval of publishing broker $SYS Messages
mqtt.broker.sys_interval = 60

# Broker Version, Uptime and Description

TopicDescription
$SYS/brokersBroker nodes
$SYS/brokers/${node}/versionBroker Version
$SYS/brokers/${node}/uptimeBroker Uptime
$SYS/brokers/${node}/datetimeBroker DateTime
$SYS/brokers/${node}/sysdescrBroker Description

# Online/Offline Status of MQTT Client

The topic path started with: $SYS/brokers/${node}/clients/

TopicPayload(JSON)Description
${clientid}/connected{ipaddress: "127.0.0.1", username: "test", session: false, version: 3, connack: 0, ts: 1432648482}Publish when a client connected
${clientid}/disconnected{reason: "keepalive_timeout", username: "test", ts: 1432749431}Publish when a client disconnected

Properties of 'connected' Payload:

ipaddress: "127.0.0.1",
username:  "test",
session:   false,
protocol:  3,
connack:   0,
ts:        1432648482

Properties of 'disconnected' Payload:

reason: normal,
ts:     1432648486

# Broker Statistics

Topic path started with: $SYS/brokers/${node}/stats/

# Clients

TopicDescription
clients/countCount of current connected clients
clients/maxMax number of cocurrent connected clients

# Sessions

TopicDescription
sessions/countCount of current sessions
sessions/maxMax number of sessions

# Subscriptions

TopicDescription
subscriptions/countCount of current subscriptions
subscriptions/maxMax number of subscriptions

# Topics

TopicDescription
topics/countCount of current topics
topics/maxMax number of topics

# Broker Metrics

Topic path started with: $SYS/brokers/${node}/metrics/

# Bytes Sent/Received

TopicDescription
bytes/receivedMQTT Bytes Received since broker started
bytes/sentMQTT Bytes Sent since the broker started

# Packets Sent/Received

TopicDescription
packets/receivedMQTT Packets received
packets/sentMQTT Packets sent
packets/connectMQTT CONNECT Packet received
packets/connackMQTT CONNACK Packet sent
packets/publish/receivedMQTT PUBLISH packets received
packets/publish/sentMQTT PUBLISH packets sent
packets/subscribeMQTT SUBSCRIBE Packets received
packets/subackMQTT SUBACK packets sent
packets/unsubscribeMQTT UNSUBSCRIBE Packets received
packets/unsubackMQTT UNSUBACK Packets sent
packets/pingreqMQTT PINGREQ packets received
packets/pingrespMQTT PINGRESP Packets sent
packets/disconnectMQTT DISCONNECT Packets received

# Messages Sent/Received

TopicDescription
messages/receivedMessages Received
messages/sentMessages Sent
messages/retainedMessages Retained
messages/storedTODO: Messages Stored
messages/droppedMessages Dropped

# Broker Alarms

Topic path started with: $SYS/brokers/${node}/alarms/

TopicDescription
${alarmId}/alertNew Alarm
${alarmId}/clearClear Alarm

# Broker Sysmon

Topic path started with: '$SYS/brokers/${node}/sysmon/'

TopicDescription
long_gcLong GC Warning
long_scheduleLong Schedule
large_heapLarge Heap Warning
busy_portBusy Port Warning
busy_dist_portBusy Dist Port

# Trace

The emqttd broker supports to trace MQTT packets received/sent from/to a client, or trace MQTT messages published to a topic.

Trace a client:

./bin/emqttd_ctl trace client "clientid" "trace_clientid.log"

Trace a topic:

./bin/emqttd_ctl trace topic "topic" "trace_topic.log"

Lookup Traces:

./bin/emqttd_ctl trace list

Stop a Trace:

./bin/emqttd_ctl trace client "clientid" off

./bin/emqttd_ctl trace topic "topic" off