Documentation

# Create and manage clusters

EMQX supports creating clusters manually and automatically. This chapter will guide you through creating and managing EMQX clusters.

Prerequisites:

# Preparations before clustering

  1. All nodes are set with a unique node name in the format of name@host, where host must be an IP address or fully qualified domain name (FQDN).
  2. If there is a firewall or security group between nodes, ensure the cluster communication port has been opened. For details, see Intra-cluster communication port.
  3. All nodes use the same cookie.

TIP

EMQX node names are immutable, as they are baked into the database schema and data files. It is strongly recommended to use static FQDNs for EMQX node names, even when the nodes have static IPs.

# Create clusters manually

If you choose a manually create a cluster, you will need to manually configure each node in the cluster, including setting up network connections among them.

Compared with automatic clustering, you can customize the network topology with manual clustering. It is especially suitable when the automatic clustering mechanism is unavailable or inappropriate.

Suppose there are two nodes, emqx@s1.emqx.io and emqx@s2.emqx.io, and we can follow the steps below to create a cluster for these two nodes manually:

  1. Set the cluster discovery strategy to manual:

    cluster {
    ## Options: manual | static | mcast | dns | etcd | K8s
    discovery_strategy  =  manual
}
    1
    2
    3
    4

  2. After the two nodes are started, run the join cluster command on one node:

    $ ./bin/emqx_ctl cluster join emqx@s1.emqx.io

Join the cluster successfully.
Cluster status: [{running_nodes,['emqx@s1.emqx.io','emqx@s2.emqx.io']}]
    1
    2
    3
    4

    TIP

    1. This command must be run on the node to join the cluster, that is, as a request rather than invite.
    2. After emqx@s2.emqx.io joins emqx@s1.emqx.io to form a cluster, it will clear the local data and synchronize the data in emqx@s1.emqx.io.
    3. When a node that already joined a cluster joins another cluster, it should first leave the current cluster.

  3. Query the cluster status on any node:

    $ ./bin/emqx_ctl cluster status

Cluster status: [{running_nodes,['emqx@s1.emqx.io','emqx@s2.emqx.io']}]
    1
    2
    3

  4. You can let a node leave the cluster with leave or force-leave:

    1. leave: let the node leave the cluster
    2. force-leave: remove the node from the cluster

    Run the command below on emqx@s2.emqx.io to let it leave the cluster:

    ./bin/emqx_ctl cluster leave
    1

    Or run the command below on emqx@s1.emqx.io to remove emqx@s2.emqx.io from the cluster:

    ./bin/emqx_ctl cluster force-leave emqx@s2.emqx.io
    1

# Pseudo-distributed cluster

For users with only one server, you can use the pseudo-distributed cluster to test the EMQX cluster.

After starting the first node, use the following command to start the second node and join the cluster manually. To avoid port conflicts, we need to adjust some listening ports:

EMQX_NODE__NAME='emqx2@127.0.0.1' \
    EMQX_STATSD__SERVER='127.0.0.1:8124' \
    EMQX_LISTENERS__TCP__DEFAULT__BIND='0.0.0.0:1882' \
    EMQX_LISTENERS__SSL__DEFAULT__BIND='0.0.0.0:8882' \
    EMQX_LISTENERS__WS__DEFAULT__BIND='0.0.0.0:8082' \
    EMQX_LISTENERS__WSS__DEFAULT__BIND='0.0.0.0:8085' \
    EMQX_DASHBOARD__LISTENERS__HTTP__BIND='0.0.0.0:18082' \
    EMQX_NODE__DATA_DIR="./data2" \
./bin/emqx start

./bin/emqx_ctl cluster join emqx1@127.0.0.1
1
2
3
4
5
6
7
8
9
10
11

# Node discovery and auto clustering

Node discovery allows individual EMQX node with different IPs or locations to be discovered and communicated with each other, and it is a crucial step when creating EMQX clusters.

EMQX's autocluster feature is based on the Ekka (opens new window) library, the cluster management library developed for Erlang/OTP applications, supporting features like Erlang node service discovery, autocluster, network partition autoheal, and autoclean.

EMQX supports multiple autocluster strategies:

StrategyDescription
manualCreate a cluster through manual command
staticCreate a cluster using a static node list
dnsCreate a cluster using DNS A records
etcdCreate a cluster via etcd
k8sCreate a cluster via Kubernetes service

By default, EMQX adopts a manual clustering strategy, which can be set in emqx.conf:

cluster {
    ## options: manual | static | mcast | dns | etcd | K8s
    discovery_strategy  =  manual
}
1
2
3
4

Note: The mcast discovery strategy has been deprecated and will be removed in future releases.

# Autocluster by static node list

The static clustering of EMQX is to use a static node list pre-configured on each node to join the cluster. After starting, the nodes will create a cluster automatically according to the node list.

Static clustering is the easiest way to create an EMQX cluster automatically with no dependencies on other network components or services. As long as each node can communicate with each other through the TCP protocol, they can form an EMQX cluster.

Configure the same cluster mode and node list in emqx.conf:

cluster {
    discovery_strategy = static
    static {
        seeds = ["emqx@s1.emqx.io", "emqx@s2.emqx.io"]
    }
}
1
2
3
4
5
6

Start all nodes one by one, and the cluster will be automatically established.

# Autocluster by DNS Record

# Working principles

DNS Stands for Domain Name System. A DNS server returns the IP addresses after receiving a domain query, that is, the A records. DNS system allows multiple A records for one domain name, this makes a one-to-many mapping.

EMQX can use this one-to-many mapping to find all nodes that belong to one cluster, so that the nodes can join a cluster automatically.

Configuration: Most public cloud services have DNS services. After assigning a domain name, you only need to add the IP address of each EMQX node to the A record of this domain to finish the configuration.

TIP

If EMQX is deployed in a private cloud or internal network, you will need to deploy your own DNS system, for example, with software BIND (opens new window).

After the preparation, we will configure in emqx.conf to let all nodes join the cluster using the same clustering method, dns.

EMQX supports DNS automatic clustering using DNS A record and DNS SRV record:

Auto clustering using DNS A record:

cluster {
    discovery_strategy = dns
    dns {
        name = "localhost"
        record_type = a
    }
}
1
2
3
4
5
6
7

Auto clustering using DNS SRV record:

cluster {
    discovery_strategy = dns
    dns {
        name = "localhost"
        record_type = srv
    }
}
1
2
3
4
5
6
7

Start all nodes one by one after the configuration, and the cluster will be automatically established.

# Autocluster using etcd

etcd (opens new window) is an open-source project initiated by CoreOS. It is widely used in distributed systems for service discovery and connection establishing, which is exactly what EMQX autoclustering needs.

If you have an etcd server(cluster) in your network, EMQX can automatically create the cluster via etcd. For how to install and configure etcd, see etcd Install (opens new window).

Configuration: You need to specify the address of the etcd server, multiple etcd servers can be separated with ,; also the directory prefix and TTL used to specify the EMQX node.

Code example：

cluster {
    discovery_strategy = etcd
    etcd {
        server = "http://127.0.0.1:2379"
        prefix = emqxcl
        node_ttl = 1m
    }
}
1
2
3
4
5
6
7
8

After completing the configuration, we can start the EMQX nodes one by one, and use the etcdctl tool to observe the changes on the etcd server:

$ etcdctl ls /emqxcl/emqxcl --recursive

/emqxcl/emqxcl/nodes
/emqxcl/emqxcl/nodes/emqx@s1.emqx.io
/emqxcl/emqxcl/nodes/emqx@s2.emqx.io
1
2
3
4
5

The result shows that all nodes are started normally and joined the cluster automatically.

# Autocluster on Kubernetes

Kubernetes (K8s) (opens new window) is Google's open source container management system. EMQ X can use kubernetes API for node discovery and auto clustering.

Configuration: You need to specify the Kubernetes API server for all nodes, the service name of EMQX on K8s, and the address type:

cluster {
    discovery_strategy = K8s
    K8s {
        apiserver = "http://10.110.111.204:8080"
        service_name = emqx
        address_type = ip
        app_name = emqx
        suffix = "pod.local"
        namespace = default
    }
}
1
2
3
4
5
6
7
8
9
10
11

Start all nodes one by one after the configuration, and the cluster will be automatically established.

TIP

Calico rather Fannel plugin is recommended.

← Cluster Security →

Edit this page
What’s on this page