# Create and manage clusters
EMQX supports creating clusters manually and automatically. This chapter will guide you through creating and managing EMQX clusters.
Prerequisites:
- Knowledge of Distributed Clusters.
- Knowledge of Deployment Architecture and Cluster Requirements.
# Preparations before clustering
- All nodes are set with a unique node name in the format of
name@host
, where host must be an IP address or fully qualified domain name (FQDN). - If there is a firewall or security group between nodes, ensure the cluster communication port has been opened. For details, see Intra-cluster communication port.
- All nodes use the same cookie.
TIP
EMQX node names are immutable, as they are baked into the database schema and data files. It is strongly recommended to use static FQDNs for EMQX node names, even when the nodes have static IPs.
# Create clusters manually
If you choose a manually create a cluster, you will need to manually configure each node in the cluster, including setting up network connections among them.
Compared with automatic clustering, you can customize the network topology with manual clustering. It is especially suitable when the automatic clustering mechanism is unavailable or inappropriate.
Suppose there are two nodes, emqx@s1.emqx.io
and emqx@s2.emqx.io
, and we can follow the steps below to create a cluster for these two nodes manually:
Set the cluster discovery strategy to
manual
:cluster { ## Options: manual | static | mcast | dns | etcd | K8s discovery_strategy = manual }
1
2
3
4After the two nodes are started, run the join cluster command on one node:
$ ./bin/emqx_ctl cluster join emqx@s1.emqx.io Join the cluster successfully. Cluster status: [{running_nodes,['emqx@s1.emqx.io','emqx@s2.emqx.io']}]
1
2
3
4TIP
- This command must be run on the node to join the cluster, that is, as a request rather than invite.
- After
emqx@s2.emqx.io
joinsemqx@s1.emqx.io
to form a cluster, it will clear the local data and synchronize the data inemqx@s1.emqx.io
. - When a node that already joined a cluster joins another cluster, it should first leave the current cluster.
Query the cluster status on any node:
$ ./bin/emqx_ctl cluster status Cluster status: [{running_nodes,['emqx@s1.emqx.io','emqx@s2.emqx.io']}]
1
2
3You can let a node leave the cluster with leave or force-leave:
- leave: let the node leave the cluster
- force-leave: remove the node from the cluster
Run the command below on
emqx@s2.emqx.io
to let it leave the cluster:./bin/emqx_ctl cluster leave
1Or run the command below on
emqx@s1.emqx.io
to removeemqx@s2.emqx.io
from the cluster:./bin/emqx_ctl cluster force-leave emqx@s2.emqx.io
1
# Pseudo-distributed cluster
For users with only one server, you can use the pseudo-distributed cluster to test the EMQX cluster.
After starting the first node, use the following command to start the second node and join the cluster manually. To avoid port conflicts, we need to adjust some listening ports:
EMQX_NODE__NAME='emqx2@127.0.0.1' \
EMQX_STATSD__SERVER='127.0.0.1:8124' \
EMQX_LISTENERS__TCP__DEFAULT__BIND='0.0.0.0:1882' \
EMQX_LISTENERS__SSL__DEFAULT__BIND='0.0.0.0:8882' \
EMQX_LISTENERS__WS__DEFAULT__BIND='0.0.0.0:8082' \
EMQX_LISTENERS__WSS__DEFAULT__BIND='0.0.0.0:8085' \
EMQX_DASHBOARD__LISTENERS__HTTP__BIND='0.0.0.0:18082' \
EMQX_NODE__DATA_DIR="./data2" \
./bin/emqx start
./bin/emqx_ctl cluster join emqx1@127.0.0.1
2
3
4
5
6
7
8
9
10
11
# Node discovery and auto clustering
Node discovery allows individual EMQX node with different IPs or locations to be discovered and communicated with each other, and it is a crucial step when creating EMQX clusters.
EMQX's autocluster feature is based on the Ekka (opens new window) library, the cluster management library developed for Erlang/OTP applications, supporting features like Erlang node service discovery, autocluster, network partition autoheal, and autoclean.
EMQX supports multiple autocluster strategies:
Strategy | Description |
---|---|
manual | Create a cluster through manual command |
static | Create a cluster using a static node list |
dns | Create a cluster using DNS A records |
etcd | Create a cluster via etcd |
k8s | Create a cluster via Kubernetes service |
By default, EMQX adopts a manual clustering strategy, which can be set in emqx.conf
:
cluster {
## options: manual | static | mcast | dns | etcd | K8s
discovery_strategy = manual
}
2
3
4
Note: The mcast discovery strategy has been deprecated and will be removed in future releases.
# Autocluster by static node list
The static clustering of EMQX is to use a static node list pre-configured on each node to join the cluster. After starting, the nodes will create a cluster automatically according to the node list.
Static clustering is the easiest way to create an EMQX cluster automatically with no dependencies on other network components or services. As long as each node can communicate with each other through the TCP protocol, they can form an EMQX cluster.
Configure the same cluster mode and node list in emqx.conf
:
cluster {
discovery_strategy = static
static {
seeds = ["emqx@s1.emqx.io", "emqx@s2.emqx.io"]
}
}
2
3
4
5
6
Start all nodes one by one, and the cluster will be automatically established.
# Autocluster by DNS Record
# Working principles
DNS Stands for Domain Name System. A DNS server returns the IP addresses after receiving a domain query, that is, the A records. DNS system allows multiple A records for one domain name, this makes a one-to-many mapping.
EMQX can use this one-to-many mapping to find all nodes that belong to one cluster, so that the nodes can join a cluster automatically.
Configuration: Most public cloud services have DNS services. After assigning a domain name, you only need to add the IP address of each EMQX node to the A record of this domain to finish the configuration.
TIP
If EMQX is deployed in a private cloud or internal network, you will need to deploy your own DNS system, for example, with software BIND (opens new window).
After the preparation, we will configure in emqx.conf
to let all nodes join the cluster using the same clustering method, dns
.
EMQX supports DNS automatic clustering using DNS A record and DNS SRV record:
Auto clustering using DNS A record:
cluster {
discovery_strategy = dns
dns {
name = "localhost"
record_type = a
}
}
2
3
4
5
6
7
Auto clustering using DNS SRV record:
cluster {
discovery_strategy = dns
dns {
name = "localhost"
record_type = srv
}
}
2
3
4
5
6
7
Start all nodes one by one after the configuration, and the cluster will be automatically established.
# Autocluster using etcd
etcd (opens new window) is an open-source project initiated by CoreOS. It is widely used in distributed systems for service discovery and connection establishing, which is exactly what EMQX autoclustering needs.
If you have an etcd server(cluster) in your network, EMQX can automatically create the cluster via etcd. For how to install and configure etcd, see etcd Install (opens new window).
Configuration: You need to specify the address of the etcd server, multiple etcd servers can be separated with ,
; also the directory prefix and TTL used to specify the EMQX node.
Code example:
cluster {
discovery_strategy = etcd
etcd {
server = "http://127.0.0.1:2379"
prefix = emqxcl
node_ttl = 1m
}
}
2
3
4
5
6
7
8
After completing the configuration, we can start the EMQX nodes one by one, and use the etcdctl tool to observe the changes on the etcd server:
$ etcdctl ls /emqxcl/emqxcl --recursive
/emqxcl/emqxcl/nodes
/emqxcl/emqxcl/nodes/emqx@s1.emqx.io
/emqxcl/emqxcl/nodes/emqx@s2.emqx.io
2
3
4
5
The result shows that all nodes are started normally and joined the cluster automatically.
# Autocluster on Kubernetes
Kubernetes (K8s) (opens new window) is Google's open source container management system. EMQ X can use kubernetes API for node discovery and auto clustering.
Configuration: You need to specify the Kubernetes API server for all nodes, the service name of EMQX on K8s, and the address type:
cluster {
discovery_strategy = K8s
K8s {
apiserver = "http://10.110.111.204:8080"
service_name = emqx
address_type = ip
app_name = emqx
suffix = "pod.local"
namespace = default
}
}
2
3
4
5
6
7
8
9
10
11
Start all nodes one by one after the configuration, and the cluster will be automatically established.
TIP
Calico rather Fannel plugin is recommended.