Back

Rapidly deploy EMQ X clusters on Kubernetes via Helm

2019-10-21

EMQ X can be easily deployed to the Kubernetes platform via Helm chart provided by EMQ X. Before you begin, be sure to follow the Helm Documentation Guide to install Helm.

About Kubernetes

Kubernetes is a widely used open source container orchestration platform for automatic deployment, expansion and operation of applications.

With Kubernetes, customer needs can be quickly and efficiently responded:

  • Deploy applications quickly and predictably
  • Ability to instantly scale applications
  • Seamlessly release new functionality without affecting existing business
  • Optimize hardware resources and reduce costs

Kubernetes has the following characteristics:

  • Portability: Full support for public, private, hybrid or multi cloud architectures
  • Scalability: It is modular, pluggable, mountable, combinable, and supports various forms of scaling
  • Self-repairing: It can self-maintain application status, self-restart, self-replicate, self-scale, and provides powerful self-repairing ability through declarative grammar.

The Kubernetes project was launched by Google in 2014. Kubernetes is based on Google's more than 10 years of experience in operations and maintenance, all of its applications run on containers , and combined with the best ideas and practices in the community, it may be the most popular container platform.

About Helm

Helm is a package management tool for kubernetes applications that manages charts - pre-configured package resources, which is a bit similar to APT in Ubuntu and yum in CentOS.

Helm chart is a yaml file used to encapsulate the kubernetes native application. It can customize some metadata of the application when deploying the application, and then distribute it with the application.

The main roles of Helm and Charts are:

  • Application Encapsulation
  • Version management
  • Dependency checking
  • Facilitate application distribution

Setup

  • Kubernetes 1.6+
  • Helm

Rapidly deploy a simple EMQ X cluster

  • Add Helm repository

    $ helm repo add emqx https://repos.emqx.io/charts
    
  • Search EMQ X

    helm search emqx
    NAME      CHART VERSION   APP VERSION     DESCRIPTION
    emqx/emqx v1.0.0-beta.1   v1.0.0-beta.1   A Helm chart for Emqx
    

    EMQ X Chart is currently in Beta, so it is necessary to add --devel when executing the command helm install. It is not necessary to add this option for Chart in the official version, please pay attention to it.

  • Start the EMQ X cluster, set service.type=NodePort, and specify image to be emqx/emqx:v3.2.3

    $ helm install --devel --name my-emqx --set service.type=NodePort --set image=emqx/emqx:v3.2.3 emqx/emqx
    
  • Check EMQ X cluster

    $ kubectl get pods
    NAME       READY  STATUS             RESTARTS  AGE
    my-emqx-0  1/1    Running            0         81m
    my-emqx-1  1/1    Running            0         81m
    my-emqx-2  1/1    Running            0         80m
    
    $ kubectl exec -it my-emqx-0 -- emqx_ctl cluster status
    Cluster status: [{running_nodes,['my-emqx@192.168.109.82',
                                   'my-emqx@192.168.71.78',
                                   'my-emqx@192.168.95.126']}]
    
  • Check EMQ X service

    $ kubectl get svc
    NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                                      AGE
    kubernetes           ClusterIP   10.100.0.1      <none>        443/TCP                                                                                      7h26m
    my-emqx              NodePort    10.100.213.62   <none>        1883:30068/TCP,8883:32463/TCP,8080:31854/TCP,8083:32057/TCP,8084:31459/TCP,18083:31539/TCP   57s
    my-emqx-headless     ClusterIP   None            <none>        1883/TCP,8883/TCP,8080/TCP,8083/TCP,8084/TCP,18083/TCP                                       57s
    

    It can be seen that the host IP corresponding to my-emqx port 18083 is 31539. (NodePort changes at each deployment, depending on the actual deployment.)

  • Access port 31539 of any node IP of Kubernetes, enter the default username: admin, default password: public, and log in to EMQ X Dashboard.

  • Delete EMQ X cluster.

    $ helm del --purge my-emqx
    

Deploy a persistent EMQ X cluster

EMQ X implements persistent pods by creating a PVC resource mount /opt/emqx/data/mnesia directory. Before deploying EMQ X, users need to deploy a load balancer such as Haproxy or Nginx-PLUS, and create PV resources or Storage Classes resources in Kubernetes by themselves.

  • Add Helm repository

    $ helm repo add emqx https://repos.emqx.io/charts
    
  • Start the EMQ X cluster

    Persistence EMQ X clusters require the following settings:

    image=emqx/emqx:v3.2.3
    emqxAddressType=hostname
    persistence.enabled=true
    

    **Note: **Persistent EMQ X only supports versions above v3.2.1

    • If the user has deployed PV resources, set persistence.existingClaim=your_pv_name
    $ helm install --devel --name my-emqx --set image=emqx/emqx:v3.2.3 --set emqxAddressType=hostname --set persistence.enabled=true --set persistence.existingClaim=your_pv_name  emqx/emqx
    
    • If the user has deployed Storage Classes resources, set persistence.storageClass=your_storageClass_name`
    $ helm install --devel --name my-emqx --set image=emqx/emqx:v3.2.3 --set emqxAddressType=hostname --set persistence.enabled=true --set persistence.storageClass=your_storageClass_name emqx/emqx
    
  • Check EMQ X cluster

    $ kubectl get pods
    NAME       READY  STATUS             RESTARTS  AGE
    my-emqx-0  1/1    Running            0         81m
    my-emqx-1  1/1    Running            0         81m
    my-emqx-2  1/1    Running            0         80m
    
    $ kubectl exec -it my-emqx-0 -- emqx_ctl cluster status
    Cluster status: [{running_nodes,['my-emqx@my-emqx-2.my-emqx-headless.default.svc.cluster.local',
                                   'my-emqx@my-emqx-1.my-emqx-headless.default.svc.cluster.local',
                                   'my-emqx@my-emqx-0.my-emqx-headless.default.svc.cluster.local']}]
    
  • Taking Storage Classes as an example, it can be seen that the PVC resource has been successfully established.

    $ kubectl get pvc
    NAME                  STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    emqx-data-my-emqx-0   Bound     pvc-8094cd75-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            2m11s
    emqx-data-my-emqx-1   Bound     pvc-9325441d-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            99s
    emqx-data-my-emqx-2   Bound     pvc-ad425e9d-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            56s
    

    The cluster will mount the /opt/emqx/data/mnesia directory of EMQ X to the PVC. When the Pods are rescheduled, EMQ X will retrieve the data from the /opt/emqx/data/mnesia directory and restore it.

  • Check ClusterIP of EMQ X

    $ kubectl get svc
    NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                  AGE
    kubernetes           ClusterIP   10.100.0.1      <none>        443/TCP                                                  24h
    my-emqx              ClusterIP   10.100.205.13   <none>        1883/TCP,8883/TCP,8080/TCP,8083/TCP,8084/TCP,18083/TCP   26m
    my-emqx-headless     ClusterIP   None            <none>        1883/TCP,8883/TCP,8080/TCP,8083/TCP,8084/TCP,18083/TCP   26m
    

    It can be seen that the clusterIP of my-emqx is 10.100.205.13 (Cluster IP changes at each deployment, depending on the actual deployment.)

  • Forward the ports 1883, 8883, 8080, 8083, 8084, 18083 of the load balancing listener to the ClusterIP of my-emqx. If there is a need for a TLS connection, it is recommended to terminate the SSL connection in the load balancer, with a TLS secure connection between the client and the load balancer, and a normal TCP connection between LB and EMQ X.

  • Access URL:18083, enter the default username: admin, default password: public, and log in to EMQ X dashboard.

  • Easily scale the EMQ X cluster with the helm upgrade command. The following shows the helm upgrade command with the addition of the EMQ X node.

    # Change the number of nodes in EMQ X to 5
    # Note: The number of nodes in EMQ X is recommended to be singular
    $ helm upgrade --set replicas=5 my-emqx emqx/emqx
    
    $ kubectl get pods
    NAME       READY  STATUS             RESTARTS  AGE
    my-emqx-0  1/1    Running            0         81m
    my-emqx-1  1/1    Running            0         81m
    my-emqx-2  1/1    Running            0         80m
    my-emqx-3  1/1    Running            0         22m
    my-emqx-4  1/1    Running            0         22m
    
    $ kubectl exec -it my-emqx-0 -- emqx_ctl cluster status
    Cluster status: [{running_nodes,['my-emqx@my-emqx-4.my-emqx-headless.default.svc.cluster.local',
                                   'my-emqx@my-emqx-3.my-emqx-headless.default.svc.cluster.local',
                                   'my-emqx@my-emqx-2.my-emqx-headless.default.svc.cluster.local',
                                   'my-emqx@my-emqx-1.my-emqx-headless.default.svc.cluster.local',
                                   'my-emqx@my-emqx-0.my-emqx-headless.default.svc.cluster.local']}]
    
  • Delete EMQ X cluster

    $ helm del --purge my-emqx
    

    Note: After the EMQ X cluster is deleted, the PVC resources will not be automatically released to restore EMQ X. It is necessary to manually delete PVC resources after confirmation that no recovery is required.

    $ kubectl get pvc
    NAME                  STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    emqx-data-my-emqx-0   Bound     pvc-8094cd75-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            84m
    emqx-data-my-emqx-1   Bound     pvc-9325441d-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            84m
    emqx-data-my-emqx-2   Bound     pvc-ad425e9d-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            83m
    emqx-data-my-emqx-3   Bound     pvc-b6c5a565-adbd-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            25m
    emqx-data-my-emqx-4   Bound     pvc-c626cafd-adbd-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            25m
    
    $ kubectl delete pvc emqx-data-my-emqx-0 emqx-data-my-emqx-1 emqx-data-my-emqx-2 emqx-data-my-emqx-3 emqx-data-my-emqx-4                    
    persistentvolumeclaim "emqx-data-my-emqx-0" deleted
    persistentvolumeclaim "emqx-data-my-emqx-1" deleted
    persistentvolumeclaim "emqx-data-my-emqx-2" deleted
    persistentvolumeclaim "emqx-data-my-emqx-3" deleted
    persistentvolumeclaim "emqx-data-my-emqx-4" deleted
    

Deploy the EMQ X Edge cluster and EMQ X enterprise edition cluster

EMQ X Edge

When deploying an EMQ x edge cluster, specify image=emqx/emqx-edge:v3.2.3. Other settings are consistent with the deployment of an EMQ x cluster.

$ helm install --devel --name my-emqx --set service.type=NodePort --set image=emqx/emqx-edge:v3.2.3 emqx/emqx

EMQ X EE

To deploy an EMQ X Enterprise edition cluster, you first need to log in emqx.io to apply for and download the license file and create the license file as a Secret resource.

$ kubectl create secret generic your-license-secret-name --from-file=/path/to/emqx.lic

Then specify emqxLicneseSecretName=your-license-secret-name, image=emqx/emqx-ee:v3.4.0 at deployment, and other settings are consistent with deploying EMQ X clusters

helm install --devel --name my-emqx --set service.type=NodePort --set image=emqx/emqx-ee:v3.2.3 --set emqxLicneseSecretName=your-license-secret-name emqx/emqx

EMQ X Helm Chart Configuration item

Parameter Description Default Value
apiserver Kubernates API server https://kubernetes.default.svc:443
namespace kubernetes namespace default
replicas It is recommended to keep EMQ X number of nodes an odd number, otherwise it will not recover automatically after brain splitting 3
image EMQ X image name emqx/emqx:latest
imagePullPolicy Image pull policy IfNotPresent
persistence.enabled whether to enable PVC false
persistence.storageClass Storage class name nil
persistence.existingClaim PV name ""
persistence.accessMode PVC access mode ReadWriteOnce
persistence.size PVC size 20Mi
resources CPU/Memory resource {}
service.type Emqx Cluster service type ClusterIP
service.mqttPort MQTT Protocol port 1883
service.mqttsslPort MQTT/SSL Port 8883
service.mgmtPort Manage API port 8080
service.websocketPort MQTT/WebSocket Port 8083
service.wssPort MQTT/WebSocket/SSL Port 8084
service.dashboardPort Dashboard Port 18083
emqxAddressType Used to get EMQ X nodes from k8s service
Optional values: ip, dns, hostname
Note: hosename only supports versions above EMQ X v3.2.1
ip
emqxConfig EMQ X configuration item,check documentation for detail {}
emqxLicneseSecretName EMQ X Enterprise Edition requires manual creation of a license file as a Secret resource ""

When complex parameters need to be set, Yaml files can be used to record parameters.

$ helm install --devel --name my-emqx -f values.yaml emqx/emqx

You can get the default values.yaml from Github


Welcome to our open source project github.com/emqx/emqx. Please visit the documentation for details.