EMQX/K8S: can't log in to dashboard

EMQX
1

Hi All,
I tried to create Deployment on k8s by yml file as the below:
apiVersion: v1
kind: Service
metadata:
name: emqx
namespace: backend
spec:
ports:

  • name: mqtt
    nodePort: 31883
    port: 1883
    protocol: TCP
    tartgetPort: 1883
  • name: mqttssl
    nodePort: 30883
    port: 8883
    protocol: TCP
    tartgetPort: 8883
  • name: mgmt
    nodePort: 30081
    port: 8081
    protocol: TCP
    tartgetPort: 8081
  • name: ws
    nodePort: 30083
    port: 8083
    protocol: TCP
    tartgetPort: 8083
  • name: wss
    nodePort: 30084
    port: 8084
    protocol: TCP
    tartgetPort: 8084
  • name: dashboard
    nodePort: 31083
    port: 18083
    protocol: TCP
    tartgetPort: 18083
    selector:
    workload.user.cattle.io/workloadselector: emqx
    type: NodePort
    status:
    loadBalancer: {}

apiVersion: apps/v1
kind: Deployment
metadata:
name: emqx
namespace: backend
spec:
replicas: 2
selector:
matchLabels:
workload.user.cattle.io/workloadselector: emqx
template:
metadata:
labels:
workload.user.cattle.io/workloadselector: emqx
spec:
imagePullSecrets:

  • name: public
    containers:
  • image: emqx/emqx
    imagePullPolicy: Always
    name: emqx
    ports:
  • containerPort: 1883
    name: mqtt
    protocol: TCP
  • containerPort: 8883
    name: mqttssl
    protocol: TCP
  • containerPort: 8081
    name: mgmt
    protocol: TCP
  • containerPort: 8083
    name: ws
    protocol: TCP
  • containerPort: 8084
    name: wss
    protocol: TCP
  • containerPort: 18083
    name: dashboard
    protocol: TCP
    resources: {}
    restartPolicy: Always
    env:
  • name: EMQX_NAME
    value: emqx
  • name: EMQX_CLUSTER__DISCOVERY
    value: k8s
  • name: EMQX_CLUSTER__K8S__APP_NAME
    value: emqx
  • name: EMQX_CLUSTER__K8S__SERVICE_NAME
    value: emqx
  • name: EMQX_CLUSTER__K8S__APISERVER
    value: “https://kubernetes.default.svc:443”
  • name: EMQX_CLUSTER__K8S__NAMESPACE
    value: backend
    When the replicas is 1, I can access to dashboard normally.
    But when I scale replicas to 2 or 3, it had the message “Login has expired, please login again” as the below:
    image
    Please support me this issue.
    Thank you very much.
2

Sorry for keep you waiting. As the error suggests, the token has expired and a new login is required. Can you log back in?

3

Hi maverick.
I already tried many times.
this message still have when I input account and press “Login” button.

4

Can you try EMQX Operator to deploy? Docs: Deploy EMQX K8S Operator.

5

My system is using rancher. So I deployed by yml file with content as the above.

6

Can you run emqx ctl cluster status and provide its result?

7

Hello,
It looks like your nodes didn’t join together into a cluster, and run as three stand-alone nodes. This is likely due to service discovery settings being wrong. Can you double-check that K8S apiserver url is correct?

8

Hi dmif, Maverick
I already checked K8S apiserver url.
It’s right.
:frowning:
This is result on cli.

image
image1264×140 6.84 KB

Do you have more cli that I can check?

9

Hello,

Could you please run the command that Maverick has provided on one of the pods: emqx_ctl cluster status and share the results?

10

Hi dmif,
I run this cmd on my node 3 (IP: 10.202.1.139 - this node has message error “Login has expired”).

image
image808×42 2.47 KB

I already tried access by IP cluster 10.42.3.110:31083 as my yml file. But can’t access it.
image
image1129×651 15.6 KB

11

hi All,
I tried create emqx cluster by helm. But when I run the cli $ helm install my-emqx emqx/emqx --set service.type=NodePort
has error as the below:
Error: unable to build kubernetes objects from release manifest: error validating “”: error validating data: ValidationError(StatefulSet.spec.template.spec.securityContext): unknown field “fsGroupChangePolicy” in io.k8s.api.core.v1.PodSecurityContext
kubectl version is v1.17
I can’t upgrade it because have many service is running on it.
Please help me.

12

So the screenshot of the emqx_ctl cluster status command confirms that the nodes didn’t form the cluster and operate one their own.
This is why you see “login is expired” errors when trying to use the dashboard: you’ve logged in on one of the nodes, but the other one isn’t aware of that.
Failure to cluster likely happens because the backplane ports are closed. EMQX uses ports 5369 and 4370 for broker-to-broker communication. These ports should be open. Unfortunately, I can’t tell how to do it in K8S.

13

Hi @dmif,
I tried the emqx image with emqx/emqx-ee. This version is running cluster normally.
But it need license for devices

image
image1193×622 25.1 KB
.
I saw EMQX free version has 100k connections.
When I change to others image like: emqx-lastest or rc, cluster can’t run normally. Just see one pod when I run emqx_ctl cluster status.
Do you know this issue?

14

Which exact versions of emqx/emqx-ee and emqx/emqx containers did you run? I’d like to see the version (for example, e4.4.1 or v5.0.11)

P.S. EMQX open source edition has no limitations on the number of connections, so it can go beyond 100k.

15

Hi dmif,
I saw this version is EMQX Enterprise 4.3.13.
I would like to use the open source edition can run cluster on k8s, connect to Mongo DB and has more 200 connections.
But I don’t know what version can do it.