# Apache Kafka

Apache Kafka is a widely-used open-source distributed event streaming platform. EMQX's integration with Apache Kafka/Confluent presents our users with reliable bi-directional data transport and processing capability under high-throughput scenarios.

Being a top IOT data infrastructure provider, EMQX currently supports authenticating with Apache Kafka/Confluent via SASL/SCRAM, SASL/GSSAPI, or TLS.

Prerequisites

  • Knowledge about EMQX data integration rules
  • Knowledge about data bridge
  • Relevant Kafka topics should be created before creating the data bridge in Producer or Consumer mode.

# Feature list

# Quick starts

# Install Kafka

This section takes macOS as an example to illustrate the process. You can install and run Kafka with the commands below:

wget https://archive.apache.org/dist/kafka/3.3.1/kafka_2.13-3.3.1.tgz

tar -xzf  kafka_2.13-3.3.1.tgz

cd kafka_2.13-3.3.1

# Use KRaft to run Kafka (optional)
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"

bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties

bin/kafka-server-start.sh config/kraft/server.properties
1
2
3
4
5
6
7
8
9
10
11
12

For detailed operation steps, you may refer to the Quick Start section in Kafka Documentation (opens new window).

# Create Kafka topics

Create two topics in Kafka: testtopic-in and testtopic-out

bin/kafka-topics.sh --create --topic testtopic-in --bootstrap-server localhost:9092

bin/kafka-topics.sh --create --topic testtopic-out --bootstrap-server localhost:9092
1
2
3

TIP

These topics must be created before we create the data bridge to Kafka.

# Create a data bridge to Kafka

  1. Go to EMQX Dashboard, click Data Integration -> Data Bridge.

  2. Click Create on the top right corner of the page.

  3. In the Create Data Bridge page, click to select Kafka, and then click Next.

  4. Input a name for the data bridge. Note: It should be a combination of upper/lower case letters or numbers.

  5. Input the connection information. Input 127.0.0.1:9092 for the Bootstrap Hosts. For the other fields set as the actual condition.

  6. Configure the data bridge in Producer mode.

    1. Topic: The MQTT topics to create the data bridge for. Here we will input t/#, indicating all MQTT messages matching this topic will be sent to Kafka. You can also leave it blank, and create a rule to specify data to be sent to Kafka.

    2. Kafka Topic Name: Input the Kafka topics we created before, that is, the testtopic-in. Note: Variables are not supported here.

    3. Kafka Message Template: Template will specify the rule or MQTT topics with messages to be sent to the Kafka topic. You can keep the default setting or use variables to create the template.

  7. Advanced settings (optional): Set the Max Batch Bytes, Compression, and Partition Strategy as your business needs. For details, see Configuration parameters.

TIP

Currently EMQX only supports creating data bridges to Kafka in Producer mode.

# Test

Use MQTTX to send messages to topic t/1:

mqttx pub -i emqx_c -t t/1 -m '{ "msg": "Hello Kafka" }'
1

Check the running status of the two data bridges, there should be one new incoming and one new outgoing message.

Check whether messages are written into the topic testtopic-in with the following Kafka command:

bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092  --topic testtopic-in --from-beginning
1