GCP Pub/Sub Bridge

EMQX GCP Pub/Sub integration can send MQTT messages and events to Google Cloud Pub/Sub (opens new window), which can help you flexibly choose services on Google Cloud and build IoT applications more easily.

TIP

EMQX Enterprise Edition features. EMQX Enterprise Edition provides comprehensive coverage of key business scenarios, rich data integration, product-level reliability, and 24/7 global technical support. Experience the benefits of this enterprise-ready MQTT messaging platform (opens new window) today.

The following steps will guide you to use this configuration.

Setup

A few configuration steps on GCP's side must be performed before configuring the GCP Pub/Sub Bridge on EMQX.

1. Create a Service Account (opens new window) in your GCP account. Ensure that the Service Account has permissions to at least publish messages to the topic of interest.
2. Create a Service Account Key for that account and download it in JSON format.
3. Create a Pub/Sub topic (remember that the Service Account must have permissions to publish to that topic).

Configure GCP Pub/Sub Bridge via Dashboard

Go to the EMQX Dashboard, select the "Data Integration" item on the menu on the left, then "Data Bridges". Then, click "+ Create".

In the dialog, select the Google Pub/Sub bridge type, click next. Fill the required fields for that bridge (those are marked with an asterisk). The payload template field, if left blank, will encode all visible inputs from the MQTT message using JSON format, such as clientid, topic, payload, etc. Otherwise, it'll use the user defined template, filling placeholders of the form ${variable_name} with the corresponding value from the MQTT context. For example, ${topic} will be replaced with my/topic if such is the MQTT message topic.

In the GCP Service Account Credentials field, upload the Service Account credentials in JSON format you exported during the setup step.

Finally, after clicking "Create", you'll be offered the option of creating an associated rule. This will allow MQTT messages matching the rule to be forwarded to GCP Pub/Sub. Refer to the Rules for more info on creating rules.

Configure GCP Pub/Sub Bridge via Configuration File

Add the following configuration to the end of the emqx.conf file, substituting the contents of service_account_json with the corresponding contents of your Service Account JSON file.

bridges.gcp_pubsub.my_gcp_pubsub_bridge {
  connect_timeout = "15s"
  enable = true
  max_retries = 2
  pipelining = 100
  pool_size = 8
  pubsub_topic = "my-topic"
  request_timeout = "15s"
  resource_opts {
    async_inflight_window = 100
    auto_restart_interval = "60s"
    batch_size = 1
    batch_time = "20ms"
    health_check_interval = "15s"
    max_queue_bytes = "100MB"
    query_mode = "async"
    worker_pool_size = 16
  }
  service_account_json {
    "auth_provider_x509_cert_url" = "https://www.googleapis.com/oauth2/v1/certs"
    auth_uri = "https://accounts.google.com/o/oauth2/auth"
    client_email = "test-516@emqx-cloud-pubsub.iam.gserviceaccount.com"
    client_id = "000000000000000000000"
    "client_x509_cert_url" = "https://www.googleapis.com/robot/v1/metadata/x509/test-pubsub.iam.gserviceaccount.com"
    private_key = "-----BEGIN PRIVATE KEY-----\nMIIEvAIBA...\n-----END PRIVATE KEY-----\n"
    private_key_id = "0000000000000000000000000000000000000000"
    project_id = "my-project-id"
    token_uri = "https://oauth2.googleapis.com/token"
    type = "service_account"
  }
}