Monitor MQTT with Prometheus and Grafana

Prometheus and Grafana are the usual stack for monitoring an MQTT broker. Prometheus scrapes metrics over HTTP and evaluates alert rules. Grafana draws the dashboards. Brokers expose their metrics in one of two ways: some have a native Prometheus endpoint (EMQX, VerneMQ, HiveMQ through an extension), while others, like Mosquitto, only publish $SYS topics and need an exporter to translate them.

This guide wires up all three data sources you are likely to need: an exporter for Mosquitto, the native EMQX endpoint, and a client-side probe that works with any broker, managed ones included. Then it adds dashboards and alerts.

Where the metrics come from

Broker-side metrics tell you what the broker is doing: connections, throughput, queues and drops. A client-side probe tells you what your devices experience: can they connect, and how long does a message take? You want both. If you only have one, choose the probe, because it is the one that catches DNS, TLS, load balancer and authentication failures.

If you are new to $SYS, read monitoring a broker with $SYS topics first. The exporter below is a thin layer on top of it.

A local stack with Docker Compose

This stack runs Mosquitto, the exporter, Prometheus and Grafana together. It is a good lab for building dashboards before you point them at production.

services:
  mosquitto:
    image: eclipse-mosquitto:2
    ports: ["1883:1883"]
    volumes:
      - ./mosquitto.conf:/mosquitto/config/mosquitto.conf:ro

  mosquitto-exporter:
    image: sapcc/mosquitto-exporter
    command: ["--endpoint", "tcp://mosquitto:1883"]
    ports: ["9234:9234"]
    depends_on: [mosquitto]

  prometheus:
    image: prom/prometheus
    ports: ["9090:9090"]
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./alerts.yml:/etc/prometheus/alerts.yml:ro

  grafana:
    image: grafana/grafana
    ports: ["3000:3000"]
    volumes:
      - ./grafana-datasource.yml:/etc/grafana/provisioning/datasources/prometheus.yml:ro

listener 1883
allow_anonymous true
sys_interval 10

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    isDefault: true

allow_anonymous true is acceptable only on a laptop. On a real broker, create a dedicated monitoring user that can read $SYS/# and nothing else, then pass it to the exporter with --user and --pass. The mosquitto.conf generator can produce the listener and authentication blocks for you. For a fuller lab with more services, see the MQTT Docker Compose stack.

The exporter turns each numeric $SYS topic into a metric whose name follows the topic path, for example $SYS/broker/clients/connected becomes broker_clients_connected. Open http://localhost:9234/metrics to see the exact names your version produces before you write queries.

A probe exporter for any broker

This Python exporter (paho-mqtt 2.x and prometheus_client) connects over TLS every 15 seconds, subscribes to a unique topic, publishes one message and waits for it to come back. It exports whether that worked and how long each step took:

import os, ssl, threading, time, uuid
import paho.mqtt.client as mqtt
from prometheus_client import Gauge, Histogram, start_http_server

HOST, PORT, INTERVAL = "broker.connectmq.com", 8883, 15
success = Gauge("mqtt_probe_success", "1 if the last connect/publish/receive cycle worked")
connect_s = Gauge("mqtt_probe_connect_seconds", "TCP + TLS + CONNACK time of the last probe")
roundtrip = Histogram("mqtt_probe_roundtrip_seconds", "Subscribe, publish and receive time",
                      buckets=(0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5))

def probe() -> bool:
    topic = f"testmqtt/probe/{uuid.uuid4().hex[:8]}"
    connected, received = threading.Event(), threading.Event()
    c = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2,
                    client_id=f"testmqtt-probe-{uuid.uuid4().hex[:6]}")
    c.username_pw_set(os.getenv("MQTT_USER"), os.getenv("MQTT_PASS"))
    c.tls_set(cert_reqs=ssl.CERT_REQUIRED)
    c.on_connect = lambda cl, ud, fl, rc, pr: connected.set() if not rc.is_failure else None
    c.on_subscribe = lambda cl, ud, mid, rcs, pr: cl.publish(topic, "ping", qos=1)
    c.on_message = lambda cl, ud, msg: received.set()
    t0 = time.perf_counter()
    try:
        c.connect(HOST, PORT, keepalive=10)
        c.loop_start()
        if not connected.wait(5):
            return False
        connect_s.set(time.perf_counter() - t0)
        t1 = time.perf_counter()
        c.subscribe(topic, qos=1)
        if not received.wait(5):
            return False
        roundtrip.observe(time.perf_counter() - t1)
        return True
    except OSError:
        return False
    finally:
        c.disconnect()
        c.loop_stop()

if __name__ == "__main__":
    start_http_server(9105)          # metrics on http://localhost:9105/metrics
    while True:
        success.set(1 if probe() else 0)
        time.sleep(INTERVAL)

Install it with pip install paho-mqtt prometheus-client. A new connection on every cycle is deliberate: it exercises DNS, TLS and authentication each time, just like a device that reboots. Use a unique client ID for every run. The client ID generator explains why duplicates kick each other off.

Scrape configuration

global:
  scrape_interval: 15s

rule_files: ["/etc/prometheus/alerts.yml"]

scrape_configs:
  - job_name: mosquitto
    static_configs:
      - targets: ["mosquitto-exporter:9234"]

  - job_name: mqtt-probe
    static_configs:
      - targets: ["host.docker.internal:9105"]   # where the probe runs

  - job_name: emqx
    metrics_path: /api/v5/prometheus/stats
    static_configs:
      - targets: ["emqx:18083"]

Remove the jobs you do not use. On recent EMQX versions the stats endpoint can be configured to require authentication. If so, add basic_auth with an API key and secret. To try EMQX locally, see EMQX in Docker. host.docker.internal works on Docker Desktop. On Linux, run the probe as another Compose service instead.

Dashboards that answer real questions

A useful MQTT dashboard has few panels, and each one answers a question someone will ask during an incident:

Use rate() on counters and never graph the raw counter value. Put the probe panels at the top, because they show what users see. Community dashboards for the Mosquitto exporter and EMQX are published on grafana.com. They are a good starting point, but check that their metric names match your exporter version.

Alert rules

groups:
  - name: mqtt
    rules:
      - alert: MqttProbeFailing
        expr: max_over_time(mqtt_probe_success[3m]) == 0
        labels: { severity: page }
        annotations:
          summary: "No successful MQTT round trip for 3 minutes"

      - alert: MqttProbeSlow
        expr: histogram_quantile(0.99, sum by (le) (rate(mqtt_probe_roundtrip_seconds_bucket[10m]))) > 0.5
        for: 10m
        labels: { severity: ticket }

      - alert: MqttClientsDropped
        expr: broker_clients_connected < 0.7 * (broker_clients_connected offset 10m)
        for: 2m
        labels: { severity: page }

      - alert: MqttExporterDown
        expr: up{job="mosquitto"} == 0
        for: 2m
        labels: { severity: ticket }

MqttClientsDropped catches the classic incident pattern: a large share of devices disconnects at once, after a certificate expiry, a network change or a broker restart. Route page alerts through Alertmanager to your on-call channel and ticket alerts somewhere quieter. For external checks and SLO math, see MQTT uptime monitoring.

Test the monitoring itself

Before you trust the alerts, make each one fire on purpose:

1. Probe failure: change MQTT_PASS to a wrong value and restart the probe. mqtt_probe_success should drop to 0 and MqttProbeFailing should fire within about three minutes.
2. Client drop: connect a batch of test clients (a device simulator or several tabs of the online client), wait ten minutes, then close them. The connected-clients panel should drop accordingly.
3. Exporter down: docker compose stop mosquitto-exporter should fire MqttExporterDown.
4. Latency: add delay with a network shaper, or compare the probe's numbers with a dedicated latency test.

An alert that never fired in a test may also fail to fire during a real outage.

Prometheus and Grafana give you history, dashboards and alerts that $SYS alone cannot. Combine broker metrics with a client-side probe, keep the dashboard small, and test every alert once before you depend on it.

Frequently asked questions