Monitor an MQTT Broker with $SYS Topics

Most MQTT brokers report on themselves over MQTT. Subscribe to $SYS/# and the broker publishes how many clients are connected, how many messages and bytes it has received and sent, how many subscriptions and retained messages it holds, and how long it has been running. That makes $SYS the fastest way to monitor a broker. You need no exporter and no agent, just an MQTT client.

This guide covers what $SYS exposes on common brokers, how to subscribe to it correctly, how to turn raw counters into useful rates, and how to log and graph them with a short Python script.

What $SYS topics are

The MQTT specification reserves topics that start with $ for broker-internal use. Clients should not publish there, and wildcards do not match them by accident: a subscription to # or +/foo never receives $SYS messages. You have to subscribe to $SYS/# (or a narrower filter) on purpose.

The content is not standardized. Mosquitto, EMQX, VerneMQ and others each publish their own tree, and some managed services hide it completely or restrict it with ACLs. For a full reference of the topic tree, see MQTT $SYS topics. This article is about using those topics for monitoring.

The topics worth watching

These are the Mosquitto names (2.x). Other brokers publish similar values under different paths.

Two kinds of numbers live here. Gauges (connected clients, subscriptions) are meaningful as they are. Counters (messages, bytes) only grow, so you need the difference between two samples to get a rate. Mosquitto also publishes ready-made moving averages under $SYS/broker/load/.... These are handy for a quick look, but you lose control over the window.

On EMQX, the tree is per node: $SYS/brokers/<node>/version, $SYS/brokers/<node>/uptime, plus stats/... and metrics/... subtrees, and optional client connect/disconnect events. Its publishing interval is set by sys_msg_interval. For dashboards on EMQX, its built-in Prometheus endpoint is usually the better source. See Prometheus and Grafana monitoring.

Subscribe to $SYS

The quickest check is mosquitto_sub. Quote the topic in single quotes, or the shell will try to expand $SYS as a variable:

mosquitto_sub -h broker.connectmq.com -p 1883 \
  -u "$MQTT_USER" -P "$MQTT_PASS" \
  -v -t '$SYS/#'

To watch only a few values, subscribe to several narrow filters:

mosquitto_sub -h broker.connectmq.com -p 1883 \
  -u "$MQTT_USER" -P "$MQTT_PASS" -v \
  -t '$SYS/broker/clients/connected' \
  -t '$SYS/broker/load/messages/+/1min' \
  -t '$SYS/broker/uptime'

You can do the same in the browser: open the online MQTT client, connect, and subscribe to $SYS/#. If nothing arrives, check these three things in order:

1. The broker has $SYS disabled. In Mosquitto, sys_interval 0 turns it off.
2. An ACL denies it. Many production configs only allow an admin user to read $SYS. MQTT 5 clients get a SUBACK reason code 0x87 Not authorized. Decode it with the reason code lookup.
3. You only waited for one interval. Some values only appear after the next sys_interval tick.

Configure the update interval

In Mosquitto, the interval is a single option:

# Publish $SYS updates every 10 seconds (default). 0 disables $SYS.
sys_interval 10

A shorter interval gives finer graphs but costs a little CPU and bandwidth for every subscriber to $SYS/#. For dashboards, 10–30 seconds is plenty. The mosquitto.conf generator includes this option if you build your config from scratch.

Turn counters into rates

Counters are only useful as deltas. If messages/received was 1,204,500 at 12:00:00 and 1,210,500 at 12:00:10, the broker received (1210500 − 1204500) / 10 = 600 messages per second in that window. Two rules keep the math honest:

• Use the time you received the sample, not the nominal interval. Network jitter and broker load shift the ticks.
• Handle resets. When the broker restarts, counters start again from zero. If a new value is smaller than the previous one, drop that delta instead of graphing a huge negative spike. A drop in uptime confirms the restart.

Log and graph $SYS with Python

This script (paho-mqtt 2.x) subscribes to a few $SYS topics, computes rates from the counters, prints one line per update and appends everything to a CSV file you can graph later:

import csv, os, time
import paho.mqtt.client as mqtt

HOST, PORT = "broker.connectmq.com", 1883
TOPICS = {
    "$SYS/broker/clients/connected": "gauge",
    "$SYS/broker/subscriptions/count": "gauge",
    "$SYS/broker/messages/received": "counter",
    "$SYS/broker/messages/sent": "counter",
    "$SYS/broker/bytes/received": "counter",
    "$SYS/broker/bytes/sent": "counter",
}
last = {}  # topic -> (timestamp, value)
out = csv.writer(open("sys_metrics.csv", "a", newline=""))

def on_connect(client, userdata, flags, reason_code, properties):
    if reason_code.is_failure:
        print("connect failed:", reason_code)
        return
    client.subscribe([(t, 0) for t in TOPICS])

def on_message(client, userdata, msg):
    now = time.time()
    try:
        value = float(msg.payload.decode().split()[0])
    except ValueError:
        return
    name = msg.topic.removeprefix("$SYS/broker/")
    if TOPICS[msg.topic] == "counter":
        prev = last.get(msg.topic)
        last[msg.topic] = (now, value)
        if prev is None or value < prev[1]:   # first sample or broker restart
            return
        value = (value - prev[1]) / (now - prev[0])
        name += "/s"
    out.writerow([int(now), name, round(value, 2)])
    print(f"{time.strftime('%H:%M:%S')}  {name:<24} {value:>12.1f}")

client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id="testmqtt-sys-monitor")
client.username_pw_set(os.getenv("MQTT_USER"), os.getenv("MQTT_PASS"))
client.on_connect = on_connect
client.on_message = on_message
client.connect(HOST, PORT, keepalive=30)
client.loop_forever()

Subscribing inside on_connect matters: after a reconnect the subscriptions are restored automatically. To graph the CSV, a few lines of pandas and matplotlib are enough:

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv("sys_metrics.csv", names=["ts", "metric", "value"])
df["ts"] = pd.to_datetime(df["ts"], unit="s")
df.pivot_table(index="ts", columns="metric", values="value").plot(subplots=True, figsize=(10, 8))
plt.tight_layout()
plt.show()

For a permanent dashboard with history and alerting, feed the same values into Prometheus or InfluxDB instead of a CSV file.

Test that your monitoring reacts

A monitor that has never seen a change has not been tested. Generate known load and check that the numbers move:

for i in $(seq 1 1000); do echo "{\"i\":$i}"; done | \
  mosquitto_pub -h broker.connectmq.com -u "$MQTT_USER" -P "$MQTT_PASS" \
  -t testmqtt/sys-monitor/load -l

During the run, messages/received/s should rise by roughly the rate you publish. Open ten extra connections in the online client or with a device simulator and clients/connected should rise by ten. If neither moves, you are probably reading a different node than you are writing to, or a cluster is reporting per-node values.

Limits of $SYS monitoring

• No history. $SYS shows the current state only. You must store it yourself.
• No alerting. Pair it with Prometheus, Grafana or a script that pages you. For outside-in checks, see broker health checks.
• Broker-specific. Topic names and meanings change between brokers, and sometimes between versions.
• Monitored by the system it monitors. If the broker is down, $SYS is silent. A silent monitor should raise an alert, not look like a quiet day.

$SYS topics are the cheapest observability an MQTT broker offers. Subscribe to $SYS/# once to learn what your broker publishes, keep a small script that turns counters into rates, and restrict access so that only your monitoring account can read the tree.

Frequently asked questions