External MQTT Uptime Monitoring and Alerting

External uptime monitoring answers one question: could a device outside your network connect to the broker and exchange a message just now? Internal metrics cannot answer that. They do not see DNS failures, expired certificates, broken load balancers or a firewall change. Only a probe that behaves like a client, running from outside, does.

This guide builds that probe, schedules it, routes alerts to Slack or PagerDuty, and shows the SLO math you need to choose a check interval and an alert threshold.

What "up" means for an MQTT broker

"The port is open" is a weak definition. A broker is useful only when every layer below works:

Check each layer separately and report which one failed. "TLS failed, certificate expired" can be fixed in five minutes. "Probe failed" starts a 30-minute investigation. For one-off manual checks, see checking MQTT broker status and testing whether an MQTT port is open.

A minimal probe with mosquitto clients

If you already have mosquitto_pub and mosquitto_sub, a shell probe takes a dozen lines:

#!/usr/bin/env bash
set -u
HOST=broker.connectmq.com; PORT=8883
TOPIC="testmqtt/uptime/$(hostname)-$$"
MSG="ping-$(date +%s)"
OUT=$(mktemp)

mosquitto_sub -h "$HOST" -p "$PORT" --capath /etc/ssl/certs \
  -u "$MQTT_USER" -P "$MQTT_PASS" -t "$TOPIC" -q 1 -C 1 -W 10 > "$OUT" &
SUB=$!
sleep 1
mosquitto_pub -h "$HOST" -p "$PORT" --capath /etc/ssl/certs \
  -u "$MQTT_USER" -P "$MQTT_PASS" -t "$TOPIC" -q 1 -m "$MSG"
wait $SUB

if grep -qx "$MSG" "$OUT"; then echo "OK"; rm -f "$OUT"; exit 0
else echo "FAIL: no round trip within 10 s"; rm -f "$OUT"; exit 1; fi

-C 1 makes the subscriber exit after one message, and -W 10 makes it give up after ten seconds. The exit code is all a scheduler or heartbeat service needs.

A layered probe in Python

This version (paho-mqtt 2.x) checks each layer, measures it, reports the days until the certificate expires, and prints one JSON line that is easy to ship to logs:

import json, os, socket, ssl, sys, threading, time, uuid
import paho.mqtt.client as mqtt

HOST, PORT, TIMEOUT = "broker.connectmq.com", 8883, 10
result = {"host": HOST, "ts": int(time.time())}

def fail(stage, detail):
    result.update(ok=False, stage=stage, detail=str(detail))
    print(json.dumps(result)); sys.exit(1)

t = time.perf_counter()
try:
    socket.getaddrinfo(HOST, PORT)
except OSError as e:
    fail("dns", e)
try:
    ctx = ssl.create_default_context()
    with socket.create_connection((HOST, PORT), timeout=TIMEOUT) as raw:
        with ctx.wrap_socket(raw, server_hostname=HOST) as tls:
            not_after = ssl.cert_time_to_seconds(tls.getpeercert()["notAfter"])
except ssl.SSLError as e:
    fail("tls", e)
except OSError as e:
    fail("tcp", e)
result["cert_days_left"] = int((not_after - time.time()) // 86400)
result["tls_ms"] = round((time.perf_counter() - t) * 1000)

topic = f"testmqtt/uptime/{uuid.uuid4().hex[:8]}"
connack, got = {}, threading.Event()
c = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id=f"testmqtt-uptime-{uuid.uuid4().hex[:6]}")
c.username_pw_set(os.getenv("MQTT_USER"), os.getenv("MQTT_PASS"))
c.tls_set_context(ctx)
c.on_connect = lambda cl, ud, fl, rc, pr: (connack.update(rc=rc),
                                           None if rc.is_failure else cl.subscribe(topic, qos=1))
c.on_subscribe = lambda cl, ud, mid, rcs, pr: cl.publish(topic, "ping", qos=1)
c.on_message = lambda cl, ud, msg: got.set()

t = time.perf_counter()
try:
    c.connect(HOST, PORT, keepalive=15)
except OSError as e:
    fail("connect", e)
c.loop_start()
ok = got.wait(TIMEOUT)
c.disconnect(); c.loop_stop()
if "rc" not in connack:
    fail("connect", "no CONNACK")
if connack["rc"].is_failure:
    fail("connect", connack["rc"])
if not ok:
    fail("roundtrip", f"no message within {TIMEOUT} s")
result.update(ok=True, roundtrip_ms=round((time.perf_counter() - t) * 1000))
print(json.dumps(result))
sys.exit(2 if result["cert_days_left"] < 14 else 0)

Exit code 0 means healthy, 1 means down (the stage field says where), and 2 means up but the certificate expires within two weeks. That last one is a warning, not a page. When the connect stage fails, the reason code (for example Not authorized) tells you whether it is a credential or an ACL problem. Look it up with the reason code tool or in MQTT connection errors.

Scheduling the probe

• cron or a systemd timer on a small VM outside your broker's network: * * * * * /opt/probe/mqtt_uptime_probe.py >> /var/log/mqtt-probe.log 2>&1. This is simple and runs every minute.
• Several locations. Run the same probe from two or three regions or providers. Alert when most of them fail. One failing location is usually that location's network.
• GitHub Actions schedule works for low-stakes checks, but the minimum interval is five minutes and runs can start late. It is not suitable for paging.
• Uptime Kuma (self-hosted) includes an MQTT monitor type and works well if you prefer a UI.

Whatever runs the probe can fail too. Use a heartbeat (dead man's switch): after every successful run, the probe pings a URL at a heartbeat service such as Healthchecks.io or an Uptime Kuma push monitor. If the pings stop, either the broker or the probe host is down, and you get an alert in both cases:

* * * * * /opt/probe/mqtt-probe.sh && curl -fsS -m 10 --retry 3 "$HEARTBEAT_URL" > /dev/null

Alert channels

Send alerts where someone will actually see them, and only after enough evidence. A common rule is three consecutive failures, from at least two locations, with a recovery message when it clears.

curl -fsS -X POST -H 'Content-Type: application/json' \
  -d '{"text":":red_circle: MQTT probe failed at stage tls (broker.connectmq.com:8883)"}' \
  "$SLACK_WEBHOOK_URL"

curl -fsS -X POST https://events.pagerduty.com/v2/enqueue \
  -H 'Content-Type: application/json' \
  -d '{"routing_key":"'"$PD_ROUTING_KEY"'","event_action":"trigger",
       "dedup_key":"mqtt-probe-broker",
       "payload":{"summary":"MQTT round trip failing","source":"mqtt-probe","severity":"critical"}}'

Send the same request with "event_action":"resolve" and the same dedup_key to close the incident automatically. Keep certificate-expiry warnings in Slack or email. They need action this week, not at 3 a.m. If you already run Prometheus, export the probe as metrics and alert there instead. See MQTT monitoring with Prometheus and Grafana.

SLO math

An availability target only makes sense with its downtime budget:

Measure it as successful probes ÷ total probes. One probe per minute gives 43,200 samples in 30 days, so 99.9% allows 43 failed probes. Three consequences follow:

• The interval limits accuracy. With a one-minute probe, a 20-second outage may go unseen or count as a whole minute. If you want to measure 99.99%, probe every 10–15 seconds.
• Detection time eats the budget. Three failures at one-minute intervals means about three minutes before anyone is paged. That is fine for 99.9% (43 minutes) and almost the entire budget at 99.99%.
• Your SLO is not the provider's SLA. The SLA covers the broker. Your probe measures the whole path, including your DNS, your certificates and your network. Read MQTT broker SLAs and uptime before you compare the two.

Track the error budget: if 30 of your 43 minutes are gone by mid-month, freeze risky broker changes until it recovers.

Test the alert path

A probe that has never failed has not been tested. Before you rely on it:

1. Set a wrong password and confirm that the connect stage fails and the alert arrives in Slack or PagerDuty.
2. Point HOST at a name that does not resolve and confirm stage: dns.
3. Stop the cron job and confirm the heartbeat service alerts you.
4. Check by hand that the probe's view matches reality: connect with the online MQTT client and publish to the probe topic while the probe runs.

External uptime monitoring is a small script, a scheduler and a heartbeat. Check every layer separately, alert only on repeated failures from more than one location, choose your probe interval from your SLO, and break the setup on purpose once to prove the alerts arrive.

Frequently asked questions