Testing MQTT Reconnect Logic and Backoff

Reconnect code is the part of an MQTT client that runs least often in development and most often in production. To test MQTT reconnect logic, break the connection on purpose in the ways the real world does — broker restart, refused connections, a silent network drop — and check four things: the client notices, it waits with growing delays, it subscribes again, and no QoS 1 messages are lost while it was away.

This guide gives you a logging test client in Python, three failure scenarios you can reproduce in minutes, and a checklist of assertions.

What "correct" reconnect behavior looks like

The design side — how to choose delays and session settings — is covered in MQTT reconnect best practices. Here we focus on proving your client does it.

How common libraries reconnect

• paho-mqtt (Python 2.x) reconnects automatically inside loop_forever() and loop_start(). reconnect_delay_set(min_delay, max_delay) controls the exponential backoff; there is no jitter.
• MQTT.js 5 reconnects automatically every reconnectPeriod milliseconds (1000 by default) — a fixed interval, not exponential. It resubscribes previously subscribed topics by default (resubscribe: true).
• Eclipse Paho Java reconnects when setAutomaticReconnect(true) is set, doubling the delay up to a configurable maximum.
• MQTTnet leaves reconnecting to your code: you handle the disconnected event and call connect again with your own delay.

Whatever the library, the defaults are rarely what you want in production, which is exactly why they deserve a test.

A test client that logs every transition

This paho-mqtt 2.x client uses MQTT 5 with a persistent session, resubscribes on every connect and prints a monotonic timestamp for each event, so you can read the backoff straight from the output:

import os, time
import paho.mqtt.client as mqtt
from paho.mqtt.packettypes import PacketTypes
from paho.mqtt.properties import Properties

HOST = os.getenv("MQTT_HOST", "localhost")
PORT = int(os.getenv("MQTT_PORT", "1883"))
TOPIC = "testmqtt/reconnect/probe/#"
T0 = time.monotonic()

def log(event, **info):
    print(f"{time.monotonic() - T0:8.2f}s  {event:<15} {info}", flush=True)

def on_connect(client, userdata, flags, reason_code, properties):
    log("connected", rc=str(reason_code), session_present=flags.session_present)
    if not reason_code.is_failure:
        client.subscribe(TOPIC, qos=1)   # always resubscribe

def on_disconnect(client, userdata, flags, reason_code, properties):
    log("disconnected", rc=str(reason_code))

def on_connect_fail(client, userdata):
    log("connect_failed")

def on_message(client, userdata, msg):
    log("message", topic=msg.topic, payload=msg.payload.decode(), dup=msg.dup)

client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2,
                     client_id="reconnect-probe-1", protocol=mqtt.MQTTv5)
if os.getenv("MQTT_USER"):
    client.username_pw_set(os.getenv("MQTT_USER"), os.getenv("MQTT_PASS"))
client.on_connect, client.on_disconnect = on_connect, on_disconnect
client.on_connect_fail, client.on_message = on_connect_fail, on_message
client.reconnect_delay_set(min_delay=1, max_delay=32)

props = Properties(PacketTypes.CONNECT)
props.SessionExpiryInterval = 3600       # keep the session for 1 hour
client.connect_async(HOST, PORT, keepalive=10, clean_start=False, properties=props)
client.loop_forever(retry_first_connection=True)

Keep the client ID fixed while testing — a persistent session belongs to a client ID, and two processes with the same ID will keep kicking each other off. A short keepalive=10 makes silent drops show up quickly; use your production value for the final run.

Scenario 1: broker restart and outage

Run a throwaway Mosquitto locally so you can stop it at will:

docker run -d --name mqtt-reconnect -p 1883:1883 eclipse-mosquitto:2 \
  mosquitto -c /mosquitto-no-auth.conf
python reconnect_probe.py            # terminal 1

docker stop mqtt-reconnect && sleep 60 && docker start mqtt-reconnect   # terminal 2

Expected output, with the delays doubling until the broker returns:

    0.03s  connected       {'rc': 'Success', 'session_present': False}
   12.40s  disconnected    {'rc': 'Unspecified error'}
   13.41s  connect_failed  {}
   15.42s  connect_failed  {}
   19.43s  connect_failed  {}
   27.44s  connect_failed  {}
   43.45s  connect_failed  {}
   75.46s  connected       {'rc': 'Success', 'session_present': False}

The exact reason code strings vary by version, but the gaps between attempts should be 1, 2, 4, 8, 16, 32 seconds. Note session_present: False after the restart: this Mosquitto has no persistence, so the session died with the process. Your client must resubscribe — if messages stop arriving after this test, that is the bug. Enable persistence true in Mosquitto (see Mosquitto persistence) and repeat to see the session survive a restart.

Scenario 2: silent network drop

A broker restart closes sockets cleanly. A dead Wi-Fi access point or a NAT timeout does not — packets simply vanish, and the client only finds out through keep-alive. On a Linux machine you can simulate it with a firewall rule:

sudo iptables -I OUTPUT -p tcp --dport 1883 -j DROP
sleep 45
sudo iptables -D OUTPUT -p tcp --dport 1883 -j DROP

Use -j REJECT --reject-with tcp-reset instead of -j DROP to simulate an immediate reset. The difference matters: with DROP, the probe should log disconnected roughly 10–20 seconds after the rule is added (one to two keep-alive intervals), then back off until the rule is removed. If it never logs a disconnect, keep-alive is not working — see MQTT keep alive.

For latency, packet loss, bandwidth limits or cross-platform setups, a proxy is more flexible than iptables; testing with Toxiproxy shows how to inject those faults between client and broker.

Scenario 3: session resume with queued messages

This is the test most teams skip and most incidents come from. Point the probe at a broker that keeps sessions — for example the shared endpoint broker.connectmq.com — and:

1. Start reconnect_probe.py and wait for connected with the subscription in place.
2. Block port 1883 with the iptables rule from scenario 2.
3. While the probe is offline, publish five QoS 1 messages to testmqtt/reconnect/probe/q from a path the rule doesn't block — for example the online MQTT client, which connects over secure WebSockets on port 8084.
4. Remove the rule and watch the probe reconnect.

The probe should log session_present: True and then all five messages. If it reconnects with session_present: False, the broker discarded the session: check clean_start, the session expiry interval and whether your client ID changed. If messages arrive with dup: True or twice, that is QoS 1 at-least-once behavior working as designed — your handler needs to be idempotent. Persistent sessions explains what the broker keeps and for how long.

More failure cases worth a test

• Credentials changed while offline. The broker answers the reconnect with "Not authorized" or "Bad user name or password". The client should log it loudly and slow down, not retry every second forever. Look codes up with the reason code tool.
• DNS failure. Disconnect the network entirely; resolution errors must go through the same backoff path as refused connections.
• Duplicate client ID. Start the probe twice. Each instance kicks the other off, and both reconnect endlessly — a classic production loop.
• Long outages. Leave the broker down for longer than max_delay and check that attempts settle at the cap instead of growing or stopping.
• Reconnect during publish. Publish QoS 1 messages in a loop while you cut the network, and count what arrives at a subscriber after recovery.

Reconnect test checklist

1. Log every connect, disconnect and failed attempt with a monotonic timestamp.
2. Restart the broker: verify detection, growing delays and recovery.
3. Drop traffic silently: verify keep-alive detection within one to two intervals.
4. Queue QoS 1 messages while offline: verify session_present and delivery after reconnect.
5. Verify resubscription after a restart without persistence.
6. Test bad credentials, DNS failure and duplicate client IDs.
7. Automate the scenarios in CI with a disposable broker container.

Reconnect logic is only as good as the last time you saw it work. A probe that prints timestamps, a broker you can stop, and one firewall rule are enough to prove backoff, resubscription and session resume — and to catch the regression before a broker restart does it for you.

Frequently asked questions