Chaos Testing MQTT: Network Partitions, Packet Loss and Broker Restarts

Chaos testing MQTT means breaking the network and the broker on purpose — latency, packet loss, partitions, restarts — while measuring whether your clients recover on their own. Most MQTT bugs that reach production are recovery bugs: a client that never notices a dead connection, a subscription that is not restored, a QoS 1 message lost because the session was not persistent. This guide shows the tools (tc netem, iptables, Toxiproxy, docker) and a probe script that turns each experiment into numbers.

Define what "recovered" means first

A chaos experiment without a pass criterion is just an outage. Write down the steady state your system must return to:

Two mechanisms do most of the work: the keep-alive interval, which bounds how long a dead connection goes unnoticed, and persistent sessions, which let the broker hold QoS 1 messages for a subscriber while it is away.

The probe: measure loss, duplicates and reconnects

This paho-mqtt 2.x script runs a publisher and a subscriber with a persistent MQTT 5 session. It publishes a numbered message every 100 ms at QoS 1 and reports every five seconds how many acknowledged messages have not arrived yet (missing), how many arrived twice (dups), reconnect count and worst latency.

import json, os, threading, time, uuid
import paho.mqtt.client as mqtt
from paho.mqtt.packettypes import PacketTypes
from paho.mqtt.properties import Properties

HOST = os.getenv("MQTT_HOST", "localhost")
PORT = int(os.getenv("MQTT_PORT", "1883"))
RUN = uuid.uuid4().hex[:8]
TOPIC = f"testmqtt/chaos/{RUN}/seq"
lock = threading.Lock()
seen, sent = set(), []            # sent: (seq, MQTTMessageInfo)
stats = {"dups": 0, "reconnects": 0, "max_ms": 0.0}

def on_connect(client, userdata, flags, reason_code, properties):
    print(f"[{userdata['role']}] connected: {reason_code}, session present={flags.session_present}")
    if userdata["ever"]:
        stats["reconnects"] += 1
    userdata["ever"] = True
    if userdata["role"] == "sub":
        client.subscribe(TOPIC, qos=1)

def on_disconnect(client, userdata, flags, reason_code, properties):
    print(f"[{userdata['role']}] disconnected: {reason_code}")

def on_message(client, userdata, msg):
    data = json.loads(msg.payload)
    with lock:
        if data["seq"] in seen:
            stats["dups"] += 1
        seen.add(data["seq"])
        stats["max_ms"] = max(stats["max_ms"], (time.time() - data["t"]) * 1000)

def make(role, session_expiry):
    props = Properties(PacketTypes.CONNECT)
    props.SessionExpiryInterval = session_expiry
    c = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id=f"chaos-{role}-{RUN}",
                    protocol=mqtt.MQTTv5, userdata={"role": role, "ever": False})
    c.on_connect, c.on_disconnect, c.on_message = on_connect, on_disconnect, on_message
    c.reconnect_delay_set(min_delay=1, max_delay=16)
    c.connect_async(HOST, PORT, keepalive=10, properties=props)
    c.loop_start()
    return c

sub = make("sub", session_expiry=600)   # session survives outages up to 10 minutes
pub = make("pub", session_expiry=0)
time.sleep(2)

seq, next_report = 0, time.time() + 5
while True:
    info = pub.publish(TOPIC, json.dumps({"seq": seq, "t": time.time()}), qos=1)
    sent.append((seq, info))
    seq += 1
    if time.time() >= next_report:
        acked = {s for s, i in sent if i.is_published()}
        with lock:
            print(f"{time.strftime('%H:%M:%S')} sent={seq} acked={len(acked)} "
                  f"missing={len(acked - seen)} dups={stats['dups']} "
                  f"reconnects={stats['reconnects']} max_latency={stats['max_ms']:.0f}ms")
            stats["max_ms"] = 0.0
        next_report += 5
    time.sleep(0.1)

While the network is down, paho keeps QoS 1 messages in its in-memory queue and sends them after reconnecting, so sent keeps growing while acked pauses. During an outage missing may rise briefly; after recovery it must return to 0. A permanent non-zero value means the broker acknowledged messages that were later lost — usually a session that did not survive.

Experiment 1: latency and packet loss with tc netem

On Linux, tc netem degrades an interface. Run it inside a VM or a container with --cap-add NET_ADMIN so you do not slow down your whole workstation.

IFACE=eth0
sudo tc qdisc add dev $IFACE root netem delay 300ms 100ms loss 5%     # bad cellular
sudo tc qdisc change dev $IFACE root netem delay 2000ms loss 30%      # nearly unusable
sudo tc qdisc del dev $IFACE root                                     # back to normal

Expected: at 300 ms and 5 % loss, TCP retransmits hide the loss — latency rises but there are no reconnects and missing stays at 0. At 2 s and 30 % loss, PINGRESP may arrive too late and clients start reconnecting; check that they settle once the rule is removed. For realistic cellular profiles see testing MQTT under cellular network conditions.

Experiment 2: network partitions with iptables

There are two very different ways for a network to fail, and clients often handle only one of them:

# Blackhole: packets silently vanish. Only keep-alive can detect this.
sudo iptables -A OUTPUT -p tcp --dport 1883 -j DROP
# ...wait 60 s, then heal:
sudo iptables -D OUTPUT -p tcp --dport 1883 -j DROP

# Hard failure: the client gets a TCP reset immediately.
sudo iptables -A OUTPUT -p tcp --dport 1883 -j REJECT --reject-with tcp-reset
sudo iptables -D OUTPUT -p tcp --dport 1883 -j REJECT --reject-with tcp-reset

With a 10-second keep-alive, the probe should log disconnected roughly 10–20 seconds into a blackhole, retry with backoff while it lasts, and reconnect with session present=True once it heals. On the broker side, other clients should receive the probe's Last Will after about 1.5 × keep-alive. A client that stays "connected" for minutes during a blackhole has keep-alive disabled or set far too high. Testing QoS with a network cut goes deeper into what each QoS level guarantees here.

Experiment 3: per-connection faults with Toxiproxy

Toxiproxy is a TCP proxy whose behaviour you change through an HTTP API. It works on macOS and Windows too, affects only connections that go through it, and is easy to script in CI.

docker run -d --name toxiproxy -p 8474:8474 -p 21883:21883 ghcr.io/shopify/toxiproxy
curl -s -X POST localhost:8474/proxies \
  -d '{"name":"mqtt","listen":"0.0.0.0:21883","upstream":"broker.connectmq.com:1883"}'

# Point the probe at the proxy
MQTT_HOST=localhost MQTT_PORT=21883 python chaos_probe.py

Then inject faults from a second terminal:

API=localhost:8474/proxies/mqtt
# 1.5 s latency with 500 ms jitter
curl -s -X POST $API/toxics -d '{"name":"lag","type":"latency","attributes":{"latency":1500,"jitter":500}}'
curl -s -X DELETE $API/toxics/lag
# Blackhole: data stops, connection stays open until the toxic is removed
curl -s -X POST $API/toxics -d '{"name":"cut","type":"timeout","attributes":{"timeout":0}}'
curl -s -X DELETE $API/toxics/cut
# Connection reset by peer
curl -s -X POST $API/toxics -d '{"name":"rst","type":"reset_peer","attributes":{"timeout":0}}'
curl -s -X DELETE $API/toxics/rst
# Take the whole "network" down and up again
curl -s -X POST $API -d '{"enabled":false}'; sleep 30; curl -s -X POST $API -d '{"enabled":true}'

The timeout toxic with 0 is the Toxiproxy equivalent of an iptables DROP; reset_peer matches REJECT. Testing MQTT with Toxiproxy covers bandwidth limits, slicing and test-framework integration.

Experiment 4: broker restarts and crashes

Restart experiments need a broker you run yourself. A local Mosquitto container is enough to find most client-side bugs:

docker run -d --name chaos-broker -p 1883:1883 eclipse-mosquitto:2 mosquitto -c /mosquitto-no-auth.conf
MQTT_HOST=localhost MQTT_PORT=1883 python chaos_probe.py     # in another terminal

docker restart chaos-broker                                   # graceful: SIGTERM, then start
docker kill --signal=KILL chaos-broker && docker start chaos-broker   # crash

What you should observe:

• Without persistence (the default config above), the broker forgets every session. The probe reconnects with session present=False, re-subscribes, and messages published during the restart can be permanently missing.
• With persistence true and a volume for the data directory, sessions and queued messages survive a graceful restart. After a KILL, anything since the last autosave can still be lost — check whether that matches what your design promises.
• Clients must not reconnect in lockstep. With hundreds of clients, a restart turns into a connection spike. Backoff with jitter spreads it out; see MQTT reconnect storms.

For clustered brokers, replace the restart with stopping one node at a time and verify that clients fail over. MQTT reconnect testing has more client-side scenarios.

An experiment plan you can repeat

Run the plan before every major client or broker release, keep the probe output as an artifact, and add a new row whenever a real incident teaches you a new failure mode.

Watch probe messages arrive in your browser

Chaos testing MQTT is mostly about recovery: detecting the failure fast, reconnecting gently and losing nothing you promised to deliver. With netem, iptables, Toxiproxy and a restartable broker you can reproduce nearly every outage on a laptop, and the probe tells you in numbers whether your clients pass.

Frequently asked questions