Simulating Bad Networks for MQTT with Toxiproxy

Toxiproxy is a small TCP proxy from Shopify that sits between your MQTT client and the broker and lets you break the connection on purpose: add latency and jitter, throttle bandwidth, freeze traffic, or reset the socket — all controlled by an HTTP API, so your tests can switch failures on and off in code.

This guide sets up Toxiproxy in front of an MQTT broker, walks through the toxics that matter for MQTT, and turns them into automated tests for keep-alive, reconnect and QoS behavior.

Why a proxy instead of pulling the cable

Unplugging Wi-Fi or killing the broker are blunt tools. They are hard to time, hard to repeat in CI, and they only simulate one failure: "everything is gone". Real networks fail in subtler ways — a mobile link that adds two seconds of latency, a NAT box that silently drops an idle flow, a congested uplink that delivers 5 KB per second. Those are the conditions that expose bugs in reconnect loops, inflight windows and timeouts.

Toxiproxy gives you each of those failures as a named, reversible "toxic" on a single TCP connection. The broker and your other clients are not affected, and the test can assert exactly what your code did while the toxic was active.

Run Toxiproxy in front of the broker

The quickest start is the official container. Port 8474 is the control API; every proxy you create gets its own listen port.

docker run -d --name toxiproxy \
  -p 8474:8474 -p 21883:21883 \
  ghcr.io/shopify/toxiproxy

Create a proxy that listens on 21883 and forwards to the broker:

curl -s -X POST localhost:8474/proxies -d '{
  "name": "mqtt",
  "listen": "0.0.0.0:21883",
  "upstream": "broker.connectmq.com:1883"
}'

Point any MQTT client at localhost:21883 instead of the broker and confirm traffic still flows:

mosquitto_sub -h localhost -p 21883 -t 'testmqtt/toxiproxy/#' -v &
mosquitto_pub -h localhost -p 21883 -t 'testmqtt/toxiproxy/hello' -m 'through the proxy'

You can watch the same topic in the online MQTT client connected directly to the broker: messages published through the proxy appear there too, which makes it easy to see when a toxic delays or blocks them.

For a test stack, declare the proxy in a config file so it exists as soon as the container starts:

services:
  toxiproxy:
    image: ghcr.io/shopify/toxiproxy
    command: ["-host=0.0.0.0", "-config=/config/toxiproxy.json"]
    volumes:
      - ./toxiproxy.json:/config/toxiproxy.json:ro
    ports:
      - "8474:8474"
      - "21883:21883"

[
  { "name": "mqtt", "listen": "0.0.0.0:21883", "upstream": "broker.connectmq.com:1883", "enabled": true }
]

If the broker runs in the same Compose file (for example a Mosquitto container in Testcontainers or a service called mosquitto), set the upstream to mosquitto:1883.

The toxics that matter for MQTT

Toxics are added per proxy and per direction. downstream is broker → client (CONNACK, PUBACK, delivered messages), upstream is client → broker (CONNECT, PUBLISH, PINGREQ). The default is downstream.

# 800 ms ± 200 ms on everything the broker sends back
curl -s -X POST localhost:8474/proxies/mqtt/toxics -d '{
  "name": "slow", "type": "latency", "stream": "downstream",
  "attributes": { "latency": 800, "jitter": 200 }
}'

# Throttle the client's uplink to 4 KB/s
curl -s -X POST localhost:8474/proxies/mqtt/toxics -d '{
  "name": "thin", "type": "bandwidth", "stream": "upstream",
  "attributes": { "rate": 4 }
}'

curl -s localhost:8474/proxies/mqtt/toxics       # list active toxics
curl -s -X DELETE localhost:8474/proxies/mqtt/toxics/slow
curl -s -X POST localhost:8474/reset              # remove all toxics, re-enable proxies

A few MQTT-specific notes:

• latency delays each chunk of data. Applied downstream it stretches every round trip: CONNACK, SUBACK and PUBACK all arrive late. Use it to find timeouts that are too tight — for instance a connect timeout of 1 s fails as soon as latency passes it.
• bandwidth (rate in KB/s) shows what happens when you publish faster than the link can carry. With QoS 1, outstanding messages pile up in the client's inflight window; see MQTT flow control.
• timeout with timeout: 0 stops all data but keeps the socket open. Neither side gets a TCP error — this is the classic half-open connection, and only MQTT keep alive will notice it.
• reset_peer closes the connection with a TCP RST after timeout ms, giving you ECONNRESET on the client.
• slicer splits data into small segments with a delay between them. A correct client reassembles MQTT packets across reads; a naive embedded parser may not.
• limit_data closes the connection after bytes bytes — handy for cutting a large PUBLISH in half.

To drop the connection entirely, disable the proxy: curl -X POST localhost:8474/proxies/mqtt -d '{"enabled": false}'. Existing connections are closed and new ones are refused until you enable it again.

Automating it: a keep-alive test in Python

The control API is plain HTTP, so no client library is required. This test (paho-mqtt 2.x) connects through the proxy with a 5-second keep-alive, freezes the link and asserts that the client notices within the keep-alive window.

import json, threading, time, urllib.request
import paho.mqtt.client as mqtt

API = "http://localhost:8474"

def toxiproxy(method, path, body=None):
    data = json.dumps(body).encode() if body is not None else None
    req = urllib.request.Request(API + path, data=data, method=method)
    with urllib.request.urlopen(req) as resp:
        return resp.read()

def test_client_detects_half_open_connection():
    toxiproxy("POST", "/reset")
    connected, disconnected = threading.Event(), threading.Event()

    client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id="toxi-keepalive-test")
    client.on_connect = lambda c, u, f, rc, p: connected.set()
    client.on_disconnect = lambda c, u, f, rc, p: disconnected.set()
    client.connect("localhost", 21883, keepalive=5)
    client.loop_start()
    try:
        assert connected.wait(5), "no CONNACK through the proxy"

        # Freeze the connection in both directions without closing it
        for stream in ("upstream", "downstream"):
            toxiproxy("POST", "/proxies/mqtt/toxics", {
                "name": f"freeze_{stream}", "type": "timeout",
                "stream": stream, "attributes": {"timeout": 0},
            })
        started = time.monotonic()

        assert disconnected.wait(15), "client never noticed the dead link"
        print(f"dead link detected after {time.monotonic() - started:.1f} s")
    finally:
        toxiproxy("POST", "/reset")
        client.loop_stop()
        client.disconnect()

Run it with pytest -s test_keepalive.py. Detection should take roughly one to two keep-alive intervals, depending on where in the ping cycle the freeze lands. If it takes much longer, or never happens, your client is not enforcing keep-alive — a bug that would leave real devices "connected" to nothing for hours. For a cleaner fixture layout, see pytest fixtures for MQTT.

The same pattern covers other scenarios:

1. Reconnect — disable the proxy, wait, enable it, and assert that the client reconnects and re-subscribes (details in MQTT reconnect testing).
2. QoS 1 retransmission — add limit_data on upstream, publish a large QoS 1 message, remove the toxic and check that the subscriber receives it exactly once or at least once, as your design expects.
3. Timeout budget — add 3 s of downstream latency and assert that your connect or request-response timeout fails cleanly with a clear error instead of hanging.

MQTT over TLS and WebSockets through Toxiproxy

Toxiproxy forwards raw bytes, so TLS (8883) and WebSockets (8083, 8084 with path /mqtt) work without changes: create a proxy whose upstream is broker.connectmq.com:8883 and the handshake passes straight through.

Hostname verification is the one thing to handle. The client connects to localhost, but the certificate names the broker. Options:

• Keep the broker hostname in the client and map it to the proxy in /etc/hosts or with Docker's extra_hosts on the test runner.
• Set the TLS server name explicitly — in MQTT.js, the servername option; in Python, wrap the socket with server_hostname set to the broker name.
• Use plain TCP for network-fault tests and keep TLS for a separate TLS connection test.

Never disable certificate verification to make the proxy work in a test that also runs against production settings — that habit tends to leak into real code.

Common mistakes

• Adding latency only upstream. Many interesting timeouts wait on the broker's reply. Test both directions, or downstream first.
• Confusing timeout: 0 with disabling the proxy. Disabling closes sockets (the client sees an immediate disconnect); timeout: 0 freezes them (only keep-alive saves you).
• Testing with QoS 0 only. QoS 0 hides what happens to unacknowledged messages. Repeat critical scenarios at QoS 1.
• Measuring through a public broker. Shared sandboxes add their own variance. For anything you plan to assert on, use a broker you control.

Watch the proxied messages in your browser

Toxiproxy turns "it sometimes disconnects in the field" into a test you can run on every commit. Start with one proxy, a frozen-link keep-alive test and a reconnect test; once those pass, add latency and bandwidth profiles that match the networks your devices really use, such as the cellular profiles in testing MQTT on flaky cellular networks.

Frequently asked questions