Testing MQTT on Slow and Flaky Cellular Networks

An MQTT client that is rock-solid on office Wi-Fi can fall apart on a cellular modem: connects that time out, keep-alive disconnects every few minutes, QoS 1 messages that arrive in bursts. The fix starts with reproducing cellular conditions on your desk — latency, jitter, packet loss and limited bandwidth — and then tuning keep-alive, payload size, QoS and session settings until the client survives them.

This guide shows how to emulate 2G, 3G and LTE links with tc netem, which MQTT settings to test under each profile, and a script that measures delivery and latency so you can compare configurations.

What makes cellular different

Cellular links are not just "slow Wi-Fi". They fail in specific ways that hit MQTT's long-lived TCP connection:

• High and variable round-trip time. Every MQTT exchange that waits for a reply — CONNACK, SUBACK, PUBACK, PINGRESP — pays the full RTT, and jitter makes it unpredictable.
• Loss bursts. Handovers and weak signal drop several packets in a row. TCP retransmits them, so MQTT sees stalls of seconds rather than lost messages.
• Radio wake-up delay. After a quiet period the modem drops to an idle radio state; the first packet afterwards waits for the radio to reconnect, adding noticeable latency to that one message.
• Carrier-grade NAT. Mobile operators put devices behind NAT gateways that forget idle flows. If nothing crosses the connection for longer than the NAT's idle timeout, it becomes half-open: the device thinks it is connected, the broker never hears from it again. See MQTT and NAT timeouts.
• Metered data. Every byte of headers, keep-alive and TLS handshake costs money and battery on a fleet of thousands.

Emulate cellular links with tc netem

On Linux, tc with the netem queueing discipline adds delay, jitter, loss and rate limits to an interface. The profiles below are rough starting points for typical conditions, not measurements of any particular network — replace them with numbers from your own devices when you have them.

The cleanest setup is to run the MQTT client inside a container and shape only that container's network, leaving the host untouched:

docker run --rm -it --cap-add NET_ADMIN --name cell python:3.12-slim bash

# inside the container
apt-get update && apt-get install -y iproute2 mosquitto-clients
pip install paho-mqtt

# 3G-like: 200 ms ± 50 ms, 1 % loss with 25 % correlation (bursty), 384 kbit/s
tc qdisc add dev eth0 root netem delay 200ms 50ms loss 1% 25% rate 384kbit

tc qdisc change dev eth0 root netem delay 400ms 150ms loss 3% 25% rate 64kbit   # 2G
tc qdisc show dev eth0                                                         # verify
tc qdisc del dev eth0 root                                                     # back to normal

netem shapes outgoing traffic only. On the client side that is enough for most tests: the delay applies once per round trip, and loss of outgoing segments triggers the same TCP retransmission behavior. If you need to shape the downlink separately, put netem on the broker side of a bridge or use Toxiproxy, which can delay and throttle each direction independently.

Check the profile before testing MQTT: ping -c 10 broker.connectmq.com should show the added delay, and a mosquitto_pub should still get through.

Measure delivery under each profile

This script (paho-mqtt 2.x) connects a subscriber and a publisher, sends a batch of QoS 1 messages and reports how many arrived, connect time and end-to-end latency percentiles. Run it inside the shaped container once per profile.

import json, sys, threading, time, uuid
import paho.mqtt.client as mqtt

HOST, PORT = "broker.connectmq.com", 1883
TOPIC = f"testmqtt/cellular/{uuid.uuid4().hex[:8]}"
N, QOS, SIZE = int(sys.argv[1]), int(sys.argv[2]), int(sys.argv[3])
latencies, done = [], threading.Event()

def on_message(client, userdata, msg):
    latencies.append((time.time() - json.loads(msg.payload)["t"]) * 1000)
    if len(latencies) == N:
        done.set()

def connect(name, keepalive=60):
    ready = threading.Event()
    c = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id=f"{name}-{uuid.uuid4().hex[:6]}")
    c.on_connect = lambda *args: ready.set()
    t0 = time.time()
    c.connect(HOST, PORT, keepalive=keepalive)
    c.loop_start()
    if not ready.wait(30):
        sys.exit(f"{name}: no CONNACK within 30 s")
    print(f"{name}: connected in {(time.time() - t0) * 1000:.0f} ms")
    return c

sub = connect("sub")
sub.on_message = on_message
sub.subscribe(TOPIC, qos=QOS)
pub = connect("pub")
time.sleep(2)

pad = "x" * SIZE
for i in range(N):
    pub.publish(TOPIC, json.dumps({"i": i, "t": time.time(), "pad": pad}), qos=QOS)
    time.sleep(0.2)

done.wait(60)
latencies.sort()
pct = lambda q: latencies[min(len(latencies) - 1, int(len(latencies) * q))]
print(f"received {len(latencies)}/{N} at QoS {QOS}, {SIZE}-byte padding")
if latencies:
    print(f"p50 {pct(0.5):.0f} ms  p95 {pct(0.95):.0f} ms  max {latencies[-1]:.0f} ms")

python cell_test.py 100 1 200    # 100 messages, QoS 1, ~200-byte payloads
python cell_test.py 100 0 200    # same at QoS 0 for comparison
python cell_test.py 20 1 20000   # few large payloads

Publisher and subscriber share the container's clock, so there is no clock-skew error in the latency numbers. Both connections go through the shaped interface, which is pessimistic — in production the subscriber usually sits in a data center — but it makes differences between configurations easy to see. Keep the online MQTT client subscribed to testmqtt/cellular/# on a normal connection to watch messages trickle in.

Tune keep-alive for the carrier, not the lab

Keep-alive has two jobs on cellular: keep the NAT entry alive and detect dead links. The keep-alive interval must be shorter than the carrier's NAT idle timeout, otherwise the connection silently dies between pings.

To find that timeout, connect, stay completely idle (keep-alive larger than the test window), then publish after increasing pauses — 1, 2, 5, 10, 20 minutes. The first pause after which the publish never reaches the broker (and the client only notices on the next ping or write) is your NAT limit. Set keep-alive comfortably below it.

Also test the opposite failure: under the 2G and "dead zone" profiles, make sure keep-alive does not cause disconnects. If PINGRESP regularly takes several seconds, a client with aggressive timeouts may drop healthy connections. MQTT allows the broker to wait one and a half times the keep-alive before giving up, so very short values (under 15–20 s) leave little room on slow links.

Payload size and protocol overhead

On a 64 kbit/s link, a 20 KB message occupies the uplink for a few seconds, and everything behind it waits. Under the 2G profile, compare small and large payloads with the script and you will see p95 latency climb with size.

• Keep payloads compact. Short JSON keys, or binary formats such as CBOR and Protobuf, help. The packet size calculator shows how topic length and properties add up per message.
• Batch thoughtfully. One message with ten readings costs less overhead than ten messages, but a lost connection mid-batch hurts more. Test both.
• Count TLS. A full TLS handshake costs several round trips and several kilobytes. On a 400 ms link that is seconds per connect, so frequent reconnects are expensive. Measure connect time with TLS on port 8883 under each profile, and see MQTT bandwidth overhead for the numbers behind it.
• Use MQTT 5 topic aliases if long topics are sent repeatedly, after checking that your broker supports them.

QoS and sessions on lossy links

For most cellular telemetry, QoS 1 with idempotent processing on the receiver is the pragmatic choice. Pair it with a persistent session — clean_start=False and a session expiry interval in MQTT 5 — so the device does not have to re-subscribe and messages queued for it survive a short outage. Details in MQTT persistent sessions.

Test reconnects explicitly under bad profiles: drop the link (tc qdisc change dev eth0 root netem loss 100%), wait, restore it, and check that the client reconnects with backoff instead of hammering the network, and that QoS 1 messages published during the outage arrive once the link is back.

A cellular test checklist

1. Smoke-test on a normal network first, then apply each netem profile.
2. Record connect time (with and without TLS), delivery count and p50/p95 latency per profile.
3. Find the carrier NAT idle timeout with idle-then-publish tests; set keep-alive below it.
4. Cut the link for 30 s and 5 min; verify reconnect backoff, session resume and QoS 1 redelivery.
5. Compare payload sizes and batching under the 2G profile.
6. Repeat the critical cases on real hardware with the production SIM.

Cellular networks will always be slower and less predictable than the lab. Emulating them on every change turns "works in the office" into "works in a moving truck", and the numbers from these runs give you a defensible basis for keep-alive, QoS and payload decisions.

Frequently asked questions