How to Read MQTT Benchmark Results Without Fooling Yourself

An MQTT benchmark produces numbers quickly. Producing numbers you can trust is harder. Most misleading results come from the same handful of mistakes: quoting averages, measuring the load generator instead of the broker, ignoring coordinated omission and comparing runs that were not set up the same way.

This guide walks through each pitfall, shows how to detect it in your own data and ends with a reporting template that makes results reproducible. It assumes you have already run a test — if not, start with MQTT broker load testing.

Averages hide the slow messages

Latency is never normally distributed. A broker under load delivers most messages in a few milliseconds and a small fraction in hundreds of milliseconds — during a garbage-collection pause, a disk flush or a retransmission. The mean mixes both groups into a number that describes neither.

(Illustrative values.) Both runs have the same average, but run B means one message in a hundred arrives more than a tenth of a second late. With a device publishing once per second, that is a visible delay roughly every two minutes.

Rules that keep you honest:

• Report percentiles: p50 (typical), p99 (the tail), max (the worst case you observed). Add p99.9 only when you have enough samples.
• Never average percentiles across runs or clients. The mean of ten p99 values is not the p99 of the combined data — merge the raw samples or histograms instead.
• Check sample size. p99 of 100 messages is one message. Collect thousands per run.

Coordinated omission

Coordinated omission is the most common reason a benchmark looks better than reality. It happens when the load generator waits for a response before sending the next request. If the broker stalls for 500 ms, a synchronous publisher records one slow message — and then simply doesn't send the 49 others that a real fleet would have sent during that half second. Those messages, which would all have seen high latency, never appear in the data.

Real devices do not coordinate with the broker. A sensor publishes on its own schedule whether the last PUBACK arrived or not. To reproduce that, measure latency from the intended send time, not from the moment the message was actually handed to the socket:

import json, statistics, time, uuid
import paho.mqtt.client as mqtt

HOST, PORT = "broker.connectmq.com", 1883
TOPIC = f"testmqtt/bench-{uuid.uuid4().hex[:8]}/latency"
N, RATE = 20000, 500                   # messages, messages per second
samples = []

def on_message(client, userdata, msg):
    intended = json.loads(msg.payload)["t"]
    samples.append((time.perf_counter() - intended) * 1000)

sub = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2)
sub.on_message = on_message
sub.connect(HOST, PORT)
sub.subscribe(TOPIC, qos=1)
sub.loop_start()

pub = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2)
pub.max_queued_messages_set(0)         # 0 = unlimited client-side queue
pub.connect(HOST, PORT)
pub.loop_start()
time.sleep(1)

start = time.perf_counter()
for i in range(N):
    intended = start + i / RATE        # the schedule a real device would follow
    delay = intended - time.perf_counter()
    if delay > 0:
        time.sleep(delay)
    pub.publish(TOPIC, json.dumps({"i": i, "t": intended}), qos=1)

time.sleep(5)
q = statistics.quantiles(samples, n=1000)   # 999 cut points
print(f"received {len(samples)}/{N}")
print(f"p50 {q[499]:.1f} ms  p99 {q[989]:.1f} ms  p99.9 {q[998]:.1f} ms  max {max(samples):.1f} ms")

If the publisher falls behind schedule — because the broker or the network pushed back — every late message carries that delay in its measurement. Compare the output with a version that stamps time.perf_counter() at publish time: if the intended-time p99 is much higher, the broker was stalling and the naive measurement was hiding it.

Are you measuring the broker or the client?

A benchmark machine that runs out of CPU, sockets or network bandwidth produces a plateau that looks exactly like a broker limit. Before you blame the broker, check the load generator:

• CPU on the generator. If one core sits at 100 % (common with single-threaded clients such as one paho network loop handling hundreds of messages per second), the ceiling is yours. Spread clients across processes or machines.
• Round-number plateaus. Connection counts that stop at 1,024, 4,096 or around 28,000–64,000 usually mean file-descriptor or ephemeral-port limits on the client host. Raise ulimit -n or add source IPs.
• Network interface. Multiply message rate by on-the-wire size (the packet size calculator gives you the exact bytes per PUBLISH) and compare with the link capacity — including TLS overhead.
• The double-machine test. Run the same load from two generators at half the rate each. If total throughput rises, the single generator was the bottleneck.

The broker side tells the rest of the story: record its CPU, memory, connection count and message rates (for example from $SYS topics or Prometheus) next to the client-side numbers. A broker at 30 % CPU with a stuck throughput graph is waiting on something else.

Clocks, warm-up and other quiet distortions

• Clock skew. If the publisher and subscriber run on different machines, end-to-end latency includes the difference between their clocks. NTP keeps clocks within a few milliseconds at best — often worse. Measure with publisher and subscriber in one process, or use round trips. More in MQTT clock and timestamp issues.
• Warm-up. The first seconds include TCP and TLS handshakes, JIT compilation and cache misses. Discard a warm-up period or report it separately.
• Too-short runs. A 30-second run can miss periodic events such as persistence snapshots or log rotation. Hold each load level for minutes; use a soak test for hours.
• Lost messages. Latency percentiles calculated only from messages that arrived look great if the slow ones were dropped. Always report received / sent next to latency.
• QoS mismatch. QoS 0, 1 and 2 have different packet counts and very different costs; never compare results across QoS levels as if they were the same test. See QoS 0 vs 1 vs 2 benchmark.

Spotting a bad result

How to report MQTT benchmark results

A number without its setup cannot be reproduced or compared. Report at least this:

Two habits make reports far more useful:

1. Plot latency over time. A histogram hides when the slow messages happened. A time series shows whether the tail is a constant property or a few events.
2. Show the saturation point. Instead of one number, run increasing load levels and report the rate at which p99 crosses your target (for example 50 ms). That rate is your usable capacity; the absolute maximum throughput is not.

A quick sanity check before every run

Before starting a long benchmark, connect with the online MQTT client, subscribe to your test topic and publish a handful of messages by hand. If one message takes visibly long or the subscription does not receive anything, fix that first — a load test won't tell you more.

Smoke-test your benchmark topic

Benchmarks are only as good as the questions behind them. Measure from the intended send time, keep an eye on the load generator, report percentiles together with delivery counts and the full setup, and your MQTT numbers will survive the next person who tries to reproduce them — including future you.

Frequently asked questions