MQTT Fan-Out Test: One Publisher, Thousands of Subscribers

In a fan-out scenario one message goes to many receivers: a configuration change pushed to a whole fleet, a price update shown on 10,000 dashboards, a "firmware available" announcement. The publisher sends one PUBLISH; the broker sends one per matching subscription. A fan-out test measures how long it takes for the last subscriber to get the message and whether every subscriber got it at all.

This guide sets up a 1 → 10,000 benchmark, adds a probe that measures delivery spread, and works through the bandwidth math so the network bill does not surprise you. For general rate testing, see the MQTT throughput test; for the reverse pattern, the fan-in test.

Why fan-out is its own test

Publish throughput tests usually pair each publisher with one subscriber. Fan-out stresses something different: the broker's ability to copy one message into thousands of outgoing queues and flush them to thousands of sockets.

The question to answer is not "how many messages per second" but "how long until every subscriber has it" — the delivery spread — at your real fleet size.

The bandwidth math first

Do the arithmetic before you start: a fan-out test can saturate a network link long before the broker struggles. A PUBLISH on the wire is roughly fixed header + 2-byte topic length + topic + packet identifier (QoS > 0) + properties + payload. See MQTT bandwidth overhead for the details.

Example: 500-byte payload on testmqtt/fanout/broadcast (25 bytes) at QoS 0 ≈ 530 bytes per PUBLISH.

Add TCP/IP and TLS overhead on top. The last row is more than many single load-generator machines can receive, which is why fan-out tests are usually spread across several machines. The packet size calculator gives you the exact per-packet size for your topic and payload.

Step 1: open the subscribers

emqtt-bench can hold thousands of subscriber connections. Split 10,000 across four machines (or containers on different hosts) to stay clear of file-descriptor and ephemeral-port limits:

ulimit -n 65535
docker run --rm -it --ulimit nofile=65535:65535 emqx/emqtt-bench sub \
  -h broker.connectmq.com -p 1883 \
  -c 2500 -i 5 \
  -t testmqtt/fanout/broadcast -q 0 \
  -u "$MQTT_USER" -P "$MQTT_PASS"

-i 5 opens one connection every 5 ms (200 per second per machine), so the four machines need a little over 12 seconds to connect everyone. Opening connections faster is a test in itself — see the connection stress test — and should not be mixed with fan-out measurements. Wait until all four report the full count before publishing.

Step 2: publish and probe

emqtt-bench's subscriber output tells you the aggregate receive rate, but not how late the slowest subscriber was. This paho-mqtt 2.x script adds 50 probe subscribers in one process, publishes timestamped messages to the same broadcast topic, and reports latency and delivery spread across the probes. Because publisher and probes share one clock, the numbers are directly comparable:

import json, threading, time, uuid
from collections import defaultdict
import paho.mqtt.client as mqtt

HOST, PORT = "broker.connectmq.com", 1883
TOPIC = "testmqtt/fanout/broadcast"
PROBES, MESSAGES, RATE, QOS = 50, 60, 1, 0
lat = defaultdict(list)          # seq -> latencies (ms) across probes
lock = threading.Lock()

def on_message(client, userdata, msg):
    try:
        m = json.loads(msg.payload)
    except ValueError:
        return                   # ignore non-probe traffic
    with lock:
        lat[m["seq"]].append((time.perf_counter() - m["t"]) * 1000)

def new_client(name):
    return mqtt.Client(mqtt.CallbackAPIVersion.VERSION2,
                       client_id=f"{name}-{uuid.uuid4().hex[:6]}")

probes = []
for n in range(PROBES):
    c = new_client(f"fanout-probe-{n}")
    c.on_message = on_message
    c.connect(HOST, PORT)
    c.subscribe(TOPIC, qos=QOS)
    c.loop_start()
    probes.append(c)

pub = new_client("fanout-pub")
pub.connect(HOST, PORT)
pub.loop_start()
time.sleep(3)                    # let all subscriptions settle

for seq in range(MESSAGES):
    body = {"seq": seq, "t": time.perf_counter(), "pad": "x" * 450}
    pub.publish(TOPIC, json.dumps(body), qos=QOS)
    time.sleep(1 / RATE)
time.sleep(5)

pct = lambda xs, q: xs[min(len(xs) - 1, int(len(xs) * q))]
all_lat = sorted(v for vals in lat.values() for v in vals)
spread = sorted(max(v) - min(v) for v in lat.values() if len(v) > 1)
print(f"delivered {len(all_lat)}/{PROBES * MESSAGES}")
if all_lat:
    print(f"latency p50 {pct(all_lat, .5):.1f}  p99 {pct(all_lat, .99):.1f}  max {all_lat[-1]:.1f} ms")
if spread:
    print(f"spread  p50 {pct(spread, .5):.1f}  max {spread[-1]:.1f} ms")

Run it three times: with no bench subscribers (baseline), with 2,500, and with all 10,000. The baseline tells you the network's contribution; the growth from there is the broker's fan-out cost. Then repeat with QOS = 1 and with a higher RATE to find where latency starts to climb.

Reading the results

• Delivered < expected on the probes at QoS 0 means the broker dropped messages for some connections — usually because their outgoing queues were full. Retry at QoS 1 and check broker logs.
• p99 latency grows with subscriber count while p50 stays flat is normal up to a point: the broker writes to sockets one after another. What matters is whether p99 stays inside your requirement.
• Spread rises sharply at higher publish rates means the broker (or its network) cannot flush all copies before the next message arrives. You are past the sustainable fan-out rate.
• Bench machines report lower receive rates than expected — check their CPU and network first. Receiving 100,000 msg/s is real work for a load generator.
• A few subscribers fall behind and get disconnected — see slow consumers. In production, one slow device must not delay everyone else.

Designing topics for fan-out

The test usually ends with a design discussion. A few patterns hold up well:

1. Broadcast where everyone really needs it. fleet/all/config for messages that every device must receive; keep them small.
2. Group topics instead of one giant broadcast. fleet/region/eu/config or fleet/model/x200/firmware reduce fan-out per message and let you roll out gradually.
3. Retain the current state. For configuration, publish with the retain flag so devices that reconnect later get the latest value immediately — see retained messages. Test that a late subscriber receives it.
4. Pick QoS by consequence. Replaceable updates at QoS 0; commands that must arrive at QoS 1, sized so the broker's per-subscriber tracking stays affordable.
5. Avoid shared subscriptions here. They load-balance; they do not broadcast.

A short checklist

1. Calculate egress bandwidth for your target fleet and publish rate.
2. Spread subscribers across enough load machines; raise file-descriptor limits.
3. Connect all subscribers first, then publish — keep connection tests separate.
4. Measure latency and spread with probes, not just aggregate receive rates.
5. Repeat at QoS 0 and QoS 1, with and without TLS.
6. Record broker CPU, memory and network out next to the client numbers.

Fan-out turns one small message into a large burst of work and traffic. Do the bandwidth math first, connect subscribers before you publish, and judge the result by the slowest subscriber — that is the one your users will notice.

Frequently asked questions