To test an MQTT Last Will, you have to make the client die badly. The broker publishes the will only when a connection ends without a proper DISCONNECT — so stopping your app normally proves nothing. Kill the process with kill -9, freeze it with kill -STOP, or cut the network, then watch the will topic from a second client and note when the will arrives. A good test also includes the negative case: a graceful shutdown must not fire the will.
This guide gives you a small device script with a will, an observer, the fault injections that trigger each code path, and MQTT 5 will-delay tests.
When the broker publishes the will
| Event | Will published? | Typical timing |
|---|---|---|
Process crash (kill -9, segfault, OOM kill) | Yes | Almost immediately — the OS closes the socket |
| Frozen process, socket still open | Yes | After 1.5 × keep-alive without traffic |
| Network cut, power loss, cable pulled | Yes | After 1.5 × keep-alive (no FIN or RST arrives) |
| Protocol error, broker closes the connection | Yes | Immediately |
Graceful DISCONNECT (reason code 0) | No | — |
MQTT 5 DISCONNECT with reason 0x04 | Yes | Immediately (plus will delay) |
| MQTT 5 with Will Delay Interval | Yes, unless the client reconnects in time | Delay expiry or session end, whichever is first |
The keep-alive row is why "my will takes ages" is so common: with a 60-second keep-alive, a dead cellular device is reported after roughly 90 seconds. That is correct behavior, not a bug — see keep alive for choosing the interval. The concept itself is covered in MQTT Last Will.
The device under test
This paho-mqtt 2.x script behaves like a well-designed device: it registers a retained offline will, publishes a retained online on connect, and on SIGTERM shuts down gracefully (publishes offline itself and disconnects cleanly). That gives you one process with a clean exit path and several dirty ones.
import os, signal, sys, threading
import paho.mqtt.client as mqtt
from paho.mqtt.packettypes import PacketTypes
from paho.mqtt.properties import Properties
DEV = sys.argv[1] if len(sys.argv) > 1 else "dev1"
WILL_DELAY = int(os.environ.get("WILL_DELAY", "0")) # seconds, MQTT 5
STATUS = f"testmqtt/lwt-test/{DEV}/status"
client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2,
client_id=f"testmqtt-lwt-{DEV}", protocol=mqtt.MQTTv5)
# client.username_pw_set("username", "password")
will_props = Properties(PacketTypes.WILLMESSAGE)
will_props.WillDelayInterval = WILL_DELAY
client.will_set(STATUS, "offline", qos=1, retain=True, properties=will_props)
def on_connect(client, userdata, flags, reason_code, properties):
print(f"connected: {reason_code}, session present: {flags.session_present}", flush=True)
client.publish(STATUS, "online", qos=1, retain=True)
client.on_connect = on_connect
stop = threading.Event()
signal.signal(signal.SIGTERM, lambda *_: stop.set())
connect_props = Properties(PacketTypes.CONNECT)
connect_props.SessionExpiryInterval = 300 # must be >= will delay
client.connect("broker.connectmq.com", 1883, keepalive=10,
clean_start=False, properties=connect_props)
client.loop_start()
print("pid", os.getpid(), flush=True)
while not stop.is_set():
stop.wait(1)
# graceful shutdown: report offline ourselves, then a normal DISCONNECT (no will)
client.publish(STATUS, "offline", qos=1, retain=True).wait_for_publish(5)
client.disconnect()
client.loop_stop()In a second terminal, start an observer that prints a timestamp with every message:
mosquitto_sub -h broker.connectmq.com -p 1883 -t 'testmqtt/lwt-test/#' -F '%I %r %t %p'%I is an ISO-8601 timestamp and %r the retain flag: the will arrives as a live message (0), while a stale offline from an earlier run shows up with 1 right after you subscribe. You can also watch testmqtt/lwt-test/# in the online MQTT client, which is handy when you cut the network on the machine running the device.
Scenario 1: crash with kill -9
python device.py dev1 & # prints: pid 41237
kill -9 41237The observer prints offline within a second or so. SIGKILL cannot be caught, so the client never sends DISCONNECT; the kernel closes the TCP connection and the broker treats that as an ungraceful disconnect. An out-of-memory kill or a segfault behaves the same way.
Scenario 2: graceful shutdown (negative control)
python device.py dev1 & PID=$!
sleep 3; kill -TERM "$PID"The observer shows offline exactly once — published by the device itself, not by the broker. If you see a second offline shortly after, the client exited without a clean DISCONNECT (for example, the process ended before the disconnect was flushed). Getting this case right matters as much as the crash case: a will that fires on every deploy floods dashboards with false alarms.
Scenario 3: freeze and network cut
A frozen process keeps its socket open, so only keep-alive can detect it:
python device.py dev1 & PID=$!
sleep 3; kill -STOP "$PID" # paho's network thread stops: no PINGREQ
# observer: "offline" after ~15 s (1.5 x keep-alive of 10 s)
kill -CONT "$PID" # the client wakes up, reconnects and publishes "online"For a real network cut, drop the traffic on the device machine. Run the observer elsewhere (or in the browser on the secure WebSocket port), because these rules also block your local mosquitto_sub:
BROKER_IP=$(dig +short broker.connectmq.com | head -1)
sudo iptables -I OUTPUT -p tcp -d "$BROKER_IP" --dport 1883 -j DROP
sudo iptables -I INPUT -p tcp -s "$BROKER_IP" --sport 1883 -j DROP
# ... wait for the will, then restore:
sudo iptables -D OUTPUT -p tcp -d "$BROKER_IP" --dport 1883 -j DROP
sudo iptables -D INPUT -p tcp -s "$BROKER_IP" --sport 1883 -j DROPIn containers, docker pause reproduces the freeze and docker network disconnect the cut. For repeatable CI runs, put Toxiproxy between client and broker and use its timeout toxic.
Measure the gap between the last online and the offline in the observer output. If it is much longer than 1.5 × your keep-alive, the client is negotiating a different keep-alive than you think, or a proxy or load balancer in between keeps the TCP connection alive on the broker side.
Scenario 4: MQTT 5 will delay
The Will Delay Interval lets a device survive a short blip without announcing itself offline. The broker publishes the will when the delay expires or the session ends, whichever comes first, and skips it entirely if the client reconnects in time. Test both outcomes:
# A: crash and come back within the delay -> no will
WILL_DELAY=30 python device.py dev2 & PID=$!
sleep 3; kill -9 "$PID"
sleep 10
WILL_DELAY=30 python device.py dev2 & PID=$! # "session present: True", no offline
# B: crash and stay away -> will about 30 s after the broker notices the loss
sleep 3; kill -9 "$PID"If test A still produces an offline, check three settings: the reconnect must use the same client ID, Clean Start must be off (a clean start ends the old session, which fires the will at once), and the session expiry must be at least the will delay — with the MQTT 5 default of 0 the session ends at disconnect and the delay is ignored. The will delay interval guide explains the interaction in detail.
Automate the crash test
import signal, subprocess, threading, time
import paho.mqtt.client as mqtt
TOPIC = "testmqtt/lwt-test/ci/status"
def test_will_is_published_on_crash():
got = threading.Event()
def on_message(client, userdata, msg):
if msg.payload == b"offline" and not msg.retain: # live will, not stale state
got.set()
obs = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2)
obs.on_message = on_message
obs.connect("broker.connectmq.com", 1883)
obs.subscribe(TOPIC, qos=1)
obs.loop_start()
dev = subprocess.Popen(["python", "device.py", "ci"])
time.sleep(3) # connect and publish "online"
dev.send_signal(signal.SIGKILL)
t0 = time.monotonic()
try:
assert got.wait(10), "Last Will not received within 10 s"
print(f"will arrived after {time.monotonic() - t0:.1f} s")
finally:
obs.publish(TOPIC, b"", qos=1, retain=True).wait_for_publish(5) # clear status
obs.disconnect()
obs.loop_stop()Add the graceful case as a second test that sends SIGTERM and asserts that exactly one offline arrives. If the crash test fails, work through Last Will not sent: wrong will topic, an ACL that denies the will topic, or a broker that silently drops it. Pair the will with a retained online message as shown here to build reliable presence detection.
A Last Will is only tested once you have seen it fire for a crash, a freeze and a network cut — and seen it stay quiet for a clean shutdown. Record the timings next to your keep-alive and will delay settings, and you will know exactly how long your dashboards take to notice that a device is gone.
Frequently asked questions
Why is my MQTT Last Will not sent when I stop the client?
Because the client stopped gracefully. A Last Will is published only when the connection ends without a normal DISCONNECT: a crash, a network failure or a keep-alive timeout. Pressing Ctrl+C in an app that calls disconnect() is a clean exit, so the broker discards the will. Use kill -9 or cut the network to test it.
How long does it take for the broker to publish the Last Will?
If the client process dies, the operating system closes the socket and the will is published almost immediately. If the network silently disappears, the broker waits until 1.5 times the keep-alive interval passes without traffic. In MQTT 5 a Will Delay Interval adds further delay on top.
Why is my will published even though the client reconnected within the will delay?
The session ended. In MQTT 5 the will is published when the delay expires or when the session ends, whichever comes first. If the session expiry interval is 0, or the client reconnects with Clean Start set, the old session ends and the will goes out. Reconnect with the same client ID, Clean Start off and a session expiry at least as long as the will delay.
