Running MQTT Client Apps on Kubernetes

Running an MQTT client on Kubernetes looks like any other Deployment until you scale it to two replicas. Then the pods start kicking each other off the broker, every replica processes every message, and rolling updates drop data. Three rules fix nearly all of it: give every pod a unique client ID, use shared subscriptions to split work between replicas, and wire the MQTT connection into readiness, not liveness.

This tutorial builds a small Node.js consumer (MQTT.js 5) and a Deployment manifest that applies all three, plus graceful shutdown and a test you can run from your browser.

Why MQTT clients behave differently on Kubernetes

HTTP services are stateless: any pod can answer any request and the Service load-balances connections. MQTT clients are the opposite. Each pod opens one long-lived connection out to the broker, and the broker — not Kubernetes — decides which client receives which message.

Rule 1: one client ID per pod

MQTT allows exactly one active session per client ID. When a second connection arrives with the same ID, the broker closes the first one (MQTT 5 reason code 0x8E Session taken over). With two replicas configured identically, they take turns disconnecting each other forever — see session taken over for the symptoms.

The cleanest fix is the downward API: inject the pod name as an environment variable and build the ID from it. Pod names are unique within a namespace and stable for the lifetime of the pod.

env:
  - name: POD_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.name

In the app, clientId = "ingest-" + process.env.POD_NAME. Keep IDs short and within the characters your broker accepts; the client ID generator and client ID best practices cover the limits.

Rule 2: shared subscriptions split the work

With a normal subscription, three replicas of the same consumer each receive every message, so you store each reading three times. A shared subscription puts the replicas in a group, and the broker delivers each message to only one member:

$share/<group>/<topic filter>
$share/ingest/testmqtt/k8s/+/telemetry

Every pod subscribes to the same $share/ingest/... filter. Scaling the Deployment up adds group members; scaling down removes them, and the broker rebalances on its own. Shared subscriptions are an MQTT 5 feature; many brokers also accept the $share/ prefix from MQTT 3.1.1 clients, but check yours. The full mechanics, including how brokers pick a member, are in MQTT shared subscriptions.

Keep two things in mind: ordering is only guaranteed per member, not across the group, and if one message must reach every pod (a config reload, for example) use a separate plain subscription for that topic.

The consumer app

The app connects with a pod-specific ID, joins the shared group and exposes /healthz (process alive) and /readyz (connected to the broker) on port 8080.

import http from 'node:http';
import mqtt from 'mqtt';

const pod = process.env.POD_NAME ?? `local-${Math.random().toString(16).slice(2, 8)}`;
const client = mqtt.connect(process.env.MQTT_URL ?? 'mqtts://broker.connectmq.com:8883', {
  clientId: `ingest-${pod}`,
  username: process.env.MQTT_USER,
  password: process.env.MQTT_PASS,
  protocolVersion: 5,
  clean: true,
  keepalive: 30,
  reconnectPeriod: 2000 + Math.floor(Math.random() * 3000), // jitter between pods
});

let ready = false;

client.on('connect', async () => {
  await client.subscribeAsync('$share/ingest/testmqtt/k8s/+/telemetry', { qos: 1 });
  ready = true;
  console.log(`[${pod}] connected and subscribed`);
});
client.on('close', () => { ready = false; });
client.on('error', (err) => console.error(`[${pod}] mqtt error:`, err.message));

client.on('message', (topic, payload) => {
  // Real work goes here: write to a database, call an API, ...
  console.log(`[${pod}] ${topic} ${payload.toString()}`);
});

http.createServer((req, res) => {
  if (req.url === '/healthz') return res.writeHead(200).end('ok');
  if (req.url === '/readyz') return res.writeHead(ready ? 200 : 503).end(ready ? 'ready' : 'not connected');
  res.writeHead(404).end();
}).listen(8080);

process.on('SIGTERM', async () => {
  console.log(`[${pod}] SIGTERM: disconnecting`);
  ready = false;
  await client.endAsync(); // waits for in-flight messages, then sends DISCONNECT
  process.exit(0);
});

The random reconnectPeriod matters: when the broker restarts, fifty pods reconnecting on the same tick produce a spike that can knock it over again. See reconnect storms for backoff strategies.

Rule 3: probes that do not make outages worse

• Readiness — returns 503 while the MQTT connection is down. Kubernetes stops counting the pod as available, so a rollout pauses instead of replacing healthy pods with disconnected ones.
• Liveness — only checks that the process answers. If the broker is unreachable, restarting the pod does not fix it, and restarting every pod at once turns an outage into a reconnect storm.
• Startup (optional) — gives slow-starting apps time before liveness kicks in.

A worker with no HTTP traffic does not need a Service; the readiness probe still controls rollouts and kubectl get pods shows at a glance which replicas are connected.

The Deployment manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingest
spec:
  replicas: 3
  selector:
    matchLabels: { app: ingest }
  strategy:
    rollingUpdate: { maxUnavailable: 0, maxSurge: 1 }
  template:
    metadata:
      labels: { app: ingest }
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: ingest
          image: ghcr.io/your-org/mqtt-ingest:1.0.0
          ports: [{ containerPort: 8080 }]
          env:
            - name: POD_NAME
              valueFrom: { fieldRef: { fieldPath: metadata.name } }
            - name: MQTT_URL
              value: mqtts://broker.connectmq.com:8883
            - name: MQTT_USER
              valueFrom: { secretKeyRef: { name: mqtt-credentials, key: username } }
            - name: MQTT_PASS
              valueFrom: { secretKeyRef: { name: mqtt-credentials, key: password } }
          readinessProbe:
            httpGet: { path: /readyz, port: 8080 }
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:
            httpGet: { path: /healthz, port: 8080 }
            periodSeconds: 10
            failureThreshold: 3
          resources:
            requests: { cpu: 50m, memory: 64Mi }
            limits: { memory: 256Mi }

Create the Secret once with kubectl create secret generic mqtt-credentials --from-literal=username="$MQTT_USER" --from-literal=password="$MQTT_PASS". Never bake credentials into the image.

Graceful shutdown and rolling updates

When a pod is terminated, Kubernetes sends SIGTERM, waits up to terminationGracePeriodSeconds, then sends SIGKILL. The handler above marks the pod unready, lets MQTT.js finish in-flight acknowledgements and sends a clean DISCONNECT. Two consequences:

• With a clean DISCONNECT, the broker does not publish the client's Last Will. Use the will for crashes, not for planned restarts — see Last Will and Testament.
• With QoS 1, a message that was delivered but not yet acknowledged when the pod died is redelivered to another group member (behaviour varies by broker). Make your handler idempotent so a duplicate does no harm.

maxUnavailable: 0 with maxSurge: 1 means a new pod must be ready — that is, connected — before an old one is removed, so the shared group never shrinks during a rollout.

Where the broker runs

The client pods do not care whether the broker is inside the cluster or outside it; they only need egress on the right port (8883 for TLS). Running the broker in the cluster gives you one more stateful workload to operate — Mosquitto on Kubernetes shows what that involves. A managed broker such as ConnectMQ keeps the cluster stateless: the same manifest works in every environment, only the URL and Secret change.

If your cluster has a restrictive NetworkPolicy, allow egress to the broker host and port explicitly; a blocked port shows up as connection timeouts in the logs and a pod that never becomes ready.

Testing it

1. Apply the manifest and wait for kubectl get pods -l app=ingest to show three ready pods.
2. Open the online MQTT client, connect to broker.connectmq.com, and publish a few messages to testmqtt/k8s/dev-1/telemetry, testmqtt/k8s/dev-2/telemetry and so on.
3. Run kubectl logs -l app=ingest --prefix -f. Each message should appear in exactly one pod's log, spread across the replicas.
4. Run kubectl rollout restart deployment/ingest while publishing. Readiness should hold the rollout until new pods are connected, and no message should go missing at QoS 1.
5. Scale to 1 and back to 3 and confirm the messages redistribute.

Publish test telemetry from your browser

The shared test endpoint does not keep persistent sessions, so to test StatefulSet-style sessions with queued messages run the same steps against your own private broker instance.

Kubernetes treats pods as interchangeable; MQTT brokers treat client IDs as identities. Bridge the two with pod-derived IDs, shared subscriptions and readiness-driven rollouts, and your consumers scale like any other stateless service.

Frequently asked questions