MQTT Client Keeps Dropping or Messages Go Missing

intermediate · 12 min

A field procedure for MQTT clients that keep dropping and messages that go missing: how to tell connect failures, keepalive timeouts, duplicate client IDs, QoS downgrades and retained ghosts apart, then how to fix each one.

ℹ️ MQTT never promised your message would reach the far end. It is a per-hop transport with a store-and-forward broker in the middle, so most faults reported as "lost data" are really something else: a client the broker decided was dead, a session wiped on reconnect, or a subscriber reading a retained ghost of a device that died without a Will.

MQTT quality of service is negotiated on each hop separately — a QoS 2 publish into the broker still reaches a QoS 0 subscriber as QoS 0, because delivered QoS is min(publish, subscribe) — and the keepalive timeline shows the broker declaring a silent client dead at 1.5 × keepalive and firing its Last Will.

Tell the four failure modes apart

Sort first, fix second.

• Never connects. Either the broker returns a CONNACK (connect acknowledgement) with a non-zero code, or the socket dies before any MQTT byte moves. A code means the broker heard you and refused — go to the CONNACK table. Silence means TCP or TLS failed first — go straight to the port/TLS checks, not through the keepalive and QoS sections.
• Drops on a rhythm. Time the gap. Near 1.5x your keepalive value? Keepalive — but read the next section before you change it, because the usual fix is to lower it, not raise it. Regular and much shorter than keepalive, on a cellular, VPN or firewalled path? Suspect a NAT or stateful-firewall idle timeout. Two devices flapping against each other? Duplicate client ID.
• Connected, messages vanish. Not a connection fault. Check QoS and session flags.
• A tag on the HMI never changes. The publisher died quietly, or you are reading a retained message.

mosquitto_sub, from the Eclipse Mosquitto client package, is the first probe: run it from a laptop on the same segment. A pass proves the broker is reachable from that segment on that port — it does not clear the device, because the laptop is using a different client ID and different credentials. To actually isolate the device, run mosquitto_sub with the device's own client ID and credentials (with the device powered down, so you are not colliding with it).

CONNACK: what the broker actually said

MQTT 3.1.1 defines six return codes. Zero is the only success.

Codes 4 and 5 are worth separating, but the split is broker-dependent, not a protocol guarantee. The intent is that 4 is credentials and 5 is authorisation, so on a broker that follows the intent, a code 5 means the identity is real but the ACL says no and retyping the password will not help. Confirm against your broker before you send anyone down that path: Mosquitto returns 5 when allow_anonymous false and the client supplied no credentials at all, and some brokers deliberately return 5 for bad credentials so they do not leak which usernames exist. The broker log tells you which case you are in.

MQTT 5.0 replaces these with reason codes: 0x00–0x7F is success, 0x80 and above is failure. Common ones are 0x85 client identifier not valid, 0x87 not authorised, 0x8C bad authentication method, 0x95 packet too large.

Keepalive: how the broker decides you are gone

Keepalive is a value in seconds the client puts in its CONNECT packet — a 16-bit field, so 65535 seconds is the ceiling.

Two halves of one rule, and most bad advice comes from conflating them:

• Client side. The client must ensure the gap between control packets it sends never exceeds Keep Alive. In the absence of anything else to send, the client MUST send a PINGREQ; the broker answers PINGRESP. Any packet resets the timer — a PUBLISH counts.
• Broker side. If keepalive is non-zero and the broker receives no packet from that client within 1.5x keepalive, it must close the connection as if the network had failed. The 1.5x is grace for network delay.

This means a publish interval longer than keepalive is not a fault. A conformant client publishing every 60 seconds with keepalive 30 pings at ~30 seconds and stays up indefinitely. If that setup drops on schedule, the defect is in the client: a hand-rolled MQTT stack with no PINGREQ, or a deep-sleep MCU that stops servicing its network loop between publishes. Fix the ping, don't paper over it by inflating keepalive.

Raising keepalive is usually the wrong direction anyway. On cellular, NAT and firewalled paths the common cause of clockwork drops is a NAT or stateful-firewall idle timeout shorter than your keepalive — the middlebox silently tears the flow down, the broker learns nothing until 1.5x keepalive elapses, and publishes in between vanish into a half-open socket. There the fix is to set keepalive below the middlebox timeout. Shorter keepalive also means faster dead-peer detection and a faster Will; the cost is more PINGREQ traffic, which matters only on metered links.

Second classic fault: a shared client ID. The specification says the broker MUST disconnect the existing client when a new one connects with the same ID, so two devices flashed from one image knock each other offline forever — a fast, regular flap, one device's connect matching the other's disconnect in the log. Enforcement is reliable on a single broker; in clustered brokers, client-ID tracking is often node-local, so two devices landing on different nodes both stay connected and neither takeover fires. Derive every client ID from a serial number or MAC.

The broker log names the client ID and the disconnect reason. On Mosquitto, connect/disconnect logging is the connection_messages option (default true) — log_type selects severity categories (error, warning, notice, information, debug, all), which is a different knob. Live counters sit under $SYS/ — $SYS/broker/clients/connected and $SYS/broker/publish/messages/dropped.

Last Will and Testament

The Will is a message you hand the broker inside CONNECT: topic, payload, QoS, retain flag. The broker publishes it when the connection ends without a clean shutdown — network failure, keepalive expiry, protocol error, or the broker closing the socket. A DISCONNECT with reason code 0x00 discards it: you left on purpose. Note the MQTT 5.0 exception — DISCONNECT reason code 0x04, "Disconnect with Will Message", closes the connection normally and still fires the Will.

MQTT 5.0 also adds a Will Delay Interval, which defers publication so a client that reconnects inside the delay never triggers its Will. Useful for flaky links; misleading if you forget you set it and wonder why the offline alarm is late.

A client with no Will fails silently. It stops publishing, nothing announces it, and every consumer keeps displaying the last value it received. An operator reads a dead tank level as a live one.

So set one on every publisher: topic plant/line1/status, payload offline, retain true, plus a retained online after connecting. Retained is right here because the status topic is a state, not an event.

Sparkplug B is the exception, not the confirmation. It formalises the pattern — NDEATH is the registered Will, NBIRTH republishes the node's metric set on reconnect — but with the opposite flags: Sparkplug 3.0 requires the NDEATH Will to be registered with retain=false and QoS=0, and requires every non-STATE Sparkplug message to be published with retain false. Do not carry the retained-Will advice above into a Sparkplug node.

QoS guarantees less than you think

QoS is per hop, not end to end. The publisher's QoS covers publisher-to-broker only. Broker-to-subscriber runs at the QoS that subscriber was granted, and the delivered level is the minimum of the two: publish at 2, subscribe at 0, and that subscriber gets QoS 0. The broker never upgrades a message above the level it was published at.

• QoS 0 — fire and forget. One attempt, no acknowledgement.
• QoS 1 — at least once. Retried until PUBACK. Duplicates are legal and will happen.
• QoS 2 — exactly once on that hop, via a four-packet handshake (PUBLISH, PUBREC, PUBREL, PUBCOMP).

None of these bound when delivery happens. Retransmission of unacknowledged PUBLISH and PUBREL packets is required on reconnect into a resumed session — clean session false in 3.1.1, or Clean Start 0 with Session Present 1 in 5.0 — not on a timer during a live session. Connect with clean session true and there is no session to resume and nothing to resend.

Retained, sessions, ports, topics

Retained. The retain flag makes the broker keep the last message per topic and hand it to a matching subscriber at subscribe time. That is why a new subscriber sees old data — the broker is doing what you asked. (MQTT 5.0 subscribers can change this with the Retain Handling option: 0 send on every subscribe, 1 only on a new subscription, 2 never.) Retained values carry no freshness, so put a timestamp in the payload — or set a Message Expiry Interval on MQTT 5.

Clear a retained value with a zero-byte publish with retain set. Be precise about what that does: the empty message is delivered immediately, as a normal message, to every currently matching subscriber, and then the stored value is removed. Live HMIs blank at once, not at their next reconnect. Do it on a running line and someone is watching a field go empty.

Sessions. Clean session true (MQTT 5: Clean Start with Session Expiry 0) discards subscriptions and queued messages. False keeps them and queues QoS 1 and 2 while the client is away. A client reconnecting with clean session true loses that queue silently — and, per the section above, resends nothing.

Ports. 1883 is registered with IANA for plain MQTT over TCP; 8883 is registered for MQTT over TLS. On 8883 the TLS handshake finishes before any MQTT byte moves, so a TLS failure produces no CONNACK at all — you get a closed socket and no protocol-level explanation. Test with openssl s_client -connect <broker>:8883. Usual causes: expired server certificate, a CA the client does not trust, or a device clock wrong enough that a valid certificate looks expired or not-yet-valid.

Topics. + matches exactly one level and must fill that level entirely; # matches many and must be the last character of the filter. A filter starting with a wildcard never matches topics beginning with $, so # will not pull in $SYS/ — subscribe to $SYS/# explicitly. Never put values in topic names: plant/line1/temp/72 mints a new topic per reading, bloating the broker's topic tree and subscription matching — and if those publishes are retained, growing the retained store without limit.

Step-by-step

1. Reproduce with mosquitto_sub -h <broker> -p 1883 -t 'plant/#' -v on a laptop on the same segment. If the fleet uses TLS, reproduce on 8883 with the device's CA instead — testing the wrong port proves nothing.
2. No CONNACK at all? Stop here and work the transport: TCP reach to the port, then openssl s_client -connect <broker>:8883 for certificate chain and dates, then the device's clock. Do not continue down this list until a CONNACK appears.
3. Read the CONNACK code in the client log. Non-zero: fix per the table — and read the broker log, which says which of credentials, ACL or client ID it actually rejected.
4. Re-run the probe using the failing device's own client ID and credentials, with the device powered down. Passing there points at the device's stack; failing points at credentials or ACL.
5. Confirm every device has a unique client ID. Check the broker log for a connect from one client ID matching a disconnect of the same ID — that is a takeover flap, not a network fault.
6. For rhythmic drops, measure the interval before changing anything. Near 1.5x keepalive means the broker timed the client out — verify the client actually emits PINGREQ when idle (packet capture, or the broker log's ping traffic) before touching any interval. Shorter than keepalive and regular means a NAT or firewall idle timeout; set keepalive below that timeout.
7. Watch $SYS/broker/clients/connected and $SYS/broker/publish/messages/dropped across the failure window.
8. Verify every publisher sets a Will and a retained online — with Sparkplug's flags if this is a Sparkplug node.
9. For missing messages, check the subscriber's granted QoS, not only the publisher's, and check clean session / Clean Start and Session Expiry at both ends.
10. Restarting a broker or client stops publishing, and consumers without a Will keep showing stale values. Announce it, then restart.

Never do this

Never close a safety or interlock loop over MQTT. It is a telemetry transport over TCP: no deadline, no jitter budget, no bounded delivery time. QoS 2 guarantees exactly-once, not on-time, and only for one hop. Brokers queue, networks retry, and a message can arrive minutes late and still be protocol-correct. Interlocks, E-stops and closed-loop control stay in the PLC on a deterministic bus. Publish the result to MQTT for people to read.

Second: never raise keepalive to "fix" a drop you have not measured. It delays dead-peer detection, delays the Will, and on a NAT'd link guarantees a longer window of half-open socket during which publishes disappear with no error anywhere.

Third: never ship a fleet with retained values and no Will. Every screen keeps showing the last good number after the device dies, so total failure looks like a steady process.