MQTT Fundamentals for the Unified Namespace

What Do You Actually Need to Know About MQTT to Run a Unified Namespace?

Four MQTT mechanisms carry almost all the weight in a unified namespace: broker-mediated publish/subscribe decoupling, three quality-of-service levels that trade bandwidth for delivery certainty, retained messages, and the Last Will and Testament plus session machinery that tells you whether a publisher is still alive. Everything else in the OASIS specification is refinement on those four ideas. Retained messages are the literal mechanism behind a claim I made in the first article of this series: a subscriber connecting at 3 a.m. sees the whole plant instantly. This article explains, packet by packet, how that promise is actually kept.

For cold readers, the series so far in one paragraph: the first article defined a unified namespace as a broker, a semantic hierarchy, and live retained state; the second showed why the interfaces a UNS replaces fail hardest at ISA-95 level boundaries; the third built the full Westbridge topic tree and deferred every protocol question to here. This article stops deferring and covers what MQTT itself guarantees on the wire.

I anchor every claim to a version. OASIS MQTT Version 3.1.1 (2014-10-29, also published as ISO/IEC 20922:2016) is what most installed industrial gear still speaks. OASIS MQTT Version 5.0 (2019-03-07) is what a new deployment should target. Where a behavior is version-specific, I say so, rather than let a 5.0 feature masquerade as universal.

Key Takeaways

• Four mechanisms do almost all the work: broker-mediated pub/sub, three QoS levels, retained messages, and Last Will plus session state.
• QoS 0, 1, and 2 are not a quality ladder to climb. Each is a genuine, deliberate design choice, and the effective QoS a subscriber gets is the LOWER of the publisher’s QoS and its own granted subscription QoS.
• MQTT’s delivery guarantee is per hop (client to broker), never end to end across a bridge or a chain of consumers.
• A retained value carries no liveness information on its own. Pair it with a Last Will and a connection-status topic, or a dashboard will render a dead machine as live.
• MQTT, AMQP, and Kafka each win in a different part of the same plant. The honest answer to “which protocol” is “all three, in different places.”

Where MQTT Came From, and Why the Origin Story Is Load-Bearing

MQTT was created in 1999 by Andy Stanford-Clark, then at IBM, and Arlen Nipper, then at Arcom (later part of Eurotech; Nipper went on to co-found Cirrus Link Solutions), to move SCADA telemetry from remote oil pipeline sites over expensive, low-bandwidth satellite links. Every design decision that makes MQTT good for a UNS traces to that brief: a two-byte minimum fixed header, publish on change instead of poll on a timer, a broker that holds last-known state so an offline client can recover it in one message, and a will message so the far end can be declared dead without a human noticing.

<!– [UNIQUE INSIGHT: personal experience] –>
That history is not trivia: it is why MQTT outfits a plant edge better than protocols built for stable datacenter networks.

What This Article Deliberately Does Not Cover

This article is about what MQTT itself guarantees, so later articles can argue about what to build on top of it. Sparkplug B’s namespace and birth/death lifecycle sit on plain MQTT and get their own article; broker selection, clustering, and high availability get their own article; edge gateways and OPC UA conversion get their own article; payload schema design gets its own article; TLS, authentication, and topic access control get their own article. None gets more than the sentence above here.

How Does MQTT Publish/Subscribe Decoupling Actually Work?

The exchange is short and worth walking once. A client opens a TCP connection and sends CONNECT; the broker answers CONNACK. A subscriber sends SUBSCRIBE with one or more topic filters and a requested maximum QoS; the broker answers SUBACK with a granted QoS per filter, which may be lower than requested. A publisher sends PUBLISH to a topic name; the broker matches that name against every current subscription and fans the message out. The publisher never names a recipient and never learns whether anyone subscribed, which is what makes adding the eleventh consumer free: one new SUBSCRIBE, zero changes anywhere else.

The failure modes catalogued in the second article of this series, schema drift, unowned interfaces, cascading outages, all require the publisher and consumer to know about each other directly. Pub/sub removes that requirement at the transport layer.

Decoupled in Space, Time, and Synchronization

Three standard decouplings apply, each grounded at Westbridge. Decoupled in space: the checkweigher on blst-01 does not know the MES, the historian, or the OEE dashboard exist, let alone their network addresses. Decoupled in time: the historian can be offline during a publish and still get the value later, through retained state or a persistent session. Decoupled in synchronization: publishing never blocks the checkweigher’s edge client waiting on a consumer to acknowledge, which matters because that client sits next to a machine running a live cycle.

The honest caveat: transport decoupling is not semantic decoupling. Every subscriber still depends on the payload shape it receives, so changing a field name or a unit still breaks every consumer that hardcoded it, exactly as in a point-to-point interface. Pub/sub solves the wiring problem, not the contract problem, which is why payload design gets its own article.

Wildcards: Subscribing to the Westbridge Tree with + and

MQTT gives a subscriber two wildcards, legal only in subscription filters and never in a published topic name. + stands in for exactly one topic segment. # stands in for any number of remaining segments and must be the last character of the filter. At Westbridge: auralis/westbridge/blister-packaging/blst-01/# subscribes to everything on that line; auralis/westbridge/+/+/checkweigher/status/oee collects the OEE of every checkweigher in every area; auralis/westbridge/blister-packaging/+/+/status/alarm catches every alarm on every blister-packaging line and cell.

These filters are readable only because the hierarchy uns-03 built is consistent and positional. A tree with mixed depth or casing makes + land on the wrong segment for half the branches, the concrete cost of the over-nesting and inconsistent-casing anti-patterns that article named. One more rule: topics beginning with $ (such as $SYS) are reserved, and neither # nor + matches them unless a subscription explicitly starts with $.

QoS 0, 1, and 2: Which Delivery Guarantee for Which Westbridge Signal?

This is the deepest section of the article, because QoS is where most MQTT deployments get sloppy. For each level: the packet exchange, the guarantee, the failure mode, the cost, and a named Westbridge signal.

QoS 0 is at most once, QoS 1 is at least once, and QoS 2 is exactly once, and the cost rises with the guarantee.

Two facts get misread constantly. First, the QoS a subscriber actually receives is the LOWER of the publisher’s QoS and the QoS granted on its own subscription, never the higher. Second, MQTT’s guarantee is per hop, publisher to broker, and separately broker to each subscriber, never one end-to-end guarantee across a bridge or a chain of consumers.

QoS 0: At Most Once, and Why That Is Usually Correct

QoS 0 deserves defending, not dismissal. For a self-superseding stream (the checkweigher’s instantaneous weight, blst-01’s line speed, a temperature trend), a dropped sample is repaired by the next sample milliseconds later, so an acknowledgement buys nothing. The one real trap: a retained message published at QoS 0 that is lost in transit leaves the broker’s retained value silently stale, which matters more than the lost live sample, because every future subscriber inherits that stale value as current.

QoS 1: At Least Once, and Designing for Duplicates

QoS 1 is the workhorse level. PUBLISH, PUBACK, and retransmission with DUP mean a consumer will receive duplicates as defined behavior, not an edge case, so any QoS 1 consumer must be idempotent or de-duplicate on an application-level identifier in the payload. At Westbridge that covers cartoner state transitions, checkweigher alarms, and the canonical OEE topic. My practical rule: QoS 1 with an idempotent consumer is cheaper and faster than QoS 2, and covers most state and event traffic.

QoS 2: Exactly Once, and When It Is Worth Four Packets

Walk the handshake in order: PUBLISH, then PUBREC (“I have it”), then PUBREL (“release it”), then PUBCOMP (“done”). The receiver holds the packet identifier throughout so a retransmitted PUBLISH is recognized rather than delivered twice. The cost is two round trips and per-message state at both ends, which shows up as latency and broker memory at scale. Reserve it for non-idempotent, non-repeating events where a duplicate is itself a defect: the serialization vision station commissioning a unique carton serial or a batch-complete event triggering a genealogy record. Many mature deployments avoid QoS 2 entirely, choosing QoS 1 with application-level de-duplication instead, a defensible choice, not a shortcut.

A QoS Decision Rule for Westbridge, and the GAMP 5 Argument for Writing It Down

A compact rule covers most decisions: if the next message repairs a lost one, use QoS 0; if every event matters but the consumer can de-duplicate, use QoS 1; if the event is unique and non-repeating and a duplicate is a data-integrity defect, use QoS 2.

The GAMP 5 point deserves to be made properly. Under a risk-based approach, the QoS assigned to a GxP-relevant topic is a design decision recorded with its rationale. Choosing QoS 0 for the serialization commissioning event creates a silent gap against the ALCOA+ “complete” attribute: a message that never arrives leaves no trace anywhere to reconstruct from. Full validation treatment belongs to a later article; here the point is that the QoS choice is documented, not an implementation detail. L1/L2 telemetry usually tolerates QoS 0; L3 operations events usually do not.

How Do Retained Messages Deliver “Current State on Connect”?

This is the mechanism that cashes the promise made in the first article of this series. A PUBLISH sent with the RETAIN flag set causes the broker to store that message as the single retained message for that exact topic name, replacing any previous one. When any client later subscribes to a filter matching that topic, the broker immediately delivers the retained message, without the original publisher being involved or even connected. That behavior is the entire mechanism behind the claim that a subscriber connecting at 3 a.m. sees the whole plant. Strip RETAIN out and a UNS only delivers whatever happens to be published while you are listening, a message bus, not a namespace.

One constraint deserves stating plainly: exactly one retained message per topic. The retained state is last-value-only and carries no history, exactly why uns-01’s division of responsibility holds: the historian is a separate subscriber that archives every change, and the namespace itself is not a history feature.

Setting, Replacing, and Clearing Retained State

Each retained publish replaces whatever was retained on that topic before it. Publishing a zero-byte payload with RETAIN set deletes the retained message entirely, the standard way to decommission a topic when a cell is removed from blst-01. MQTT 5.0 adds a Retain Handling subscription option (send retained messages at subscribe, send only if the subscription did not already exist, or send none), which lets a reconnecting client avoid reprocessing the whole tree, plus a Retain Available flag in CONNACK by which a broker can declare it does not support retention at all.

The Stale Retained Value Trap

Here is the strongest practitioner point in this article. A retained value carries no liveness information. If the checkweigher’s edge client died three hours ago, the broker still serves its last OEE value to every new subscriber, and that value looks exactly as current as a live one. A dashboard renders it; an operator believes it.

The protocol-level fixes are in the next section: a Last Will message and a separate connection-status topic, plus a payload-level timestamp a consumer can age out, which is a payload design decision deferred to a later article. The design rule I hold to: never let a retained value be the only evidence that a publisher is alive.

Last Will and Testament and Session State: How Do You Know a Publisher Is Still Alive?

The Last Will and Testament is configured at connect time, not failure time. The CONNECT packet carries a Will Flag plus a Will Topic, Will Message, Will QoS, and Will Retain. The broker holds that message for the life of the connection and publishes it on the client’s behalf only if the connection closes ungracefully, such as through network loss, a keep-alive timeout, a client crash, or a protocol error. A client that sends a proper DISCONNECT packet has left cleanly, and its will is discarded, exactly the behavior an engineer wants: a maintenance shutdown should not raise the same alarm as a dead gateway.

At Westbridge, the checkweigher’s edge client connects with a retained will of “offline” on its own connection-status topic, then publishes a retained “online” right after CONNACK, so any subscriber reads liveness directly from the tree instead of inferring it from silence.

Keep Alive Sets Your Death-Detection Latency

The client declares a Keep Alive interval in CONNECT and must send some packet, a PINGREQ if it has nothing else to say, within that interval. The broker declares the client dead if it hears nothing for one and a half times that interval and only then publishes the will. Detection latency is a number an engineer chooses: a 60-second keep-alive means up to 90 seconds where a dead checkweigher still looks healthy. Shorter intervals detect failure faster but cost more traffic and connection churn on a marginal link, a real trade-off on Westbridge’s palletizing area, where wireless coverage is inconsistent.

Clean Start, Session Expiry, and Persistent Sessions

In MQTT 3.1.1, CONNECT carries a single Clean Session flag: set, and the broker discards any prior session and starts fresh; cleared, and it restores the client’s subscriptions plus any queued QoS 1 and QoS 2 messages. In MQTT 5.0 that flag splits into Clean Start (should the session begin fresh) and a separate Session Expiry Interval (how long the broker keeps the session after disconnect), a genuine improvement, since 3.1.1 gave no way to bound how long a broker holds state for a client that never returns.

Session state includes subscriptions and in-flight QoS 1/2 message state, which lets a bottle-line gateway on a flaky link reconnect and continue rather than re-subscribing and losing queued events. Persistent sessions consume broker memory per offline client, so session expiry is a sizing decision covered in the broker-selection article later in this series. MQTT 5.0 also adds a Will Delay Interval, delaying the will so a client that reconnects within the window never appears to have died, suppressing false alarms on a marginal link.

Which MQTT 5.0 Features Actually Matter on a Plant Floor?

Five MQTT 5.0 additions have a real industrial payoff; the rest are refinements a plant engineer rarely touches. Shared subscriptions ($share/<group>/<filter>) let the broker load-balance matching messages across a named group instead of fanning out to all members, how a consumer such as an OEE calculation service scales horizontally. Message Expiry Interval discards a queued message not delivered within its lifetime, preventing a reconnecting subscriber from being handed a flood of stale events. User Properties carry arbitrary key/value pairs in the packet header, useful for a content type or schema version without contaminating the topic path, tying directly back to the anti-pattern uns-03 named against encoding metadata into the path.

Reason Codes on acknowledgements replace 3.1.1’s near-silent failures, so a client learns why a publish or subscribe was rejected. Topic Aliases substitute a numeric alias for a long topic string after first use, cutting overhead on a deep hierarchy like Westbridge’s over a metered link. The caveat governing all five: they exist only where BOTH broker and client library implement them, and plenty of installed industrial gear still speaks 3.1.1 only. Verify support on both ends; never assume it.

MQTT vs AMQP vs Kafka at the Plant Edge: Which Protocol Actually Wins Where?

The thesis here is not that MQTT is better than the alternatives. It is that MQTT is optimized for a different problem: many constrained, intermittently connected publishers on unreliable links that need last-known state on demand, the edge problem specifically. AMQP and Kafka each win clearly in a different part of the same plant, and I want an AMQP or Kafka engineer reading this to recognize their own tool described fairly, not caricatured. One security note applies to all three: MQTT itself defines no authorization model beyond a username and password field in CONNECT, leaving transport security and access control to the deployment, covered properly in the security article later in this series.

Where MQTT Wins at the Edge

Four properties are concrete, not hand-waved: a two-byte minimum fixed header and tiny per-message overhead on a metered or wireless link; client libraries small enough to run on a constrained edge device or PLC-adjacent gateway; a connection model that assumes disconnection is normal, with will messages, persistent sessions, and retained state all designed around recovery; and the last-value-per-topic retained semantic itself, which no other mainstream broker offers as a first-class, connect-time behavior. That last property is what makes MQTT a natural UNS substrate, not merely a viable one.

What AMQP Does Better

AMQP deserves a fair hearing. AMQP 1.0 is an OASIS Standard (2012-10-29, also published as ISO/IEC 19464:2014); AMQP 0-9-1, a different and earlier protocol, is the widely deployed dialect behind RabbitMQ, and the two should not be conflated. AMQP offers far richer broker-side semantics: explicit queues with their own lifecycle, exchange-based routing (topic, direct, fanout), per-message flow control with credit-based back-pressure, and transactional publishing. For enterprise workflow messaging with durable queues, complex routing, and per-consumer back-pressure over reliable networks, AMQP beats MQTT and it is not close. The cost at the edge: a heavier protocol, a larger client footprint, and a connection model built for stable datacenter links rather than a satellite or cellular backhaul.

Where Kafka Legitimately Wins

This is the subsection where credibility is won or lost. Kafka is a distributed, partitioned, durable commit log, not a message broker in the MQTT sense, and it wins clearly on sustained high-throughput ingestion, durable retention with replay from an arbitrary offset, strict ordering within a partition, consumer groups with independently tracked offsets, and a mature stream-processing ecosystem. MQTT has no replay, no offsets, and no history: once delivered, a message is gone, and the retained state is exactly one value per topic.

The closest comparison deserves precision, not a loose analogy. Kafka’s log compaction guarantees at least the last value per key (a tombstone record removes prior values and is itself cleaned up after delete.retention.ms), which resembles a retained message on the surface, but it is a key-scoped retention policy a consumer reads by walking the log, not a broker pushing current state to a client the instant it subscribes: similar in appearance, different in behavior. Kafka’s honest weaknesses at the edge: heavier clients, an assumption of stable long-lived connections, and a poor fit for tens of thousands of intermittently connected constrained publishers. Pairing MQTT at the edge with Kafka in the enterprise is a common, legitimate architecture, and designing that hybrid is the subject of a later article.

The Comparison in One Table

Dimension	MQTT	AMQP	Kafka
Primary design target	Constrained, intermittently connected edge devices	Enterprise workflow messaging, durable queues	High-throughput distributed log, stream processing
Per-message overhead	Two-byte minimum fixed header	Moderate, richer envelope	Higher, batched for throughput
Connection assumption	Disconnection is normal	A stable, long-lived connection expected, with credit-based flow control managing back-pressure	Stable, long-lived connection expected
Delivery guarantees offered	Per-hop QoS 0/1/2	At-most-once, at-least-once, or exactly-once, chosen per link via the settlement model	At-least-once or exactly-once with idempotent producers
Current-state-on-connect	Yes, via retained messages	No native equivalent	No native equivalent (compaction is key-scoped, read on demand)
History and replay	None; last value only	Queue depth only, no replay	Full replay from any retained offset
Ordering guarantee	Per topic, per client, within the in-flight window	Per queue, broker-managed	Strict within a partition
Consumer scaling model	Shared subscriptions (5.0) or fan-out	Competing consumers on a queue	Consumer groups with partition assignment
Typical plant placement	Cells to the site broker	Enterprise workflow queues, where they already exist	Enterprise analytics backbone downstream

At Westbridge, that placement is literal: MQTT carries data from the cells up to the site broker, Kafka ingests it into the enterprise analytics backbone downstream of that broker, and AMQP fits wherever enterprise workflow queues already exist independent of the plant floor.

Frequently Asked Questions (FAQ) on MQTT for a Unified Namespace

Does QoS 2 guarantee my MES receives every message exactly once?

No, and this is the most common misreading of the specification. QoS 2 guarantees exactly-once transfer for a single hop, between one client and the broker. The publisher-to-broker hop and the broker-to-subscriber hop are separate agreements: the subscriber’s effective QoS is the lower of the published QoS and its own granted subscription QoS, and any bridge or chained consumer beyond that sits outside MQTT’s guarantee entirely. End-to-end exactly-once is an application property built with idempotent consumers and message identifiers, not something QoS 2 hands you by itself.

If retained messages give me current state, do I still need a historian?

Yes. A topic holds exactly one retained message: the last one. There is no history, no replay, and no way to ask what the value was an hour ago. The historian subscribes to the namespace and stores the time series, which is the same division of responsibility set out in the first article of this series: the UNS holds now, and the historian holds then.

Should every UNS topic be published as retained?

State topics, yes; event topics, usually not. A retained state topic (equipment state, OEE, a setpoint) is what makes the namespace browsable on connect. A retained event topic (an alarm raised, a batch completed, a carton commissioned) means every new subscriber is delivered a historical event as if it just happened, which triggers duplicate downstream action. The test I apply: would a subscriber connecting right now be correct to act on this message as the current situation?

Why not just use Kafka for the whole unified namespace?

Because the edge half of this problem is not the one Kafka was built for: many constrained, intermittently connected publishers that need last-known state pushed to them the moment they connect. Kafka has no equivalent of a retained message delivered at subscribe time, and its client and connection model is a poor fit for a machine-side gateway on a marginal link. Kafka is a strong choice for the enterprise ingestion and analytics tier downstream of the namespace, which is why the hybrid pattern exists and gets its own article later in this series.

Do I need MQTT 5.0, or is 3.1.1 good enough for a UNS?

3.1.1 is sufficient for a working UNS: pub/sub, all three QoS levels, retained messages, wills, and persistent sessions are all present. 5.0 adds operational quality that matters at scale, session expiry, will delay, shared subscriptions, message expiry, user properties, and reason codes. I target 5.0 for anything new, and I verify support on both the broker and every client library before depending on a 5.0-only feature, because a lot of installed industrial gear still speaks only 3.1.1.

References

• OASIS, MQTT Version 5.0, OASIS Standard, 2019-03-07, docs.oasis-open.org
• OASIS / ISO/IEC, MQTT Version 3.1.1, OASIS Standard, 2014-10-29 (also published as ISO/IEC 20922:2016), docs.oasis-open.org
• OASIS / ISO/IEC, Advanced Message Queuing Protocol (AMQP) Version 1.0, OASIS Standard, 2012-10-29 (also published as ISO/IEC 19464:2014), docs.oasis-open.org
• ISA-95 / IEC 62264, Enterprise-Control System Integration, International Society of Automation, isa.org
• ISA-88 / IEC 61512, Batch Control, International Society of Automation, isa.org
• ISPE, GAMP 5: A Risk-Based Approach to Compliant GxP Computerized Systems, Second Edition, 2022, ispe.org
• IEC / ISA, IEC 62443 Series, Security for Industrial Automation and Control Systems, isa.org
• Eclipse Foundation, Eclipse Mosquitto Documentation, mosquitto.org
• HiveMQ, MQTT Essentials (practitioner reference, cross-checked against the OASIS text for every normative claim), hivemq.com
• Apache Software Foundation, Apache Kafka Documentation, kafka.apache.org
• Eclipse Foundation, Sparkplug Specification, Version 3.0.0 (named only as a deferred topic), sparkplug.eclipse.org
• ValidatedState, What Is a Unified Namespace? A Ground-Up Definition, validatedstate.com
• ValidatedState, Why Point-to-Point Integration Fails at Scale, validatedstate.com
• ValidatedState, Designing Your UNS Topic Hierarchy with ISA-95, validatedstate.com