Reading Telemetry Is Easy. Sending Commands Is Not.

Most IoT projects start with telemetry, and telemetry is forgiving. A reading that arrives late is an inconvenience. A reading that never arrives leaves a gap in a chart. Nobody gets hurt.

Commands are a different problem entirely. When your cloud tells a battery inverter to change its power setpoint, or a pump to stop, "I sent it" is not an acceptable answer to "did it happen?" You need to distinguish four genuinely different outcomes — the device never received it, received but rejected it, received and accepted but failed to apply it, or applied it successfully — and you need to be able to prove which one occurred, after the fact, to someone who was not in the room.

This is where a lot of otherwise solid IoT platforms quietly under-deliver. The command gets published, the UI shows a green tick, and nobody notices for six weeks that a percentage of commands never actually executed.

This article covers how to design a cloud-to-device command path that cannot silently fail, based on a system TTPL built and verified end to end against live energy hardware in the field.

Why You Cannot Just Use Request/Response

The intuitive design is a synchronous call: cloud sends command, device responds, done. Some IoT platforms offer exactly this — direct methods, RPC-style invocations, synchronous device calls.

There are two reasons it often is not available to you.

The first is transport constraints. Synchronous mechanisms typically use their own reserved topic namespace. If your device integration contract fixes the topic structure — common when firmware is supplied by a third-party manufacturer, or when the device stack is already certified — that namespace may simply be off limits.

The second is more fundamental: a synchronous response only tells you the device received and parsed the message. It does not tell you the command executed. A battery does not change its setpoint in the milliseconds available before an RPC times out. So even with synchronous transport, you still need an asynchronous result path for the outcome that actually matters.

The Two-Reply Model

The design that works is one command down, two replies up:

1. Cloud dispatches the command to the device over the cloud-to-device channel, carrying a unique request identifier and an explicit expiry time.
2. Device replies with an acknowledgement as soon as it has received and validated the message. This is a parse-and-accept signal: "I got it, it is well-formed, I am going to attempt it" — or a rejection with a reason.
3. Device replies with a result once it has actually attempted execution: completed, failed, or expired.

Both replies travel as ordinary device-to-cloud messages on the same connection the device already uses for telemetry, distinguished by a message-type property and routed to their own stream. The unique request identifier is echoed in both replies — that is what lets the cloud correlate everything back to the originating command.

The two replies answer different questions, and you need both:

An acknowledgement without a result is the most dangerous state in the system. The device said yes and then went quiet — and that is exactly the case a naive implementation reports as success.

Validate Before Dispatch, Not After

Every command should pass through a registry before it leaves the cloud. The registry is configuration, not code: a declarative list of every command the platform is allowed to send, and for each one, its parameters with types, ranges, allowed values and whether they are required.

This gives you several things that are hard to retrofit:

• A whitelist. If a command code is not in the registry, it cannot be dispatched. Not by an API caller, not by a bug, not by a misconfigured integration.
• Parameter validation at the boundary. A setpoint outside its safe range is rejected in the cloud, where you can return a clear error, rather than on the device where the failure mode is unpredictable.
• A per-command safety gate. Each entry can carry a flag controlling whether it may physically actuate equipment — which lets you ship the message path before you ship the ability to move anything.
• Onboarding as configuration. Adding a new command becomes a reviewed configuration change with explicit ranges and limits, not a code deployment.

That last point matters more than it first appears. When the hardware manufacturer finally delivers the physical command contract — codes, parameters, units, ranges, safety limits, readback rules — you want to encode it in one reviewable place, not scatter it across handlers.

Ship With Physical Actuation Disabled

This is the single most useful practice in this article, and it costs almost nothing.

Build the full command lifecycle. Deploy it to production. Verify it end to end against the real device in the field. But enable only mock commands — commands that traverse the entire path and return a real result, while doing nothing physical.

A mock ping proves the whole chain: cloud validation, dispatch, device receipt, acknowledgement, result, correlation, audit write. A mock command carrying a parameter additionally proves parameter serialisation and device-side parsing. Neither one touches an inverter.

So you get the hardest part of the system — the part with the most failure modes and the most integration risk — verified against production hardware, under real network conditions, with zero risk to equipment. Then physical commands are enabled one at a time, each with its own verification cycle, never as a blanket switch.

Timeouts Are Not Error Handling — They Are the Design

Every asynchronous command has three ways to end: the acknowledgement arrives, the result arrives, or neither does. Only the third needs deliberate design, and it is the one most often left implicit.

Run a timeout monitor with three independent clocks:

The point is that every command reaches a terminal state, always, whether or not the device cooperates. A command stuck in "sent" forever is a bug, not a state. If you ever query your command table and find rows that have been pending for days, the timeout monitor is missing or broken.

Tune the values against real hardware. Defaults chosen against a simulator are usually too aggressive — a real controller on a real network, possibly behind a constrained industrial link, is slower than a local mock. Make them configuration rather than constants so they can be adjusted per environment without a rebuild.

Expiry deserves specific attention. If a device is offline, the command queues at the broker with a time-to-live. Your command expiry should be less than or equal to that queue TTL — otherwise you get commands that expire logically in your database while still sitting in a queue waiting to be delivered, and a device that executes something the cloud already wrote off.

The Clock Skew Trap

Here is a failure mode that will cost you a day if you have not seen it before.

A command carries an expiry timestamp. The device compares that expiry against its own clock and rejects the command as already expired. Everything in the cloud looks correct. The payload is well-formed, the timestamps are valid, the logic is right — and every single command comes back rejected.

The device clock was hours ahead of UTC. On live hardware, we observed an eight-hour skew.

Two rules follow, and both are worth writing into your integration contract:

• Require NTP synchronisation on the device, explicitly, as a contractual precondition rather than an assumption. Any expiry comparison performed device-side depends on it.
• Never use device-reported timestamps for cloud-side decisions. Ordering, liveness, offline detection and replay sequencing should all use broker-side sequence numbers and enqueue times — values stamped by infrastructure you control.

Idempotency and One-In-Flight

Two rules prevent the most common classes of command duplication.

Idempotency by request identifier

The request identifier is generated once, when the command is issued, and is unique. If the same identifier is submitted again — a retried API call, a double-clicked button, a client retry after a timeout — the platform returns the existing command rather than dispatching a second one. The same identifier appears in the dispatched payload, the acknowledgement, the result and every audit entry, which makes end-to-end tracing trivial.

One command in flight per device

While a command is pending for a device, refuse to dispatch another to that device. This avoids an entire category of ambiguity: two setpoint changes racing, replies that cannot be reliably attributed, and devices applying commands in an order the cloud did not intend. If your use case genuinely needs concurrent commands, scope the constraint more narrowly — per sub-device or per command family — but keep the constraint.

The Audit Trail Is the Product

For any command that touches physical equipment, the audit trail is not a nice-to-have. It is what you produce when someone asks what happened.

Store two things:

The command aggregate — current state, target device and sub-device, command code, parameters, timestamps, and critically the exact payload that was dispatched. Not a reconstruction from the current parameters; the literal bytes sent. If serialisation was wrong, that is the only place you will see it.

An append-only event log — one row per state transition, with timestamp, resulting state, reason code and any message the device returned. Append-only matters: rows are never updated or deleted, so the trail cannot be quietly rewritten.

A healthy trail for a successful command reads as four entries — approved, sent, acknowledged, completed — each with its own timestamp. From that you can derive dispatch latency, device acknowledgement latency, execution time, and the total round trip, per command, without any additional instrumentation. In a verified production test on live hardware, that sequence took 1.77 seconds from issue to completed, with roughly one second to acknowledgement.

Reliability of the Reply Path

One implementation detail worth flagging, because it is easy to get wrong and hard to notice.

Command replies come back as events on a stream, consumed by your command service. If that consumer is configured for slow, conservative partition rebalancing — a sensible default for a high-volume telemetry consumer — replies can be dropped or delayed during ownership changes. On a telemetry stream that costs you a few rows. On a command reply stream it means commands that actually completed get recorded as timed out.

Give the command reply consumer its own stream, its own consumer group and its own configuration, tuned for prompt ownership acquisition rather than throughput. The volumes are tiny compared to telemetry; the reliability requirement is much higher. These are different workloads and they deserve different settings.

A Design Checklist

1. Two replies, not one. Acknowledgement answers "received and accepted"; result answers "executed". Absence of either is meaningful information.
2. Registry-driven validation. Whitelist every command code; validate every parameter by type, range and allowed values before dispatch.
3. Physical actuation disabled by default. Prove the path with mock commands in production, then enable real commands individually.
4. Three timeouts. Acknowledgement, result and expiry, each producing a deterministic terminal state. No command stays pending forever.
5. Command expiry at or below the broker queue TTL. Otherwise offline devices execute commands the cloud has already abandoned.
6. Never trust device clocks. Require NTP; use broker-side time for every cloud-side decision.
7. Idempotency by request identifier, and one command in flight per device.
8. Append-only audit, storing the exact dispatched payload alongside every state transition.
9. A dedicated, correctly tuned consumer for the reply stream.
10. A device simulator with a mock responder, so the whole lifecycle can be regression-tested without field hardware.

Closing Thought

The difference between a command system that works and one that merely appears to work is almost entirely in how it handles the cases where nothing comes back. Success paths are easy; every implementation gets those right.

What separates them is whether a command that was accepted and then vanished ends up in a clearly labelled terminal state with a timestamp and a reason — or sits in a table marked "sent", indistinguishable from the ones that succeeded.

Design for the silence, and the rest follows.

TTPL designs and builds industrial IoT control systems — device integration, command lifecycles, audit trails and safety gating for energy, manufacturing and utility platforms. If you are adding remote control to a device fleet, talk to our team.