Why Most IoT Telemetry Pipelines Fail in the Same Three Places

Connecting one device to the cloud is a weekend project. Connecting a fleet of them, keeping every message, and still being able to answer "what did site twelve do at 03:40 last Tuesday" eighteen months later is an architecture problem — and it is usually the same three things that go wrong.

The first is data loss during ordinary operations. Not dramatic outages: a deployment, a database failover, a worker restart. If devices publish directly into something that writes straight to a database, every one of those routine events is a hole in the record.

The second is coupling ingestion to the API. Device traffic scales with how many devices you have installed. API traffic scales with how many people have a dashboard open. Put them in one process and a busy Monday morning competes with the field.

The third is trusting delivery guarantees that do not exist. Every mainstream event platform gives you at-least-once delivery. Teams that design as though they have exactly-once eventually discover duplicate rows in production, usually while explaining a billing discrepancy.

This article walks through a pipeline that addresses all three, built on Azure IoT Hub, Event Hubs and PostgreSQL. It is the architecture TTPL used to ingest three-second telemetry from energy management controllers in the field, and the reasoning generalises to any high-frequency device fleet.

The Shape of the Pipeline

Four stages, each with one job:

1. Azure IoT Hub — the MQTT front door. Terminates TLS, gives every device its own revocable identity, and routes messages onward by type.
2. Azure Event Hubs — the durable buffer. Holds every accepted message for a retention window regardless of whether anything downstream is reading.
3. An ingestion worker — a headless background service that consumes the stream and writes to the database at its own pace.
4. PostgreSQL — the system of record, holding both the raw frame and typed, indexed time-series rows.

A separate REST API sits beside this, reading from the database to serve web and mobile clients. It is deliberately not in the ingestion path.

Stage 1: Why a Managed IoT Broker Instead of Your Own MQTT Server

The obvious alternative is to run Mosquitto or EMQX yourself. That is a legitimate choice, and for some workloads it is the right one — particularly if you need free-form topic structures or you are already operating a broker fleet.

What a managed IoT broker buys you is per-device identity with revocable credentials. One compromised device should not mean rotating credentials for everyone else. You also get TLS termination, authentication and message routing as managed capabilities, with no server for you to patch.

The cost is a fixed topic scheme. Devices publish to a broker-defined path rather than whatever structure you would have chosen. In practice this means the device vendor’s preferred topic format has to be mapped onto the broker’s scheme, and that mapping belongs in your integration contract, documented, before anyone writes firmware against it.

Stage 2: The Buffer Is the Whole Point

This is the stage teams are most tempted to skip. Why not have the broker write straight to the database?

Because the buffer is what makes everything downstream disposable. Every accepted message is durably retained for a fixed window — commonly 24 to 72 hours — whether or not any consumer is alive. That single property converts an entire class of incidents into non-events:

• Deploying a new worker version? Messages queue. The new revision picks up from the last recorded position.
• Database failover or maintenance window? The worker stalls, messages accumulate, and it catches up afterwards.
• Worker crashed at 2am and nobody noticed until 9am? Seven hours of telemetry is still sitting in the stream, not lost.
• Found a mapping bug? Replay from retention rather than asking devices to resend.

Partition count is the other decision here. Partitions are the unit of parallelism — you can run at most one consumer per partition within a consumer group. Choose based on the peak throughput you expect a single consumer to handle, then divide, and add headroom, because partition counts are awkward to change later. A high-frequency telemetry stream might get eight partitions while a low-volume alarm stream gets two.

Consumer groups are independent read cursors over the same events. This is how you add a second consumer — an analytics sink, a real-time alerting service — without touching the first one, and without the two competing for the same messages.

The buffer is not a performance optimisation. It is the difference between "we lost four hours of data during the upgrade" and "the upgrade was uneventful."

Stage 3: The Ingestion Worker — Four Decisions That Matter

Decision 1: A Worker Service, Not a Serverless Function

An event-triggered serverless function and a long-running worker use the same underlying consumer SDK. The difference is hosting and control.

Functions are excellent for bursty, mostly-idle workloads — you pay for invocations and scale to zero between them. Telemetry ingestion is the opposite shape: a message every few seconds, always. There is no idle to scale down to, so the billing model works against you, and the trigger’s batching abstraction takes away the checkpoint control you want.

A long-running worker gives you explicit control over checkpoint cadence and batching, and runs the same way on a laptop, a container platform or a VM. The genuinely function-shaped component in most IoT systems is something like a scheduled heartbeat sender: wake on a timer, send one small message, go idle.

Decision 2: Write Raw and Typed, Not One or the Other

Store each frame twice, in one pass.

The raw write is the payload exactly as received, in a JSONB column, with stream metadata attached — partition, sequence number, enqueue time. This is your contract with the device vendor and your audit record. When a mapping assumption turns out to be wrong six months in, the typed tables can be rebuilt from raw. Without it, the only recovery is asking devices to resend data they no longer hold.

The typed write parses the payload into indexed numeric columns. Dashboards query time ranges filtered by device; that needs real columns and real indexes, not JSON scans across millions of rows.

A useful refinement: when the message type is unrecognised, still write raw and log a warning rather than rejecting the frame. A vendor adding a new message type in a firmware update should never break your ingestion.

Decision 3: Idempotency Instead of Exactly-Once

Event streaming gives you at-least-once delivery. If a worker processes an event and dies before recording its position, that event is redelivered on restart. This is not a flaw to engineer around — it is the contract.

The correct response is to make writes idempotent. Put a unique constraint on the message identifier the device supplies, and write with an on-conflict-do-nothing clause. Replays become no-ops by construction. You stop caring about duplicate delivery entirely, which also means you stop needing stronger — and slower — delivery guarantees anywhere else in the stack.

Decision 4: Dead-Letter Anything You Cannot Parse

One malformed payload must never stop the stream. If parsing fails, write the payload and the error text to a dead-letter table and carry on to the next message.

Two things follow from this. A firmware change that breaks the envelope degrades into a growing dead-letter count rather than a full fleet outage. And the dead-letter table becomes a genuinely useful operational signal — its row count is one of the first metrics worth alerting on, because in healthy operation it should stay flat at zero.

Checkpointing: The One Number Worth Tuning

A checkpoint records "this consumer has processed up to sequence N on this partition." On restart, the worker resumes there rather than replaying the entire retention window.

Checkpoint frequency is a direct trade-off:

Because writes are idempotent, replay is harmless — which means you can checkpoint infrequently and take the cheaper option. A cadence in the tens of events per partition is a reasonable default: the storage cost disappears into the noise, and the worst case after an unclean shutdown is a few dozen redundant no-op inserts.

Stage 4: Modelling Device Data So Queries Stay Simple

A subtlety that catches people out: in most industrial deployments, the thing that connects to the broker is not the thing you want to chart.

A single site gateway or controller authenticates as one device identity, but reports on behalf of many physical units behind it — inverters, battery racks, meters, sensors. The vendor payload identifies those with its own internal position or slot identifier.

So every row needs two identifiers: the authenticated device identity (which site or controller) and the in-payload device identifier (which unit at that site). With both stored and indexed together with the timestamp, per-device time series fall out of a trivial filtered query, and one controller reporting three units lands as three independently queryable series.

Keep the Read API Out of the Ingestion Path

The final structural decision. The REST API that serves dashboards should read from the database and take no part in ingestion.

The argument is scaling axes. Ingestion load is a function of device count — thousands of controllers reporting every few seconds, completely indifferent to whether any human is awake. API load is a function of user traffic, which is bursty and follows office hours. These are different curves and they want different scaling policies.

Coupling them means a traffic spike in one degrades the other, and a bug in either risks taking down both. Separated, they deploy, scale and fail independently. They can still share one database — cleanly partitioned by schema, so the ingestion worker owns its tables and the API owns its own — which keeps a single source of truth without a single point of failure.

What to Monitor Once It Is Running

Four signals cover most of what goes wrong:

• Rows written per minute — a drop means devices stopped publishing, the worker stopped consuming, or a route broke. Alert on a floor, not a ceiling.
• Dead-letter count — should be flat at zero. Any sustained increase means a payload contract changed.
• Consumer lag — how far behind the stream head each partition is. A steadily growing lag means the worker cannot keep up and needs more replicas or a faster write path.
• Time since last message per device — the basis of fleet liveness. Judge this on broker-side enqueue time, never on a timestamp the device reports.

Scaling: What Changes and What Does Not

The useful property of this architecture is that growth is configuration rather than redesign:

Worth noting on the third row: consumers in the same group coordinate partition ownership automatically through the checkpoint store, so adding a replica requires no code change and no manual partition assignment. The ceiling is one active consumer per partition, which is why partition count deserves thought up front.

A Pre-Build Checklist

Before writing ingestion code for a device fleet, settle these:

1. Write the payload contract down. Envelope shape, message types, identifiers, required fields, timestamp semantics, units. Ambiguity here becomes a production incident later.
2. Build a device simulator first. Use the raw protocol rather than a cloud SDK, so it exercises the exact contract real firmware implements. It lets you build and regression-test the whole pipeline without field hardware or vendor scheduling.
3. Decide your idempotency key. Which field uniquely identifies a message? If the vendor does not supply one, that is a contract negotiation, not an implementation detail.
4. Size partitions for peak, not average. They are hard to change later.
5. Agree the retention window. How long must you be able to be broken before data is genuinely lost? That number is your retention setting.
6. Plan for the unknown message type. Store and warn — never reject.

Closing Thought

None of the individual pieces here are exotic. A managed broker, a durable stream, a background worker, a relational database — all mature, all well documented.

What makes the difference is where you place the boundaries: a buffer between the broker and the database so downstream components become disposable, idempotent writes so at-least-once delivery stops mattering, dual raw and typed storage so today’s mapping bug is recoverable, and separated ingestion and read paths so device growth and user growth never collide.

Get those four right and the pipeline stops being something you worry about. It becomes infrastructure — which is exactly what telemetry ingestion should be.

TTPL builds industrial IoT platforms on Azure and AWS — telemetry ingestion, device management, time-series storage and cloud-to-device control. If you are designing a device fleet and want the architecture reviewed before you commit to it, get in touch.