🍿
IoT Snacks
Part 7by Muhammad

MQTT vs Kafka for IoT Data Pipelines: What to Use and When

MQTT vs Kafka for IoT data pipelines

MQTT vs Kafka for IoT data pipelines is a practical choice you make when you move sensor data from devices to applications, analytics and storage. This comparison targets intermediate IoT and embedded engineers who already know basic pub/sub concepts and want concrete selection criteria, tradeoffs and integration patterns.

You will learn how MQTT (Message Queuing Telemetry Transport) and Apache Kafka differ in reliability, scaling, ordering, retention and operations, plus when you should use both together in a single IoT data pipeline.

Table of Contents

Quick summary (one paragraph)

Use MQTT close to devices when you need lightweight connectivity over unreliable links, simple pub/sub fanout and device-friendly features like retained messages and Last Will and Testament. Use Kafka in your backend when you need high-throughput ingestion, durable log retention, replayability, stream processing and multiple downstream consumers that can reprocess data independently. For many production systems, the best answer to MQTT vs Kafka for IoT data pipelines is “both”: MQTT for device ingress and Kafka as the durable, scalable backbone for analytics, storage and event-driven services.

Core mental model: where each technology fits

Think about an IoT data pipeline as three zones:

  1. Device and edge zone: constrained devices, cellular links, intermittent Wi-Fi, NAT, low power, sometimes store-and-forward gateways.
  2. Ingress and normalization zone: authentication, tenant routing, topic naming, schema normalization, enrichment, rate limiting and dead-lettering.
  3. Backend and data platform zone: durable storage, replay, batch and stream processing, serving layers, data sharing across teams.

MQTT is optimized for the device and edge zone and can also cover ingress for smaller systems. Kafka is optimized for the backend zone and often sits behind an ingress layer (which can be MQTT, HTTP, gRPC or custom gateways). When you compare MQTT vs Kafka for IoT data pipelines, the biggest mistake is trying to force one of them to cover every zone.

Protocol vs platform

  • MQTT is a client-broker protocol (plus a broker implementation) designed for low bandwidth and unreliable networks.
  • Kafka is a distributed event log platform (protocol plus brokers plus storage replication) designed for throughput, retention and many independent consumers.

MQTT basics for IoT pipelines

MQTT (Message Queuing Telemetry Transport) uses a broker and lightweight clients. Devices publish to topics and subscribers receive messages for the topics they subscribe to. MQTT is widely used in embedded IoT because it keeps client code small and works well over lossy networks.

Key MQTT concepts that matter in data pipelines

  • QoS (Quality of Service): QoS 0 (at most once), QoS 1 (at least once), QoS 2 (exactly once at the protocol level, higher overhead).
  • Sessions and queued delivery: persistent sessions can buffer messages for clients while they are offline (broker dependent, bounded by policy).
  • Retained messages: broker stores the last retained message per topic and delivers it to new subscribers (useful for state like “current config”).
  • Last Will and Testament (LWT): broker publishes a will message if a client disconnects unexpectedly (useful for online status).
  • Topic hierarchy and wildcards: simple routing and fanout. Great for “tenant/site/device/telemetry”.

What MQTT brokers do well

  • Handle millions of lightweight connections (with the right broker and tuning).
  • Support TLS, client certificates, username/password, token auth (varies by broker).
  • Bridge to other brokers and sometimes to Kafka or cloud services via plugins/connectors.

Common MQTT limitations in pipelines

  • Retention is not a durable event log by default. “Retained” is not “historical”.
  • Replay for analytics is not a built-in concept in most MQTT deployments.
  • Backpressure handling and consumer lag visibility are usually weaker than Kafka.
  • Topic and authorization design can get complex at scale without strict conventions.

Kafka basics for IoT pipelines

Apache Kafka is a distributed commit log where producers append records to topics and consumers read them at their own pace. Kafka stores data for a configured retention time or size and enables replay. It is a strong fit for backend event-driven architectures and data platforms.

Key Kafka concepts that matter in data pipelines

  • Topics and partitions: partitions scale throughput and determine ordering. Ordering is guaranteed only within a partition.
  • Consumer groups: horizontal scaling for a logical application. Each partition is consumed by at most one consumer in a group.
  • Offsets: consumers track progress. Rewind offsets to replay or reprocess.
  • Retention and compaction: retain for N days, or compact by key to keep latest value per key (useful for device state).
  • Replication and durability: data replicated across brokers, controlled by ack settings and ISR (in-sync replicas).

What Kafka does well

  • High throughput ingestion and fanout to many downstream systems.
  • Durable storage with replay and independent consumers.
  • Stream processing ecosystems (Kafka Streams, ksqlDB, Apache Flink integrations).
  • Operational tooling around lag, throughput, partition health, quotas (depending on distro).

Common Kafka limitations in pipelines

  • Not device-friendly: heavy client libraries, TCP behavior, and operational complexity on constrained endpoints.
  • Connection scaling is possible but not typically the first choice for millions of intermittently connected devices behind NAT.
  • Exactly-once semantics exist in Kafka, but end-to-end exactly-once across device ingress, transformations and sinks still requires careful design.

MQTT vs Kafka for IoT data pipelines: decision criteria

This section frames MQTT vs Kafka for IoT data pipelines as a set of engineering questions. The right choice depends less on “which is better” and more on where your bottlenecks and requirements live.

1) Are your producers constrained devices or backend services?

  • Constrained devices: prefer MQTT. Smaller packets, simpler state, better tolerance for flaky links.
  • Backend services: Kafka fits naturally, especially when multiple teams consume the same events.

2) Do you need durable retention and replay?

  • If you need replayable history for analytics, debugging or model training, Kafka is the default tool.
  • MQTT retained messages are great for latest state, not for historical event streams.

3) How many downstream consumers will you have?

  • MQTT fanout is straightforward for online subscribers, but scaling many independent consumers with different processing rates and backlogs is harder.
  • Kafka excels when you have multiple consumer groups: storage, alerting, dashboards, anomaly detection and billing, each with its own lag.

4) What are your ordering requirements?

  • MQTT ordering depends on session, QoS and broker behavior but it is usually “good enough” per connection for telemetry streams.
  • Kafka ordering is explicit: guaranteed only within a partition, so you must choose partition keys carefully (often deviceId).

5) What is your operational tolerance?

  • MQTT clusters can be complex, but many brokers are simpler to run than Kafka for small to medium systems.
  • Kafka adds operational surface area: partitions, replication, rebalancing, storage IO, upgrades and quotas. Managed Kafka reduces this but increases cost.

6) Do you need stream processing close to the data log?

  • If you plan to do joins, windowed aggregations, enrichments and stateful processing at scale, Kafka plus a stream processor is a strong baseline.
  • MQTT alone is not a stream processing platform, although you can attach rules engines or bridge into processing systems.

Side-by-side feature comparison

Dimension MQTT Kafka What it means for IoT
Primary role Device messaging protocol + broker Distributed event log + streaming backbone MQTT fits device ingress, Kafka fits backend distribution and replay
Client footprint Small (embedded-friendly) Larger (Java, C/C++, Go, Python libs) Devices typically prefer MQTT
Network tolerance Excellent on lossy links Better on stable links Cellular and intermittent Wi-Fi favor MQTT
Fanout model Broker pushes to subscribers Consumers pull at their pace Kafka handles slow consumers without blocking producers
Retention Not a durable log by default Durable retention (time/size) and compaction Kafka supports replay and long-term pipelines
Replay Not inherent First-class via offsets Critical for analytics and debugging
Ordering Generally per connection/topic, broker dependent Guaranteed within a partition Partition by deviceId for per-device ordering
Delivery semantics QoS 0/1/2 At-least-once by default, exactly-once possible within Kafka End-to-end exactly-once still requires idempotency
Backpressure Limited, broker queues can fill Consumers lag, producers keep appending (within quotas) Kafka is safer for many slow consumers
Observability Broker metrics vary by implementation Rich metrics around lag, throughput, partitions Kafka shines for data platform ops
Security model TLS, authN/authZ by topic, certs common TLS, SASL, ACLs, quotas Both can be hardened, but designs differ

Reference architectures (MQTT-only, Kafka-only, hybrid)

Below are three practical topologies. Each includes a text diagram you can map to your own stack (EMQX, Mosquitto, HiveMQ, Kafka, Redpanda, MSK, Confluent Cloud, etc).

Topology A: MQTT-only (small to mid systems)

Text diagram: Devices (MQTT) → MQTT Broker → Rules engine/webhooks → Time series DB + Alerting + Dashboard

  • When it works: tens of thousands of devices, modest data rates, a small set of consumers, minimal replay needs.
  • Strength: simple device integration and fewer moving parts.
  • Risk: analytics team later asks for replay and multiple downstream consumers, you add complexity under pressure.

Topology B: Kafka-only (backend producers, not edge devices)

Text diagram: Services → Kafka → Stream processing → Storage + APIs

  • When it works: your “devices” are actually gateways or backend services that can run Kafka producers reliably.
  • Strength: one backbone for everything, strong replay and governance.
  • Risk: direct-to-Kafka from constrained devices creates support and network headaches.

Topology C: Hybrid (common in production)

Text diagram: Devices (MQTT) → MQTT Broker → Ingress service/connector → Kafka → Stream processing/consumers → Storage/ML/alerts

  • When it works: you have real devices and you also need a durable event backbone.
  • Strength: MQTT handles edge realities, Kafka handles data platform requirements.
  • Key design choice: where you perform authentication, normalization, schema validation and topic mapping.

Delivery semantics, ordering and duplicates

Most production IoT systems must handle duplicates, out-of-order events and occasional loss, regardless of protocol. Design for it explicitly.

MQTT delivery semantics (QoS) in practice

  • QoS 0: fastest, can lose messages. Great for high-rate telemetry where you can tolerate gaps.
  • QoS 1: at-least-once. You can receive duplicates, so your backend should be idempotent.
  • QoS 2: protocol-level exactly-once between client and broker, but it increases latency and state. End-to-end exactly-once is still not guaranteed once you forward data beyond the broker.

Kafka delivery semantics in practice

  • Producer acks control durability tradeoffs. For example, acks=all reduces data loss risk but can increase latency.
  • At-least-once is the common baseline. Duplicates can occur during retries.
  • Exactly-once within Kafka is possible using idempotent producers and transactions, but once you involve external systems (databases, HTTP APIs) you still need idempotency or transactional outbox patterns.

Practical strategy: idempotency keys

For telemetry events, include a stable tuple like (deviceId, bootId, sequenceNumber) or a UUID per message. On the consumer side, deduplicate within a time window (in memory) or with a compacted Kafka topic keyed by that id (for higher durability).

Security and multi-tenancy considerations

Security requirements often drive the architecture more than throughput.

MQTT security checklist

  • TLS everywhere: require TLS for device connections, consider mutual TLS (mTLS) with per-device certificates.
  • Topic-based authorization: enforce publish/subscribe ACLs so a device can only access its own topics.
  • Credential rotation: certificates with rotation workflows, or short-lived tokens minted by a device identity service.
  • Tenant isolation: topic namespace design like t/{tenantId}/d/{deviceId}/telemetry.

Kafka security checklist

  • TLS and SASL: enforce encrypted transport and authenticated clients.
  • ACLs and quotas: enforce per-producer and per-consumer permissions, limit noisy tenants.
  • Schema governance: consider Schema Registry and compatibility rules (Avro, Protobuf, JSON Schema) to prevent breaking downstream consumers.

Operations: scaling, upgrades, cost and failure modes

Operational reality is where MQTT vs Kafka for IoT data pipelines becomes concrete: disk fills, partitions explode, certificates expire and a single misconfigured client can create a storm.

Scaling characteristics

  • MQTT: scaling often centers on number of concurrent connections, subscription fanout and broker clustering/bridging. Watch CPU for TLS and memory for sessions and queued messages.
  • Kafka: scaling often centers on partitions, disk throughput, replication traffic and consumer lag. Watch broker disk, page cache, network and controller health.

Failure modes to plan for

  • MQTT broker overload: persistent sessions queue too much for offline clients, memory pressure builds. Mitigation: limits per client, message expiry (MQTT 5), bounded queues and gateway buffering.
  • Kafka disk pressure: retention too high or topic explosion fills disks. Mitigation: explicit retention per topic, tiered storage (if available), capacity planning.
  • Hot partitions: a few devices dominate traffic. Mitigation: partitioning strategy, per-tenant topics, backpressure at ingress, batching.

Cost and complexity

  • MQTT can be cheaper and simpler for pure device messaging, especially when paired with a time series database and a small rules layer.
  • Kafka becomes cost-effective when many systems need the same data, you need replay or you need strong data platform primitives. Managed Kafka shifts cost from ops time to cloud bill.

Working code examples

These examples show a common hybrid approach: publish telemetry over MQTT, then bridge it into Kafka via a small Python service. This is not the only way (you can also use broker plugins, Kafka Connect or managed integrations), but it makes the data flow explicit and testable.

Prerequisites

  • An MQTT broker (for local tests: Mosquitto on localhost:1883)
  • A Kafka cluster (for local tests: Kafka on localhost:9092)
  • Python 3.10+ and pip

Example 1: Publish simulated device telemetry to MQTT (Python)

Step 1: Install dependencies.

# Install the MQTT client library for Python
pip install paho-mqtt

Step 2: Run this publisher. It publishes JSON telemetry every second to a device topic.

# Publishes simulated IoT telemetry to an MQTT broker using paho-mqtt
import json
import time
import random
import paho.mqtt.client as mqtt

MQTT_HOST = "localhost"
MQTT_PORT = 1883
DEVICE_ID = "device-123"
TOPIC = f"t/demo/d/{DEVICE_ID}/telemetry"

client = mqtt.Client(client_id=f"pub-{DEVICE_ID}")
client.connect(MQTT_HOST, MQTT_PORT, keepalive=60)
client.loop_start()

seq = 0
try:
    while True:
        seq += 1
        payload = {
            "deviceId": DEVICE_ID,
            "ts": int(time.time() * 1000),
            "seq": seq,
            "tempC": round(20 + random.random() * 5, 2),
            "rpm": random.randint(1200, 1800),
        }
        client.publish(TOPIC, json.dumps(payload), qos=1)
        time.sleep(1)
except KeyboardInterrupt:
    pass
finally:
    client.loop_stop()
    client.disconnect()

Example 2: Bridge MQTT topics into Kafka (Python)

Step 1: Install dependencies.

# Install MQTT and Kafka client libraries for Python
pip install paho-mqtt kafka-python

Step 2: Run this bridge. It subscribes to the MQTT telemetry wildcard and produces each message into a Kafka topic. It uses the deviceId as the Kafka message key to preserve per-device ordering (assuming a stable partitioner).

# Bridges MQTT telemetry topics into Kafka using paho-mqtt and kafka-python
import json
from kafka import KafkaProducer
import paho.mqtt.client as mqtt

MQTT_HOST = "localhost"
MQTT_PORT = 1883
MQTT_SUB = "t/demo/d/+/telemetry"

KAFKA_BOOTSTRAP = "localhost:9092"
KAFKA_TOPIC = "iot.telemetry.demo"

producer = KafkaProducer(
    bootstrap_servers=KAFKA_BOOTSTRAP,
    key_serializer=lambda k: k.encode("utf-8"),
    value_serializer=lambda v: json.dumps(v).encode("utf-8"),
    linger_ms=20,
    acks="all",
    retries=10,
)

def on_connect(client, userdata, flags, rc):
    if rc != 0:
        raise RuntimeError(f"MQTT connect failed with rc={rc}")
    client.subscribe(MQTT_SUB, qos=1)

def on_message(client, userdata, msg):
    try:
        event = json.loads(msg.payload.decode("utf-8"))
        device_id = event.get("deviceId", "unknown")

        # Produce to Kafka. The key helps keep per-device ordering within Kafka partitions.
        producer.send(KAFKA_TOPIC, key=device_id, value=event)
        producer.flush(timeout=2)
    except Exception as e:
        # In production, route failures to a dead-letter topic and include the raw payload
        print(f"Bridge error: {e}")

client = mqtt.Client(client_id="mqtt-to-kafka-bridge")
client.on_connect = on_connect
client.on_message = on_message
client.connect(MQTT_HOST, MQTT_PORT, keepalive=60)

try:
    client.loop_forever()
finally:
    producer.close()

How to validate end-to-end quickly

  1. Start Mosquitto (or your broker) locally.
  2. Start Kafka locally.
  3. Run the bridge script.
  4. Run the publisher script.
  5. Consume from Kafka using your preferred tool (for example, kafka-console-consumer) and verify JSON messages arrive.

Practical recommendations and a selection checklist

Use this checklist to choose a direction quickly, then validate with a short load test and failure injection (disconnect devices, slow consumers, fill disk, rotate certificates).

Choose MQTT first when

  • You connect real devices over unreliable networks.
  • You need simple pub/sub for commands and telemetry.
  • You need retained messages, LWT and session behavior.
  • Your downstream requirements are simple (few consumers, limited replay needs).

Choose Kafka first when

  • Your producers are gateways or services, not constrained devices.
  • You need retention and replay as a core capability.
  • You expect multiple independent consumer groups and evolving analytics.
  • You need a data platform backbone with stream processing.

Choose a hybrid design when

  • You need device-friendly ingress and data-platform-friendly distribution.
  • You want MQTT for commands (southbound) and Kafka for event distribution (northbound).
  • You plan to scale teams and consumers over time.

Topic and key design tips (avoid pain later)

  • MQTT topic convention: t/{tenant}/d/{deviceId}/telemetry, t/{tenant}/d/{deviceId}/cmd, t/{tenant}/d/{deviceId}/state.
  • Kafka topic convention: separate raw vs normalized: iot.telemetry.raw, iot.telemetry.normalized.
  • Kafka partition key: use deviceId (or tenantId|deviceId) to preserve per-device ordering.
  • Schema strategy: version fields in payloads and enforce compatibility rules early.

Answering the common question: which is “better”?

In most real deployments, “better” means “better for a specific zone.” If you treat MQTT as the edge protocol and Kafka as the backend event log, MQTT vs Kafka for IoT data pipelines stops being a rivalry and becomes a clean separation of concerns. If you must pick one technology end-to-end, let your hardest constraint decide: edge connectivity favors MQTT, replay and multi-consumer data platforms favor Kafka.

Final note: If you are planning for long-term analytics, auditing and reprocessing, do not skip the retention and replay discussion. Those requirements often arrive later and they are expensive to retrofit without Kafka or a similar durable log.

Conclusion

MQTT is a strong fit for device connectivity, command and control and lightweight telemetry ingress, while Kafka is a strong fit for durable retention, replay and scalable backend consumption. For many teams, the most robust approach to MQTT vs Kafka for IoT data pipelines is a hybrid pipeline: MQTT at the edge, Kafka as the backend backbone, with a clear normalization and security boundary between them.