SQS vs Kafka vs Redis Streams: Choosing the Right Event Bus for a Dockerised Stack

Every team hits this fork eventually: the background jobs queue is groaning, a cron-triggered webhook handler is dropping events under load, or product wants an “activity feed” that three different services need to read independently. Someone says “let’s just add Kafka.” Someone else says “isn’t that overkill, can’t SQS do this?” And increasingly, a third voice points out that Redis is already in the docker-compose file, so why not use Streams?

The direct answer: this isn’t really a “which tool is best” question — it’s a “what shape is your problem” question. If you need to hand off discrete tasks for exactly one worker to process and forget about, you want a broker — SQS or RabbitMQ. If you need multiple independent consumers to read the same sequence of events, possibly replay it, and reason about history, you want a log — Kafka or Redis Streams. Picking the wrong model doesn’t just cost you performance; it costs you architecture — you end up bolting replay logic onto a broker or fake-routing logic onto a log, and both are miserable.

This article walks through the architectural models first, then gives you the comparison and a decision framework you can actually apply to a Dockerised Next.js/FastAPI/Django stack on AWS.

Core architecture models: broker vs log

The single most useful mental model for this entire decision is broker vs log.

A message broker (SQS, RabbitMQ) follows a “route and forget” pattern: a producer sends a message, the broker routes it to a consumer, the consumer processes it and acknowledges it, and the message is gone. RabbitMQ is the clearest example of this — it’s a traditional broker where a message is deleted once it’s delivered and acknowledged, which means it cannot replay history, and it can’t let multiple independent consumers each read the same stream of events from the beginning. What RabbitMQ is genuinely excellent at, in exchange for giving up replay, is precise routing — exchanges and routing keys let you direct messages to exactly the queues that should receive them, with per-message acknowledgment semantics that suit fine-grained workflow routing.

A log (Kafka, Redis Streams) flips this. Producers append events to an ordered, retained log, and consumers track their own read position — nothing is deleted just because one consumer read it. That means ten different services can all read the same topic independently, at their own pace, and a new consumer that comes online next month can still replay everything from the start if retention allows it. This is the core reason Kafka and Redis Streams get grouped together despite being very different pieces of infrastructure: they share the log abstraction, not the implementation.

Worth noting: Redis Streams isn’t a dedicated distributed streaming platform the way Kafka is — it’s a log-like data structure that lives inside Redis, offering ordered, retained, replayable messages with consumer groups, but running as part of a general-purpose in-memory data store rather than a purpose-built distributed commit log. That distinction matters more than it sounds like it should, because it shapes retention, durability, and operational cost, which we’ll get to.

Apache Kafka fundamentals

Apache Kafka is an open-source distributed event streaming platform that works as a distributed, partitioned, and replicated commit log. Topics are divided into partitions, and each partition is an ordered, immutable sequence of events that gets appended to on write. That partitioning is what gives Kafka its parallelism: within a consumer group, each partition is consumed by exactly one consumer, which enables load balancing across workers, while separate consumer groups can each independently consume the full topic. This is a genuinely elegant design — it’s how you get both “ten workers share the processing load” and “the analytics team and the billing team both see every event” out of the same topic.

Kafka was originally built at LinkedIn and is designed to handle very high message volumes while retaining every message for as long as you configure it to, which is why it’s the natural fit for event sourcing and CQRS patterns, real-time analytics, log aggregation, and change data capture.

ZooKeeper, KRaft, and operational complexity

Kafka historically depended on Apache ZooKeeper for metadata management and leader election. More recent Kafka architectures move to a self-managed metadata quorum, known as KRaft, specifically to simplify the architecture and reduce operational overhead that ZooKeeper introduced. Either way, you’re running a distributed cluster: partition rebalancing, retention tuning, broker scaling, and metadata quorum management are all things you — or your managed service provider — now own. This is the single biggest thing people underestimate when they reach for Kafka “because it’s the industry standard.” It is a powerful system. It is also a distributed system, with all the 2am-pager implications that phrase carries.

Kafka stores events durably on disk for a configurable retention period, which is what enables replay, and it achieves its performance by leveraging sequential disk I/O through the OS file system rather than fighting it with random access patterns.

Redis Streams fundamentals

Redis Streams were introduced in Redis 5.0 as an append-only log data structure with native consumer group support, where every message gets an auto-generated ID made up of a millisecond timestamp plus a sequence number. Producers write with XADD, and consumer groups read with XREADGROUP, which gives you queue-like semantics — multiple workers pulling from the same group, each getting a distinct slice of messages, with backpressure built in. That combination makes Redis Streams genuinely good at absorbing spiky traffic and responding in milliseconds, because you’re appending to and reading from memory rather than going to disk on the hot path.

The catch, and it’s a real one: without a MAXLEN cap, a busy stream will keep growing in memory indefinitely, and an uncapped high-throughput stream is a reliable way to exhaust Redis’s memory and trigger an out-of-memory kill. If you take one operational rule away from this section, make it this: always set a MAXLEN cap on a production stream. It’s a one-line decision that prevents a very bad on-call night.

Amazon SQS fundamentals

Amazon SQS is a fully managed message queuing service from AWS, built specifically to decouple and scale microservices, distributed systems, and serverless applications. “Fully managed” is doing real work in that sentence — you administer no messaging infrastructure yourself, and it integrates directly with AWS services like Lambda, which makes it a near-zero-setup choice if you’re already living in AWS.

SQS gives you two queue types: standard queues, tuned for maximum throughput, and FIFO queues, which add strict ordering guarantees at some throughput cost. That’s the whole mental model — there’s no partitioning scheme to design, no replication factor to pick, no broker count to size. You create a queue, you send and receive messages, and AWS deals with the rest.

RabbitMQ as the broker reference point

RabbitMQ deserves its place in this conversation as the canonical self-hosted broker, because it’s the clearest illustration of what you give up and gain by choosing the broker model over the log model. It routes messages through exchanges using routing keys to the queues that should receive them, with per-message acknowledgment and the Erlang runtime underneath handling concurrency and fault tolerance. If your real requirement is “route this specific message to this specific handler based on type or priority,” RabbitMQ’s routing model will usually produce cleaner code than bending a log-based system to do the same thing.

Throughput and latency compared

Here’s where the architectural differences turn into effects you’ll actually feel in production. Redis Streams is typically lower-latency because it operates primarily in memory, while Kafka’s durability guarantees come from writing to disk before an event is considered committed — though real numbers depend heavily on configuration and workload, so benchmark on your own stack rather than trusting rules of thumb. If your event needs to trigger a UI update or a real-time notification within a tight latency budget, that difference is not academic — it’s the difference between “feels instant” and “feels fine.”

The trade for that speed is retention capacity. Redis Streams are memory-limited via MAXLEN, while Kafka’s retention is configurable for days, weeks, or effectively indefinitely because it’s disk-backed. So the throughput-latency story isn’t simply “Redis is faster” — it’s “Redis is faster for a bounded, memory-sized window of recent events, and Kafka sustains high throughput over a much larger durable window.” Which one wins depends entirely on whether your consumers need last week’s events or just the last five minutes.

Durability, retention, and replay semantics

This is the dimension that most cleanly separates the three systems, and it’s worth treating as its own decision axis rather than folding it into “performance.”

• SQS is built for message delivery and handoff, not long-term replay — once a message is consumed and deleted (or expires), it’s gone, and there’s no concept of rewinding to an arbitrary earlier point in a feed.
• Redis Streams retain what fits in memory, bounded by whatever MAXLEN you set, which makes replay possible but only across a recent, memory-sized window.
• Kafka retains events on disk for however long you configure, which can be short, long, or effectively permanent, making genuine historical replay — rebuilding a service’s state from day one, auditing a full event history — a first-class capability rather than a workaround.

If replay is a feature your product needs — not “nice to have,” but something a consumer will actually invoke — that alone should weight you toward Kafka over Redis Streams, and away from SQS or RabbitMQ entirely.

Delivery guarantees and ordering

All three systems can give you at-least-once delivery, but the path to stronger guarantees differs. Kafka offers at-least-once delivery as its default posture, with the option to layer in exactly-once semantics using Kafka’s transactional support, which coordinates producer writes and consumer offset commits so that a crash-and-retry doesn’t produce duplicate side effects. It’s worth being precise about what “exactly-once” means here — it describes exactly-once processing within Kafka’s own transactional boundary, not an unconditional guarantee that nothing downstream can ever see a duplicate. If your consumer has side effects outside that boundary — calling a third-party API, say — you still need idempotency there.

Redis Streams provide at-least-once delivery by default through their consumer group acknowledgment model. SQS gives you the choice between standard queues (at-least-once, highest throughput, no ordering guarantee) and FIFO queues (strict ordering, with deduplication), which is the same “ordering costs you something” trade-off you see across every one of these systems. The practical takeaway: don’t assume “exactly-once” anywhere in this landscape means “idempotency is no longer your problem.” Design consumers to be idempotent regardless of which system you choose — it’s cheap insurance against the retry behavior every one of these guarantees still permits at the edges.

Operational overhead: managed vs self-hosted

This is where a lot of teams make the decision emotionally rather than rationally, so let’s be blunt about it. Kafka requires manual configuration and ongoing cluster management — partition rebalancing, retention tuning, and ZooKeeper or KRaft quorum management are now your team’s job. Per AWS documentation, SQS is a fully managed queue with no servers to provision. Redis sits in between in practice — you’re already likely running it for caching, so adding Streams doesn’t mean standing up new infrastructure, but it does mean taking on memory sizing, persistence configuration, and MAXLEN discipline as an ongoing operational consideration.

If you already manage Redis for session storage or caching in your stack, Redis Streams is close to “free” operationally. Managed Kafka reduces operational load, but you still design partitions, consumer groups, and schema evolution.

Use-case decision guidance

Stitching the evidence together into recommendations:

• SQS suits applications needing straightforward queuing with minimal setup — sending an order confirmation email after checkout is the textbook case — but it’s a poor fit for complex streaming workloads like real-time data processing in a trading system, where ordering, replay, and sustained throughput at scale matter more than simple handoff.
• Redis Streams are a strong fit for low-latency job queues, short-lived event pipelines, and workloads under roughly 100 million messages a day where keeping events in memory is practical.
• Kafka is the right call for event sourcing, audit logs, a cross-team event backbone shared by ten or more services, and anywhere you need exactly-once semantics for something as unforgiving as financial transactions.
• RabbitMQ earns its keep when the real requirement is complex routing logic between many small, independent consumers rather than a shared, replayable event history.

Ecosystem and stream processing

Kafka’s gravitational pull partly comes from what’s built around it rather than the broker itself — Kafka Streams and ksqlDB for in-place stream processing, Kafka Connect for piping data in and out of other systems, Flink for heavier stream analytics, and Schema Registry for keeping producer and consumer contracts honest as topics evolve. Neither SQS nor Redis Streams has a comparably broad stream-processing ecosystem, which is a legitimate reason to choose Kafka even before you need its raw throughput — if you know analytics, CDC, or complex stream transformations are on your roadmap, the ecosystem is the asset, not just the log.

A direct SQS vs Kafka comparison

Dimension	SQS	Kafka
Model	Message broker (queue)	Distributed partitioned log
Management	Fully managed service (AWS)	Self-managed cluster, or managed service still requiring config
Delivery after read	Message deleted/expires	Retained per configured retention, re-readable
Ordering	FIFO queues only; standard queues unordered	Per-partition ordering
Exactly-once	Not a design goal	Available via transactions, within Kafka’s boundary
Best fit	Task handoff, serverless decoupling, simple job queues	Event sourcing, audit trails, multi-consumer backbones

The pragmatic hybrid pattern

You don’t have to pick exactly one and live with it forever. A sensible pattern at real scale is to run Redis at the edge for low-latency buffering and burst absorption, and Kafka as the durable backbone that holds the system of record and feeds the services that need replay or long retention. One illustrative way this could look, in a Dockerised Next.js/FastAPI stack: an API route writes an event to a Redis Stream for immediate, low-latency fan-out to a WebSocket layer or a live dashboard, while a background worker also forwards that same event into Kafka for durable storage, analytics, and any service that needs to replay history later. You get Redis’s latency where the user feels it, and Kafka’s durability where the business needs it — without forcing either system to do the other’s job.

If you’re running everything in docker-compose today and traffic is still modest, don’t reach for this hybrid prematurely — start with whichever single system matches your dominant access pattern (SQS or a simple broker for task handoff, Redis Streams if you already run Redis and need light replay) and introduce Kafka only when a second independent consumer genuinely needs the same event history. Architecture debt from under-building is cheaper to pay down than the operational tax of running a Kafka cluster nobody yet needs.

Choosing with confidence

Strip away the feature lists and the decision comes down to the question this article opened with: is this a task to hand off once, or an event multiple independent readers need to see — possibly more than once? Answer that, factor in your tolerance for operational overhead and how much history you actually need to retain, and the choice between SQS, Redis Streams, and Kafka stops being a brand debate and starts being an engineering decision you can defend in a design review.

Frequently asked questions

Is Kafka always better than SQS because it’s more powerful? No — power isn’t free. Per AWS documentation, SQS is fully managed, meaning no cluster to administer, while Kafka’s partition rebalancing, retention tuning, and metadata quorum management are ongoing work for your team or a managed provider. Choose based on whether you need log semantics and replay, not based on perceived sophistication.

Can Redis Streams replace Kafka entirely? For workloads under roughly 100 million messages a day where memory-bounded retention is acceptable, Redis Streams can be a lean, low-latency substitute. Beyond that, or where long-term replay and a broader ecosystem of stream processing tools matter, Kafka is the better fit.

Does “exactly-once” mean I can skip idempotent consumer logic? No. Kafka’s exactly-once semantics describe processing guarantees within its own transactional boundary, not a guarantee against every possible downstream duplicate. Build idempotent consumers regardless of which messaging system you choose.

What happens if I forget to cap a Redis Stream’s length? An uncapped stream under real traffic will keep consuming memory indefinitely and can trigger an out-of-memory kill in Redis. Always set MAXLEN before a stream goes anywhere near production traffic.

Is RabbitMQ obsolete now that log-based systems are popular? No — it solves a different problem. RabbitMQ’s exchange and routing-key model is still a strong fit when you need precise message routing between consumers rather than a shared, replayable history.