Message Queues vs Event Logs: Why Kafka Isn't a Queue

"We need a queue" is where a lot of asynchronous designs start, and the choice that follows — RabbitMQ or SQS on one side, Kafka or Pulsar or Kinesis on the other — is usually made on familiarity. They are genuinely different data structures, and picking the wrong one shows up months later as "we can't reprocess that".

The difference is what happens to a message after it's consumed.

Delete on acknowledge, or keep and track

A queue hands a message to a consumer and deletes it once acknowledged. Its state is "what work is left". A queue with nothing in it is a queue that has done its job.

A log is an append-only sequence. Consuming doesn't remove anything — it advances that consumer's offset, a bookmark saying how far it has read. The data leaves on a retention policy, not on consumption.

Every practical difference falls out of that one design decision.

What the log buys you

Replay. Rewinding an offset reprocesses history. When a consumer ships a bug that mangles three hours of events, you fix the code, reset the offset, and reprocess. With a queue, those messages are gone — the only recovery is whatever you happened to persist elsewhere.

Fan-out without coordination. A new service reads the same topic from offset 0 with no change to the producer and no effect on existing consumers. Adding a consumer to a queue means adding a new queue and a routing rule, and it still gets nothing that happened before it existed.

Lag as a first-class metric. Log end offset minus committed offset is a precise, per-consumer number: how far behind, in messages. Queue depth tells you the backlog exists but not who is behind.

Ordering within a partition. A log partition is strictly ordered, and a single consumer reads it in order. Getting the same guarantee from a queue means one consumer and no parallelism.

What the queue buys you

Per-message work distribution. Ten consumers on one queue share the work message by message; a slow message doesn't block the others. In a log, parallelism is bounded by partition count, and one slow message blocks its whole partition until it's handled.

Per-message acknowledgement and redelivery. A failed message is redelivered, often after a visibility timeout, and eventually lands in a dead-letter queue. In a log there is no per-message ack — an offset is a single number, so a poison message means either blocking the partition or committing past it and handling the failure yourself.

How redelivery works is worth knowing precisely. In SQS, receiving a message doesn't remove it — it hides it for a visibility timeout. If the consumer deletes it before the timeout, it's done; if the consumer crashes or takes too long, the message reappears for someone else. Set the timeout comfortably above your worst-case processing time, or extend it while working; otherwise a slow-but-healthy consumer and a second consumer will both process the same message. RabbitMQ does the same with unacknowledged messages when a consumer's connection drops.

Simpler operations. No partitions, no consumer group rebalancing, no retention sizing. For "send the welcome email", that simplicity is the correct answer.

Priority and delay. Priority queues and scheduled delivery are natural in a queue and awkward in a log, where order is fixed by append order.

Partitions and the ordering trap

A log's ordering guarantee is per partition, not per topic. The partition is chosen by the message key:

await producer.send({
  topic: 'orders',
  messages: [{
    key: order.userId,          // ← this decides the partition
    value: JSON.stringify(event),
  }],
});

With userId as the key, all events for one user land in one partition and stay ordered relative to each other, while different users spread across partitions for parallelism. Change that key to a random UUID and you get perfect balance and no ordering guarantee for anything.

This is where most Kafka ordering bugs come from: OrderCreated and OrderCancelled for the same order sent with different keys, arriving on different partitions, processed out of order.

Partition count also caps consumer parallelism. Eight partitions means at most eight consumers doing useful work in a group; a ninth sits idle. And increasing the partition count later changes the key-to-partition mapping, which breaks ordering across the change — so size it with room to grow.

Sizing partitions

A rough method: measure how many messages a second one consumer instance can process, divide your peak rate by that, and double it for headroom and growth. If one consumer handles 500 messages a second and peak is 4,000, you need at least 8 consumers working in parallel — so 16 partitions. Don't overshoot into the thousands per topic "just in case"; every partition costs memory, file handles and rebalancing time on the brokers.

Consumer groups and rebalancing

In a log, "a set of consumers sharing the work" is a consumer group. The broker assigns each partition to exactly one member of the group, and the group commits one offset per partition. A second group reading the same topic gets its own assignment and its own offsets — that's how fan-out works without copying data.

When a consumer joins, leaves, or stops sending heartbeats, the group rebalances: partitions are reassigned. Two consequences matter in practice:

• Processing pauses for the group during a classic rebalance. Deploying 20 consumer instances one by one can trigger 20 rebalances. Newer protocols (incremental cooperative rebalancing) move only the partitions that need to move, and static membership lets a restarting consumer reclaim its partitions without a rebalance at all.
• Uncommitted work is redone. A partition moved mid-batch is picked up by its new owner from the last committed offset, so the batch is processed again. Another reason consumers must be idempotent.

A consumer that takes too long on one batch — longer than the max.poll.interval — is presumed dead and kicked out, triggering a rebalance, after which it's still processing a batch it no longer owns. Keep batches small enough to finish comfortably within that interval.

Poison messages in a log

A queue has an answer for a message that always fails: redeliver it a few times, then move it to a dead-letter queue. A log has no per-message state, so you build the equivalent:

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    try {
      await handle(message);
    } catch (err) {
      if (isRetryable(err) && attempts(message) < 3) {
        // park it on a retry topic, consumed later with a delay
        await producer.send({
          topic: `${topic}.retry`,
          messages: [{ key: message.key, value: message.value,
                       headers: { ...message.headers, attempts: String(attempts(message) + 1) } }],
        });
      } else {
        await producer.send({ topic: `${topic}.dlq`, messages: [message] });
      }
      // either way, commit past it so the partition keeps moving
    }
  },
});

The trade-off is explicit: moving a failed message to a retry topic lets the partition continue, but it also breaks ordering for that key — a later event for the same order can be processed before the retried one. If per-key order matters, the alternative is to block that partition and retry in place, accepting that one bad message stalls everything behind it. Choose per topic, deliberately.

Retention and compaction

A log keeps messages by time or size — seven days, or 500 GB per partition — after which old segments are deleted, whether or not anyone read them. Retention is your replay window: if a bug is noticed after the data is gone, replay can't help. Size it to cover at least your realistic "time to notice and fix" plus a margin.

Log compaction is the other mode: instead of deleting by age, keep only the latest message for each key. A compacted topic of userId → profile becomes a durable, replayable snapshot of current state — a new service can read it from the start and build a full copy of every user's latest profile. Compaction is how logs double as the backbone for event sourcing and CQRS read models and for change data capture.

Delivery semantics, honestly

Both systems advertise "at least once" by default, and both mean it: duplicates happen. The consumer crashed after processing and before committing; the offset commit failed; a rebalance replayed a batch.

"Exactly once" exists in Kafka via transactions, but it's exactly-once processing within Kafka — read, transform, write, commit offsets, all in one transaction. The moment your consumer calls an external API or writes to another database, that guarantee ends at the boundary.

The durable answer is the same in both systems: make consumers idempotent. Key writes by event ID, use upserts instead of inserts, record processed IDs. Idempotency is cheaper to build than exactly-once delivery is to guarantee.

When the consumer writes to a relational database, the cleanest version records the event ID in the same transaction as the effect:

async function onOrderPaid(event) {
  await db.tx(async (t) => {
    const fresh = await t.result(
      `INSERT INTO processed_events (event_id) VALUES ($1)
       ON CONFLICT DO NOTHING`, [event.id]);
    if (fresh.rowCount === 0) return;             // seen it — skip
    await t.none(
      `UPDATE orders SET status = 'paid' WHERE id = $1`, [event.orderId]);
  });
}

Either both the effect and the "processed" marker commit, or neither does. A redelivered event finds its ID already recorded and does nothing. The producer side has the mirror-image problem — writing to the database and publishing the event without one getting lost — and the outbox pattern solves it the same way.

Backpressure

What happens when consumers fall behind is different in each model, and it matters for capacity planning:

• A queue grows. Most brokers keep messages in memory and on disk, and a very deep queue can slow the broker itself — RabbitMQ in particular degrades when queues hold millions of messages. Producers may get throttled or rejected.
• A log doesn't care how far behind a consumer is: messages are sequential files on disk either way, and a lagging consumer reads them later. The risk is different — fall behind by more than the retention period and the oldest unread messages are deleted before you get to them.

So queue backlogs are an operational emergency sooner, and log backlogs are a data-loss risk later. Alert on queue depth growth for one, and on consumer lag as a fraction of retention for the other.

The tools, mapped

The hybrids are worth noticing. Redis Streams and Pulsar keep a log but add per-message acknowledgement for consumers that want queue semantics. The two models aren't rival products so much as two ends of a design space, and newer systems let you pick per subscription.

Choosing

Reach for a queue when work needs doing once, consumers are interchangeable, and history has no value after the work completes — sending emails, resizing images, generating PDFs, background jobs.

Reach for a log when the events are the data: several services care about them, you'll want to replay them, the order matters, or you expect to add consumers you haven't thought of yet — user activity, order lifecycle events, change data capture, analytics pipelines.

And running both is normal. An event log as the backbone for facts about what happened, with plain queues for the fire-and-forget jobs, is a common and reasonable shape. A common bridge is a small consumer that reads the log and enqueues jobs for the slow, per-item work.