Apache Kafka & Event Streaming Architecture

Picture your checkout service. A customer clicks "Place Order," and your code calls the inventory service, then the payment service, then the email service, then the analytics service — one after another, synchronously, waiting for each to respond before moving to the next. Then one day the email provider has a slow afternoon. Now every checkout in your system is waiting an extra six seconds on a service that has nothing to do with whether the order actually succeeded.

That's the problem. The order service shouldn't need to know or care how many downstream systems want to react to "an order was placed." It should just announce the fact and move on. That's the entire idea behind Apache Kafka: instead of calling services directly, you write an event to a durable, append-only log, and anyone who cares can read it — at their own pace, independently of each other, without blocking the thing that produced the event.

This is different from a traditional message queue like RabbitMQ, which deletes a message the moment a consumer acknowledges it. Kafka keeps messages around for a configured retention period, which means multiple independent systems — inventory, email, analytics, fraud detection — can all read the same stream without competing for the same message.

The Distributed Log Anatomy

At its core, a Kafka topic is divided into one or more Partitions. A partition is an ordered, immutable sequence of records that is continually appended to.

Partition 0:  [Msg 0][Msg 1][Msg 2][Msg 3][Msg 4] ---> (Append Only)
Partition 1:  [Msg 0][Msg 1][Msg 2]
Partition 2:  [Msg 0][Msg 1][Msg 2][Msg 3]

• Each record in a partition is assigned a sequential ID called an Offset.
• Offsets are monotonically increasing numbers that uniquely identify a message's position within a partition.
• Consumers track their reading progress by periodically saving ("committing") the offset of the last read message.

Why partitions at all, instead of one giant log? Because a single log can only be read as fast as one consumer can read it. Splitting the topic into partitions is what lets you parallelize consumption — which brings us to consumer groups.

Consumer Groups & Horizontal Scaling

A Consumer Group is a collection of consumers that cooperate to consume data from one or more topics. Kafka guarantees that each partition is consumed by exactly one consumer within a consumer group — this is what lets you scale out processing without two workers double-handling the same order.

Topic A (4 Partitions)
  P0 -------------> Consumer 1 (Group A)
  P1 -------------> Consumer 2 (Group A)
  P2 -------------> Consumer 3 (Group A)
  P3 -------------> Consumer 3 (Group A)

If you have more consumers than partitions in a group, the extra consumers sit idle doing nothing. This trips people up constantly — adding a fourth consumer to a 3-partition topic gives you zero extra throughput. To actually scale consumption, you have to increase the partition count, not just add consumers.

Consumer Group Rebalancing

When a consumer leaves or joins a group (a deploy, a crash, an autoscaler kicking in), Kafka triggers a Rebalance, shifting partition assignments to spread the load evenly again. In production, this is usually where things get painful: a rebalance pauses consumption for every member of the group while assignments are recalculated, not just for the consumer that changed. If your deploys restart consumers one at a time without care, you can end up in a rebalance storm — each restart triggering another rebalance before the last one finishes. Tuning session.timeout.ms and max.poll.interval.ms correctly is usually the fix.

Data Replication & High Availability

Kafka partition logs are replicated across multiple Brokers (servers) so that losing one machine doesn't mean losing data.

• Leader: The broker that handles all read and write requests for a given partition.
• Follower: Brokers that replicate log entries from the leader passively.
• In-Sync Replicas (ISR): The subset of followers that are caught up with the leader's log.

Write Guarantees (acks)

This is where the real decision-making happens. acks isn't a performance-tuning knob you set once — it's a per-topic business decision about how much data loss you're willing to risk in exchange for speed.

1. acks=0: Producer doesn't wait for any broker acknowledgement. Fastest option, but if the leader dies before the write lands, that message is just gone.
2. acks=1: Producer waits until the partition leader writes the record to its local log. Reasonable middle ground — you can still lose data if the leader crashes before followers replicate it.
3. acks=all (or -1): Producer waits for every in-sync replica to acknowledge the write. Slowest option, but the safest.

In practice, I've seen teams use acks=all for payment and order events — losing one of those is a customer-facing incident — and acks=1 or even acks=0 for clickstream or analytics events, where losing a handful of page-view events during a broker failover is a non-issue nobody will ever notice.

Key Kafka Performance Features

How does Kafka handle millions of messages per second on hardware that isn't particularly exotic? Three specific design decisions:

1. Sequential Disk Access

Disk seeks are slow, but sequential writes to disk are extremely fast — comparable to memory speeds. By using an append-only structure, Kafka avoids random disk seeking entirely.

2. Zero-Copy Operations

Normally, sending data from a file to a network socket requires copying that data four times between kernel space and user space. Kafka uses the Linux kernel's sendfile API to transfer bytes directly from the OS page cache to the network socket, skipping most of those copies.

Normal Copy:  Disk -> Page Cache -> User Space Buffer -> Socket Buffer -> NIC
Zero-Copy:    Disk -> Page Cache -----------------------------------------> NIC

3. Record Batching

Kafka groups multiple messages together into batches before sending them. This reduces network round-trip overhead and improves compression ratios, since similar records compress better together than one at a time.

Key Takeaways

• Kafka's core value isn't "faster messaging" — it's decoupling the producer of an event from however many consumers eventually need to react to it, so a slow downstream service can never block the thing that triggered it.
• Adding consumers beyond your partition count buys you nothing; partition count is the real throughput ceiling for a consumer group.
• Rebalances pause the entire group, not just the consumer that changed — badly tuned rolling deploys can trigger rebalance storms.
• acks is a per-topic business decision, not a global performance setting. Payment and order events usually need acks=all; clickstream and analytics events usually don't.

Next Steps

Kafka solves how services talk to each other asynchronously. The last piece is what those services actually expose to the outside world — the API contract that clients (mobile apps, frontends, partner integrations) depend on. That's next.