AEM Workflows & Asset Management

Traditional message queues (like RabbitMQ) delete messages as soon as consumers acknowledge them. In contrast, Apache Kafka is a distributed append-only commit log that persists messages for a configured retention period, allowing multiple independent systems to read the same stream at their own pace.

The Distributed Log Anatomy

At its core, a Kafka topic is divided into one or more Partitions. A partition is an ordered, immutable sequence of records that is continually appended to.

Partition 0:  [Msg 0][Msg 1][Msg 2][Msg 3][Msg 4] ---> (Append Only)
Partition 1:  [Msg 0][Msg 1][Msg 2]
Partition 2:  [Msg 0][Msg 1][Msg 2][Msg 3]

• Each record in a partition is assigned a sequential ID called an Offset.
• Offsets are monotonically increasing numbers that uniquely identify a message's position within a partition.
• Consumers track their reading progress by periodically saving ("committing") the offset of the last read message.

Consumer Groups & Horizontal Scaling

A Consumer Group is a collection of consumers that cooperate to consume data from one or more topics. Kafka guarantees that each partition is consumed by exactly one consumer within a consumer group.

Topic A (4 Partitions)
  P0 -------------> Consumer 1 (Group A)
  P1 -------------> Consumer 2 (Group A)
  P2 -------------> Consumer 3 (Group A)
  P3 -------------> Consumer 3 (Group A)

,[object Object],

Consumer Group Rebalancing

When a consumer leaves or joins a group (e.g., during scale-up or crash), Kafka triggers a Rebalance, shifting partition assignments to ensure even distribution. Rebalances can cause a brief pause in consumption, so tuning timeouts (like session.timeout.ms and max.poll.interval.ms) is critical.

Data Replication & High Availability

Kafka partition logs are replicated across multiple Brokers (servers) to ensure fault tolerance.

• Leader: The broker that handles all read and write requests for a given partition.
• Follower: Brokers that replicate log entries from the Leader passively.
• In-Sync Replicas (ISR): The subset of followers that are caught up with the leader's log.

Write Guarantees (acks)

Producers can customize their write acknowledgement configuration to trade off performance for data safety:

1. acks=0: Producer doesn't wait for any broker acknowledgement. Maximum throughput, high risk of data loss.
2. acks=1: Producer waits until the partition leader writes the record to its local log. Moderate safety, moderate speed.
3. acks=all (or -1): Producer waits for all In-Sync Replicas (ISR) to acknowledge the write. Absolute safety, highest latency.

Key Kafka Performance Features

How does Kafka handle millions of messages per second on modest hardware?

1. Sequential Disk Access

Disk seeks are slow, but sequential writes to disk are extremely fast—comparable to memory speeds. By using an append-only structure, Kafka avoids disk seeking.

2. Zero-Copy Operations

Usually, sending data from file to socket requires copying data four times between kernel space and user space. Kafka uses the Linux kernel sendfile API to transfer bytes directly from the OS page cache to the network socket.

Normal Copy:  Disk -> Page Cache -> User Space Buffer -> Socket Buffer -> NIC
Zero-Copy:    Disk -> Page Cache -----------------------------------------> NIC

3. Record Batching

Kafka groups multiple messages together in batches. This reduces network round-trip overhead and enhances compression ratios since similar text compresses better in larger blocks.