Kafka architecture, explained from the original paper

A Kafka topic is a named event stream, such as pageviews or clicks. Kafka splits each topic into partitions so data can be spread across brokers and consumed in parallel.

This is the main scaling lever. More partitions allow more brokers to store data and more consumers to process data. Kafka preserves order within a partition, not across the entire topic.

Producers append messages to the end of a partition. Each message is addressed by an offset, which is just its logical position in the log.

This design avoids heavy per-message indexes. It also makes replay natural: a consumer can move its offset backward and read the same data again.

Kafka does not push messages to consumers. Consumers ask brokers for records from a specific offset and fetch batches of data.

Within a group, a partition is owned by one consumer at a time. That keeps ordering simple and avoids locking between consumers. Different groups can read the same stream independently.

Kafka brokers store bytes and serve fetch requests. They do not track every consumer's delivery state. In the original design, consumer offsets and partition ownership were coordinated through ZooKeeper.

LinkedIn ran Kafka clusters in live datacenters for real-time consumers, then copied streams to an analysis datacenter near Hadoop and the data warehouse.

The result was one pipeline for online and offline consumers, with about 10 seconds of end-to-end latency in their production setup.

Kafka became fast because it narrowed the problem. It traded heavyweight queue semantics for append-only logs, batched I/O, pull-based reads, consumer-managed offsets, and partition-level parallelism.

The architecture is not just a broker cluster. It is a storage model: durable streams with independent bookmarks for every consumer group.