Distributed Message Queue

Intermediate

Overview

A distributed message queue accepts messages from producers, stores them durably, and delivers them to consumers, decoupling the two sides in time and in load. This is a design-a-Kafka question: the defining challenges are durability (never lose an acknowledged message), high throughput (millions of messages per second), horizontal scalability, and configurable ordering and delivery guarantees.

High-level architecture for Distributed Message Queue

Functional Requirements

Producers publish messages to a named topic.
Consumers subscribe to a topic and read messages, individually or as a consumer group that shares the load.
Messages within a partition are delivered in order.
Messages are retained for a configurable window (time or size) and can be re-read (replay).
Support at-least-once delivery, with idempotent/transactional options for stronger guarantees.

Non-Functional Requirements

Durability: an acknowledged message survives broker crashes (replicated to disk).
High throughput: millions of messages/sec via horizontal scaling.
Scalability: add brokers and partitions without downtime.
Low latency: end-to-end delivery in tens of milliseconds.
High availability: the cluster tolerates broker failures without data loss.

Capacity Estimation

Assume 1M messages/sec at an average size of 1 KB, retained for 7 days.

Throughput / bandwidth:
- Ingress: 1M/s x 1 KB = 1 GB/s write.
- With replication factor 3, disk write traffic ~ 3 GB/s across the cluster.
- If each message is read by an average of 3 consumer groups, egress ~ 3 GB/s.
Storage:
- Per day: 1M/s x 1 KB x 86,400 s = ~86 TB/day.
- Over 7 days: ~86 TB x 7 ~ 600 TB; with replication x3 ~ 1.8 PB.
Brokers: at ~150 MB/s sustained sequential write per disk and multiple disks per node, on the order of tens of brokers carry the write load; more are added for headroom and replication.

The scale forces partitioning (one topic spans many machines) and cheap sequential disk I/O as the core design driver.

High-Level Architecture

The system consists of Producers that send data, Consumers that read data, and a cluster of Brokers that store it. A Coordination Service (like ZooKeeper or KRaft) manages the metadata, detecting broker failures and electing partition leaders. Topics are divided into partitions, which are distributed across the brokers to parallelize load.

Data Model

Entity	Fields / Schema	Storage Choice
topic	name, partition_count, retention, replication_factor	Metadata store (coordination service)
partition	topic, partition_id, leader_broker, replica_brokers	Metadata store
message	offset (monotonic per partition), key, value, timestamp	Append-only segment files on disk
consumer_offset	group_id, topic, partition, committed_offset	Compacted internal topic / KV store

Detailed Design

The Log is the Core Abstraction

Each partition is an append-only commit log written sequentially to disk - the reason a queue built on cheap disks can outrun many in-memory systems, since sequential I/O and the OS page cache do the heavy lifting. Messages are grouped into segment files; retention deletes whole old segments cheaply.

Partitioning for Scale and Ordering

A topic's partitions spread across brokers so writes and reads parallelize. The producer chooses a partition by hashing the message key (e.g., user_id), which guarantees per-key ordering without needing a global order. Order is guaranteed within a partition, never across partitions - the standard trade-off that makes horizontal scaling possible.

Replication for Durability

Each partition has one leader and N-1 follower replicas on other brokers. Producers write to the leader; followers pull and replicate. The leader tracks the in-sync replica (ISR) set. A producer's acks setting tunes durability vs. latency: acks=1 (leader only, fast, small loss window) or acks=all (all ISR must persist, safe). Leader failover is handled by the coordination service via leader election.

Consumer Groups and Offsets

Consumers in a group divide the partitions among themselves - each partition is read by exactly one consumer in the group, so throughput scales by adding consumers up to the partition count. Each consumer tracks its committed offset; it reads forward and periodically commits progress. Because the broker just serves a byte range from a file and does not track per-message acks, it stays simple and fast.

Bottlenecks & Solutions

The number of partitions is the hard upper limit on consumer parallelism. If a topic has 10 partitions, having 11 consumers in a group means 1 consumer sits idle. However, having too many partitions (e.g., 100,000) degrades cluster performance due to ZooKeeper metadata overhead and the OS having too many open file handles. Network egress is also a common bottleneck if many consumer groups are reading from the same cluster.

Interview Follow-up Questions

Q: How does the message queue achieve 'exactly-once' semantics?

Exactly-once is incredibly hard. It requires the Producer to be idempotent (using a producer ID and sequence number so retries aren't duplicated) and the Consumer to commit its read offset and its output (e.g., a DB write) in a single atomic transaction. Kafka supports this via its Transactional API, but it adds latency and complexity.

Q: What happens if a consumer crashes halfway through processing a batch of messages?

The consumer will fail to commit its offset. The broker's coordination service detects the consumer's heartbeat timeout, removes it from the group, and rebalances its partition to another healthy consumer. The new consumer starts reading from the last committed offset, meaning some messages will be redelivered. This requires the consumer logic to be idempotent.