Chapter 4: Distributed Message Queue Design

Although traditionally message queues are distinct from event streaming platforms, this design incorporates advanced features like long message retention (two weeks), the capacity for repeated message consumption, and guaranteed message ordering within a partition. The system organizes messages into topics, which are sharded using partitioning and distributed across servers known as brokers for high scalability and fault tolerance. The underlying persistence layer relies on the Write-Ahead Log (WAL), an append-only structure optimized for the system's write-heavy, sequential access traffic pattern, with log files divided into manageable segments. High throughput is a critical non-functional requirement achieved by minimizing expensive data copying and aggressively promoting batching across the producer, broker, and consumer flows. Producers use a routing mechanism to send messages to the leader replica of the target partition. Consumers are structured into consumer groups, which manage parallel message consumption using the pull model, allowing consumers to control their processing rate. To maintain message ordering, a critical constraint dictates that only one consumer within a group may read from a single partition. Centralized system coordination, including service discovery and broker leader election, is handled by an external service like ZooKeeper, which also stores metadata (topic properties) and state storage (tracking consumer group offsets). Replication ensures high availability, utilizing In-Sync Replicas (ISR) that synchronize data from the leader replica. Message durability is managed by configurable acknowledgment (ACK) settings (ACK=0 for lowest latency but potential loss, ACK=1 for middle ground, and ACK=all for highest durability). The chapter concludes by discussing data delivery semantics—at-most once, at-least once (allowing duplicates), and the complex exactly once—as well as advanced features such as message filtering using metadata tags and processing delayed or scheduled messages.