Chapter 9: Failure Detection in Distributed Databases

Essential properties of failure detectors are examined, including liveness guarantees that ensure eventual detection of failed nodes and safety requirements that prevent false declarations of node death, both crucial for maintaining the integrity of consensus protocols and distributed transactions. The chapter systematically presents various detection strategies, starting with basic ping mechanisms and heartbeat protocols that provide simple health monitoring capabilities. Advanced approaches are then introduced, including outsourced heartbeat systems that leverage neighboring node perspectives to improve detection reliability, and gossip-based protocols where nodes periodically exchange heartbeat information to build collective awareness of cluster health. A significant portion focuses on the phi-accrual failure detector, an innovative approach that abandons binary alive-or-dead classifications in favor of continuous suspicion levels calculated from statistical analysis of heartbeat arrival patterns, enabling adaptive responses to varying network conditions and load scenarios. The chapter concludes with an examination of group-based failure propagation mechanisms, exemplified by services like FUSE, which transform individual node failures into cluster-wide notifications to ensure comprehensive failure awareness even during complex network partition scenarios, thereby maintaining system consistency and enabling appropriate recovery actions across distributed database architectures.