Chapter 10: Batch Processing & Data Pipelines

The fundamental principles are rooted in the Unix philosophy, emphasizing the composition of small tools—like awk or sort—using a uniform interface (pipes/files) to achieve powerful data analysis, a method that elegantly handles datasets larger than available memory through sorting techniques that spill data efficiently to disk. This philosophy is scaled up in MapReduce, a foundational distributed batch processing framework that operates on distributed filesystems such as HDFS, employing a shared-nothing architecture for massive parallelism. MapReduce jobs execute in stages: mappers transform input records into key-value pairs, which are then partitioned, sorted, and shuffled to the reducers, where all data associated with a single key is aggregated. Key data manipulation tasks, such as joins and grouping (sessionization), are often implemented as reduce-side joins (like sort-merge joins), leveraging the framework's sorting mechanism to collocate related records by a common key, thus maximizing local computation and minimizing random network requests to external systems. Performance challenges arise from skew or hot keys (linchpin objects), where disproportionate data volumes for a single key slow down execution; advanced algorithms compensate by randomizing assignment to multiple reducers. Faster alternatives, such as map-side joins (including broadcast hash joins for small datasets and partitioned hash joins), avoid the costly shuffle and sort steps under specific data organization assumptions. The output philosophy of batch systems mirrors Unix's immutability: inputs are never changed, and outputs are fully replaced, which is crucial for achieving human fault tolerance and enabling safe automatic retries of failed tasks by the framework. While historically preceded by massively parallel processing (MPP) databases, MapReduce gained prominence by supporting diverse data formats (schema-on-read or the "sushi principle") and heterogeneous workloads, avoiding the upfront modeling constraints of traditional databases. Furthermore, its design specifically catered to environments with high rates of non-hardware-related task termination (preemption) by prioritizing task-level fault recovery. Moving beyond MapReduce, modern dataflow engines like Spark, Tez, and Flink improve efficiency by treating entire workflows as a single job, reducing the need to fully materialize intermediate state to the distributed filesystem between every stage. These dataflow systems model computations as flexible operators in a Directed Acyclic Graph (DAG) and use techniques like Resilient Distributed Datasets (RDDs) or state checkpointing for rapid fault recovery through recomputation. For specific problems like PageRank or transitive closure, iterative computation models like Pregel (or Bulk Synchronous Parallel, BSP) are used for graph processing, allowing vertices to maintain state across fixed rounds of message passing. Ultimately, high-level APIs (Hive, Spark SQL) introduce declarative programming to batch processing, allowing query optimizers to select the most efficient join strategies and apply vectorized execution, bridging the gap between general-purpose frameworks and high-performance MPP databases.