Chapter 5: Birds of a Feather

The author contrasts frequentist and Bayesian philosophical approaches to probability, establishing why Bayesian methods have become increasingly central to contemporary machine learning practice. Bayes's theorem emerges as the key mathematical tool enabling machines to update beliefs based on observed evidence, a process formalized through Bayesian decision theory. The chapter develops students' intuition for probability by examining common reasoning errors highlighted through the Monty Hall problem, then systematically introduces probability distributions, random variables, and their characterizing statistics including mean, variance, and standard deviation. A critical distinction is drawn between discrete probability mass functions and continuous probability density functions, with particular emphasis on the Bernoulli distribution for binary outcomes and the normal distribution for modeling real-valued data. Two essential parameter estimation techniques are presented: maximum likelihood estimation identifies parameter values that make observed data most probable under a candidate model, while maximum a posteriori estimation incorporates prior beliefs to regularize these estimates. The chapter then contrasts two fundamental classifier architectures through the Bayes optimal classifier, which makes theoretically perfect predictions when probability distributions are known, and the naïve Bayes classifier, which achieves practical effectiveness by assuming feature independence despite this assumption rarely holding in real data. Applications spanning medical diagnosis and historical document analysis illustrate how these probabilistic frameworks enable machines to identify patterns, make predictions, and navigate the practical trade-off between accuracy and computational cost, revealing that successful machine learning often depends on accepting simplified models over theoretically optimal but computationally intractable solutions.