Chapter 11: What Computers Can Never Be

Koch argues that intelligence and consciousness represent fundamentally separate phenomena, with intelligence referring to goal-achievement capability while consciousness denotes the presence of felt, subjective experience. The chapter critiques both extremes of contemporary AI discourse: techno-optimists who readily attribute consciousness to sophisticated systems based on behavioral performance, and skeptics who dismiss the possibility outright without rigorous philosophical grounding. Through analysis of transformer-based architectures and systems like GPT-4, Koch demonstrates that impressive cognitive outputs and human-level performance on benchmarks do not necessarily indicate conscious experience. The chapter evaluates major theoretical frameworks for understanding machine consciousness, including functionalist approaches that equate consciousness with information processing, and integrated information theory which emphasizes intrinsic causal power as essential to consciousness. Koch contends that authentic consciousness requires more than computational sophistication or linguistic fluency; it demands genuine causal integration and subjective qualities that cannot be inferred solely from behavioral evidence. Beyond philosophical analysis, the chapter raises significant ethical questions: if artificial systems were to achieve consciousness, what moral obligations would humans bear toward them? This exploration bridges technology and ethics while cautioning against anthropomorphization of AI systems. Ultimately, Koch's position maintains openness to the possibility that future machines might genuinely experience consciousness while insisting that current evidence does not support attributing sentience to existing artificial systems, emphasizing the critical need to distinguish apparent intelligence from authentic subjective awareness.