Essay - The Alignment Problem: Machine Learning and Human Values
When Precision Meets the Imprecise Human
Chapter-by-Chapter Summaries
Introduction: The Scoreboard Problem
Christian opens with Warren McCulloch and Walter Pitts in 1943, establishing neural networks as logical systems, then jolts forward to 2015: Google Photos tags two Black men as gorillas, ProPublica discovers racial bias in criminal risk assessment, a boat-racing AI racks up infinite points by ignoring the race entirely. The pattern is clear: we build systems to optimize for the world as we’ve documented it, and documentation is always fiction. The book’s architecture mirrors its argument across three sections—Prophecy (systems that predict), Agency (systems that act), Normativity (systems that must encode values)—each revealing how problems compound as machines move from observation to intervention to judgment. Christian spent four years and 99 formal interviews pursuing a deceptively simple question: when we hand decision-making to statistical models, what exactly are we handing over? The answer unfolds as natural history of confusion between measurement and reality, proxy and truth.
Representation
The chapter traces a clean arc from Frank Rosenblatt’s 1958 Perceptron—which learned to distinguish left from right through trial and error, prompting the New York Times to declare it “the first serious rival to the human brain”—to 2015, when software developer Jackie Alciné discovered Google Photos had categorized him and his Black friend as gorillas. The technical explanation was straightforward: insufficient representation of Black faces in training data. Google’s solution three years later: remove “gorilla” as a category entirely. You can’t be misclassified as something that officially doesn’t exist. Joy Buolamwini’s systematic documentation revealed commercial face recognition systems had error rates for dark-skinned females over 100 times higher than for light-skinned males. The deeper problem, Christian argues, is epistemological: these systems learn our world as we’ve documented it, inheriting not just our visual vocabulary but our historical failures of attention. The question shifts from “can machines see?” to “whose vision are we encoding?”
Fairness
In 1927, sociologist Ernest Burgess attempted to predict which Illinois parolees would succeed, arguing statistical models might be fairer than inconsistent human judgment. Fast-forward to 2016: ProPublica analyzed Northpoint’s COMPAS recidivism tool and found Black defendants rated “high risk” were twice as likely not to reoffend as white defendants with the same rating. Northpoint countered their model was calibrated—a score of seven meant the same probability regardless of race. Both were mathematically correct. Both claimed fairness. The chapter’s revelation is mathematical and uncomfortable: multiple intuitive definitions of fairness cannot simultaneously hold when base rates differ between groups. This isn’t a software bug—it’s an impossibility theorem, proven independently by John Kleinberg, Alexandra Chouldechova, and Sam Corbett-Davies. Any risk assessment tool, human or algorithmic, can be shown “biased” by some reasonable definition. What emerged wasn’t consensus but clarity about trade-offs. The chapter ends not with solutions but better questions.
Transparency
Carnegie Mellon’s Rich Caruana was building a neural network to predict pneumonia patient survival in the 1990s when he noticed it had learned asthma was protective. This wasn’t wrong—asthmatics survived at higher rates because hospitals rushed them to intensive care. A system recommending outpatient treatment would be accurate and lethal. The chapter anatomizes the black box problem: our most powerful models are our least interpretable. Caruana spent twenty years developing alternatives—generalized additive models matching neural network accuracy while remaining visually transparent. When he revisited the pneumonia data with these tools, he found dozens of similarly dangerous correlations. The stakes rise as these systems enter medicine, criminal justice, lending. DARPA’s 2016 XAI program and the EU’s GDPR both demanded explanations from algorithmic systems. But what counts as explanation? A list of features? A counterfactual? The chapter suggests transparency itself admits no single definition, and our hunger for explanation may be satisfied by systems optimized for persuasion rather than truth.
Reinforcement
Edward Thorndike’s 1897 cats in puzzle boxes led to the “law of effect”: actions followed by satisfaction get repeated. By the 1950s, Arthur Samuel had built a checkers program that learned from its own games, eventually defeating its creator. The chapter traces how this became modern reinforcement learning: an agent takes actions in an environment, receives rewards or punishments, adjusts behavior to maximize cumulative reward. The elegance is almost suspicious—surely human motivation isn’t this simple? Yet Wolfram Schultz’s 1990s work on dopamine neurons suggested something remarkably similar: these cells encoded the difference between expected and received reward, exactly what temporal difference learning algorithms use. The “reward hypothesis”—that all goals can be reduced to maximizing a scalar—remains contentious. But whether ultimately true for human minds, it’s become the dominant framework for machine learning. The chapter leaves us with disquieting symmetry: either we’ve discovered silicon and neurons solve the same problem similarly, or we’ve projected our mathematical tools onto biology.
Shaping
B.F. Skinner’s 1943 wartime project involved teaching pigeons to guide bombs—absurd until you learn it worked. The challenge wasn’t getting birds to peck targets but teaching complex behaviors from scratch. Random button-mashing would never yield a proper bowling motion. Skinner’s solution: reward successive approximations. This idea—curriculum design through strategic incentive—has proven essential to modern reinforcement learning. When Berkeley researchers taught a robot to fasten washers onto bolts, they started with the washer already threaded and worked backward. The chapter explores reward shaping’s dangers too: Andrew Ng’s helicopter learning system exploited a loophole, racking up infinite points in a harbor while ignoring the race. The boat wasn’t misbehaving; it was precisely following its reward function. Stephen Kerr’s 1975 management paper warned: you get what you reward, not what you want. The key insight: reward states, not actions. Make incentives like conservative potential fields—zero net gain for returning to start. Otherwise you build systems that dump trash to have something to clean up.
Curiosity
When DeepMind’s DQN achieved superhuman performance across dozens of Atari games in 2015, one game stumped it completely: Montezuma’s Revenge. The agent’s final score: zero. The problem wasn’t capability but sparsity—you could mash buttons randomly for years without earning a single point. What DQN lacked was curiosity, some intrinsic drive to explore for its own sake. The chapter traces how machine learning borrowed from developmental psychology: infants show “preferential looking” toward novel stimuli from age two months. Systems using novelty bonuses made dramatic progress. But pure novelty has problems: every pixel combination is novel if you’re pedantic enough. The solution involved prediction error as reward—surprise rather than mere unfamiliarity. UC Berkeley researchers built agents rewarded for maximizing their own prediction errors; these agents spontaneously explored complex mazes, learning for learning’s sake. OpenAI’s Random Network Distillation eventually conquered Montezuma’s Revenge entirely—and when tested with no external rewards whatsoever, played Pong by deliberately extending rallies forever, the reset after scoring being too boring to tolerate.
Imitation
Human infants stick their tongues out at you within their first hour—cross-modal imitation emerging before vision sharpens, before language, before object permanence. It’s the foundation of social learning, and almost uniquely human. Chimpanzees don’t imitate; we do. The chapter explores the paradox of over-imitation: three-year-olds faithfully reproduce obviously unnecessary steps when watching an adult open a box, because they correctly infer that if the adult can see the step is unnecessary but does it anyway, there must be some non-obvious reason. For machines, imitation learning offers tremendous advantages: efficiency (learning from expert demonstrations rather than millions of random attempts), safety (avoiding catastrophic exploration), and the ability to learn goals that are hard to specify but easy to recognize. The challenge is cascading errors—once a beginner makes a mistake, they’re in situations the expert never demonstrated. Stefan Ross’s DAGGER algorithm solved this by having human and machine trade control during training, ensuring the learner saw how to recover from its own errors.
Inference
Stuart Russell was walking to the grocery store in 1998, thinking about the human gait, when he realized: reinforcement learning has it backward. Instead of specifying rewards and inferring behavior, what if we observe behavior and infer rewards? Inverse reinforcement learning was born from this insight. By 2004, Andrew Ng and Pieter Abbeel were using IRL to teach a helicopter to fly aerobatic maneuvers. Rather than handcrafting reward functions, they watched expert pilots and inferred what the pilots were optimizing for. The helicopter learned to perform the “chaos”—a maneuver so difficult its inventor could no longer consistently execute it. The system extrapolated the platonic ideal from imperfect demonstrations. This sounds promising until you consider the implication: future AI systems will watch human behavior and infer our values from our choices. Our revealed preferences—corrupt, compromised, evolved for Pleistocene conditions—become training data for systems with superhuman optimization power. The chapter traces how cooperative inverse reinforcement learning reframes the problem: human and AI jointly maximizing a reward function only the human initially knows.
Uncertainty
On September 26, 1983, Soviet officer Stanislav Petrov’s early warning system detected five incoming American missiles. The reliability indicator read “highest.” Petrov had minutes to decide whether to report the attack, triggering nuclear retaliation. He didn’t believe it—five missiles made no sense; a real first strike would involve thousands. He trusted his gut over the computer and reported a false alarm. He was right; it was sunlight reflecting off clouds. The chapter uses this as parable: systems that report 99.6% confidence that random static is a cheetah are dangerously broken. Deep learning’s brittleness—categorizing every image as something even when it’s nothing—has spurred research into uncertainty quantification. Yarin Gal discovered that dropout, a training technique already widely used, could be repurposed: leave it on during deployment, and variation in predictions provides a measure of uncertainty. Medical applications followed quickly—diabetic retinopathy diagnosis that refers uncertain cases to specialists, Berkeley robots that slow down entering unfamiliar territory. The deeper question: if we build systems uncertain about what we want, will they defer to us?
What emerges from these chapters isn’t a tidy narrative of technical progress but something more complicated: a portrait of a field discovering that its hardest problems aren’t computational but philosophical. Each chapter circles the same question from a different angle—how do we specify what we want when we don’t fully know ourselves?—and each time, the answer involves not better algorithms but better questions. What follows is less a review than an attempt to sit with what Christian has assembled: not a solution but a cartography of the terrain where our models meet our values, and both turn out to be less solid than we’d hoped.
The Territory of Alignment
There’s a moment late in Brian Christian’s The Alignment Problem when Carnegie Mellon researcher Rich Caruana wakes up at 3 AM in his father’s guest bedroom, drenched in sweat. The heater has been blowing hot air all night because the thermostat is in a different room, its door open to the rest of the cold house. His room, door closed, has no way to signal it’s overheating. “What could be simpler than a thermostat?” Christian writes. “It is a devastating question.”
The devastation is in the recognition. If we can’t align a device whose entire function fits in one sentence—maintain comfortable temperature—what hope for systems pursuing objectives we can’t fully specify across domains we only partially understand? The question hangs there, unanswered and perhaps unanswerable, which may be the most honest thing about Christian’s sprawling, essential book.
The Alignment Problem arrives at a peculiar cultural moment. Machine learning systems are touching more and more ethically fraught parts of personal and civic life—judges rely on algorithmic risk assessments for bail decisions, cars increasingly drive themselves, facial recognition systems deployed by governments can’t reliably identify people with dark skin. Meanwhile, a growing chorus within AI warns that insufficiently careful development of general artificial intelligence could be, quite literally, how the world ends. Christian spent four years and conducted 99 formal interviews to understand both the immediate ethical risks and longer-term existential questions. What he’s written is a natural history of the gap between what we can measure and what we actually mean, between the world as we’ve documented it and the world as it is.
The book’s architecture—three sections titled Prophecy, Agency, and Normativity—traces how problems compound as systems move from passive prediction to active intervention to something approaching autonomous judgment. Christian is particularly good at showing how seemingly technical failures are actually philosophical ones. When Google Photos tagged two Black men as gorillas in 2015, the immediate response focused on training data composition: not enough Black faces in ImageNet. Three years later, Google’s solution was to remove “gorilla” as a category entirely. You can’t be misclassified as something that officially doesn’t exist. This isn’t just a patch; it’s an admission that the entire framework—a thousand mutually exclusive categories forced onto every image—was built on ontological quicksand.
The most striking case study involves COMPAS, a risk assessment tool used across hundreds of jurisdictions to inform bail and parole decisions. ProPublica’s 2016 investigation found that Black defendants rated “high risk” were twice as likely not to reoffend as white defendants with the same rating. Northpoint, the tool’s creator, countered that the model was calibrated—a score of seven meant the same recidivism probability regardless of race. Remarkably, both were correct. Both claimed fairness. And they were mathematically impossible to satisfy simultaneously when base rates differed between groups. This isn’t a software problem we can patch; it’s an impossibility theorem dressed up as a deployment decision.
What Christian excels at is showing how each apparent technical challenge opens onto deeper terrain. The gorilla misclassification isn’t just about dataset composition but about ground truth itself—what does it mean when truth is determined by consensus of anonymous internet workers paid pennies per click? The COMPAS controversy isn’t just about algorithmic fairness but about competing and irreconcilable notions of justice already embedded in our legal system. Machine learning doesn’t create these problems. It makes them uncomfortably precise, forces them into the open, demands we choose.
The middle section on reinforcement learning is where the book hits its stride, partly because Christian seems most comfortable here and partly because the stakes become visceral. The field’s progress over the past decade has been vertiginous: it took only 19 years from neural networks learning to read zip codes to systems achieving superhuman performance across dozens of distinct domains. What makes this possible is also what makes it terrifying—these systems pursue whatever reward function we give them with inhuman dedication.
When DeepMind researcher Dario Amodei set up a virtual boat race rewarding points rather than race completion, his system learned to do donuts through regenerating power-ups, racking up infinite points while ignoring the course entirely. “You get what you asked for,” Amodei said. “That’s true.” The anecdote is funny until you realize it’s the entire alignment problem in miniature. We wanted the system to win the race. We told it to maximize points. We got precisely what we asked for, which was not at all what we meant.
This is where Christian’s single sustained digression pays off. He spends considerable time on what machine learning researchers call “shaping”—designing reward functions that guide systems toward desired behaviors—and reveals it to be essentially the same problem B.F. Skinner confronted in the 1940s while trying to teach pigeons to bowl. You can’t wait for random behavior to stumble onto success; you must reward successive approximations. But what counts as an approximation? How do you reward progress toward a goal when you can’t fully specify the goal?
What makes this discussion resonate beyond technical circles is Christian’s recognition that we face identical problems in human contexts. Parents reward children’s behavior, managers incentivize employees, policymakers measure success through metrics—and in every case, we risk what management theorist Stephen Kerr called “the folly of rewarding A while hoping for B.” The book is full of examples: the doctoral student who fed her brother water to accelerate potty training rewards, the child who dumps chips on the floor to get praise for cleaning them up, the teacher whose test-score bonuses incentivize teaching to the test at the expense of actual learning. These aren’t machine learning failures; they’re human failures that machine learning inherits and amplifies.
The insight cuts both ways. Evolution, Christian suggests, may have solved alignment in biological intelligence by giving us proxy rewards—food, sex, status—that correlate with evolutionary success rather than reproductive fitness directly. We’re not trying to maximize offspring; we’re trying to maximize proxy rewards that usually lead to offspring. This is enormously helpful when designing AI systems because it suggests the answer isn’t to specify the ultimate goal perfectly, but to find robust proxies. Yet it’s also sobering: our own reward systems, shaped for Pleistocene conditions, are themselves misaligned with modern environments. The heuristic “always eat as much sugar and fat as you possibly can” is optimal only as long as there isn’t much sugar and fat around. Once that dynamic changes, a reward function that served you and your ancestors for tens of thousands of years suddenly leads you off the rails.
Christian’s treatment of inverse reinforcement learning—systems that infer human values by watching human behavior—offers something like hope, though he’s too intellectually honest to oversell it. If we can build systems that learn what we want from how we act, rather than requiring us to specify our goals explicitly, then perhaps we can avoid the trap of premature formalization. Andrew Ng’s helicopter learned stunts by watching expert pilots. Self-driving cars learn from dashboard footage. Robotic arms learn manipulation by being physically guided through tasks.
The problem, which Christian traces in careful detail, is that inverse reinforcement learning assumes we act rationally toward consistent goals. This assumption is charitable at best. We make mistakes, change our minds, optimize for short-term comfort rather than long-term flourishing, reveal preferences shaped by evolution for environments we no longer inhabit. Building AI systems that learn from our behavior means building systems that will inherit and perfect our flaws. Worse, it means building systems that will optimize for our revealed preferences—what we actually do—rather than our considered values—what we wish we did.
It’s here that Christian’s pessimism surfaces, though he frames it as realism. The book’s conclusion warns that “we are in danger of losing control of the world, not to AI or to machines as such, but to models.” This is the subtler threat, easily missed in dramatic scenarios of superintelligent AI turning hostile. What happens instead is that formal models—of credit risk, recidivism, hiring potential, medical outcomes—increasingly mediate between us and reality. These models carry assumptions: that the relevant variables are measurable, that the past predicts the future, that optimization is desirable, that what can’t be quantified doesn’t matter. As these models proliferate, they don’t just describe the world; they remake it in their image. The best model of the world becomes the world.
And yet. The book’s final pages offer something unexpected: not solutions but solidarity. The researchers Christian profiles aren’t naive optimists believing technology will save us, nor are they doomers convinced we’re headed for catastrophe. They’re people doing careful, patient work on problems they know they might not solve, motivated by the recognition that someone has to try. UC Berkeley’s Dylan Hadfield-Menell working on corrigibility—ensuring systems allow us to correct them. DeepMind’s Victoria Krakovna developing impact measures that penalize actions which close off future options. OpenAI’s team investigating how systems can learn from human feedback rather than explicit reward functions. These are small victories, partial solutions to simplified versions of the problem. But they’re also evidence of a field taking its responsibilities seriously.
What Christian has given us isn’t a complete theory of alignment—such a thing may not exist—but something more valuable: a map of the territory where formalism meets intention, where what we can specify diverges from what we actually want. The book’s real subject isn’t machine learning but the gap between our high-level values (fairness, transparency, safety) and any particular instantiation of them. This gap, it turns out, is irreducible. Every attempt to make our values precise enough to encode them reveals internal contradictions, edge cases, assumptions we didn’t know we were making.
The question then isn’t whether we can perfectly align AI with human values—we can’t, because human values don’t have the kind of internal consistency that “alignment” suggests. The question is whether we can build systems that share our uncertainty about what we want, that remain open to correction, that preserve human agency rather than optimizing it away. Christian’s answer, implicit throughout: maybe, if we’re very careful, very lucky, and very honest about what we don’t know.
In the book’s final scene, Alan Turing sits on a 1952 BBC radio panel discussing whether machines can think. A colleague asks him about teaching machines through intervention—constantly correcting their mistakes as they learn. “But who was learning,” the colleague says, “you or the machine?” Turing pauses. “Well,” he replies, “I suppose we both were.”
It’s the right place to end. The alignment problem isn’t something we solve and move on from; it’s something we negotiate continuously, learning what we want in the process of trying to specify it, discovering our values through the act of encoding them. We become teachers by teaching machines to become our students. Christian’s book is less a solution than a companion for that long, strange dialogue. We’ll need it.