AI pioneer Stuart Russell argues that the standard way of building artificial intelligence, giving machines fixed objectives, is flawed and dangerous, and proposes that we design machines that are uncertain about human preferences and learn them by watching us.
Overview
Human Compatible: Artificial Intelligence and the Problem of Control was published in 2019 by Stuart Russell, a professor of computer science at the University of California, Berkeley, and co-author of the leading textbook on artificial intelligence. Unlike many writers on the subject, he is a practitioner who has spent decades building AI. The book is aimed at general readers. In it, Russell reviews the progress and promise of AI, explains why it might go wrong, and offers a new approach, which he thinks can make AI beneficial. The tone is clear, measured and, in places, wry.
Full Summary
1. The promise and progress of AI
The idea: Russell begins with the history of AI and recent achievements, such as self-driving cars and personal assistant software. He explains that the goal of the field has been to create machines that can achieve their objectives, and that success would be hugely valuable, since intelligence is the basis of civilization, and human-level AI could be worth many trillions of dollars and could speed up science. But he warns that the same success carries risk, since a machine that is more capable than us could be hard to control.
In real life: Better translation and medical tools are the early benefits of AI.
2. The gorilla problem
The idea: Russell uses the “gorilla problem” to illustrate the danger: gorillas are stronger than humans, but their fate depends on us, because we are more intelligent. If we create machines that are more intelligent than ourselves, we may be in the position of the gorillas. He says that we should ask whether this is wise, and that we need to be sure of retaining control. He rejects the notion that the risk is impossible, or that we can simply switch a machine off if it misbehaves, and he argues that many common reasons for dismissing the risk do not hold up.
In real life: The group that is less intelligent often depends on the good will of the more intelligent one.
3. The flaw in the standard model
The idea: The heart of the book is a critique of the “standard model” of AI, in which we specify a fixed objective, and the machine optimizes it. If the objective is wrong or incomplete, a highly capable machine will pursue it relentlessly, with unwelcome results. He gives an example from social media algorithms, which are designed to maximize clicks, and argues that one way for such a system to do so is to change users’ preferences, making them more extreme and predictable. Specifying the right goal is very hard, and we will rarely get it perfectly right.
In real life: A system rewarded for clicks may feed you content that makes you angry, because it keeps you watching.
4. A new approach: three principles
The idea: Russell proposes that machines should be designed on three principles. First, the machine’s only objective is to maximize the realization of human preferences. Second, the machine is initially uncertain about what those preferences are. Third, the ultimate source of information about human preferences is human behavior. A machine built this way would defer to humans, ask permission, and allow itself to be switched off, since it knows that it might be wrong about what we want. Uncertainty makes it humble.
In real life: A helpful assistant who is unsure of your wishes will ask before acting.
5. Learning preferences and the off-switch
The idea: The author describes technical ideas to put the principles into practice, such as “inverse reinforcement learning,” in which a machine infers what people want from watching what they do, and the “off-switch game,” a mathematical analysis showing that a machine that is uncertain about the human’s preferences will allow itself to be turned off, since the human’s choice gives it information. He also discusses the problems that arise: humans are inconsistent, irrational and have different, conflicting preferences, which makes it hard for machines to learn what to do.
In real life: People often say one thing and do another, which makes it hard to know what they truly want.
6. Misuse, jobs and governance
The idea: Russell also discusses nearer-term problems, which he treats earlier in the book than his proposal: surveillance and manipulation, lethal autonomous weapons, which he opposes and wants banned, and the loss of jobs to automation. He also warns of overreliance on machines, which could lead to a loss of human knowledge and skills. Near the end he calls for tighter governance of AI, saying that societies must cooperate on rules and standards, and that AI researchers have a responsibility to think about safety, as engineers do in other fields. He suggests that we need to change the culture of the field.
In real life: Rules for the safe design of bridges exist, and similar rules are needed for AI.
Why It Matters
Human Compatible was widely praised, and became a leading statement of the case for AI safety from within the field of AI itself. Because Russell is a respected computer scientist, his views carried weight among researchers who had been skeptical of concerns raised by philosophers. The three principles, and the concept of machines uncertain about our preferences, introduced an original approach to the alignment problem, and influenced research. The book also reached policymakers and the public, in a time of rapid progress in AI, and added to calls for regulation, including limits on autonomous weapons.
Key Takeaways
- Fixed objectives are dangerous. A capable machine will pursue them literally.
- Machines should be uncertain about our goals. Humility makes them safer.
- Learn from human behavior. Machines should infer our preferences.
- Allow the off switch. Uncertainty means a machine will accept correction.
- Act on near-term risks too. Weapons, surveillance and jobs need attention.
What We Can Learn
You can use the book to think about the goals that are built into the technology you use, and who decides them. Ask whether a system optimizes for what it measures, not for what you want. Support rules and public debate about AI, including limits on autonomous weapons. Learn about the basics of how machine learning works. Be wary both of hype and of dismissal. And remember that technologists, like Russell, are thinking about how to make AI beneficial, and that public input matters.
Criticism and Counterpoints
Some critics say that the proposal to have machines learn human preferences faces enormous difficulties, as human values are diverse, conflicting and changeable, and it is unclear whose preferences count. Others doubt that a truly intelligent machine would behave as the book’s thought experiments suggest, since general intelligence would come with common sense and social judgment. Some say that Russell focuses on long-term risks, though others think that present harms deserve more attention. Skeptics question whether superintelligence is near. One reviewer found the technical parts too difficult and the philosophical parts too easy. Still, many consider the book valuable.
Who Should Read It
This book is for general readers who want a clear, authoritative account of AI risk by an expert, including older teenagers. It is accessible, with some technical parts that are explained simply. Those who want the philosophical arguments in more depth may read Bostrom’s Superintelligence.
Want the whole book?
A summary is a taste, not the meal. Here is where to find the full book.
You reached the end 🎉
Nicely done. Mark this one as read, and we will suggest what to read next.
Read this next
Superintelligence
Nick Bostrom
Philosopher Nick Bostrom argues that if machines become far more intelligent than humans, they could become very powerful, and that, unless we solve the problem of giving them goals that are safe for humanity, the result could be catastrophic.
💡 TechnologyLife 3.0
Max Tegmark
Physicist Max Tegmark explores what happens when artificial intelligence matches and then surpasses human intelligence, lays out a range of possible futures, and argues that we should think carefully, now, about which one we want.
🔬 ScienceThe Coming Wave
Mustafa Suleyman with Michael Bhaskar
Mustafa Suleyman, a co-founder of the AI company DeepMind, argues that a coming wave of artificial intelligence and synthetic biology will bring huge benefits and grave dangers, and that containing these technologies looks impossible yet must somehow be made possible.
💡 TechnologyWeapons of Math Destruction
Cathy O'Neil
Mathematician and former hedge-fund analyst Cathy O'Neil shows how secret, unaccountable algorithms used in schools, workplaces, policing, credit and insurance can quietly reinforce unfairness while looking objective, and she calls them "weapons of math destruction."
💡 TechnologyCo-Intelligence
Ethan Mollick
Wharton professor Ethan Mollick offers a practical guide to working with generative AI as a kind of "co-intelligence," explaining what it does well and badly, and giving four rules for using it effectively and responsibly in work, study and life.
💡 TechnologyThe Innovators
Walter Isaacson
Walter Isaacson tells the history of the digital revolution through the people who invented the computer, the internet and the web, and argues that innovation is mostly the product of collaboration, not the work of lone geniuses.
💡 TechnologyWant something different?
Pick another topic and we will show you three books to start with.