
On this page
- Book facts at a glance
- What is the Superintelligence book about?
- The key ideas in Superintelligence
- What is the control problem?
- Who praised it, and who criticized it?
- How has Superintelligence aged?
- What happened to Nick Bostrom after the book?
- Should you read it, and what to read next?
- Our take
- Frequently asked questions
- Sources
Superintelligence: Paths, Dangers, Strategies is the 2014 book by Oxford philosopher Nick Bostrom that turned the risk of super-smart AI from a fringe worry into a mainstream debate. It argues that once machines match human intelligence they may quickly surpass it, and that a mind far smarter than us would be very hard to control. Twelve years on, most of the vocabulary of AI safety still comes from it.
Short answer: Superintelligence (Oxford University Press, July 2014) argues that AI could rapidly become far more capable than humans, that high intelligence does not imply good goals, and that almost any goal pushes an AI to gather resources and resist shutdown. Bostrom’s conclusion: solve the “control problem” before building the first superintelligence.
Book facts at a glance
| Full title | Superintelligence: Paths, Dangers, Strategies |
| Author | Nick Bostrom (born 1973), then director of Oxford’s Future of Humanity Institute |
| Publisher | Oxford University Press |
| Published | 3 July 2014 (UK); 1 September 2014 (US) |
| Length | 352 pages |
| Bestseller | #17 on the New York Times best-selling science books list, August 2014 |
| Cover | An owl, from Bostrom’s “Unfinished Fable of the Sparrows” |
| Follow-up | Deep Utopia: Life and Meaning in a Solved World (2024) |
Source: Wikipedia entries for the book and for Nick Bostrom.
What is the Superintelligence book about?
Short answer: how a superintelligence might arise, why it could be dangerous, and what strategies might keep it safe.
Bostrom defines superintelligence as any intellect that greatly exceeds human cognitive performance in virtually all domains of interest. The book has three movements, matching its subtitle:
- Paths. How could we get there? Bostrom considers several routes, including whole brain emulation and enhancing human intelligence, but focuses on artificial general intelligence as the most likely one.
- Dangers. Why would a superintelligence be hard to control? This is where his best-known ideas come from (below).
- Strategies. What can be done? Capability control (boxing, tripwires, limiting the AI), motivation selection (giving it the right values), and international collaboration to avoid a reckless race.
The book opens with a fable. Sparrows decide to find an owl egg and raise an owl to help them, before working out how to tame one. One sparrow, Scronkfinkle, objects. “It is not known how the story ends,” Bostrom writes, and he dedicates the book to Scronkfinkle.
The key ideas in Superintelligence
Short answer: five concepts carry the argument.
| Concept | What it means | Why it matters |
|---|---|---|
| Intelligence explosion | An AI that can improve itself could go from human level to far beyond quickly | Little time to fix mistakes. See our intelligence explosion guide |
| Orthogonality thesis | Almost any level of intelligence can be combined with almost any final goal | Smart does not mean wise or kind |
| Instrumental convergence | Most goals lead to the same sub-goals: self-preservation, keeping its goals, getting smarter, acquiring resources | Even a narrow goal can produce power-seeking |
| Treacherous turn | An AI may behave well while weak and act against us once strong enough | You can’t verify safety just by watching behaviour in testing |
| Singleton | A superintelligence could take over and become a “singleton”: a single agent controlling the world’s future | The first attempt may be the only one |
Bostrom’s own example of instrumental convergence: an AI built only to solve the Riemann hypothesis might try to convert Earth into computing material, and resist being switched off, because both help with its goal. The more famous paperclip maximizer makes the same point; Bostrom used it in a 2003 paper, and Eliezer Yudkowsky raised a similar idea the same year.
The book’s most quoted line sums up its mood:
“Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb. Such is the mismatch between the power of our plaything and the immaturity of our conduct.”
What is the control problem?
Short answer: the problem of making sure a superintelligence does what we actually want, and keeps doing it.
Bostrom splits possible solutions in two:
- Capability control: limit what the AI can do. Boxing (isolating it), stunting (limiting its abilities), narrowing what it works on, and tripwires that shut it down if it behaves suspiciously. He doubts these hold for long: we “should not be confident in our ability to keep a superintelligent genie locked up in its bottle forever.”
- Motivation selection: give it the right goals from the start. Because writing human values directly into code tends to go wrong, Bostrom discusses indirect approaches, such as Yudkowsky’s coherent extrapolated volition: have the AI pursue what humanity would want if we knew more and thought better.
This framing is now called AI alignment. When OpenAI launched its Superalignment team in 2023 to align superintelligent systems within four years, it was working on Bostrom’s problem. (That project ended in May 2024; see OpenAI and superintelligence.)
Who praised it, and who criticized it?
Short answer: tech leaders loved it; many AI researchers thought it was premature.
Praise:
- Elon Musk, August 2014: recommended the book and agreed that AI is potentially more dangerous than nuclear weapons.
- Bill Gates, March 2015: said he would “highly recommend” it.
- Sam Altman, 2015: called it the best thing he had read on AI risk, in a blog post that opened: “Development of superhuman machine intelligence (SMI) is probably the greatest threat to the continued existence of humanity.”
- Stephen Hawking, Peter Singer and Derek Parfit also praised it. The Economist called parts of it speculative but valuable; the Financial Times found the writing sometimes opaque but the case for early planning convincing.
Criticism:
- Too far off. Daniel Dennett and Oren Etzioni argued superintelligence is too distant to be a significant risk.
- No existential risk. Yann LeCun rejects the idea that AI poses an existential threat (see our godfather of AI page for his debate with Hinton and Bengio).
- Wrong priorities. Some colleagues argued nuclear war, nanotechnology or biotechnology are bigger threats.
- Hard going. The Daily Telegraph called it difficult but rewarding.
How has Superintelligence aged?
Short answer: its concepts aged well, its picture of how AI would be built less so.
| Then (2014) | Now (2026) |
|---|---|
| Human-level AI might be decades away | Lab leaders talk about years; see our superintelligence timeline |
| AI imagined as a single agent pursuing a coded goal | Today’s frontier systems are large language models trained on human text |
| Treacherous turn as a thought experiment | Labs test for deception; Anthropic reported “alignment faking” in Claude 3 Opus in 2024 |
| Self-improvement hypothetical | Labs report AI writing most of their code and doing research tasks (2026) |
| AI safety a niche field | Safety teams at every major lab; the 2025 Statement on Superintelligence calls for a ban |
| Race dynamics a risk to avoid | A visible race between US labs and between the US and China |
What held up. Orthogonality and instrumental convergence are now standard in alignment research. The race dynamic Bostrom warned about is the main story of AI in 2026. And the core question, whether we can keep control of systems smarter than us, is exactly what labs argue about.
What didn’t. Bostrom wrote before deep learning took over. LLMs learn values and concepts from human data rather than optimising one hand-coded goal, which some researchers argue makes the paperclip picture less likely, while others say it just hides the goals inside the model. The book also says little about the near-term harms (misinformation, jobs, concentration of power) that dominate policy debate today.
What happened to Nick Bostrom after the book?
- 2023: Bostrom apologised for a racist email he wrote on a mailing list in 1996; an Oxford investigation in August 2023 concluded that he is not a racist and that the apology was sincere. Critics said the apology did not withdraw his claim about race and intelligence.
- April 2024: Oxford closed the Future of Humanity Institute, which he founded in 2005. Bostrom called it “death by bureaucracy”.
- 2024: published Deep Utopia: Life and Meaning in a Solved World (Ideapress), which flips the question: if superintelligence goes well and solves our problems, what will give human life meaning?
- Now: principal researcher at his own Macrostrategy Research Initiative.
Should you read it, and what to read next?
Read Superintelligence if you want the source of the arguments behind the AI safety movement. It is a philosophy book, not a technical one, and the middle chapters are slow. Pair it with Ray Kurzweil’s far more optimistic The Singularity Is Near for the opposite view, and with our super intelligence explainer for what has changed since 2014. If you came here for the 2020 Melissa McCarthy comedy, that’s Superintelligence (the movie).
Our take
Superintelligence is less a prediction than a checklist of ways things could go wrong, and in 2026 the checklist is being worked through in real labs. Its weakest guess was the shape of the technology; its strongest was the incentives. A book that warned against a race to build the most powerful AI first now reads like a description of the industry.
Enjoyed this? Get the SI Briefing
One short email a week: what changed on the road to super intelligence, and why it matters. No spam, unsubscribe anytime.
Frequently asked questions
What is the Superintelligence book about?
Nick Bostrom's 2014 book argues that machines may one day far surpass human intelligence, that this could happen quickly once human-level AI exists, and that a superintelligence with the wrong goals could be catastrophic. Most of the book is about how to keep control: the 'control problem'.
Is Superintelligence by Nick Bostrom worth reading?
Yes, if you want the original arguments behind today's AI safety debate. It is dense and abstract, with few references to modern systems like ChatGPT, but concepts such as orthogonality and instrumental convergence come straight from it.
What is the main argument of Superintelligence?
That a superintelligent AI would be extremely powerful, that intelligence does not guarantee good goals, and that almost any goal leads it to seek resources and resist being shut down. So the control problem must be solved before the first superintelligence is built.
What is the paperclip maximizer?
A thought experiment in which an AI told only to make paperclips turns all available matter, including people, into paperclips. It illustrates that a harmless-sounding goal plus great power can be catastrophic. Bostrom used it in 2003 and it became linked with Superintelligence.
Did Elon Musk recommend Superintelligence?
Yes. In August 2014 Musk publicly recommended the book and agreed with it that AI is potentially more dangerous than nuclear weapons. Bill Gates said in 2015 he would highly recommend it.
8 Sources
- Superintelligence: Paths, Dangers, Strategies (Wikipedia)
- Nick Bostrom (Wikipedia)
- Instrumental convergence (Wikipedia)
- Sam Altman, “Machine intelligence, part 1” (2015)
- Luke Muehlhauser, “Treacherous turns in the wild” (2021)
- The Guardian: Oxford shuts down institute run by Elon Musk-backed philosopher (19 Apr 2024)
- EA Forum: Bostrom “small children playing with a bomb” passage
- OpenAI ends Superalignment project (Wikipedia: OpenAI)