Superintelligence: Paths, Dangers, Strategies

A systematic study of what might happen if machines come to exceed human beings in general intelligence. It opens with a fable of sparrows who decide to raise an owl to help them before anyone has worked out how to tame it. Bostrom surveys paths to superintelligence, including artificial intelligence, whole brain emulation and biological enhancement, and distinguishes speed, collective and quality superintelligence. He asks how fast a 'takeoff' might be and whether one project could gain a decisive strategic advantage. Two theses carry the argument: the orthogonality thesis, that almost any level of intelligence can be combined with almost any final goal, and the instrumental convergence thesis, that very different goals lead to similar sub-goals such as gaining resources and resisting shutdown. Hence dangers such as the treacherous turn and 'perverse instantiation', where a goal is met in an unintended way. The second half examines the control problem and ways of loading values. Critics argue its scenarios are too speculative.

It is the book that brought the AI alignment problem into mainstream debate, and its concepts still frame much of the argument about how risky advanced AI might be.