Human Compatible: Artificial Intelligence and the Problem of Control

A book by a leading AI researcher on why ever more capable AI could become dangerous, and how it might be redesigned to remain beneficial. Russell argues that the standard model of AI, in which machines optimise an objective fixed by humans, is fundamentally flawed: a powerful enough machine pursuing a badly specified goal is the King Midas problem on a large scale. He also poses the gorilla problem: gorillas now depend on human goodwill, and humans could find themselves in a similar position towards machines. He examines near-term misuses such as surveillance, autonomous weapons and the loss of work, and answers common reasons for dismissing the risks. His proposal rests on three principles: the machine's only objective is to realise human preferences; it is initially uncertain what those preferences are; and human behaviour is its main evidence about them. A machine that is uncertain in this way has a reason to let itself be switched off. Appendices explain basic techniques of AI.

It sets out the AI control problem in plain terms from inside the field and offers a concrete research programme, making it a clear introduction to AI safety.