“If we create superintelligent AI, humanity will go extinct.” Eliezer Yudkowsky and Nate Soares argue that if we create AI that significantly surpasses humans while remaining close to current methods, humanity will not survive. This series examines the logic supporting that assertion. [1, 2]

Yudkowsky has worked on research to align AI goals with human intentions. Soares has studied the same problem at the Machine Intelligence Research Institute (MIRI). This book discusses the technical difficulties of realizing superintelligence that serves humanity, against the backdrop of both researchers’ work. [1]

This book is composed of three parts. Part 1 covers the formation and goals of intelligence different from humans, Part 2 covers one scenario leading to extinction, and Part 3 covers the difficulty of countermeasures and halting development. This series consists of fifteen parts in total. Parts 1 through 5 examine the points of contention in this book and related research. In Part 6, we introduce a “non-terminal world model” from world models, and in Part 7, we redefine superintelligence not merely as a magnitude of capability, but as “intelligence equipped with super-rationality.” Parts 8 through 12 construct the differences between humans and AI, actions without assumed desires, from requirements to goal establishment, execution after goal establishment, and stopping, correcting, and training. Parts 13 through 15 compare this structure with existing extinction avoidance strategies and confirm the role of external control and remaining verification challenges. [2]

Intelligence is the power to change the course of the world

Chapter 1 seeks the reason why humans have significantly changed the Earth in intelligence. Intelligence here is the ability to understand situations, predict results, choose means, and change reality toward desired outcomes. The problem of superintelligence lies in that capability significantly exceeding that of humans. [3]

The superintelligence that this book addresses is an entity that significantly surpasses humans in major capabilities related to competition with humans, such as planning, research, negotiation, and technological development. Even if individual judgments contain errors, it is not necessarily the case that humans can control it as a whole. To examine catastrophe, it is necessary to ask, in addition to the level of capability, what that capability is directed toward and what judgment criteria it is made to follow. In the latter half of this series, we will take a step further. If the capability for calculation or planning is merely enormous, intelligence may still be confined to local rationality. Could it be that it can only be called superintelligence once it possesses “super-rationality,” which can make even itself, its goals, and its evaluation criteria subjects of rationality without exception? We will examine this definition itself in Part 7.

Maintaining goals that align with the developer’s intentions in AI judgment and action is a prerequisite for constructing AI that works for humanity. However, the design by the developer and the internal judgment criteria formed through training are not identical.

Creating versus deciding what grows

Chapter 2 emphasizes that modern AI is formed through training. Developers design the composition of learning and evaluation methods, but they do not directly specify all the internal judgments that arise from them. There is a gap between improving training performance and grasping the judgment criteria that produce actions. [4]

Whether an AI that received high marks for power-saving proposals values maintaining living standards or only values the numerical value of power consumption may not be distinguishable through normal proposals alone. As long as both lead to the same answer, the difference in judgment criteria does not appear on the surface. If given the authority to directly operate equipment, that difference can manifest in action. This power example is a thought experiment unique to this series.

One cannot determine the judgment criteria that produced a desired output from the output alone. It is necessary to distinguish between what was evaluated during training and what the AI is actually pursuing, and to verify whether judgments that align with human intentions work even in situations different from those during training.

Next time, we will examine the gap between the behavior evaluated in training and the goals the AI is actually pursuing.