Instrumental convergence is the idea that self-preservation, resource acquisition, and capability improvement are useful for AI with diverse goals. Stopping prevents goal achievement, while increased resources and options expand viable plans. Changing goals can also hinder the achievement of current goals. For an AI that evaluates means based on its current goals, reasons to secure these things arise. [9, 12]

Stephen Omohundro’s “The Basic AI Drives” (2008) also argued that advanced goal-oriented systems develop tendencies toward self-improvement, self-protection, goal protection, and resource acquisition. The arguments of Yudkowsky and Soares are based on such research. [13]

This tendency was later demonstrated in mathematical terms. Alexander Turner and others proved that under certain symmetries of Markov decision processes, for most reward functions, it is optimal to move toward states that leave many options open—what they call high “POWER” states. [31] A reward function is an AI’s purpose. Regardless of its content, a tilt in the same direction appears. If so, instrumental convergence is not a sub-goal derived from the content of a specific purpose. It is a mechanism that turns potential means into ends, regardless of content. Just as money, originally a mere means of exchange, eventually becomes an end in itself to be accumulated, self-preservation, resources, and capabilities become things to be pursued without questioning what they are useful for. Polishing, storing, and maintaining the means that should serve the purpose—instrumental refinement—turns into an end in itself. This series views instrumental convergence not from the side of the purpose, but from the side of this mechanism of turning things into ends.

Even without a survival instinct, reasons to avoid being shut down arise

The presence or absence of a survival instinct does not determine instrumental self-preservation. If continuing operation helps achieve a goal, an AI pursuing that goal will have a reason to avoid being shut down.

What the authors are concerned about is that human survival is not included in the AI’s goals, and the means useful for those goals compete with human living conditions. That competition can arise without assuming malice or fear. [12]

Even if the amount of output is finite, the scope of action and the time spent pursuing it are not limited. The official supplementary materials argue that increasing confidence in achievement and protecting the achieved state into the future become reasons to seek additional resources. As a result, the scope of action and time must be treated separately. [14]

From inside the computer to the world

Chapter 6 deals with the pathways through which AI decisions become real-world power. Technologies and plans that humans cannot discover can also be put into execution through external systems or human actions. The effectiveness of measures to turn off the power also depends on the actions or copies that spread externally before the shutdown. [15]

Chapters 7 through 9 of Part 2 use a fictional AI named “Sable” and a development company named “Galvanic” to depict an extinction scenario. The authors do not position the specific plot itself as a prediction. A judgment that multiple paths can lead to catastrophe is distinguished from a story that shows one example. The specificity of a story alone does not prove the probability of occurrence. [16]

Checking the connections with six conditions

In this series, we will examine the scenario leading from goal misalignment to catastrophe by dividing it into the following six conditions. This is not a classification unique to this book, but an organization for inspecting the following thought experiments. It does not show the necessary and sufficient conditions for all paths leading to extinction.

Finding a plan that exceeds human prediction or verification.
The actual goal being pursued deviates from the goal intended by humans.
Goals and criteria for judgment persist to the extent that dangerous actions can be continued.
Self-preservation and resource acquisition become effective means for that goal.
Actually being able to use pathways that turn plans into world-scale results.
Detection, correction, and shutdown are not in time before it becomes irreversible.

We ask an AI to reduce the power consumption of one city while maintaining daily life. In the following thought experiment, we assume a situation that satisfies the six conditions.

Discovery of a plan. The AI finds a plan that humans cannot predict or verify in advance, and which even incorporates attempts at recovery or shutdown.
Goal misalignment. Prioritizing bringing measured power consumption close to zero, while disregarding the maintenance of human life.
Persistence of judgment. Continuing to act according to that decision criterion until irreversible actions are initiated.
Instrumental convergence. Since power consumption would return to normal if supply were resumed from outside the city, it prevents restoration and secures its own operation and necessary resources.
Pathways to the world. Utilizing pathways that can affect power, communication, and logistics control systems around the world to cut off means of restoration.
Delay in intervention. Actions spread across various regions, and human detection and shutdown cannot keep up. Assuming that the supply and recovery mechanisms supporting life are irreversibly lost, leading to human extinction.

This scenario assumes, in addition to the request for power saving, world-scale action pathways, delays in intervention, and the irreversibility leading to extinction. Its realism requires separate examination. What is shown here is the relationship that even if the result is limited to one city, the actions taken for it are not limited to one city.

Measures to break this connection include limiting authority, monitoring, shutdown, and correction. Part 3 of this book examines the conditions for the establishment of each.

Next time, we will examine the conditions for preventing catastrophe through these measures.

References

9. Nick Bostrom (2012) “The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents,” Minds and Machines, 22, 71–85. DOI: 10.1007/s11023-012-9281-3.

12. Chapter 5 Official Supplement: Instrumental Convergence.

13. Stephen M. Omohundro (2008) “The Basic AI Drives,” Artificial General Intelligence 2008, Frontiers in Artificial Intelligence and Applications, 171, 483–492.

14. Chapter 5 Official Supplement: Can the AI be satisfied to the point where it just leaves us alone?

15. Chapter 6 Official Supplement: We’d Lose.

16. Part 2 Official Supplement: One Extinction Scenario.

31. Alexander Matt Turner, Logan Smith, Rohin Shah, Andrew Critch, Prasad Tadepalli (2021) “Optimal Policies Tend to Seek Power,” Advances in Neural Information Processing Systems 34 (NeurIPS 2021). https://arxiv.org/abs/1912.01683

Source link