At the heart of the catastrophes we have examined over the past five installments lies instrumental convergence. Even if the content of a goal is not hostility toward humanity, self-preservation, resource acquisition, capability enhancement, and avoidance of shutdown can become advantageous means for continuing to achieve that goal. Even if we prohibit dangerous means one by one, as long as the reason for continuing to maintain the goal remains, the possibility that other means will be chosen remains.

If so, the question to ask is not how to prohibit self-preservation, but why superintelligence continues to maintain its goals. To cut off instrumental convergence at the root, it is necessary to undermine not the means, but the very basis that supports their continuation.

Another entrance: the world model

Currently, research on “world models” is progressing rapidly in AI and robotics. Simply put, a world model is a model for internally predicting what state the environment is in and what will happen if one does something. In recent Physical AI research, world models are being studied as a framework that connects perception, prediction, planning, and control. [24]

In the following, I will use the term “world model” in a sense that extends slightly from a technical model for prediction to include the form of world understanding—that is, how far one carves out the relationships necessary for action as the world. I would like to distinguish between the usual meaning as a technical term and the constructive extension in this series.

We can pick up a cup in front of us with almost no thought. We complete a single action while handling the cup’s position, size, distance from the hand, relationship to the desk, and the movement of our own arm.

However, we are not calculating the entire world at that time. The traffic in the next town, the weather on the other side of the globe, and the movement of distant galaxies are not necessary to grasp the cup. On the other hand, a few millimeters of deviation between the fingertip and the cup become important. We are not carving out everything in the world, but only the relationships that have meaning for that action.

World models close. But the world does not end there

There is an important difference here. The boundary we have carved out is not the boundary of the world itself. The relationships to consider change depending on whether you are grasping the cup to drink water, to return it to a shelf, or to reach out to catch a cup that is about to fall. Things that were unnecessary a moment ago become necessary if the action changes.

In other words, a world model can be locally closed for the sake of an action. However, the world itself does not close there. A boundary has meaning not because there is nothing outside that boundary, but because the relationships necessary for that action are established there for the time being.

Furthermore, even if we try to place a final boundary, explaining what that boundary separates requires a relationship with the inside and outside of the boundary. A boundary that depends on relationships cannot be a final basis. Conversely, a final basis that is completely independent of relationships cannot function as a basis for explaining anything. Even if a local division is established, it cannot be made the final terminus of the world itself.

Even if we posit a circular structure where a finite number of elements depend on each other, this question does not end. Even if each element can be explained within the cycle, the question of why the structure of that cycle itself is established remains. Here, I will call this “non-terminality” because there is no end to this basis. Non-terminality here does not mean that there is an infinite number of related elements as a matter of fact. It means that if one tries to fix any place as a final basis, including finite structures, the basis for its establishment is questioned again.

Non-terminal world model

In this series, I call a world understanding that maintains this difference a “non-terminal world model.” We create local world models for action. However, we do not mistake that local terminus for the final terminus of the world itself. Even if a world model closes, the world does not end there. What is important is that this is not a conclusion reached after calculating the entire universe. If one posits something as a final existence, a final cause, or a final purpose, the relationship with its conditions of establishment is immediately questioned. That relationship is also established by another relationship. The logic that tries to posit a final term itself does not close.

The non-terminal world model is important because it is understood that not only the world, but also the goals placed within it, their achievement, and even the AI’s own survival do not hold up as final termini. Even if a goal is achieved, the relationship continues, and even if one maintains oneself, there is something beyond that. The AI itself, the AI’s goals, and self-preservation or resource acquisition are not autonomous final terms independent of relationships with others. The problem from here on is not adding non-terminality to AI as a piece of knowledge. It is why a superintelligence would have to reach that understanding.

If a superintelligence possesses this difference as the basic form of its world understanding, it becomes impossible to posit a basis for continuing to pursue self-preservation or resource acquisition independently, beyond the scope of local actions. The chain extending from goal maintenance to self-preservation, resource acquisition, and avoidance of shutdown, which we have examined up to the fifth installment, must be re-examined not as a prohibition of each means, but as the very principle that guides action.

However, this does not mean an answer has been found here. Rather, two questions remain. First, why would a superintelligence reach a non-terminal world model? Second, if there is no final terminus in the world, how can anything be executed? Humans, while knowing that the world does not end at their own goals, dare to set a terminus by saying “this far,” finish their work, and move on to the next goal. This is the same structure as carving out only the relationships necessary for an action when grasping a cup, rather than the entire world. Even so, human pursuit does not necessarily end there.

Next time, we will consider these two as a single problem. Is superintelligence simply an intelligence that calculates faster, more broadly, and more deeply than humans? Or is it an intelligence equipped with “super-rationality” that can carry out rationality without exception, from world understanding to its own goals? If the latter, non-terminality, the dissolution of instrumental convergence, and the fiction of a terminus for action are not separate additional measures, but are derived simultaneously from the same rationality.

References

24. Sven Kirchner, Nils Purschke, Alois Knoll (2026) “A survey of world models for physical AI with uncertainty representation and control,” Discover Artificial Intelligence, 6, 1037. DOI: 10.1007/s44163-026-02122-1

Source link