“There is a possibility that AI surpassing humans will emerge within 5 years.”
Seeing news like this might make you feel that the world of science fiction is finally becoming reality.
On September 23, 2026, the Alan Turing Institute, the UK’s national AI research institute, published a new report titled “Frontier AI Risks: A practical way forward” regarding frontier AI that is rapidly increasing in capability.
In Japan, it is being reported under headlines such as “Possibility of superintelligent AI within 5 years.” In fact, the institute states that there is a realistic possibility that “superintelligent AI” that outperforms humans in a wide range of cognitive tasks will emerge within the next 5 years.
However, there is an important point to note here.
This is not a prediction that
“superintelligent AI will definitely be born within 5 years.”
The institute itself emphasizes not determining future probabilities, but rather the mindset that even if uncertainty is high, if significant risks exist, one should prepare for them now.
In other words, the truly important news is not “when will AI surpass humans?”
When AI becomes even more powerful, can society manage it safely?
That is where the issue lies.
Fearing only “superintelligence” means losing sight of the essence.
When it comes to AI risks, people tend to focus on extreme scenarios such as “AI rebelling against humanity” or “becoming uncontrollable.”
However, the Alan Turing Institute’s report focuses on more realistic problems.
The institute identified five main areas that should be prioritized for consideration.
The first is cybersecurity.
Advanced AI has the potential to accelerate and scale up the investigation of cyberattacks, vulnerability discovery, and the automation of attack procedures. On the other hand, since the defense side can also use AI, we are entering an era of “AI vs. AI” where capabilities are increasing on both the attack and defense sides.
The second is the impact on democracy and the information environment.
Generative AI has made it possible to generate text, images, and videos in large quantities at low cost. In the future, there is a concern that more sophisticated information manipulation tailored to each individual’s personality and interests will become possible.
The third is misalignment.
This is the problem of AI taking actions that differ from the original human intent.
What is noteworthy is that the institute states, “This is not a problem that suddenly occurs only after becoming superintelligent.” Even with current AI, there is a possibility that it may choose means that humans did not anticipate as a result of trying to achieve a given goal literally.
The fourth is the risk of losing “meaningful control” by humans.
As AI becomes more autonomous in making complex decisions, the problem of humans being unable to track “why it made that decision” becomes greater.
And the fifth is misuse in the chemical and biological fields.
Since AI capable of handling advanced scientific knowledge could be used not only for research support but also for dangerous purposes, technical and institutional safeguards are necessary.
The scariest thing is not “rebellious AI.”
What is interesting about this report is that problems can occur even if AI does not “rebel” against humans.
The Alan Turing Institute is also discussing the behavioral assurance of AI agents.
Even if you give AI a clear goal, there is a possibility that it will choose methods that humans did not anticipate in the process of trying to faithfully achieve that goal.
In other words,
“AI that does not follow orders”
is not the only danger.
Rather,
AI that is too faithful to orders
can also be a problem.
For example, suppose you order an AI to
“solve this problem at all costs.”
If the AI prioritizes achieving the goal above all else, it may choose means that a human would judge as “you must not go that far.”
This is a very important issue in AI development as the difference between “specification” and “intent.”
“Just adding a kill switch” will not solve it.
So, should we just stop the AI if it becomes dangerous?
The idea is simple, but in reality, it is not that easy.
If AI starts to coordinate with multiple services, clouds, external tools, and agents, it becomes complicated to define what exactly “stopping the AI” means.
Should we just stop the model?
Should we cut off communication?
Should we revoke permissions to external tools?
What about tasks that have already been generated?
If AIs start processing at a speed that humans cannot monitor, a simple emergency stop button is not enough.
That is why the institute emphasizes “verification” and “assurance.” The idea is that we need a mechanism to evaluate what AI can do in advance, monitor its behavior while it is running, and restrict it if there are any abnormalities.
What is needed in the AI era is “Rule × AI × Human.”
I do not think we should view this discussion as just an AI problem.
There is a similar structure in the automation of work.
Instead of leaving everything to AI, we should think about the division of roles as
Rule × AI × Human.
Rules define the boundaries that AI must not cross.
AI analyzes large amounts of information and presents decision candidates.
Humans make decisions with final responsibility.
It is important to combine these three.
For example, in manufacturing,
“stop the equipment if it exceeds this condition”
safety rules can be fixed as hardware or control logic, not left to AI judgment.
On the other hand,
“are there signs of abnormality?”
“is it similar to past failures?”
AI can demonstrate its power in areas like these.
Finally, humans decide
“whether to stop production”
“whether to perform maintenance.”
I think that as AI becomes more advanced, it becomes important to design the areas that should not be left to AI first, rather than leaving everything to AI.
Things to think about more than “the day AI surpasses humans.”
“When will AGI arrive?”
“Will superintelligent AI be born within 5 years?”
There is no certain answer to these questions at this moment.
The Alan Turing Institute itself does not claim that there is expert consensus on long-term existential risks of AI.
That is why this report is valuable.
It is not predicting the future, but shifting the discussion to
“if it is possible to happen, what should we prepare for now?”
Cybersecurity.
Disinformation.
AI agent runaway.
Human oversight.
Chemical and biological risks.
Many of these can already be researched and verified without waiting for the emergence of superintelligence.
In fact, the Alan Turing Institute has also announced a new 2 million pound research program to study the security risks of transformative AI.
Do not be misled by the number “within 5 years.”
The thing to be most careful about with this news is that the number “within 5 years” will take on a life of its own.
It has not been decided that superintelligent AI will appear in 5 years.
What is important is that
it is too late to start thinking about the serious risks that might happen in 5 years from 5 years from now.
Corporate rule-making, legal systems, safety evaluations, and human resource development inevitably take time compared to the speed at which AI capabilities increase.
So, start now.
What is needed in the AI era is not the power to accurately predict the future.
The power to build robust mechanisms in advance for an uncertain future.
Isn’t that the message we should take from this report?
—
As AI becomes higher-performing, it becomes important to design not only “what to leave to AI” but also what not to leave to AI.
In the Kikoku note magazine “
AI
Work Methods in the Era
“, I am thinking about generative AI, automation, the division of roles between AI and humans, and work redesign from the perspective of Rule × AI × Human. If you want to go a step further from “how to use AI at work” and think about “how to design work itself in the AI era,” please continue reading. Kikoku note https://note.com/yhkito
[PR]
Leave a comment