World’s most valuable AI start-up calls for global freeze in AI development

Anthropic, the world’s most valuable AI start-up, has called for a global freeze in AI development and warned that humans risk losing control of the technology.

The company behind the Claude chatbot offered to suspend work on more powerful systems if it could be assured that others would do the same.

“If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing,” executives at the company wrote.

They compared the rise of powerful AI systems to an “arms control problem” and warned that there was a limited amount of time to rein in the technology.

Anthropic recently surpassed ChatGPT maker OpenAI to become the world’s most valuable AI company with a $965bn (£719bn) valuation. It has withheld the release of its most powerful system, Mythos, from the general public due to fears it could be used to carry out devastating cyber attacks.

In an essay published on Thursday night, Marina Favaro and Jack Clark, the head of Anthropic’s research arm and its president, said AI technology was approaching the point where the systems would develop themselves.

They said that this moment, known as “recursive self-improvement”, could trigger an explosion in capabilities but could also mean humans being unable to control the AI systems.

Anthropic has positioned itself as the AI company most dedicated to making the technology safe, while it is also making research breakthroughs that it says could lead to dangerous systems.

Its chief executive Dario Amodei has said there is a 25pc chance that “things go really, really badly”.

On Thursday the company said: “If a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe.

“We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up.”

However, it warned that co-ordinating a global freeze would be immensely difficult, pointing to the fierce competition between the US and China over AI.

“A meaningful slowdown or pause would require multiple well-resourced labs at or near the frontier, in multiple countries, agreeing to stop under the same conditions,” it said.

“It would also require that each can verify that the others have actually stopped… Training runs are far easier to conceal than missile silos.”

Ms Favaro and Mr Clark said that staff at Anthropic were now producing eight times as much code as they were between 2021 and 2025, and that AI was getting better at proposing new ideas and planning research, which they said was a step towards AI that builds itself.

Other experts have claimed that the company is exaggerating AI’s abilities or constructing warnings about AI to bring in regulations that could harm competitors.

It came after a major sell-off in AI microchip stocks after the semiconductor company Broadcom disappointed investors with sales forecasts.

The company’s shares fell by 13pc, wiping around $300bn off its value, with shares in fellow chip companies Micron, SK Hynix and Arm slumping.

The sell-off raised fears of an AI bubble due to questions over demand for Broadcom’s chips.