Anthropic urges a way to pause AI development as risks grow with the tech advances

Anthropic is proposing that the world's top artificial intelligence companies come up with a coordinated way to pause development of advanced AI systems, warning the technology is improving so quickly there's a risk humans would lose control.

The company behind the Claude chatbot said in a blog post Thursday that as cutting-edge AI gets increasingly faster at carrying out tasks, “it would be good for the world to have the option to slow or temporarily pause” its development.

Anthropic said its internal research institute plans to explore the issue in collaboration with others and “take actions" to help build the systems for a credible slowdown or pause, without being more specific.

AI models are getting faster, with rapid increases in how quickly they can carry out software tasks like coding on their own, the company said. Based on current trends and given enough computing power, an AI system could be able to design and develop its own successor, in what is known as “recursive self-improvement.”

Self-building AI would be a major technological milestone that would bring benefits in science, healthcare and other areas, Anthropic said, but it “also might increase the risks of humans losing control over AI systems.”

Some tech industry figures have long warned of such a scenario.

Anthropic’s post comes after a different warning this week from a team of researchers at the University of Toronto who showed how AI tools could be used to create a new kind of AI “worm” that adapts its hacking strategy as it spreads from device to device and takes over a vast computing network.

“I think it’s really important that people understand that it’s not just the biggest, most powerful language models that pose the security concerns,” lead researcher Nicolas Papernot said in an interview.

The authors of the Anthropic post, company co-founder Jack Clark and Marina Favaro, head of the research institute, said the pause would be used to enable “societal structures and alignment research" to keep up with AI advances. Alignment is industry shorthand for making sure the technology matches human values and intentions.

The proposed coordination would let advanced AI labs verify that global rivals have actually stopped or slowed their work, “and that a bad actor could not use the auspices of a coordinated slowdown to jump ahead in secret.”

The company said a coordinated global mechanism is needed because without it a slowdown in AI development could let the “least cautious” players catch up and add to pressure on companies and governments as they make tough choices about AI safety.

Anthropic's post comes as the company itself is racing with ChatGPT-maker OpenAI to sell shares on the stock market, in an IPO that could value it at nearly a trillion dollars.