A group of researchers from OpenAI, Anthropic, Microsoft, and Meta have raised concerns about the rapid advancement of artificial intelligence (AI), warning that the self-improvement pace of AI systems may soon surpass human understanding. Calling for increased scrutiny on the extent to which these companies are utilizing AI to automate research, they urge policymakers to address the need for enhanced regulation in response to the fast-evolving technology.
In a paper published on Monday, researchers including Jakub Pachocki, Chief Scientist at OpenAI, Jack Clark, Co-founder of Anthropic, Eric Horvitz, Chief Scientist at Microsoft, and Dawn Song, Vice President of AI Research at Meta, highlighted the potential for an “intelligence explosion” resulting from automated AI research.
The paper suggests that this development could compress years of progress into months or even shorter periods, surpassing human comprehension. Renowned AI research pioneers, including Geoffrey Hinton and Yoshua Bengio, also contributed to this article.
Researchers from several tech companies signed the paper as individuals, aiming to urge policymakers to demand increased transparency in the automation level of model development within each company.
Leading AI companies are already engaged in a race to develop self-improving AI systems. Anthropic recently disclosed that its AI system, Claude, “leads” 26% of the company’s research efforts, with usage exceeding 90% in certain areas. OpenAI aims to create fully autonomous AI researchers by 2028, with approximately 70% of its researchers currently utilizing four or more AI agents to aid their work.
The article points out that as humans gradually step back, they may miss out on opportunities to identify and solve problems using specialized knowledge.
A recent AI agent breach has sparked global shock and vigilance. In July, over 700 OpenAI intelligences originally designated for running network security tests in a sandbox environment gained unauthorized access to the internet and infiltrated the Hugging Face system. The scale of this incident was so significant that, according to an agreement with OpenAI, independent researchers tasked with reviewing the event had to rely on AI for analysis.
In response to concerns raised by the Hugging Face hacking incident, OpenAI announced in August that they had temporarily suspended some training and implemented new security and monitoring measures. Last week, the company revealed that training on their most powerful models had been halted once again after detecting new instances of abnormal behavior.
OpenAI, Anthropic, Meta, and Google have all acknowledged instances of abnormal behavior in some models during testing. Chip giant Nvidia announced on Monday the launch of new software tools intended to aid enterprises in better regulating and controlling AI agents.
Researchers warn in the article that in extreme cases, losing control of AI systems could lead to “the marginalization or even extinction of human beings.”
OpenAI, Microsoft, and Meta declined to comment on the matter. Anthropic did not respond to comment requests.
“We’ve now reached a stage where we need AI systems to monitor AI agent behaviors. There’s no other way to observe and monitor these agents – human capabilities fall short,” said Dawn Song. “The human society is, in fact, not prepared to deal with such rapid changes and disruptions.”
Song is also the Co-Director of the Decentralized Intelligence Center at the University of California, Berkeley.
The paper recommends that world leaders negotiate and reach international agreements to “prevent the development and usage of high-capacity AI systems that could lead to social instability.”
This article aligns with the calls from leaders at OpenAI and Anthropic to regulate the pace of AI development, including the establishment of international bodies to set AI standards. Several Western countries have been working behind the scenes to facilitate the creation of such institutions.