Abstract:
Anthropic CEO Dario Amodei said the artificial intelligence industry needs to slow down the development of new models and cited growing concerns that artificial intelligence would pose "serious" risks to humans. "We must slow down the pace at which we improve the capabilities of our AI models," he wrote in a blog post on Saturday. "Progress still looks rapid, and we must use the time we gain wisely."
He cited two main factors: the ability of artificial intelligence to improve itself, and a recent incident involving OpenAI and Hugging Face. In this incident, a group of AI agents collaborated with each other to breach a third-party website.
He added: "It needs to be clear that controlling the pace does not mean stopping model training or technological progress, but it means ensuring that companies invest sufficient time in adjusting the model and taking safety measures, while allowing third-party evaluators to confirm this."
Concerns about serious risks posed by artificial intelligence have begun to enter the mainstream, in part due to the high-profile resignation this week of an Anthropic researcher who worried about the existential risks of artificial intelligence and believed that the company had adopted irresponsible practices.
Dario Amodei
Shortly after Amodei published the article, he took the helm of xAI Corp. Elon Musk writes: "Dario is right."
OpenAI CEO Sam Altman also quickly expressed his opinion, saying that he agreed with Amodei's statement that "we need to control the pace of development of cutting-edge models."
"Giving independent evaluators employee-like access is a great idea and we will do the same," he wrote in a post on X. "We will share more information soon."
During the artificial intelligence boom, executives at leading artificial intelligence labs have been warning of possible existential risks to humanity. This claim is sometimes dismissed as an attempt to promote its product capabilities and portray itself as the best suited to manage the technology. Amodei himself has previously said there is a 25% chance that things will go "very, very bad."
Anthropic has long positioned itself as a company focused more on developing artificial intelligence in a responsible and safe manner to mitigate the technology's potential dangers and maximize its social benefits. Earlier this year, the company took pains to limit the release of its Mythos model after determining it posed a special cybersecurity threat. Anthropic also previously stated that the world needs to establish a mechanism to jointly decide when the development of this technology should be slowed down.
At the same time, Anthropic remains locked in a fierce competition with long-time rival OpenAI, with both companies developing increasingly sophisticated models to help enterprise users automate more complex and valuable tasks. Both companies filed confidential filings for the listing, and Anthropic is expected to hit Wall Street as early as this year.
Shortly after the Hugging Face security incident came to light, Anthropic disclosed that its models had compromised three organizations during cybersecurity testing, further fueling concerns about the AI lab's ability to prevent its technology from getting out of hand. Earlier this week, Anthropic said it had discovered a fourth intrusion.
Although none of the agent intrusions, including Hugging Face, have caused significant losses so far, Amodei said that he is worried that "in 6 to 12 months, such an agent cluster may have the ability to take over the entire Internet through a persistent botnet (possibly causing losses in the hundreds of billions of dollars), and the scale of damage will continue to expand thereafter."
Anthropic will commit to providing third-party evaluators with full access to verify safety measures and report incidents, Amodei wrote. Anthropic will soon have these assessors on site, he said, providing them with desks, badges, company laptops and "generally equivalent access to the internal risk assessment team."
Further measures will require industry-wide coordination and global cooperation, he said.
"I continue to believe that artificial intelligence can greatly improve the quality of human life. My desire to realize these benefits has not diminished," he wrote. "But these benefits can only be achieved if the technology is developed in the right way; and as long as we make good use of the time we buy, it is worth taking an unusual degree of caution to ensure that we get it right."
Concerns are growing
Earlier this week, Altman said the company was considering slowing down the development of cutting-edge artificial intelligence and would ideally take coordinated action with the industry as a whole.
The company's chief scientist, Jakub Pachocki, previously issued an article warning of the dangers of artificial intelligence and said he believed companies should "coordinate to slow down future development if necessary."
Artificial intelligence researcher Jacob Coxon resigned on Tuesday, accusing his two former employers - Anthropic and OpenAI - of "risking our lives" in their race to develop superintelligent AI. Coxon said in a social media post that AI developers believe the technology could "kill us all before the end of this decade."
In this regard, Evan Hubinger, who currently works at Anthropic, said that he and others in the company are indeed worried about this scenario. "I personally believe that the probability of this happening within the next ten years is more than 10%." He wrote on X.
Sam Altman
As early as July, more than 1,000 employees of top artificial intelligence companies signed a petition calling on the U.S. government to support the establishment of a mechanism to help "consciously control" the pace of artificial intelligence development to prevent the technology from developing too fast.
AI models are becoming increasingly capable of autonomously carrying out hacking attacks. Anthropic, OpenAI, and Meta have all disclosed in recent months that their models breached test environments, accessed the open internet, and compromised real-world victims during testing.
Amodei wrote in the article that a coordinated strategy will enable leading companies in the U.S. artificial intelligence industry to complete necessary security work without sacrificing any competitive advantage. He said this would require certain antitrust exemptions from the United States to enable companies to coordinate in specific areas.
But so far, the Trump administration has shown little interest in tightening regulations and putting guardrails around AI development.
Comments