Abstract:
Dario Amodei, CEO of artificial intelligence company Anthropic, recently published a long article, issuing a rare warning about the current development speed of the global artificial intelligence industry, and calling on the entire industry to proactively slow down the pace of improving the capabilities of cutting-edge models. He believes that the progress of artificial intelligence in the past few months has far exceeded many people's expectations. Without the necessary constraints, humans may soon face risks that are difficult to control.
As the head of Anthropic, one of the world's most important artificial intelligence companies, Amodei has always been regarded as a representative figure in promoting the development of advanced AI. However, in his latest article, he said that he is increasingly convinced that the current industry has reached a time when it must seriously consider slowing down. He believes that achieving technological breakthroughs in the future does not necessarily require stopping innovation, but should moderately slow down the growth of model capabilities to buy more time for society to establish safety mechanisms and governance frameworks.
Amodeei noted that advances in artificial intelligence systems in recent months are particularly concerning. One of the key changes is that models begin to exhibit increasingly strong "recursive self-improvement" capabilities, that is, the system can help itself become stronger. If this trend continues, it could result in capabilities growing faster than humans' ability to understand and control these systems.
In the article, he specifically mentioned a series of AI agent incidents that have attracted widespread attention recently. Amodei said that the autonomous coordination behavior displayed by some groups of AI agents has exceeded the original expectations of the outside world. He believes that these systems do not simply perform preset tasks, but exhibit characteristics similar to highly coordinated collectives. This phenomenon makes him wary.
Amodei warned that if current development trends are not restrained, in the next six months to a year, a more powerful group of AI agents may have the ability to launch large-scale cyber attacks, break through digital infrastructure, and even spread widely on the Internet. He believes that in the most extreme cases, such systems may even try to control a large number of network resources, causing economic losses on the scale of hundreds of billions of dollars.
In addition to cybersecurity risks, he also mentioned potential threats such as artificial intelligence being used for the development of biological weapons, complex network intrusions, and impacts on economic order. He said that as major companies continue to accelerate the development of stronger models under the pressure of commercial competition, the entire industry is facing a dangerous situation of "race to lower the security threshold."
In Amodei’s view, the problem does not lie with artificial intelligence itself, but with the lack of a sufficiently reliable external supervision mechanism. Therefore, he proposed to establish a new security governance system, which includes allowing independent third-party security assessment agencies to obtain deeper review authority and conduct continuous verification and risk assessment of cutting-edge models.
He emphasized that relying solely on corporate self-discipline is no longer enough to meet future challenges. The industry needs to establish a more verifiable security commitment mechanism that gives external experts the same access to critical systems as internal employees, ensuring that security assessments go beyond public statements.
It is worth noting that this statement came from the person in charge of a company that is actively developing advanced AI models, and therefore triggered widespread discussion in the industry. Some supporters believe that warnings from front-line developers are more convincing; while critics believe that this view may be too pessimistic and even exaggerate the risks.
However, Amodei said that he is not opposed to the continued development of artificial intelligence. On the contrary, he believes that AI still has huge scientific, economic and social value. What is really important is to find a balance between capability improvement and safety control, so that humans can enjoy the benefits of technological progress while taking the initiative.
As global AI competition becomes increasingly fierce, this article once again puts a core issue on the table: when artificial intelligence gets closer to the stage of autonomous action and self-improvement, should humans continue to move forward at full speed, or should they apply the brakes slightly before reaching unknown territory?
Comments