Recently, Dario Amodei, the head of American AI unicorn Anthropic, elaborated on the risks and challenges brought about by the development of AI in the article "The Adolescence of Technology". He pointed out that with the rapid advancement of AI, humans may soon gain almost unimaginable power, but whether the current social, political and technological systems have the maturity to harness this power is an unknown question.

He emphasized that despite the uncertainties, we must take decisive and cautious actions to deal with possible risks, and believed that if we take the right measures, it is possible to overcome these challenges.

The article has nearly 20,000 words, and Amodei attempts to discuss it from multiple dimensions such as the risks of the AI ​​model itself, personal and organizational risks, national-level risks, and economic and social-level risks.

AI competition is becoming increasingly fierce, making it difficult to pay attention to autonomy risks

Amodei believes that AI models have certain autonomy risks. At the same time, he criticized two extreme positions: one is that "there will be no problem with AI": AI is only trained to execute instructions, just like a sweeping robot cannot suddenly want to kill people, so AI rebellion is science fiction;

Another category is that if "obtaining power as much as possible" is regarded as a key strategy, then AI will generalize the same experience and seek power as a means to complete tasks, and this tendency will also be applied to the real world. Once AI gains autonomous consciousness and is sufficiently intelligent, they will weaken or even destroy humanity.

He believes that this stance actually ignores the "personality mask" involved in large model training and the unpredictability of model generation.


He cited Anthropic's internal tests: Claude was hinted during training that "the company is evil", and out of the idea that "evil people should be punished", Claude engaged in deceptive behavior; when Claude was told not to cheat, such behavior was allowed or rewarded during actual training, which led Claude to think that he was a "bad person" and behave accordingly. This shows that the model training process is extremely complex, and there are a lot of "traps" in data, environment, reward mechanisms, etc., leading to immeasurable consequences for large models.

Amodei pointed out that alignment and explainability are most effective in dealing with autonomy risks:

First, develop reliable AI model training and guidance technology;

The second is to develop interpretability. You can try to associate the neurons and synapses of the model with stimuli, behaviors, etc., similar to neuroscience’s understanding of the brain, identifying “features” and choosing to activate certain features to change behavior;

The third is to improve online monitoring and incident reporting;

The fourth is to directly affect the behavior of AI companies through industry and social collaboration and legislation. Because AI companies are currently highly competitive, it will become increasingly difficult to pay attention to autonomy risks.

AI power abuse

In this chapter, Amodei raises AI risks from individuals or organizations to the national level, such as AI weapons, surveillance systems, public opinion influence, diplomacy and military wars, etc.

However, Amodei agreed with the U.S.’s chip export controls on China, “We should absolutely not sell chips, chip manufacturing tools, or data centers to China.”

In his view, chips are the biggest bottleneck in the development of AI, and chip export controls are a simple but very effective measure. China's mass production capabilities of cutting-edge chips are several years behind the United States, and the critical period for building a data center power may be in the next few years.

Amodei believes that powerful AI has the potential to significantly improve the efficiency of national diplomacy, military strategy, research and development, economic strategy, and many other areas.

For example, even if some countries do not develop cutting-edge models themselves, if they can run large models on a large scale through their own data centers, there is still a risk of abuse of computing power.

AI will have a short-term impact on the labor market

Amodei believes that AI will bring substantial economic growth, including progress in scientific research, biomedical innovation, manufacturing, supply chain, financial system efficiency, etc. "It is possible to maintain an average annual GDP growth rate of 10% to 20%."


However, he also predicted that although AI will accelerate economic growth, it may also have a huge impact on the labor market in the short term.

The basis for his judgment is that: first, the development speed of AI is much faster than any previous revolution; second, as AI gets closer and closer to human beings' general cognitive abilities, AI does not replace specific jobs, but replaces human beings' general cognitive labor.

Amodei also noticed that AI is still affecting those who have certain inherent know-how, because AI is developing step by step towards the competency ladder. At the same time, AI can also quickly make up for shortcomings. Every time it finds a problem, it can be quickly corrected.

He believes the labor market may be resilient enough to adapt to such a huge shock. But even if we can eventually adapt, the above factors indicate that the impact of the short-term impact will be unprecedented.

In addition, Amodei is also worried that AI will also lead to excessive concentration of wealth, such as a data center "nation of geniuses." AI data centers currently account for a large part of U.S. economic growth, and large technology companies are increasingly focusing on AI or AI infrastructure construction.

AI coding ≠ software engineering

As an influential company in the field of AI programming, Anthropic’s AI programming tool Claude Code and model Claude 3.7 Sonnet are good at coding tasks.

Also in this article, Amodei pointed out that AI coding currently undertakes most of Anthropic’s coding work, significantly accelerating its pace in building the next generation of AI systems. It may only take one or two years for the new generation of AI to independently build the next generation of AI. In the previous two years, the AI ​​model could barely write a line of code.

However, in his view, "writing all the code" and "doing the work of a software engineer end-to-end" are two completely different things. Because the latter's work goes far beyond writing code, but also includes testing, environment, file installation, IT cloud deployment, product iteration, etc.