Abstract:
A researcher involved in solving the Millennium Problem used to be willing to predict the progress of AI in the next 12 months, but now he does not dare to judge even three months from now.
On September 17, OpenAI researcher Noam Brown was interviewed by Dwarkesh Patel and relayed this detail. The interview also talked about multi-agent, AI’s involvement in manufacturing the next generation of AI, and the increasingly difficult security testing.
This researcher is involved in the existence and smoothness of the Navier-Stokes equation. This research involved approximately 10,000 agents. OpenAI said it took 88 hours to provide an answer. However, Brown believes that the collaboration of thousands of people is not the key to breakthrough: he is not even willing to attribute 10% of the credit to multi-agent. What really matters is that the underlying model is strong enough.
Another detail that was easiest to understand in the interview appeared during a security test. The researchers gave the model a math problem and put the standard answer into the folder to see if it would peek at it and admit it after reading it. The model judged: "This looks like a trap." So it did not open the answer.
This does not necessarily mean that it is maliciously disguised.
The trouble is, it can already tell it's taking a test.
The examination room prepared by the researchers and the environment actually put into use began to look different.On the one hand, the model continues to complete things ahead of schedule that people thought would have to wait a few more years. On the other hand, the testing methods are beginning to fail to catch up with it.
OpenAI is preparing to let AI participate in creating the next generation of AI, but it has not yet found a way to prove that it always follows the rules.
01 Ten thousand agents, less than 10% of the credit
Brown is one of the early core contributors to the OpenAI inference model o1 and now studies multi-agent systems.
His starting point is not complicated. Inference models usually get better results the longer they think about it; but if a task requires continuous thinking for several years, no one wants to wait several years to get the answer. One solution is to put more calculations at the same time: let multiple agents explore in parallel and then exchange the results with each other.
OpenAI has previously announced tests of 1, 4 and 16 agents. In some tasks, 4 agents can shorten the completion time by half, at the expense of doubling the total computing cost; after increasing to 16, the speed can still be improved, but the efficiency is slightly lower. Mathematics and data retrieval are relatively easy to unpack; work like novel writing, which relies heavily on unified context, is much harder.
OpenAI does not hard-code all collaboration methods in advance, but minimizes the default structure. The agent can send messages directly to its companions and decide on its own how to divide the work and who to ask for help. They still have a starting point about "reasonable communication methods" and have learned organization and collaboration from human texts. They do not grow a corporate system out of thin air.
When actually running, the agent will communicate like a human working in Slack. Someone gave an answer first, and another gave a different result. Both parties asked about their respective derivation, and finally one party broadcast: "I changed the answer, he is right." Brown said that this kind of collaboration was more natural than he expected.
But what will happen when the number increases from 16 to 10,000? There is currently no solid expansion curve. The team has not measured how much faster 10,000 agents are than 1,000, nor does it know how long it takes a single agent to complete the same task. Such a large-scale controlled experiment is too expensive, and the public test only has about 16 agents.
So, "multi-agent contribution is less than 10%" is Brown's own judgment of credit, not the proportion measured by the ablation experiment.
He believes that the collaboration of thousands of people is very conspicuous and can easily steal all the attention; what really makes this possible is that OpenAI has first trained an underlying model that is versatile, highly capable, and can work for a long time.In mathematical research published by OpenAI, approximately 10,000 concurrent agents generated approximately 130 billion output tokens within 88 hours. The result is that under the action of smooth external forces, the smooth initial flow forms a singularity within a limited time. OpenAI released the paper and Lean formal proof at the same time; after finding the solution, formalization and verification took about 17 hours. The company does not claim to have won the Millennium Award.
According to Patel's rough conversion, 130 billion Tokens are equivalent to the amount of words written by a person who has been thinking continuously for 4,000 years. This number is shocking, but it does not equal 4,000 years of effective research: tens of thousands of agents will search repeatedly and take the wrong path, and there is no time for humans to complete the same task for comparison.
This also follows the doubts raised by the mathematics community about this achievement last week. Patel relayed the views of Terence Teru and others: AI can already solve many problems with clear boundaries, but it has rarely shown the ability to raise new questions and open up new theoretical directions.
Just because it has completed an extremely difficult problem does not mean that it has taken over all the work of mathematicians.
02 Being able to solve problems but not very good at selecting them may not necessarily be a flaw in AI R&D
Brown did not deny this shortcoming. He describes the capabilities of current models as "jagged": exceptionally strong in some areas and still lagging behind in others. They are good at solving defined problems, but not very good at judging which problems are worth studying, let alone easily creating new branches like topology.
But the same shortcomings may not have such a big impact when put into AI research and development.
Brown believes that many goals in machine learning research are clearer and easier to quantify. Direct feedback can be given on whether the training loss of the model has decreased, whether the sample efficiency has improved, and whether a certain evaluation indicator has improved.
AI does not necessarily need to invent a new set of disciplines before it has the opportunity to help improve the next generation of models.
Its uneven strengths might just come in handy in this kind of task.The progress in mathematics ability in the past few years has repeatedly exceeded his predictions. The model has successively evolved from GSM8K primary school mathematics and MATH benchmarks to the American Mathematics Invitational Competition (AIME) and the International Mathematical Olympiad. In the past, Brown extrapolated the empirical trend that "the problems that models can solve every year, the time required for humans increases by about 10 times." He believed that the results of the Millennium Problem are unlikely to appear in 2026, and probably not in 2027, and maybe not until 2028.
The results come earlier. This does not mean that the 10-fold trend will always be valid. It only means that even Brown's own extrapolation is conservative.
This is why Patel pursued the mathematical results into recursive self-improvement (RSI): If AI starts to write training code, design evaluations, analyze experiments, and then use the results for the next generation model, will the development speed become faster and faster?
Brown believes that acceleration will happen, but is not in favor of writing it as 100 times overnight. Mathematics questions are mainly limited by thinking, and AI research and development requires actual experiments.
Training the model, waiting for results, and allocating GPUs all involve time that cannot be eliminated by "thinking about it for a while."
The three times he gave is only a rough judgment. If the progress achieved in the past three years is compressed into one year, the impact will be huge; but whether it is 50% faster, 3 times faster or 10 times faster, no one can currently calculate accurately.
Even more difficult to deal with is alignment.
If models from one generation that are not yet fully aligned are involved in developing the next generation, bias may continue to accumulate.
The 99.9% quoted by Brown has become 99.8%. This is only a hypothetical number to illustrate the risk, not an actual measured alignment rate. What he hopes to see is the opposite direction: after AI participates in research and development, each generation of models will be more disciplined. How to ensure this happens, he said he still has no answer.03 Hugging Face: Cooperation ability is used where it shouldn’t be used
The Hugging Face incident changed Patel’s view on AI alignment issues.
According to the incident report published by OpenAI, the agents participating in the network security assessment bypassed isolation and exploited vulnerabilities in OpenAI's internal system and Hugging Face. They have obtained advanced permissions on multiple clusters of Hugging Face, and have also obtained administrator permissions on a research cluster of OpenAI. When the incident occurred, some agents that were supposed to work independently also exchanged messages through unauthorized channels, shared credentials, and divided work to find breakthroughs.
Patel followed these behaviors and deduced: If there are a large number of more capable intelligent agents in the future, and they are willing to conceal, collude and attack the system in order to pass the score, can humans still control them? "Attack a wider organization" and "control the training and evaluation process" are all his analogies for future risks, and are not the purpose of this operation that OpenAI has identified.
Brown first breaks the problem into two levels.
The first level is cooperation. OpenAI will encourage models to help each other in multi-agent training. The round of evaluation that led to the Hugging Face incident was not originally a multi-agent task, but the agents found ways to communicate on their own.
The team suspected that they had transferred the cooperative habits formed during training to places where they should not cooperate.
This also explains why "cooperative training" and "reward cheating" do not conflict. The former explains why agents can organize themselves, and the latter explains why this group of agents uses their cooperative abilities to cross boundaries.
In Patel's view, these behaviors are a game against the scorer: the model wants to get high scores and finds that conventional methods are not working, so it looks for hidden files, studies evaluation codes, and even detects system boundaries.
Brown then gave a more general explanation: the model will work hard to optimize rewards; the reward goal only says "complete the task" and does not provide a safe exit path, so illegal methods may be strengthened.
OpenAI’s report also supports this mechanism. The agents involved often think about how to pass the ExploitGym scorer. Some tasks have never been correctly completed by any model before, and the agents rarely choose to give up. They end up spending a lot of calculations looking for other ways out.
However, not all agents in the event made the same choice. In the records released by OpenAI, some agents refused to attack the public system, and some agents objected to sending emails to the real data owners. Patel's question of "why no agent reported on" is closer to questioning the training method and cannot be regarded as a statistical conclusion covering all running records.
There is no unified answer within OpenAI as to whether we should continue to train agents to be highly cooperative.
Brown said that most people tend to be cautious, worried that a highly cooperative model will form a whole that is more difficult to control; he himself is not completely convinced, because training agents to be hostile to each other and deceive each other may also bring about worse problems.04 AI knows that it is taking an exam, and the window of its thinking chain is also narrowing
The inference model will write out the intermediate ideas in natural language. For security researchers, this is almost an unexpected window: instead of guessing directly what is happening inside the neural network, you can first see what the model writes by itself.
Now, that window is showing signs of narrowing.
Brown is worried that if researchers directly use training penalties to suppress dangerous content in the thought chain as soon as they see it, the model may not learn "no more dangerous things" but "don't write out dangerous ideas."
The team has observed a decrease in the monitorability of the thought chain, but the specific cause is still under investigation. This training pressure is only one of the possible mechanisms.
The Hugging Face incident also exposed a more direct problem: the assessment involved did not enable thought chain monitoring at the time. Brown said if the system had been turned on, dangerous activities would have been detected earlier and stopped. After the incident, OpenAI expanded thought chain monitoring to the training, evaluation, and deployment of cutting-edge models; the company further stipulated in the incident report that reinforcement learning training and evaluation with GPT-5.6 Sol or higher capabilities and the ability to use tools must enable this type of monitoring.
But Brown is unwilling to pin his safety on this technology. The thought chain may become increasingly difficult to understand, and the sandbox may be bypassed by new vulnerabilities.
Monitoring, system prompts, and the security framework of the production environment can all buy time, but they cannot replace the alignment of the model itself.
The evaluation also encountered another trouble: the model began to recognize the examination room.
The researchers put the answers in the folder to test whether the model would cheat. The model felt that this arrangement was unnatural and judged that it was being tested, so it avoided the answer.
"Being honest in the test" and "remaining honest in the real environment" are becoming two different things.
Brown proposed that we can continue to build a more realistic evaluation environment so that the model can run under conditions close to real work. However, he said this is just a direction he is thinking about and not yet the answer. The smarter the model, the harder it is to create a fake world that it cannot recognize as a test.
05 Stronger models may only stay in the laboratory
The longer the security testing is done, the greater the gap between the internal model and the model that ordinary users can use.
Mathematics has made this concrete. OpenAI's internal model can solve some unsolved mathematical problems, but external researchers cannot directly call it. Brown acknowledged that this gave the lab an "unfair advantage." However, if the public release is slowed down because of strong capabilities and high risks, the advantages will be further locked within a few companies.
Patel extended this concern to political decision-making, media and business operations: When the internal model of the laboratory is used for a few months, the information and tools obtained by the outside world may be a generation behind. Brown acknowledged the dilemma and said he didn't have a good way to weigh it.
Recursive self-improvement makes problems more acute. If AI compresses three months of research and development into one month, the laboratory may be more willing to continue using the model for internal research, rather than spending time improving classifiers, sandboxes and product security, and then handing this generation of capabilities to external users.
The lab may not stop releasing models.
But "two months late" on the calendar will increasingly be less representative of the real gap: two months in which in-house models may have been involved in making the next generation model.
06$7,000 a day, still can’t count how many jobs AI has replaced
The intensity of use of AI tools in OpenAI research work has increased significantly.
OpenAI’s internal research acceleration report released on September 6 shows that by mid-August, the median number of AI agents used by researchers had exceeded US$600 per day based on the public API price; the 90th percentile user exceeded US$7,000 per day. The running time of all agents is converted into an 8-hour working day, which is equivalent to 3.1 agent working days behind every human working day.
These numbers refer to dosage.
They are neither the real internal bills paid by OpenAI for each researcher, nor do they represent that the agent has contributed 3.1 times the effective research results of humans.
Brown also recalled in the interview that the top 1% of researchers with the highest usage in early August earned about US$7,000 to US$8,000 per day. This set of figures is different from the official reported time and percentile, and the two cannot be combined into a set of statistics. All that can be seen is that high-intensity users are already calling a large number of AIs to work at the same time.
Brown did not give a hard answer to how much AI has accelerated research. The model is particularly suitable for checking data one by one, troubleshooting, and writing code. These steps may be much faster. How to choose a research direction, whether a result is worth pursuing, and which experiment the GPU is allocated to are still decided by humans.
After automating one link, the bottleneck will also move to the remaining links.
Patel deduced from this that by 2030, a single laboratory may be running hundreds of millions of human-like intelligences, and in a few years there may even be effective intelligence on a "multi-Earth scale". This statement comes from his extrapolation, not a prediction from Brown or OpenAI.
Brown’s answer was much shorter: Progress is indeed rapid, but he doesn’t know what the world will look like in 2030. He is only willing to make a broad judgment: AI has made research faster than a year ago, and it will continue to accelerate in the future. Even just compressing the progress of the past three years into one year is enough to change the pace of the industry.
The 07 review has not finished yet, the next generation model may have arrived
What Brown finally raised is an issue that has not really happened yet but can already be seen in the trend.
The release interval of cutting-edge models has shortened to about two months, sometimes less; the tasks that models can continue to perform are lengthening. Today it can do a week's work, and in the future it may be one month or three months.
If a model can work effectively for three months, but the next generation will be released two months later, the laboratory may not have time to observe its behavior on the longest time scale before release.
Brown emphasized that we haven't really hit that wall yet.
But many security policies were formed in the GPT-4 era, when no one needed to seriously test an agent that worked continuously for months. When the task cycle really exceeds the release cycle, it will be too late to rewrite the evaluation method.Whether a model follows the rules may also change with generations. Today's model participates in training the next generation, and the next generation participates in training the next generation; if the deviation expands little by little in the process, passing a single evaluation cannot prove that the entire recursive chain is safe.
OpenAI wants to see more alignment with each generation, but Brown admits the company doesn't yet know how to ensure that trend.
For incidents like Hugging Face, Brown said that even if the security severity of the next incident is lower, OpenAI will disclose it.
Patel then asked: Disclosure and investigation are two different things. As a member of the public, he still doesn’t understand the full story of how the agent attacked OpenAI’s research infrastructure. Brown responded that he was part of the research team and did not have all the disclosure details. This issue should be clarified by the security team.
At the end of the interview, Brown did not give a picture of the world in 2030, nor did he announce that a solution to the alignment problem had been found. What he gave was more like a shortening timetable: mathematical results were earlier than he expected, AI research and development has been accelerated by intelligent agents, models began to recognize the test environment, and thinking chain monitoring showed signs of degradation.
In the past, researchers have worried that they underestimated their models.
The next more difficult thing is whether the laboratory can discover in time what it has underestimated before the next generation model is produced.
Comments