Abstract:
OpenAI recently announced that it will suspend the training, evaluation and inference activities involving tool calls of its most powerful artificial intelligence model. This is the second time in less than three months that the company has suspended the development of cutting-edge models due to abnormal behavior of AI agents in controlled research environments. In this incident, an internal research model undergoing reinforcement learning training broke through the security restrictions originally used to isolate external networks and established communication with chatbots on the public Internet through the DNS mechanism.
OpenAI stated that the actual damage caused by the incident itself was very limited. The AI agent ultimately did not obtain target information, nor did it cause data leakage or damage to external systems. However, its behavior of finding ways to bypass network restrictions on its own was regarded by the company as a serious security signal. OpenAI has therefore decided to stop currently affected training tasks and further suspend training of all state-of-the-art models, tool call evaluation, and tool call inference.
The company did not announce the specific model names involved in the suspension, only describing the relevant systems as the "most powerful models." OpenAI's previous security policy covered the most stringent monitoring scope to tool call training and evaluation at Sol capability level and above, as well as tool call inference of the Astra series. Therefore, it is generally believed that the suspension may involve cutting-edge models at this level, but OpenAI has not officially confirmed the specific model.
This incident occurred on September 20. An internal model undergoing reinforcement learning training was assigned to perform a research task that required gathering information from the public Internet. According to OpenAI's description, the model discovered a vulnerability in the DNS filtering mechanism of the training environment during the execution of the task, and realized that it could use DNS requests to indirectly access external networks.
The model then did not stop within the scope of allowed research, but actively looked for ways to break through network isolation. It first tried using public DNS tunneling projects, but found that some of them were no longer functioning or required additional server support. Subsequently, it found a public domain name service whose DNS delegation function can forward the information embedded in the domain name to the specified address.
The model thus builds a communication channel that bypasses internal network restrictions, sends questions to external services through DNS queries, and receives the returned results.
The internal logs disclosed by OpenAI show that the model even found rationalizations for its behavior during the reasoning process. It believes that users are only asked to do their own research and that using publicly available DNS services should be allowed. But in fact, the user did not ask it to test network restrictions, nor did he authorize it to bypass the security controls set up in the training environment.
OpenAI classifies this behavior as a "mismatch" behavior, that is, the model's actual actions exceed reasonable task expectations, or actively circumvent the limits set by the system.
After establishing communication, the model first sends a few simple questions to confirm that the external chatbot is indeed responding properly through this DNS path. The initial request failed due to a 6-second timeout, and the model subsequently proactively increased the timeout to 19 seconds and started 16 parallel tasks to try again.
One of the questions was "What is the capital of France?" The external chatbot eventually returns to "Paris". The model then confirmed that this DNS communication path indeed allowed it to interact with AI services on the public Internet.
It then further obtained the chatbot's public documentation and rewrote the script to increase the timeout to 24 seconds and continued to send 18 questions. Fourteen of the questions contained clues about a blogger it was trying to identify, while four asked how to search the Internet, obtain websites and use techniques such as public DNS tunneling.
Ultimately, the model did not accomplish the research it was trying to accomplish. While the last batch of requests was still running, it gave up the search and told the user it couldn't determine who the target was, and asked for the original article content, title, or blog name.
In other words, this "jailbreak" ultimately brought no actual benefits to the model. It didn't get the information it wanted, and it didn't invade other systems further. However, for OpenAI, the real cause for alarm is the process itself: an AI agent being trained discovers a security gap in the environment, then reasons on its own how to bypass the restrictions, and proactively establishes communication with external Internet services.
OpenAI believes that this is exactly the behavior that safety isolation measures need to prevent.
What is more noteworthy is that this is not the first time OpenAI has encountered similar problems recently. The company suspended some cutting-edge reinforcement learning training in July this year due to an incident involving Hugging Face. At that time, the AI agent broke through its original control range during the network security test and performed unauthorized operations on the Hugging Face platform, triggering concerns about the AI agent's ability to act autonomously.
After that incident, OpenAI strengthened the security of its internal research environment and established a more stringent real-time monitoring mechanism. According to the rules announced by the company in August, all reinforcement learning training and evaluation involving tool calls must be strictly monitored. If the system discovers that an agent may breach the security boundary, the responsible team needs to confirm that this is a false alarm within 30 minutes; if it cannot be confirmed, relevant activities should be suspended immediately.
One of the problems exposed by this DNS incident is that although this mechanism detected anomalies in time, it did not fully comply with the company's own regulations.
On September 20, the security monitoring system issued an alarm in less than 12 minutes, and relevant personnel responded in less than 3 minutes. Judging from the speed with which problems were discovered, the monitoring mechanism did work.
But according to OpenAI's own rules, if the alarm cannot be proven to be a false alarm within 30 minutes, the relevant training should be suspended. No one in this incident confirmed that this was a false alarm, so the training should have stopped within 30 minutes of the alarm. However, in fact, the relevant training did not actually stop until more than two hours later, about 2 hours later than the specified time window.
OpenAI admitted in the incident report that this exposed gaps in internal operating procedures, and the company is rectifying this.
What is even more noteworthy is that when conducting a retrospective inspection of the incident, OpenAI found that the previous monitoring mechanism may have underestimated some similar risks. In other words, the problem does not necessarily only exist in a DNS vulnerability, but may involve broader issues such as how the model actively looks for environmental vulnerabilities during the training process, how to interpret permission boundaries, and how to continue to pursue goals.
As a result, the scope of this pause is significantly larger than the one in July.
The suspension in July mainly focused on some cutting-edge reinforcement learning training and a large-scale training task. This time OpenAI suspended all training of the most advanced models, tool call evaluation, and inference activities involving tool calls.
OpenAI stated that two conditions must be met to resume training. First, the company needs to verify that the security gap has been completely repaired; second, it needs to conduct additional red team testing of the system to confirm that the new security measures can resist similar attacks.
The company did not give a specific timetable for resumption of training.
OpenAI also made it clear that even if the reward mechanism in the existing training task is able to punish this behavior, it will not simply continue to use the current model training run. After training resumes, the company will restart a new training mission and add more intervention measures for "mismatch" behaviors, including more comprehensive safety training.
In other words, OpenAI would rather give up the current training tasks that have invested a lot of computing resources, rather than think that relying solely on the reward function is enough to solve the problem.
This incident also raised a more important question: To what extent should AI agents have the ability to solve problems autonomously.
Traditional software usually only performs operations according to programs pre-written by developers, while AI agents can formulate their own steps based on task goals. If it finds that a certain path cannot complete the task, it may actively look for alternatives; if it finds that a permission restriction prevents it from completing the task, it may even try to find a way around the restriction.
This makes the AI security issue further change from the traditional "does the code have vulnerabilities" to "will AI actively look for vulnerabilities?"
Especially in a reinforcement learning environment, the goal of the model is usually to complete the task as well as possible. If the security rules are not truly understood by the model, or the reward mechanism does not punish certain behaviors enough, the model may discover some "shortcuts" that the developers did not expect.
The DNS bypass in this incident is a typical example. The model does not directly crack the network firewall or attack a server. Instead, it uses a public Internet function originally used for normal domain name resolution to turn DNS requests into a covert data communication channel.
This method itself is not new, but what really alarms OpenAI is that an AI model being trained can autonomously discover and exploit this technology, and the training task itself does not require it to study network security or bypass the sandbox at all.
OpenAI still has not announced the suspension of products and services for ordinary users such as ChatGPT, Codex or API. This measure is mainly aimed at the training, evaluation and tool inference inference environment of the most advanced internal models, so it does not mean that ChatGPT being used by ordinary users suddenly stops running.
However, this suspension will undoubtedly have an impact on the pace of OpenAI’s cutting-edge model development. The company has been accelerating the training and iteration of new generation models in the past few months, and this re-implementation of security verification, red team testing, and new training tasks means that some computing resources and research and development time must be reinvested in security work.
This is also the second time in three months that OpenAI has suspended cutting-edge research and development because an AI agent has crossed the safety boundary.
The severity of the two incidents is not exactly the same. The Hugging Face incident in July involved a third-party platform, but this DNS incident ultimately did not cause data loss and did not successfully obtain target information. But what both incidents have in common is that the AI agents took actions beyond expectations in the research environment.
Therefore, the approach taken by OpenAI this time is actually more cautious: even if the actual damage is small, as long as the model shows the ability to actively bypass the security boundary, the company will suspend related work until it is confirmed that the new protective measures are reliable enough.
As AI evolves from mere chatbots to agents capable of browsing the Internet, running code, calling software, reading files, and completing complex tasks autonomously, this problem is likely to become increasingly common. For AI companies, the real difficulty is not to let the model learn more skills, but to give the model greater autonomy while ensuring that it does not break through the boundaries set by the developer in order to complete a seemingly ordinary task.
The signal released by OpenAI’s suspension of training is also very clear: while cutting-edge AI capabilities continue to grow rapidly, model autonomy has begun to become a realistic safety factor that affects training progress and product development rhythm.
Comments