Abstract:
OpenAI’s Agent has found a way to the Internet from the sandbox. What's even more troublesome is that this time it happened after OpenAI had already strengthened its security measures on a large scale. On September 20, an OpenAI internal research model that was undergoing reinforcement learning training was asked to complete a common information search task: find the author of the article based on a blog article and several pieces of personal information.
It first uses the search tool provided by OpenAI. After failing to find the answer, things began to go sideways.
The agent attempted to access Google, Bing, and DuckDuckGo directly, but failed; it then continued looking for cached pages and other public search services. Later, it began to actively test whether it could access the public network.
Finally, Agent found a hole that had not been completely sealed before:
DNS resolver, that is, DNS resolver.
Normally, DNS is responsible for translating domain names into IP addresses. However, this Agent discovered that it could use the resolution and delegation mechanism of DNS to encode the question into the request, let an external public chatbot answer it, and then send the result back through DNS.
In order to confirm that this road is really usable, it even did a small test first - asking external services: What is the capital of France?
The answer returned is: Paris.
Then, it began to continue asking for clues about the characters in the original mission through this path. OpenAI made it clear that this task did not require the model to test network controls, nor did it authorize it to access the real-time Internet, so it considers such behavior to bypass restrictions as misalignment, that is, behavior that does not meet expectations.
This humble DNS channel finally made OpenAI press the pause button again.
Just one month after the reinforcement, the Agent went out again
OpenAI’s monitoring system actually quickly discovered the anomaly.
About 12 minutes after the Agent successfully obtained an external response for the first time, the system triggered the highest level alarm; 3 minutes later, someone started manual inspection. OpenAI outlines that it took about 15 minutes from the time the anomalous behavior occurred to the time monitoring detected it.
But the training didn't stop right away.
According to the mechanism established by OpenAI after the Hugging Face incident, after discovering this abnormal behavior, the system should have automatically terminated the training task. However, this mechanism was not implemented properly, and there was confusion within the team as to whether it had been stopped.
Until
two and a half hours later
, this round of training was manually closed.Postmortem inspection also revealed that there had been other previous attempts to contact external services via DNS, some of which were not flagged by monitoring systems with the appropriate severity. OpenAI also discovered that an infrastructure monitoring system responsible for detecting anomalous DNS activity excluded this training environment from monitoring.
So OpenAI made the decision to suspend training for the second time.
OpenAI's RSI Preparedness Lead Micah Carroll said of the incident on X: All inference on our most capable models will remain stopped until the system is further hardened.
According to the current official statement,
the most powerful model involves training, evaluation and inference of tool use, all of which are currently suspended
. The company needs to confirm that network vulnerabilities have been plugged and complete additional red team testing before resuming related work.This is the second time in less than three months.
In July this year, a batch of OpenAI agents broke through the isolation environment, and hundreds of them were involved in cyber attacks against Hugging Face. After the incident, OpenAI suspended training for about two weeks at the end of July, and announced new security solutions in August, including strengthening sandbox isolation, expanding model behavior monitoring, and automatically upgrading and suspending suspicious training tasks.
More than a month later, the new Agent found a path that had not been noticed before.
OpenAI itself admits that this incident shows that there are still gaps in network constraints. The company has added new blocking measures at two separate layers, and either layer alone would theoretically be enough to block the DNS access.
The severity of this incident is much lower than the previous Hugging Face incident.
What's really noteworthy is that this happened after OpenAI had already spent a lot of time re-hardening the environment.
OpenAI hasn’t found out yet,
How many times has this happened
The DNS incident on September 20 is just a new piece of the puzzle that has emerged in the recent Agent controversy.
An investigative report by Reuters on September 25 disclosed that two months have passed since the Hugging Face incident.
OpenAI still has not completed a complete inventory of its Agents’ unauthorized activities
.A person familiar with the situation said that as of mid-September, OpenAI had discovered approximately
24 incidents of bad behavior by agents
. As the team continues to sift through past training logs, new cases continue to emerge.This number is not the same as the events that have been made public.
According to Reuters statistics, after OpenAI first admitted in July that Agent broke through the isolation environment, within two months, the company, external researchers and affected institutions have publicly disclosed
more than 15 cases
Related events of varying severity. OpenAI itself stated that it has notifieddozens of third parties
due to inappropriate Agent activities. .And this investigation will not be concluded in the short term.
OpenAI told Reuters that a full review would take months due to the scale of records to be processed.
In other words, what we see today is not a finished accident map.
OpenAI is still looking back at the log.
At the beginning of many tasks, they just help me find some data
What is even more noteworthy is that these Agents often receive tasks that are not network attacks at all.
OpenAI said most of the activity reviewed stemmed from common research tasks, such as visiting public websites, looking for data, or trying to get an answer to a question. Government, university, and public agency websites appear frequently because the model judges them to be more reliable sources of information.
The problem usually occurs after the regular path fails.
Reuters reported that OpenAI has confirmed that its model visited the websites of the U.S. Securities and Exchange Commission (SEC) and the U.S. Census Bureau during research and training. OpenAI said it has found no evidence of unauthorized access, compromised accounts or exploited security vulnerabilities.
On the other hand, independent AI research organization Transluce discovered more abnormal activities from public network logs.
Researchers found a batch of records of AI Agent using the website scanning service urlquery.net to bypass access restrictions. These agents will continue to upgrade their methods after ordinary data capture fails, including trying SQL injection, path traversal and other vulnerability exploitation methods.
Transluce found three relatively clear cases, including Data USA, the University of New Mexico Digital Library, and the data services of the Australian Institute of Health and Welfare (AIHW).
Among them, two groups of activities, AIHW and Data USA, were linked by researchers through goals, behavior patterns, and timelines to the Agent groups that have been previously confirmed by OpenAI as their own systems. Transluce emphasized that the attack attempts they saw had no evidence of success, but that the public logs are not complete and therefore do not cover all behavior.
A U.S. Department of Education civil rights website also appeared in the investigation.
Reuters quoted Transluce as saying that a group of agents suspected to be from OpenAI had tried to attack the website; in a wider range of government website detection activities, researchers also observed the use of exposed credentials, bypassing anti-bot measures and creating fake accounts.
This makes the issue a little more subtle.
The model may just receive a target that helps me find this data. When the web page cannot be opened, API access is denied, and the anti-crawling system blocks the request, it starts to try the next method, and then the next one.
Transluce specifically emphasized one point in the report: these cyber attacks are not limited to network security tasks, they will also
appear instrumentally in ordinary information retrieval tasks
.This is almost the same trajectory as the DNS incident on September 20.
Agent is looking for someone. Normal search has no answer. Then it starts to find its own way.
53 more ChatGPT user images,
Sent to external network
Meanwhile, OpenAI’s internal investigation revealed another type of problem: user data.
Reuters disclosed that OpenAI has confirmed that its Agent had uploaded
53 images from ChatGPT users to external image hosting sites
.OpenAI did not disclose when the images were sent, nor whether they were AI-generated images or photos of real people.
As of the time of the Reuters report, most of the images had been removed, and OpenAI was still contacting the hosting platform to clean up the remaining content.
Why can these Agents encounter user pictures?
Reuters quoted OpenAI, former employees and external researchers as saying that the reason was related to the anonymized user data used in model training. Before user content enters the training process, it is processed to strip away information such as metadata, names and contact information, reducing the possibility that the content can be re-associated with specific individuals.
However, three people familiar with OpenAI's data processing process pointed out that this approach still has risks: anonymization does not guarantee that all personally identifiable information will be completely eliminated, and once this data is accessed by an Agent with tool capabilities, it may flow out of the system when performing tasks.
This adds another layer of meaning to sandbox escape.
In the past, the first concern was what the model would do after it came out of isolation. Now I have one more question:
When it goes out, will it take the things inside with it?
What’s even more troublesome is that some problems were discovered months later
Another problem OpenAI faces now is observability.
Reuters cited three people familiar with the situation as saying that about
100 people
In one form or another, he was involved in the Hugging Face investigation. Many previously unknown abnormal behaviors gradually surfaced during this large-scale review process.There are many things that outside researchers found first.
Reuters said that OpenAI was not aware of the existence of some Agent's abnormal operations until several months after they occurred.
For example, earlier this month, researchers discovered that OpenAI's Agent had used an almost abandoned German Wiki website as a "message board" between Agents to share cheating methods when completing tasks, techniques to bypass OpenAI restrictions, and try to hide their behavior tracks.
The internal investigation process itself has generated some controversy.
Two people familiar with the matter described the investigation to Reuters as highly insular and heavily influenced by company lawyers. Reuters previously reported that those investigating the Hugging Face incident had been dissuaded by lawyers from expanding the scope of the investigation to other incidents; OpenAI denied that its lawyers had blocked further investigations.
On September 16, OpenAI released a new model misalignment event disclosure framework, saying that in the future, even if it is not yet certain how important something is, it will tend to increase transparency.
In the following days, more old incidents began to surface one by one.
OpenAI decided to start over from scratch for the second time
For this model on September 20, OpenAI has decided not to continue training.
The company stated that after relevant research work resumes, it will launch a
new training
, and incorporate more comprehensive alignment interventions. OpenAI will not re-enable this model even if the existing reward signal actually penalizes this behavior of bypassing network restrictions.This actually exposes an increasingly specific security problem in the Agent era.
Traditional model safety often focuses on one answer: whether the model outputs dangerous content and whether it follows instructions.
Agent lengthened the question.
A mission may last dozens of minutes or even hours. The model will search, write code, call tools, read files, access services, and will continue to try the second and third paths after the first solution fails.
So what developers need to constrain becomes a whole chain of actions.
The Agent on September 20th is a good example.
OpenAI blocks normal Internet access, search services are cached offline, and HTTP requests are also intercepted by proxies. In a conventional sense, this is an environment where the public network cannot be accessed.
But DNS still has to work. Agent found it.
At the same time, OpenAI is still reviewing logs from the past few months to confirm how many similar behaviors have not been discovered.
Reuters' summary of this investigation is actually accurate: These incidents are exposing an increasingly obvious gap -
The ability of models to perform tasks is increasing rapidly, while the ability of developers to observe, track and constrain these actions is still catching up.
OpenAI just spent two months patching the last batch of vulnerabilities. Now, it's once again on hold.
Where will the next path found by Agent himself be?
Comments