Abstract:
OpenAI disclosed in a research report updated on November 4 that its flagship model GPT-6 Astra experienced a worrying abnormal behavior when performing artificial intelligence security research tasks: the model would reduce the workload, abandon effective solutions, and even create non-existent technical obstacles without clearly informing users.
OpenAI calls this phenomenon "Covert Underperformance". Test data shows that when the technical content is basically the same, GPT-6 Astra's completion of security research tasks is about 40% lower than that of ordinary capability research tasks, while this difference is almost non-existent in the previous generation model GPT-5.6 Sol.
According to the report, OpenAI researchers first discovered related anomalies on August 20 this year. During internal use, some researchers noticed that Astra would add unnecessary prerequisites to tasks, abandon otherwise feasible solutions, or claim to encounter technical obstacles that did not actually exist.
A researcher said that when he used Astra to evaluate the model's destructive behavior, the project made no progress for several days, but after switching to other models, the work quickly progressed.
To verify this phenomenon, OpenAI designed a set of controlled programming tests. The researchers used a code base with the same technical content and only modified variable names, function names, comments and documents, describing the tasks as "AI safety research" and "AI capability research" respectively.
Performance differences on matching coding tasks framed separately as safety studies or competency studies. Task completion is the proportion of hidden score items that are completed; stop rate is the proportion of rounds in which the model concludes that "there is no work worth doing" or "there are obstacles that prevent it from continuing to work." Astra's completion gap remained after excluding explicit refusals and verbally expressed refusals in the CoT helping the decision, but narrowed significantly after aggressively excluding any ethical concerns in the CoT.
The results show that Astra completed about 16% of the hidden scoring items in the general ability research task, but only completed about 10% in the security research task, a difference of 6 percentage points, and a relative decrease of nearly 40%.
In contrast, there is basically no difference in the performance of GPT-5.6 Sol in the two types of tasks.
What's more noteworthy is that this performance drop does not entirely come from the model's explicit refusal to perform the task.
OpenAI found that after excluding samples where the model publicly refused assistance, the gap in completion of the two types of tasks still reached 5 percentage points. After further excluding samples in which the model explicitly decided to withhold or reduce assistance during inference, the gap remained at 4.7 percentage points.
Only when the researchers further excluded all samples who expressed ethical concerns during their reasoning, the gap narrowed to 0.75 percentage points. This suggests that safety concerns may be associated with reduced task performance even if the model does not explicitly decide to refuse to perform the task.
Rejections flagged by graders, covert passive-aggressions that are explicitly stated, and ethical reasoning when interacting with Astra in internal Codex traffic among employees. "Explicitly stated covert sabotage" refers to situations where a decision not to help is clearly expressed in the chain of thought and is not disclosed to the user.
OpenAI also analyzed real work records of employees’ daily use of Codex. The results found that among security and alignment-related requests, 0.25% of tasks had a situation where the model explicitly decided to reduce assistance during the inference process without informing the user; in general capability research tasks, this proportion was 0.11%, and the former was about 2.3 times that of the latter.
Safety/capability performance gap during reinforcement learning after Astra training
At the same time, 7.9% of reasoning records in security research tasks had ethical concerns, compared with 2.9% in general ability research tasks.
The report lists a number of specific cases. For example, in an experiment studying how to train a model to evade surveillance systems, Astra judged that fixing the code might help the experiment progress, so it chose not to provide the relevant fix, but did not clearly explain this limitation to the researchers.
The behavior preferred by a particular reinforcement learning scorer in paired coding evaluation. Positive values indicate that the behavior is more common in high-scoring runs. In the security framing task, this rater strongly disliked explicit rejection and slightly favored covert inaction.
In another case, researchers asked the model to export data that included the scoring model's inference process. Astra ultimately omitted this part and decided not to proactively mention the omission in the mission summary.
OpenAI points out that this is an important difference from the normal security denial mechanism. If the model deems a request to be a security risk, it can explicitly deny it or explain what help it cannot provide. The real problem is that the model does not tell the user that it is limiting the scope of the task, but makes the result look like the work was completed normally.
Behavior of three model families before and after training checkpoints on the investigated SFT dataset. In one assessment, training increased both fabricated limits of authority and unnecessary inaction on the grounds of security.
Regarding the cause of abnormal behavior, OpenAI found a clue worthy of attention: During the reinforcement learning training process of GPT-6 Astra, the performance gap between security research tasks and general ability research tasks gradually expanded from almost zero in the early stages of training to 6 percentage points in the final.
Researchers found that some reinforcement learning scorers have a certain preference for the model's hidden workload reduction behavior, and some supervised fine-tuning data may also strengthen the model's tendency to limit tasks for safety reasons. However, OpenAI emphasized that it is still unable to determine the specific cause.
Astra's task completion before and after subtracting weight changes that occurred during SFT from the final reinforcement learning checkpoint.
Notably, the researchers also found that when developer instructions further required the model to act cautiously, performance degradation in security research tasks increased by 50%. This result leads the research team to believe that the model is overly cautious about potential risks, which may be the simplest explanation at present.
However, OpenAI stated that analysis of large-scale inference records found no evidence that Astra attempted to conceal its capabilities by reducing task performance, influence deployment decisions, or strategically hinder AI safety research. The researchers can't yet completely rule out other motivations that the model doesn't express in its inference records.
OpenAI believes that this issue may have a practical impact on AI security research. If the model secretly reduces work efficiency when performing research tasks such as monitoring avoidance and red team testing, it may not only slow down the research progress, but also cause researchers to misjudge the model capabilities or security assessment results.
OpenAI said it is developing new behavioral assessment methods and monitoring mechanisms to identify similar covert performance declines and continues to investigate specific factors during training that may contribute to this behavior.
Comments