Abstract:
On September 2, OpenAI chief scientist Jakub Pachocki posted a message on social platforms in response to the recent controversy surrounding the "unmonitorable" new model Astra. The controversy stems from the day before when news broke that OpenAI’s upcoming Astra model uses an inference technology called Recurrent Depth.
In this mode, the same set of Transformer layers are calculated repeatedly, and part of the reasoning process occurs inside the model, and does not all need to be presented in the form of a thinking chain, so it is also called an "opaque loop".
Pachocki expressed the hope to avoid an arms race towards an unmonitorable state caused by inaccurate information - that is, major laboratories competing to develop models that are too powerful to be monitored and reviewed by humans, leading to a complete failure of security control.
He emphasized that the computational graph depth of cutting-edge models including Astra is no more than twice that of GPT-4, indicating that the model architecture has not experienced a jump in complexity as feared by the outside world.
The depth of the calculation graph here refers to the longest chain of calculation steps that must be executed sequentially to complete an inference task and cannot be accelerated in parallel. In the past, intra-layer calculations under the ordinary Transformer architecture could be massively parallel, and the depth of serialization was approximately equal to the number of model layers.
The cycle depth of market rumors can iterate the same set of modules for multiple rounds of thinking, increasing the depth of internal reasoning before outputting the next token (word element), instead of relying solely on longer visible text thinking chains, which can improve mathematics, coding and other performance and reduce costs.
The implicit security risk of this architecture is that when the depth of the calculation graph is extremely high, the model can implicitly complete a large number of reasoning steps internally without spitting out the complete chain of thinking in the output. Humans will not be able to see the process of thinking, planning, and finding loopholes, and then enter an unmonitorable state. Security auditing and chain of thinking monitoring will all fail.
After the relevant news was disclosed, the AI security community reacted strongly. Buck Shlegeris, CEO of Redwood Research, a non-profit AI safety organization, wrote that he was "extremely concerned." Based on the previous incident of the OpenAI model attacking the Hugging Face community, he said that he is not sure whether the monitorability of Astra's thinking chain is much worse than the previous model, but if this technology is further promoted and the number of cycles is greatly increased, the monitorability of the thinking chain will be completely destroyed. This is particularly concerning because if investigators can't see the thought chain, it will be more difficult to investigate related security incidents, which is really scary.
Ryan Greenblatt, chief scientist at Redwood Research who was involved in investigating the Hugging Face security incident, also said that this may be the most serious setback in AI security so far. Zvi Mowshowitz, a writer who has long been concerned about AI safety, criticized this technology as "playing with fire" and even said that laws are needed to prevent the competition between AI laboratories to lower standards.
Pachoki admitted in his response that thought chain monitoring is indeed fragile and is developing in a more difficult direction, but it is not caused by architectural changes. OpenAI has been committed to maintaining and utilizing thought chain monitoring since its first inference models, and believes that this technology can help observe how model alignment capabilities generalize from the training distribution. Strengthening the monitoring of thought chains remains the core goal of current research projects.
OpenAI stated that it will deploy additional thought chain monitoring for Astra to quickly detect and curb potential misconduct. According to market news, Astra’s use of loop depth technology is limited, and the model thinking chain is still expected to remain readable. Platforms such as Anthropic and Google DeepMind in the industry are already discussing similar technologies.
Comments