OpenAI's goal is to develop an AI tool that can optimize itself. Now the company is preparing for the risks that come with it. The potential of AI systems for so-called "recursive self-improvement" has become a focus of AI industry leaders after huge advances in programming tools from OpenAI and Anthropic over the past six months.

Demis Hassabis said this week that humanity is now standing at the foot of the "singularity" - the moment when AI begins to develop on its own and surpass human intelligence.

OpenAI, which plans to go public this year, recently posted an unusual job opening: the company is looking for a security researcher to study the consequences "when AI can train a better version of itself."

The job posting was posted this month for OpenAI’s Preparedness security team. The total annual salary for the position is between 250,000 and 380,000 euros, and the recruitment targets "experienced technical experts to support preparations for recursive self-improvement."

The recruitment information reads:

“This work is based on thinking about problems that don’t exist now but may arise in the future.”

“It is therefore particularly important for this position to have a measured and strategic mindset.”

Top AI labs are racing to develop “self-training” models

Recently, labs like OpenAI and Anthropic have continued to improve their model capabilities at an astonishing rate—an improvement that is reflected in the complexity of the problems they can solve.

Researchers at METR, a laboratory that studies model capabilities, wrote in March this year: The length of tasks that AI models can complete is doubling approximately every seven months—these models are increasingly able to take on tasks that would have previously taken humans a large amount of time to complete.

They concluded that AI agents would be able to take on a "significant portion" of the software development work that would otherwise take human programmers days or even weeks.

OpenAI is actively promoting this vision - selling its Codex programming tools to enterprises has become an important source of revenue.

The company also hopes to automate its own research work. CEO Sam Altman said in October last year that the company's goal is to have an "automated AI research intern" running on hundreds of thousands of chips by September this year; and to achieve a "true automated AI researcher" by March 2028.

“It is entirely possible that we will fail on this goal,” Altman wrote on X, “but given the enormous potential impact, we believe it is in the public interest to discuss this matter openly and transparently.”

In April this year, Anthropic released a study: using AI models to supervise more powerful AI models. The results are encouraging, but there are clear limitations.

In May of this year, Anthropic co-founder and policy director Jack Clark said that he believed that the probability of achieving “AI R&D without human participation” by the end of 2028 was about 60%.

OpenAI is preparing for "self-improving AI"

If AI models could train themselves, a sci-fi dystopia could emerge: their capabilities rapidly grow, escape control, and cause widespread harm—a long-standing concern of the AI ​​safety movement.

Elizabeth Barnes wrote on Friday that in her view, "any 'rational' civilization would advance AI development in a significantly slower and more cautious manner."

OpenAI’s recruitment information also reveals how the company is preparing for a world where “AI models can rapidly improve themselves.”

The job posting mentions that the researcher may focus on protecting OpenAI models from “data poisoning” attacks—the act of corrupting an AI model through manipulated training data.

The employee may also develop tools for explaining the model's thought process or conduct experiments to understand the safety and potential dangers of these models.

The job posting also mentions that the researcher may be required to "track the progress of automation in technical positions" - including measuring the use of AI programming tools.

OpenAI’s Preparedness team is responsible for preventing AI from causing serious harm. Other roles on the team include automated red team testing to assess cybersecurity, biological and chemical risks, and threats posed by "agent AI."

The job posting from the Preparedness team reads:

“This is urgent, fast-paced work whose impact will ripple across the company and society as a whole.”