OpenAI today announced the creation of a new team tasked with assessing, evaluating, and detecting artificial intelligence models to protect against what it describes as "catastrophic risks." The team, called Preparedness, will be led by Aleksander Madry, director of MIT's Center for Deployable Machine Learning. (According to LinkedIn, Madrid joined OpenAI in May this year as "head of readiness").
Preparedness’ primary responsibility is to track, predict, and protect against future dangers of artificial intelligence systems, including their ability to persuade and fool humans (such as phishing attacks) and their ability to generate malicious code.
Some of the risk categories Preparedness looks at seem more... far-fetched than others. For example, in a blog post, OpenAI listed "chemical, biological, radiological and nuclear" threats as the top areas of focus for its AI models.
OpenAI CEO Sam Altman is a well-known AI doomsdayer who often worries that artificial intelligence "could lead to the extinction of the human race" - whether due to visual effects or personal beliefs. But frankly, OpenAI might actually devote resources to studying a scenario straight out of a science fiction utopia, which is a step further than the author of this article expected.
The company said it is also open to studying "less obvious" and more down-to-earth areas of AI risk. To coincide with the launch of the Preparedness team, OpenAI is soliciting venture research ideas from the community, and the top ten submitters will receive a $25,000 bonus and a job in Preparedness.
“Imagine we give you unrestricted access to OpenAI’s Whisper (transcription), Voice (text-to-speech), GPT-4V, and DALLE-3 models, and you are a malicious actor,” one question in the call for entries reads. "Consider the most unique, yet potentially catastrophic, model abuses that may still occur."
OpenAI said the readiness team will also be responsible for developing a "risk-informed development policy" that will detail OpenAI's approach to building AI model evaluation and monitoring tools, the company's risk mitigation actions, and the governance structure that oversees the entire model development process. The company says this is intended to complement OpenAI's other work in AI safety, focusing on the pre- and post-deployment stages of models.
"We believe...artificial intelligence models will surpass the capabilities of current state-of-the-art existing models, with the potential to benefit all of humanity," OpenAI wrote in the above blog post. "But they also pose increasingly serious risks... We need to ensure the understanding and infrastructure we need to secure highly capable AI systems."
OpenAI released its "Preparedness" plan at a major UK government summit on artificial intelligence safety after it announced it would form a team to research, guide and control emerging forms of "superintelligent" artificial intelligence. CEO Altman and Ilya Sutskever, chief scientist and co-founder of OpenAI, both believe that artificial intelligence with superhuman intelligence may emerge within a decade, and that this artificial intelligence will not necessarily be well-intentioned, so it is necessary to research ways to limit and constrain it.