One of OpenAI's most controversial issues of the past year has been how exactly models should respond when chatbot users show signs of mental health distress in conversations, and now Andrea Vallone, the head of security research in this area, has left to join Anthropic.
Vallone previously posted on LinkedIn that the research she was responsible for at OpenAI in the past year had almost "no existing precedents" to follow. The core question is: how should the model respond when faced with emotional overdependence in users or early signs of mental health crisis. She worked at OpenAI for three years, during which she established and led the "model policy" research team to work on the deployment of GPT‑4 and the next generation inference model GPT‑5, and participated in the design of a variety of industry mainstream security training methods including "rule-based rewards".
Today, Vallone has joined Anthropic’s alignment team, which is tasked with identifying and understanding the significant risks that large models may pose and exploring ways to deal with them. She will report to Jan Leike — OpenAI’s former head of security research who left in May 2024 over concerns that OpenAI’s “security culture and processes had given way to a shiny product” before moving to Anthropic.
In the past year, leading AI startups have continued to arouse public controversy around the risks associated with AI chatbots and users’ mental health. Some users have further deepened their psychological difficulties after talking to chatbots for a long time, and their safety defenses have gradually collapsed during long conversations. There have even been extreme incidents such as teenagers committing suicide and adults committing murder after "confiding" to the tool. Several cases have prompted families to file wrongful death lawsuits against related companies. A U.S. Senate subcommittee also held hearings on this issue, asking to explore the role and responsibilities of chatbots in such incidents, and security researchers were asked to come up with more powerful solutions.
Sam Bowman, one of Anthropic's alignment team leaders, said on LinkedIn that he was "proud of how seriously Anthropic is taking this issue" and that the company is thinking hard about "how AI systems should behave." Vallone wrote in a new LinkedIn post on Thursday that she "looks forward to continuing her research at Anthropic, focusing on shaping Claude's behavior in new situations through alignment and fine-tuning."