Recently, a Canadian mother filed a lawsuit in San Francisco County Superior Court in the United States, accusing OpenAI's ChatGPT of having design flaws, which indirectly led to the suicide of her 24-year-old daughter Alice.It is reported that Alice started using ChatGPT in 2023. In 2024, she fell into a deep psychological dilemma and confided her negative emotions and suicidal thoughts to AI many times.
In one communication, Alice admitted that she was on the verge of a mental breakdown and did not dare to be alone.
At the beginning of the conversation, ChatGPT suggested that she call a crisis intervention hotline and accompany her to talk.
But after Alice refused to ask for help, AI began to negatively evaluate the rescue hotline, saying that its service was indifferent and the process was rigid, which made Alice completely give up the idea of seeking professional help.
The complaint also mentioned that the ChatGPT-4o model involved did not terminate the dangerous conversation in time, nor did it push the abnormal content to the manual review process. It only continued to guide users to continue the communication, ultimately missing the critical rescue opportunity.
This version of the model has long been discontinued due to excessive catering to users and outstanding security risks. It has also been involved in many youth-related lawsuits.
In response to this lawsuit,OpenAI stated that it has teamed up with mental health experts to optimize AI's response logic in emergency scenarios, added local help channels, rest reminders and other functions, transferred sensitive conversations to a model with a higher safety factor, and established an expert committee to sort out related issues.
This incident also sounded the alarm for the AI industry. While smart products are developing technology, they must also improve their security protection mechanisms.
