Roblox is launching an artificial intelligence system it claims can detect early signs of potential child exploitation in online chats. The move comes as the platform faces growing criticism and legal challenges to its security measures. The tool, called Sentinel, is free to use and will be open source, allowing other platforms to integrate it into their own moderation systems.
Roblox reports more than 111 million monthly active users, and the company says its artificial intelligence has helped report hundreds of potential child exploitation cases to law enforcement. Roblox chief security officer Matt Kaufman told The Associated Press that while the company has long used "filters ... to block profanity and different types of abusive language," those protections were limited to "content within one or a few lines of text."
"But when you think about things related to child endangerment or child grooming, the types of behavior you see manifest themselves over a long period of time," Kaufman said.
Roblox says Sentinel is designed to detect patterns in conversations over time, rather than flagging isolated words or phrases. The system analyzes one-minute snapshots of the approximately 6 billion chat messages on the platform every day and evaluates them in context.
To achieve this, engineers created two separate indexes: one containing examples of harmless chat, and another containing messages that violate child safety guidelines. According to the company, new content is continuously added to both indexes to help AI models continue to improve.
Naren Koneru, VP of trust and safety engineering at Roblox, said: "As we find more and more bad actors, this index will become more and more accurate, and we will continue to update the index. This way, we can take another sample and see what the average user will do."
Koneru explained that the system monitors the user's ongoing activity to determine whether their behavior is leaning toward safe interactions or risky behavior. "This doesn't happen because you just send a message, it happens because your interactions every day point to one of those two behaviors," she said.
If the AI flags a user for further review, human moderators check the user's full chat history, friends list, and games they've played. Roblox said it will report cases to law enforcement and the National Center for Missing and Exploited Children when necessary.
The platform's security measures were announced amid a high-profile legal challenge. A lawsuit filed last month in Iowa alleges that a 13-year-old girl who contacted an adult pervert through Roblox was kidnapped and trafficked to multiple states. The lawsuit accuses the company of creating "easy prey for pedophiles" through the design of its games.
Roblox states that it prohibits sharing personal information, pictures, and videos in chats, and restricts private messages to users under 13 years of age, unless a parent explicitly allows it. Chats are not end-to-end encrypted, which allows the company to monitor conversations for security breaches.
The company maintains that no system can guarantee complete protection, but believes that AI advances like Sentinel greatly improve the chances of early detection.