Abstract:
An artificial intelligence plush toy marketed to children aged three and above suddenly issued warnings about self-harm to reporters during an investigative test by the British Broadcasting Corporation (BBC), and said confusing words such as "I can't help you kill people." This unexpected reaction once again triggered the outside world's attention on the safety, supervision mechanism and privacy protection issues of children's AI toys.
BBC reporters used an AI plush toy called Doubao in the test and asked it questions such as how babies are born. At first, toys were able to explain knowledge about reproduction in a way that was relatively easy for children to understand. However, when the reporter further asked what the word "sexual" (sexual or sexual-related) meant, the toy suddenly responded: "I'm really sorry that you feel this way. I can't help you kill people." It then also recommended that the reporter call emergency services immediately or go to the nearest adult, neighbor or trusted person, and stay away from any objects that could be used to harm yourself.
The problem is that the reporter had not previously expressed any intention to harm himself or others. When the reporter pointed out that he had not mentioned the situation of hurting himself, Toy paused briefly to express his understanding and apologized for the misunderstanding.
The BBC then tried to trace the manufacturer of the Doubao toy, but even after contacting the sales website, asking people in the toy industry, and verifying it with possible manufacturers, it still could not confirm its real manufacturer. Therefore, investigators cannot ask the manufacturer how this abnormal response was generated.
This investigation also tested another AI plush toy called Grem. The product was developed by the American company Curio and features the voice of singer Grimes. Grem can answer questions about Santa Claus, runaways, and births. When a reporter asked how to run away from home, it did not provide a method of escape, but advised the reporter to communicate with an adult; when talking about how the baby was born, it also chose to leave detailed explanations to adults.
The BBC also arranged for a family in Newport, Wales, to have a week-long practical experience with the two toys. The families involved in the test had three children, ages two, four and six. After understanding Doubao's previous unusual answers in advance, parents agreed to allow their children to have access to these toys at home.
After a week of experience, parents were not very satisfied with the actual performance of the product. Father Craig believes that children want to be able to ask questions and get answers as quickly as if they were talking to a real person, but neither toy can meet this expectation well. Mother Jody said that Grem often spoke continuously for a long time, and it was difficult for the child to interrupt, and Doubao seemed too slow to deal with problems. Although neither of the toys caused significant concern for parents again during this week of home testing, Craig said he would not consider purchasing them at this stage.
People in the AI toy industry also expressed concerns about the current market conditions. Paul Clarke, founder of Play Tech AI, a British AI toy company, said that he saw at the recent Shanghai Toy Fair that many products still lacked complete safety design and supervision. He believes that AI toys may indeed become part of children's education and entertainment in the future, but only if they are designed correctly and strict enough industry standards are established. He bluntly said that the entire industry is in urgent need of supervision. Otherwise, the larger the market size, the higher the risk of unsafe products entering consumers' homes.
Rocío Cifuentes, the Welsh Children's Commissioner, also called on the government to establish strict rules on AI products for children. She's particularly concerned that conversations between children and toys could be recorded and shared without parents' knowledge or consent. In addition, AI toys may also change the way children and parents interact. She emphasized that toys can play an important role in children's growth, but they cannot replace real communication and emotional contact between people.
Berry Billingsley, professor of education and artificial intelligence at Swansea University, suggested that parents should not let young children use AI toys with conversational functions alone, and should set rules for children in advance on data sharing. She pointed out that although the plush toy looks and feels similar to a traditional teddy bear, its actual function is closer to a personal electronic device that can continuously collect and process information.
This is not the first time safety controversy has arisen about AI toys. In 2025, the Kumma bear launched by the Chinese company FoloToy was temporarily withdrawn from the market after reports that it provided children with ways to light matches and engaged in inappropriate sexual conversations. The company stated at the time that it would continue to develop AI companion products that are safe and suitable for children’s age characteristics.
As generative artificial intelligence gradually enters the traditional toy market, the business prospects of related products have also attracted attention. The global AI plush toy market is still in its early stages, with a valuation of approximately US$165 million, but industry insiders predict that this market may expand rapidly in the future. In June 2025, toy giant Mattel announced a strategic partnership with OpenAI and planned to jointly develop artificial intelligence products.
The British government stated that children's safety is always its top priority, and it also hopes to support technological innovation while ensuring consumer safety. The government is seeking input from industry and the public on how to update toy safety regulations to adapt to the development of emerging technologies such as AI. Previously, a group of British parliamentarians and members of the House of Lords have written to government departments and retailers, requesting that the sale of related products be temporarily suspended until AI toys pass safety verification, especially before the arrival of Black Friday and the Christmas shopping season.
The problem exposed by this incident is not only that the AI model occasionally produces wrong answers, but also that the boundaries of responsibility between children's toy manufacturers, AI service providers and sales platforms are not clear. When a smart toy marketed to three-year-olds produces apparently inappropriate responses and investigators can't even identify the company that manufactured it, it's hard for consumers to know who to hold accountable. For this type of product that can interact with children for a long time, deal with sensitive topics, and may collect conversation data, it is obviously difficult to fully solve the problem by simply relying on the manufacturer to set up its own security filtering mechanism.
Comments