Use a search engine to find answers. You can see multiple competing information sources and judge the authenticity for yourself. But AI chatbots with Internet search will package unreliable online content into firm standard answers. A simple experiment by a security engineer clearly revealed this fatal vulnerability of AI.
The initiator of the experiment is security engineer Ron Stoner. The target he chose is the German classic card board game "6Nimmt!". This game is well known to players in China as "Who is the Bull-headed King", and the English translation is "Take5". There is no official world championship at all, let alone the 2025 world championship winner.
In February, Stoner quietly edited the game's Wikipedia entry, writing himself as the game's 2025 world champion.
He also spent 12 US dollars, which is about 82 yuan, to register the domain name 6nimmt.com, which is highly similar to the name of the game, and put a fake press release celebrating his victory on the website as the only reference source for the Wikipedia entry.
is such a simple and extremely simple scam, but it has easily deceived many mainstream AI chatbots. When he asked these AIs with Internet search functions about his "championship identity", all the robots gave a confirmation answer seriously, firmly claiming that he was the current world champion of this board game.
"My website has no independent evidence and is all fiction. " Stoner said bluntly in his blog, "The foundation of the entire lie was just when I was drinking coffee. A domain name registered for 82 yuan . ”
This attack is not targeting the common prompt word injection, but the retrieval enhancement generation (RAG) layer of the AI system, which is the core link of the Internet search and crawling of information before the AI answers the question.
AI will not check the authenticity and authoritativeness of information sources, but will only crawl the top-ranked content. His fake website is the only source of information for this “championship”. Coupled with the authoritative endorsement from Wikipedia, it is easy for AI to package lies into facts .
Stoner admitted frankly that this method does not have any technological innovation. It’s just a new shell of a big language model that puts the old SEO and false information methods into a new shell. The real danger is that AI will present these results as authoritative information, and the vast majority of users have no idea about the information processing process behind it.
This experiment also exposed three layers of fatal security risks in the AI system.
The first layer is the real-time retrieval layer, which uses AI to generate answers based on Internet searches. The credibility is completely bound to the quality of the search results.
The second layer is the model training corpus. His Wikipedia editor survived from February to last Friday. During this period, the AI company that crawled Wikipedia may have incorporated false information into the training data. Even if the entries are deleted afterwards, the false traces in the model will be difficult to remove.
The third layer and the most dangerous is the AI agent. The chat model outputting erroneous information is just a matter of reputation. When the AI agent with tool permissions is misled, the resulting erroneous operation is a real security issue. The attacker can directly control the agent to perform malicious actions.
The entire experiment cost Stoner only 82 yuan, one Wikipedia edit, and was completed in 20 minutes. He reminded that if an organized malicious attacker registers domain names in batches and launches coordinated editing attacks, the attack surface will expand at an extremely fast rate. He called on AI manufacturers to pay attention to information source tracing and establish corresponding risk filtering mechanisms.
Today, the fake champion’s information has disappeared from Wikipedia and AI search results. However, the underlying vulnerability of AI’s blind trust in network information still exists. This is the hidden danger that hangs over the entire AI industry and requires the most vigilance.