There is a growing consensus that generative AI has the potential to make the open web worse than before. Currently, all the big tech companies and AI startups rely on scraping all original content from the web to train their AI models. The problem is, the vast majority of sites don't like it and don't give it permission. But just ask the Microsoft AI CEO, who believes content on the open web should be free to crawl.
Just last week, an Akamai report once again confirmed that bots account for a very large proportion of the entire network traffic, and artificial intelligence makes it easier for cybercriminals and dishonest enterprises to succeed.
Websites and content creators using the content delivery and firewall services provided by Cloudflare now have an easy-to-use solution to curb the ability of big tech companies to unleash bots and harvest website content without explicit authorization.
Most of the popular artificial intelligence companies (such as OpenAI) have started to provide a way to block crawling robots by adding custom rules in the robots.txt file on the server. However, these solutions only work if the bots are designed to actually comply with these rules - the problem is that: 1) not all companies are willing to comply with the robots.txt directive; 2) many AI companies have given up on all possibilities before offering this "opt-out" - Cloudflare says that the vast majority of its customers (as many as 85%) have chosen to block AI bots in this way.
The new one-click solution from Cloudflare, available to both free and paid customers, appears to be effective in combating AI bots that don't follow robots.txt rules. Cloudflare can identify bots and create individual fingerprints for each one, and it vows to automatically update its fingerprint database over time.
The company compiled a list of the most active artificial intelligence bots on the Internet today, with Bytespider, GPTBot and ClaudeBot being the three most visited bots. Bytespider, run by Chinese company and TikTok owner ByteDance, likely uses content scraped from the 40% of websites protected by Cloudflare to train its large language model.
GPTBot is visiting 35% of websites and collecting data for training ChatGPT and other generative AI services provided by OpenAI. Cloudflare said ClaudeBot has recently seen an 11% increase in requests and is being used to train a family of LLM algorithms of the same name developed by Anthropic.
While these well-known bots are easier to identify through static analysis, Cloudflare can also detect bots pretending to be real people browsing the web.
The company developed its own global machine learning model, which essentially uses artificial intelligence technology to identify AI bots pretending to be something else. Cloudflare says its model is able to "appropriately label" traffic from evasive AI bots and will be used in the future to detect new scraping tools and fake bots without first generating new bot fingerprints.