Abstract:
In July this year, a company called Cloudflare issued an announcement that directly caused X to explode. What does this announcement mean? Blogger @iyici_ commented: "The verification code company used by all websites wants to block the search engines used by everyone."
New rules, new fees, it’s time to charge for AI｜Cloudflare
The most interesting part of the announcement | Cloudflare
Charles Petersen, a well-known technology writer and editor in San Francisco, believes that this is the end of the Internet. He tweeted at X "
RIP World Wide Web (1993-2026)
. ”
@iyici_'s original comment｜X.com
Why can an announcement from this technology company, which most people have never heard of, cause such a big shock? Because this company made a decision that is likely to change the Internet ecosystem:
It will help websites collect "tolls" from AI
.Cyber Bodhisattva: You don’t know it, but you use it every day
What exactly does Cloudflare do?
Cloudflare is one of the world's largest network infrastructure service providers, and its CDN and DDoS protection market share has long been at the forefront. You can simply understand it as the "front warehouse" and "moat" of the website.
For example, you are a small website owner in China. If users from Singapore, the United States, and Japan visit your website, they will get images, scripts, and content directly from the Chinese server every time, which will be extremely slow. So you connect your website to Cloudflare, which has many nodes around the world and caches commonly used content on the server closest to the user, so that users from other countries can access your website faster.
And because when users visit your website, they first visit Cloudflare's node, it can also provide protection services against malicious attacks and abnormal traffic robots.
Still feel abstract? Then you must have seen the following icon:
Click this button and wait for about four or five seconds before the website can be opened: this is Cloudflare's human-machine verification service at work, commonly known as the "five-second shield". It determines whether you are a real person or a robot by analyzing the user's mouse movement trajectory, click frequency and other behavioral patterns.
Today,
Cloudflare handles approximately 102 million HTTP requests per second, and approximately 20% of the world’s websites use its services
, ranging from personal blogs to Internet giants such as Discord and LinkedIn, are its service targets.Its service covers 125 countries and regions, 95% of global Internet users, and the average delay does not exceed 50 milliseconds
.This is why if there is any disturbance, the Internet world will be shaken. In November 2025, Cloudflare experienced a large-scale outage that lasted for about 6 hours, directly causing a large number of websites and services such as X, ChatGPT, Spotify, Uber, and Discord to be paralyzed at the same time.
So what rules did it set this time?
What does Cloudflare do?
To put it simply, Cloudflare’s new rule is this sentence:
Starting from September 15, 2026, all websites using Cloudflare services will block “hybrid AI crawlers” by default
.What is a "hybrid AI crawler"?
Crawlers are robot programs that automatically crawl web content. The reason why search engines can search web pages is because they release a large number of crawlers to crawl web content online, build indexes, and then provide them to users.
Google, Bing, and Baidu have been doing this for more than 20 years.
Later, AI came, and AI training required a lot of data, so large model companies such as OpenAI began to rely heavily on crawlers. At the beginning of this year, AI Agent became popular. Let Codex, Kimi work and other AI tools help you check information. Behind the scenes, a crawler is running errands for you to grab content from web pages.
Like this | via Kimi Work
Now, AI crawlers have occupied half of the Internet. According to Cloudflare's official data, in June 2026,
all non-human traffic (including search engine crawlers, AI crawlers, etc.) accounted for 57.4% of the total web page traffic
. In other words, more than half of the people who “read” web pages on the Internet today are no longer humans.So starting in 2025, Cloudflare will launch the "AI Crawl Control" function, allowing website owners to manually block AI crawlers. A "Pay Per Crawl" mechanism was also launched. If an AI crawler wants to access content, it must include a payment intention in the request, otherwise access will be directly denied.
By the end of 2025, this function has intercepted more than 416 billion robot requests
.This function works well for pure AI crawlers such as OpenAI's GPTBot and Anthropic's ClaudeBot - but it gets stuck when it comes to Google.
Google has a feature called "AI Overviews": when users search, Google no longer just gives links, but directly generates an AI summary of answers from other websites at the top of the results page. The problem is that when this function generates summaries, it does not use the data captured by a separate AI crawler, but directly calls the search index - that is, it shares the same Googlebot with search, and this crawler is called a "hybrid AI crawler".
Once a website owner blocks Googlebot, the AI overview is indeed blocked, but his website also disappears from Google search results and receives less traffic. Not blocked? That AI Overview continues to get your content in vain.
Faced with this dilemma, it is indeed difficult for a single website owner to negotiate directly with a giant like Google, so Cloudflare became the big brother to stand up.
The new regulations in September are aimed at this hybrid crawler, and the policy is straightforward: it has been upgraded from manual blocking by website owners to default blocking, forcing Google to make a choice, either to stop fishing in troubled waters, or to bear the consequences of a significant drop in search content.
According to media reports, while launching the policy of banning mixed crawlers, Cloudflare is also negotiating with Google, asking it to separate the AI overview crawler from the search crawler, so that the website can only block the AI overview crawler and retain search traffic.
Google disagrees. It is worried that once separated, a large number of websites will directly block AI overview crawlers.
Then why does Cloudflare take such a heavy hand? Because of the emergence of AI crawlers, the logic of the Internet that has been running for decades is changing.
The Changing World Wide Web
In the past thirty years, information search on the Internet has worked like this: you search for "how to make braised pork", the search engine gives you ten links, you click on the first food blog, browse the content, and exit.
In this process, the website writes content, the search engine sends crawlers to capture it, and users search and click in; both the website and the search engine gain traffic, which can be monetized by advertising or affiliate links, and advertisers receive more effective delivery. Everyone is happy.
This tacit understanding can last for thirty years. It is basically based on a tacit exchange of equal value. I let you capture my content for free, and you give me free traffic. No one owes anyone.
When search engines have AI, everything changes.
When you search for "how to make braised pork", Google no longer just gives you ten links. It directly summarizes the recipes, steps, and precautions and puts them at the top of the search results.
The user leaves after reading the answers without clicking on any website
.Google still gets traffic, but the website owner’s traffic is gone.
Cloudflare announced in mid-2025 the "crawling and return ratio" of a set of crawlers from various manufacturers:
Google is about 10:1 (crawling 10 web pages, bringing in 1 visitor), OpenAI is 3700:1, and Anthropic has reached an astonishing 500000:1
. In other words, Google is at least bringing back some traffic, and AI companies are getting almost nothing.Academic research also confirms this change.
A 2026 study found that more than half (51.5%) of search queries trigger a Google AI overview.
Data from the Pew Research Center shows that on search pages without an AI overview, the proportion of users clicking on links is 15%. When there is an AI overview, it drops directly to 8%; as for the link cited by the AI overview... the click-through rate is only a pitiful 1%.
There are already platforms that are losing real money.
In September 2025, the American publishing group Penske Media took Google to court. Penske Media owns well-known media brands such as "Rolling Stone", "Hollywood Reporter" and "Billboard", with more than 120 million monthly visitors, and a large part of its revenue comes from website visits. However, since Google launched its AI overview, Penske Media’s revenue from affiliate links has dropped by more than a third.
This is not just a story across the ocean. Xiaohongshu launched an AI search function called “Ask” in 2025. The logic is exactly the same as Google AI Overview: the AI sorted results are directly displayed at the top of the search.
The impact of this is exactly the same as that of Google: in the past, users searched for keywords, browsed notes one by one, and filtered the information themselves; now users ask questions, and AI integrates relevant note content and directly gives answers. Users no longer click on notes one by one, and bloggers lose the opportunity to be seen, followed, and converted.
According to media reports, Xiaohongshu initially only allowed “Ask Questions” to cover 1%-2% of search requests internally. The reason was that “the boss was afraid of having too much impact”, and later it was increased to 3%-4%. Obviously the platform itself knows how much impact this incident has on creators.
The old era ends and the new era begins
The game between Cloudflare and Google has no answer yet. But one thing is certain:
The tacit understanding of "free content in exchange for traffic" that has operated on the Internet for thirty years can no longer go back
.In fact, the Chinese Internet has already gone through this road. You can’t search for articles from WeChat public accounts on Baidu, and you can’t search for content from Xiaohongshu. It’s not that search engine technology can’t do it, but that the platform has actively built walls to trap content in its own ecosystem. Now Cloudflare is building a “paywall” for websites to prevent AI from getting content for free. The methods are different, but the results are the same.
If more and more websites block AI crawlers, AI search will obtain less and less high-quality content, and the quality of the answers given will decline; and it will be difficult to maintain the website if it loses search traffic. Eventually, both sides will collapse together, resulting in a "content desert".
AI search is really convenient, with an answer in three seconds. But all summaries are based on someone writing the original text.
If those people can no longer write, the things that AI can summarize will become less and less
. Sooner or later, someone has to pay this bill.
Comments