Abstract:
Matthew Prince, CEO of network service provider Cloudflare, recently said that as AI agents increasingly replace user browsers to browse the web, some websites are facing new cost pressures: agents visit websites, read content and consume server resources, but may ultimately give answers directly to users, and the website does not receive actual visitors and real traffic, nor does it gain revenue.
Matthew Prince believes that this scale of access is changing the original business model of the Internet, and website operators and AI service providers need to explore new cost-sharing methods to avoid more and more websites being forced to close in the future.
For example, Matthew Prince said that if a user asks an AI agent to recommend a certain restaurant, the agent may visit the websites of 1,000 restaurants and read 1,000 menus, but the end user will only choose one of them. In this hypothetical scenario, the other 999 restaurant websites would also need to handle requests, incur bandwidth, server and other resource costs, but not necessarily gain customers from them.
Similar problems also exist for news and content websites. AI agents read articles and directly generate summaries. Users may get answers on the AI interface, and websites lose access, ad display, or paid subscription opportunities that they might otherwise obtain.
Cloudflare is providing tools to identify, limit and manage access to AI crawlers and agents, and is also exploring allowing websites to charge for related services based on the content they crawl or use. Cloudflare proposes two solutions: The first is to improve crawling efficiency. For example, Cloudflare cooperates with OpenAI to provide prompts when web page content changes and reduce repeated crawling of unchanged pages, which can reduce resource consumption of websites and AI service providers.
The second item is the payment mechanism that Cloudflare is promoting. AI services require payment to content providers such as websites. Different websites can set different strategies according to different scenarios. For example, some agents search for information on behalf of users, while others may help users purchase goods or book services. For the type of information being searched, websites can set fees to obtain content.
Currently Cloudflare has adopted pay-per-crawl and pay-per-use as one of its product directions.
Related articles:
After Cloudflare publicly criticized Perplexity’s crawling strategy, some began to defend it
Cloudflare's new policy urges AI companies to pay publishers for content and separate search from training crawlers
Comments