CloudFlare, nicknamed the Cyber ​​Bodhisattva by netizens, has recently launched AI crawler protection settings for all free users. This feature has actually been available before but needs to be set according to rules. The rule function requires a subscription plan such as CloudFlarePro to be used.

The newly launched AI crawler protection (AIScrapers and Crawlers) is a one-click setting for all users. Whether they are free or subscribed users, they only need to turn on this option with one click.

CloudFlare said that turning this option on will prevent robots and crawlers from crawling website content for training artificial intelligence models, which can prevent some artificial intelligence companies from directly grabbing website content without authorization.

The operating principle behind it is not yet clear, but there is a high probability that CloudFlare has sorted out and identified common AI crawlers such as OpenAI's GPTBot, and only needs to intercept them based on the names of these crawlers.

CloudFlare provides security protection and distribution services for millions of websites, so this feature does have practical significance. After all, many websites, especially news media websites, are copyright protected.

But the problem is that crawler identification itself is also a way to protect against bad guys and not villains. It is easy to deal with it when OpenAI publishes its own crawler name and states that it will abide by the robots.txt protocol. The problem is that many companies do not abide by the agreement and do not publish their own crawler names at all, and they crazily crawl websites to obtain content.

In this case, if you want to intercept, you need to think of some countermeasures, such as directly banning UA that frequently crawls, turning on CloudFlare's real-person verification and other functions to prevent various crawlers from bypassing CloudFlare's AI crawler protection.

How to turn on this feature: CloudFlare, console, go to the corresponding website, security, automatic program, and turn on AIScrapers and Crawlers.