Cloudflare推出AI流量管理新选项 允许网站所有者精细控制爬虫访问

Cloudflare推出了新的AI流量管理工具，使网站所有者能够对不同类型的AI爬虫进行精细化控制。1该系统将AI使用场景分为三类：Search（用于索引内容以回答问题）、Agent（代表用户实时执行任务）和Training（用于训练或微调模型）。1

根据新规则，Cloudflare将于2026年9月15日设置新的默认值，其中Training和Agent爬虫在显示广告的页面上将被默认阻止，而Search爬虫保持允许状态。1同时，Cloudflare引入了内容使用信号机制，网站所有者可在robots.txt中声明三种偏好：immediate（不存储重用）、reference（索引摘录并链接回）以及full（总结和重现）。1

为加强对爬虫的可见性和管理，Cloudflare推出了BotBase功能，企业Bot管理客户可以查看所有已知爬虫的完整目录及其分类。1此外，Cloudflare更新了Verified标签的定义，使其不再意味着默认允许，而是表示该爬虫在其相关类别中是可允许的。1技术层面上，该系统采用RFC 7239 Forwarded头实现传递信任机制，允许通过中介层追踪真实运营者身份。1

Cloudflare has introduced enhanced controls for managing artificial intelligence traffic, enabling website operators to make granular decisions about how different categories of AI bots interact with their content 1. The new system categorizes AI activity into three distinct use cases: Search, which indexes content to answer questions; Agent, which performs real-time tasks on behalf of users; and Training, which gathers data for model development or refinement 1.

Starting September 15, 2026, Cloudflare will implement updated default policies for these traffic types 1. On pages displaying advertisements, Training and Agent bots will be blocked by default, while Search activity will remain permitted 1. Website owners will also gain the ability to specify content usage signals through their robots.txt files, choosing from three options: immediate (data not stored for reuse), reference (indexed excerpts with links back to the source), and full (content summarized or reproduced) 1.

The company has also rolled out BotBase, a feature for enterprise Bot Management customers that provides a complete directory of known bots along with their classifications 1. Additionally, Cloudflare has refined the meaning of its Verified label, which now indicates that a bot is permitted within its relevant category rather than automatically allowing all verified bots 1. The platform now implements transitive trust using RFC 7239 Forwarded headers, enabling website owners to trace the actual operators behind bot traffic passing through intermediary layers 1.