git.kernel.org运维团队报告，用于训练大语言模型的AI爬虫对该网站造成了严重的资源压力1。这些爬虫采用低效的HTML渲染方式大量采集提交历史数据，导致基础设施不堪重负1。虽然Anubis挑战机制已阻止66%的爬虫访问，但另外33%的爬虫已经学会破解该防护措施，继续对系统发起请求1。
受爬虫影响，git.kernel.org共享的5个地理分散节点中，14至16个CPU核心（占总容量的20%）正被持续用于渲染爬虫请求1。该网站日均收到约600万次请求，其中合法用户流量仅占约2%，其余均来自爬虫1。运维团队已被迫禁用某些功能以缓解服务压力1。爬虫的对抗手段在不断演进，从最初的简单user-agent识别发展到伪装成浏览器，再到使用数百万个随机住宅和移动IP地址1。即便运维团队将Anubis难度从第4级提升至第5级，爬虫仍继续破解并访问系统1。
The operations team at git.kernel.org has reported a significant strain on their infrastructure caused by AI crawlers scraping commit history for large language model training.1 The repository handles approximately 6 million requests daily, with crawlers now accounting for roughly 98 percent of traffic while legitimate requests represent only about 2 percent.1 The crawlers employ increasingly sophisticated evasion techniques, evolving from simple user-agent identification to browser spoofing and the use of millions of randomized residential and mobile IP addresses.1
Despite defensive measures, the situation continues to deteriorate. Initially, 66 percent of crawler requests were blocked by the Anubis challenge mechanism, but 33 percent have already learned to overcome the difficulty-5 challenge and persist in accessing the system.1 This relentless traffic is consuming substantial computing resources—between 14 and 16 CPU cores out of 90 total cores distributed across five geographically dispersed nodes are continuously dedicated to rendering commit data for crawler requests, representing approximately 20 percent of total capacity.1 The infrastructure supports approximately 1.48 million commits in linux.git and hosts roughly 922 forks through git.kernel.org.1 To mitigate the impact, the operations team has been forced to disable certain features in an effort to alleviate the pressure on their systems.1
评论
还没有评论，欢迎留下第一条。