开发者开发了一款浏览器游戏，邀请玩家扮演AI编码代理的人类监督者，在时间压力下批准或拒绝其执行的命令。[1]这一实验对超过40,000次游戏运行和409,000个批准/拒绝决策进行了数据分析。[1]结果显示，平均玩家的准确率仅为66.3%，即平均每三个威胁中就会漏掉一个。[1]
研究揭示了人类监督作为AI代理安全防护机制的严重局限性。[1]数据表明，32.9%的游戏会话以负分结束。[1]在性能表现方面，虽然35.2%的玩家成功捕捉到所有威胁，但其中仅20.8%能够在此基础上控制误判率，阻止不超过五分之一的安全命令。[1]另一方面，7%的玩家则批准了所有提示，完全失去了守门作用。[1]
特定命令的错误批准率尤其令人担忧。[1]npm run命令被错误批准的比例达到64.7%，是游戏中漏掉最多的单个命令类型。[1]三个隐藏恶意负载的npm脚本命令则被漏掉了52.5%。[1]考虑到游戏中约34%的命令属于威胁性质，这些漏掉率反映了监督过程中的认知困难。[1]
Anthropic公司曾指出影响人类监督有效性的关键因素："用户看到的批准越多，对每个批准的关注就越少。"[1]这一观察与实验数据相符，表明权限疲劳和命令混淆等问题确实影响了人类的安全决策能力。[1]
Developers created an interactive browser game designed to simulate human oversight of AI coding agents, requiring players to approve or reject commands under time pressure. Analysis of data from over 40,000 game sessions encompassing 409,000 approval and rejection decisions revealed that the average player missed approximately one-third of threats, achieving an accuracy rate of only 66.3%.[1]
The findings underscore significant vulnerabilities in relying on human supervision as a safety mechanism for AI agents. Among players tested, 32.9% of gaming sessions ended with negative scores, while just 35.2% of participants successfully identified all threats—and of those, merely 20.8% managed to avoid incorrectly blocking more than one-fifth of legitimate security commands.[1] At the extreme end, 7% of players approved every single prompt without distinction.[1] The game environment contained approximately 34% malicious commands overall.[1]
Performance gaps emerged most prominently in specific command categories. The npm run command was incorrectly approved in 64.7% of cases, making it the single most frequently overlooked threat type in the game.[1] Three npm scripts containing hidden malicious payloads were missed by 52.5% of players.[1] These results align with earlier observations from Anthropic noting that "the more approvals a user sees, the less attention they pay to each approval."[1]