语言模型Agent在生产应急响应中能力存在显著差距——ORCA-Bench基准揭示挑战

研究团队发布了ORCA-Bench基准，用于评估大语言模型Agent在生产环境应急响应中的根因分析能力。1该基准包含1,079个系统化的根因分析任务，涵盖报告具体性、检测时间和并发故障场景等变化因素。1

测试在模拟真实生产环境的系统中进行，该系统规模为50 GB、时间跨度为六天的微服务架构，集成了Prometheus、Jaeger、OpenSearch等实际使用的监测接口。1评估结果表明，当前领先模型的表现远未达到生产就绪水平。五个前沿模型在中等难度任务上的最高准确率仅为25.3%，在困难任务上更是下降至10.0%，即使包括Claude 3.5 Sonnet在内的模型也未能实现显著改进。1此外，最弱的模型在40%的事件报告中产生了不可信的根因分析幻觉。1研究进一步发现，移除源代码访问权限会导致所有性能指标下降。1人工验证的一致性系数（Cohen's κw）达到0.90，说明评估的可靠性得到保证。1

A research team has released Orca-Bench, a new evaluation framework designed to assess the root cause analysis capabilities of large language model agents in production on-call scenarios 1. The benchmark comprises 1,079 systematically constructed root cause analysis tasks that vary across factors including report specificity, detection timing, and concurrent failure scenarios 1.

The evaluation environment simulates a substantial microservices system with 50 GB of data spanning six days, providing access to real telemetry interfaces including Prometheus, Jaeger, and OpenSearch 1. Testing results reveal significant limitations in current models' readiness for production incident response. The best-performing model achieved an accuracy rate of 25.3 percent on medium-difficulty tasks and only 10.0 percent on difficult tasks 1. Even Claude Fable 5 failed to deliver substantial improvements over earlier models 1. The weakest models generated unreliable root cause analysis hallucinations in 40 percent of incident reports 1. Additionally, removing access to source code caused performance degradation across all metrics 1. Human validation of the benchmark results demonstrated strong consistency with a Cohen's kappa coefficient of 0.90 1.