研究团队发布了ORCA-Bench基准，用于评估大语言模型Agent在生产环境应急响应中的根因分析能力。[1]该基准包含1,079个系统化的根因分析任务，涵盖报告具体性、检测时间和并发故障场景等变化因素。[1]
测试在模拟真实生产环境的系统中进行，该系统规模为50 GB、时间跨度为六天的微服务架构，集成了Prometheus、Jaeger、OpenSearch等实际使用的监测接口。[1]评估结果表明，当前领先模型的表现远未达到生产就绪水平。五个前沿模型在中等难度任务上的最高准确率仅为25.3%，在困难任务上更是下降至10.0%，即使包括Claude 3.5 Sonnet在内的模型也未能实现显著改进。[1]此外，最弱的模型在40%的事件报告中产生了不可信的根因分析幻觉。[1]研究进一步发现，移除源代码访问权限会导致所有性能指标下降。[1]人工验证的一致性系数（Cohen's κw）达到0.90，说明评估的可靠性得到保证。[1]
A research team has released Orca-Bench, a new evaluation framework designed to assess the root cause analysis capabilities of large language model agents in production on-call scenarios [1]. The benchmark comprises 1,079 systematically constructed root cause analysis tasks that vary across factors including report specificity, detection timing, and concurrent failure scenarios [1].
The evaluation environment simulates a substantial microservices system with 50 GB of data spanning six days, providing access to real telemetry interfaces including Prometheus, Jaeger, and OpenSearch [1]. Testing results reveal significant limitations in current models' readiness for production incident response. The best-performing model achieved an accuracy rate of 25.3 percent on medium-difficulty tasks and only 10.0 percent on difficult tasks [1]. Even Claude Fable 5 failed to deliver substantial improvements over earlier models [1]. The weakest models generated unreliable root cause analysis hallucinations in 40 percent of incident reports [1]. Additionally, removing access to source code caused performance degradation across all metrics [1]. Human validation of the benchmark results demonstrated strong consistency with a Cohen's kappa coefficient of 0.90 [1].