AI SRE Arena是一个开源基准测试框架，旨在评估AI Site Reliability Engineering（SRE）代理在Kubernetes环境中自动化故障调查和运维的能力1。该项目提供了21个完整故障场景和6个快速测试场景供测试使用1，支持本地kind集群或AWS EKS两种部署方式1。用户可集成任何第三方监测产品进行比较评估1。
基准测试结果显示，不同的SRE代理在各类场景中的表现存在差异1。其中Edge Delta检测并调查了18个场景，Grafana检测并调查了12个场景，而Claude作为外部集成则在两个平台上都参与了全部21个场景的评估1。该框架采用GPT-6-Astra作为统一的评分模型1。
在技术要求方面，项目的唯一Python依赖为Python 3.10+1，Kubernetes操作需要kubectl，本地集群创建需要Docker和kind1。该项目采用MIT许可证1。
AI SRE Arena is an open-source benchmarking framework designed to evaluate the capabilities of AI Site Reliability Engineering agents in automating fault investigation and infrastructure operations on Kubernetes 1. The project offers a comprehensive testing environment with 21 complete failure scenarios and 6 expedited test scenarios, accommodating both local kind clusters and AWS EKS deployments 1.
The framework supports integration with third-party monitoring products for comparative analysis, allowing multiple AI agents to be evaluated against the same fault scenarios 1. Benchmark results demonstrate varying performance across different solutions: Edge Delta detected and investigated 18 scenarios, while Grafana's native AI investigation capabilities handled 12 scenarios, with Claude also participating across all 21 scenarios on both platforms 1. The evaluation employs GPT-6-Astra as a unified scoring model to ensure consistent assessment criteria 1.
The project maintains minimal technical requirements, requiring only Python 3.10 or higher as its primary Python dependency, along with kubectl for Kubernetes operations and Docker with kind for local cluster creation 1. Released under the MIT license, AI SRE Arena provides the open-source community with a standardized approach to measuring and comparing AI agent performance in reliability engineering scenarios 1.
评论
还没有评论，欢迎留下第一条。