一项名为Real-SWE的新基准测试发布，用于衡量前沿AI模型在私有、真实企业代码库中的表现1。该基准采用来自实际生产环境的任务，涵盖具有商业影响的复杂工程问题，被测试的企业包括拥有200K+用户的社交平台竞争对手、处理100K+银行账单的消费金融平台，以及企业AI销售平台1。
测试结果显示AI模型面临显著挑战1。短期执行（10分钟以内）失败率达71.4%，长期执行失败率为73.4%1。最常见的失败原因是遗漏需求，占所有失败中最大的比例1。另一个关键发现是，99%的真实企业代码库中的token对前沿模型均为隐藏的1，这意味着这些模型在训练过程中未曾接触过相关代码。基准测试预估单个任务的执行成本范围从$2.50到$6.961。
A new benchmarking framework called Real-SWE has been introduced to evaluate cutting-edge AI models against private, real-world enterprise code repositories 1. The benchmark draws tasks from production codebases at actual companies facing genuine user and business requirements, including a competitor to Luma/Partiful with over 200,000 users, a consumer finance platform processing more than 100,000 bank statements, and an enterprise AI sales platform 1. These tasks encompass complex engineering challenges with tangible business consequences that test whether AI systems can navigate production environments and meet concrete operational needs.
The testing results reveal significant performance gaps for current frontier models. Short-term execution failures—defined as failures occurring within 10 minutes—reached 71.4%, while long-term execution failures reached 73.4% 1. Analysis of the failures identified missing requirements as the leading cause, indicating that AI models frequently overlook specifications necessary for task completion 1. An additional challenge is that 99% of the tokens present in real enterprise codebases remain hidden from frontier AI models, reflecting the proprietary nature of such code 1. The estimated execution cost for these tasks ranges from $2.50 to $6.96 1.
评论
还没有评论，欢迎留下第一条。