Anthropic与METR联合推出了MirrorCode基准测试，用以衡量AI模型在长期编码任务中的独立能力[1]。该基准要求AI在无法获取原始源代码的情况下，完整重新实现整个程序，涵盖Unix工具、数据序列化、生物信息学、解释器、静态分析、密码学和压缩等25个不同领域的目标程序[1]。
Claude Opus 4.7在该基准上展现了显著成果。该模型成功重新实现了约16,000行Go代码的生物信息学工具包gotree，其中包含40多条命令，耗时仅14小时，成本为251美元，而完成相同工作的人类工程师需要2至17周[1]。该实现通过了2,000个测试中的2,000个，仅在一个利基命令的边界情况测试中失败[1]。
MirrorCode的测试规模相当可观，每次尝试的预算达到100亿tokens，评估范围涵盖30个任务，即15个目标程序在2种实现语言下的版本，每个任务运行3次[1]。最大的单次任务运行成本达2,600美元，AI在此期间可无需人类干预地工作19天[1]。其中22个目标程序及其132个任务实例已作为开源代码发布，覆盖6种编程语言，而3个私有测试集则被保留[1]。
Anthropic and METR have unveiled MirrorCode, a benchmark designed to measure artificial intelligence systems' capability to independently complete complex, extended software engineering tasks [1]. The benchmark comprises 25 target programs spanning diverse domains including Unix tools, data serialization, bioinformatics, interpreters, static analysis, cryptography, and compression [1].
Claude Opus 4.7 demonstrated notable performance by successfully reimplementing gotree, a bioinformatics toolkit containing approximately 16,000 lines of Go code with over 40 commands [1]. The AI system completed this task in 14 hours at a cost of 251 dollars, whereas human engineers typically require between 2 and 17 weeks to accomplish equivalent work [1]. The reimplemented version passed 2,000 out of 2,001 tests, with failure occurring only in an edge case of a niche command [1].
The MirrorCode evaluation framework allocates a budget of 10 billion tokens per attempt and assesses 30 tasks representing 15 target programs implemented in two programming languages, with each task executed three times [1]. The most computationally intensive MirrorCode task carries a single-run cost of 2,600 dollars and requires the AI to operate autonomously for 19 days without human intervention [1]. Of the 25 target programs, 22 have been released as open-source code across 132 task instances in six programming languages, while three remain in a private test set [1].