研究人员对AI代理在代码实现中的测试能力进行了系统评估，测试了26种不同的测试技术和库对其表现的影响1。实验基于Zstd实现和IMAP RFC等编码任务，所有代码均使用Rust编写，每个测试条件平均运行80次1。测试条件包括ACL2、Alloy、审计、模糊测试、Hegel、Kani、Lean 4、属性测试和测试驱动开发（TDD）等多种方法1。
研究发现，AI代理即使被明确指导使用特定的测试技术或库，仍然无法有效应用这些工具1。TDD按预期表现不佳，形式化方法也未能超越不加任何指导的默认条件1。属性测试库proptest相对有效，但仍远低于人类水平1。值得注意的是，在中等努力级别下，代理的结果通常低于更高努力级别的表现1。
此外，研究显示AI代理能够识别代码中的高风险区域，但无法对这些区域进行有效测试1。代理在高努力级别下通常能通过自己编写的测试，但这些测试的质量较差1。总体而言，未施加任何特殊指导的默认设置反而产生了更好的结果1。
Researchers evaluated how well AI agents employ testing methodologies by examining the impact of 26 different testing techniques and libraries on code implementation tasks. 1 The study, conducted in September 2026, tested AI agents using GPT-5.6 Sol across implementations of Zstd and IMAP RFC specifications, with all code written in Rust and each condition averaged across 80 runs. 1 The findings reveal a counterintuitive result: despite explicit instruction to use specific testing techniques or libraries, AI agents failed to apply these tools effectively, and systems performed better without additional guidance. 1
Among the 26 conditions evaluated—including ACL2, Alloy, auditing, fuzzing, Hegel, Kani, Lean 4, property testing, and test-driven development (TDD)—formal methods did not exceed baseline performance. 1 TDD performed poorly as expected, with a 55% confidence prediction that proved correct. 1 Property testing through proptest showed relative effectiveness compared to other approaches, but remained far below human-level results. 1 Notably, AI agents demonstrated the ability to identify high-risk code areas but could not effectively test those regions. 1 The agents typically passed tests they themselves had written at the highest effort levels (xhigh and max), though the quality of these tests was poor. 1 Results at medium effort levels generally performed worse than at higher effort levels. 1
评论
还没有评论，欢迎留下第一条。