开发者在Hacker News上发布了Parsimony，一个开源基准测试工具，用于衡量AI模型生成代码的冗余程度1。现有的基准测试主要关注解决率和token成本，但未能对冗余代码进行惩罚1。Parsimony通过引入"编码单元"指标填补了这一空白，该指标包括语句、调用、名称和字面量等语法元素1。在计算中，删除的代码会被扣除，最终衡量的是净单元数1。
该工具使用DeepSWE、SWE-bench Verified和SWE-bench Live已发布的补丁进行测试1。由于GPT-6和Claude 5.5尚无公开补丁可用，作者在自己的订阅上运行了3-10个任务来进行测试1。
A developer has introduced Parsimony, an open-source benchmarking tool designed to evaluate how much unnecessary code artificial intelligence models generate when solving programming tasks 1. The tool addresses a gap in existing benchmarks, which typically measure solution rates and token costs but do not account for code quality or the volume of extraneous code added by AI systems 1.
Parsimony quantifies code redundancy by counting "coding units"—syntactic elements including statements, function calls, variable names, and literal values 1. The benchmark deducts deleted code from this tally, yielding a net unit count that reflects the true footprint of generated solutions 1. Testing was conducted using patches from DeepSWE, SWE-bench Verified, and SWE-bench Live 1.
The evaluation includes models such as GPT-6 and Claude 5.5, though public patches for these systems are not yet available; the author ran between three and ten tasks on personal subscriptions to test these models 1.
评论
还没有评论，欢迎留下第一条。