研究人员对两款AI模型在代码审查中的表现进行了对比测试。GPT-5.6 Luna的定价为输入令牌每百万0.20美元、输出令牌每百万1.20美元，而GPT-6 Astra的定价分别为10美元和50美元 1。在50个来自Cal.com、Sentry、Discourse、Keycloak和Grafana的公开拉取请求测试中，Luna单次审查成本为0.0041美元，Astra为0.113美元，前者便宜28倍 1。
在错误检测能力上，两个模型各有优劣。Luna发现69个已验证错误，Astra发现92个，但Luna的误判率高达26%（93个发现中24个失败验证），而Astra的误判率仅为4%（96个发现中4个失败验证） 1。在日常代码缺陷上，两者表现相近——Luna在数据和逻辑错误检测上发现39个，相比Astra的47个 1。然而在安全敏感领域存在明显差距，Luna在安全漏洞检测中发现9个，Astra则发现19个，且Luna在Keycloak身份认证代码审查上的差距最为显著（6个对14个） 1。
将两个模型联合使用进行审查可以发现117个错误，总成本仅为5.86美元 1。
Researchers from the Entelligence team conducted a comparative study of two AI models' performance in code review tasks, testing GPT-5.6 Luna and GPT-6 Astra across 50 public pull requests from Cal.com, Sentry, Discourse, Keycloak, and Grafana.1 Luna identified 69 verified bugs at a cost of $0.0041 per review, while Astra found 92 bugs at $0.113 per review, representing a 28-fold difference in per-review expenses.1 The pricing structures reflect this disparity: Luna costs $0.20 per million input tokens and $1.20 per million output tokens, compared to Astra's $10 and $50 rates respectively.1
The models demonstrated notably different accuracy profiles in the assessment. Luna produced a false positive rate of 26 percent, with 24 unverified findings out of 93 total detections, whereas Astra maintained a 4 percent error rate across 96 findings.1 When evaluating security-sensitive code, the performance gap widened considerably: Astra detected 19 vulnerabilities in authentication and security contexts, while Luna identified only 9 out of 24 tested security-related defects.1 Both models showed comparable performance on routine logic and data errors, with Luna discovering 39 against Astra's 47, though Luna lagged substantially in Keycloak identity authentication code analysis, finding 6 vulnerabilities compared to Astra's 14.1 A combined review strategy employing both models uncovered 117 total errors—representing 82 percent coverage—at a cumulative cost of $5.86.1
评论
还没有评论，欢迎留下第一条。