一项技术实验验证了利用决策模型Jev与前沿模型Opus组合,可在大幅降低成本的同时维持接近顶级模型的准确率。该研究聚焦于检测身份解析中的“过度合并”现象,结果显示,通过让Jev先行筛选、Opus随后复核的混合策略,系统能够以低于纯Opus方案四倍以上的成本实现前沿级别的准确性 1。
作为基础组件的决策模型(包括Jev和GPT-6 Luna)相比前沿模型具有显著的成本和速度优势,价格可低至90分之一,处理速度快10倍以上 1。具体而言,这类模型每处理1,000个身份记录的成本为0.5至0.1美元,响应时间约为0.15秒 1。在简单测试场景下,模型准确率的范围介于88.8%至97.3%之间 1。
尽管GPT-6 Luna在开箱即用时表现优于Jev,但在高难度案例中其置信度校准较差,特别是在90%至100%的高置信度区间存在过度自信问题,导致其在混合架构中效果不佳 1。相比之下,Jev展现了更优的置信度校准能力,其最佳置信度阈值设定为60%,而Luna为70% 1。
该基准测试数据由华盛顿大学的pseudopeople工具生成,模拟了美国小镇居民的记录情况 1。研究指出,若采用人工抽样审核方式,速度仅约为每小时50个身份,且要估算罕见错误率需要超过1000个样本,这进一步凸显了自动化混合模型方案的优势 1。
A technical experiment involving identity resolution has demonstrated that a hybrid approach combining the decision model Jev with the large language model Opus can achieve accuracy levels comparable to frontier models while reducing costs by more than four times. 1 This method addresses the challenge of detecting "over-merging" in identity data, leveraging the specific strengths of each model to optimize both performance and economic efficiency. 1
While the GPT-6 Luna model outperformed Jev in immediate, out-of-the-box testing scenarios, it exhibited poor confidence calibration in high-difficulty cases, leading to overconfidence in its highest certainty ranges (90–100%). 1 In contrast, Jev displayed superior confidence calibration, making it more suitable as a primary filter. 1 The optimal confidence threshold for Jev was determined to be 60%, whereas Luna required a higher threshold of 70%. 1 By implementing a strategy where Jev first screens the data and Opus subsequently reviews complex cases, the system successfully balanced speed and precision. 1
Decision models like Jev and Luna offer significant operational advantages, costing up to 90 times less and operating more than 10 times faster than frontier LLMs. 1 Processing 1,000 identities with these decision models costs between $0.50 and $0.10, with an average response time of approximately 0.15 seconds. 1 In simple test scenarios, the accuracy range for these models was recorded between 88.8% and 97.3%. 1 The benchmark data used for this evaluation was generated using the University of Washington’s pseudopeople tool, which simulates records of residents in American small towns. 1 For context, manual human sampling audits operate at a much slower pace of roughly 50 identities per hour and require more than 1,000 samples to accurately estimate rare error rates. 1
评论
还没有评论,欢迎留下第一条。