Jevstiller是一个开源系统，通过学习大型模型Jev的答案来训练本地小模型，从而在保证与原模型一致性的前提下实现显著加速1。本地模型推理延迟约15毫秒，相比API调用的300毫秒延迟1。
该系统的核心创新在于提供数学化的不一致性保证1。用户可以设定目标一致率（如98%），系统承诺至少在该比例的请求上返回与Jev相同的答案，通过固定序列检验、Clopper-Pearson置信界和持续审计机制来维持这个契约1。系统将超预算概率控制在5%以内，并预留85%的预算余地用于二次检验1。引入confidence_floor参数来处理Jev的不确定性标记，但这会带来4-12个百分点的覆盖率下降1。
在生产环境保障方面，系统采用固定2%的请求抽样进行持续审计以检测环境漂移1。在24小时压力测试中，当Jev改变所有答案后，系统在4分钟内检测到异常、49分钟内完成重新训练1。在5个基准任务的评估中，该方案相比点估计方法损失4-8个百分点覆盖率；若将目标一致率从99%放宽至90%可显著提升覆盖率1。Jevstiller以Apache 2.0许可开源发布，支持Docker容器部署1。
Jevstiller, an open-source system announced on Hacker News, enables the training of compact local models by learning from the outputs of a large model called Jev, achieving inference latency of approximately 15 milliseconds compared to 300 milliseconds for direct API calls 1. The framework's distinguishing feature is a mathematically rigorous disagreement bound: users can set a target consistency rate—such as 98%—and the system contractually guarantees that it will return answers identical to Jev's at least that proportion of the time, with a budget tolerance of 1 minus the target rate 1.
To maintain this statistical guarantee, Jevstiller employs a combination of fixed-sequence testing and Clopper-Pearson confidence intervals, reducing the probability of exceeding the allocated budget to below 5% 1. The framework implements four critical design decisions: treating the consistency contract itself as the loss function, applying strict hypothesis testing thresholds, prohibiting parameter tuning on calibration data, and reserving 85% of the budget for secondary verification 1. A confidence floor parameter was introduced to handle uncertainty markers from Jev, though this approach incurs a coverage reduction of 4 to 12 percentage points 1. The system performs continuous auditing through fixed 2% request sampling to detect production environment drift; during a 24-hour stress test where Jev changed all answers, the system detected the shift within 4 minutes and completed retraining in 49 minutes 1.
Performance evaluations across five benchmark tasks show that the framework sacrifices 4 to 8 percentage points of coverage compared to point-estimate baselines, yet relaxing the consistency target from 99% to 90% significantly improves coverage 1. Jevstiller is released under the Apache 2.0 license and supports deployment via Docker containers 1.
评论
还没有评论，欢迎留下第一条。