PostHog与TypeSafe AI团队分享了在生产环境中使用结构化“决策模型”(如Jev)替代大型语言模型(LLM)处理简单任务的工程经验1。数据显示,这类决策模型相比前沿LLM具有显著的成本和速度优势,价格最高可降低444.6倍,速度快达193.6倍1。尽管具备高效能,但该模型并不适用于需要多步推理或大量上下文的任务,此类复杂场景仍需依赖LLM1。
在具体应用场景中,该团队利用决策模型作为网关以过滤昂贵的LLM调用并保护隐私1。例如,在其Replay Vision产品中,通过两个简单的“是/否”问题检查个人身份信息(PII),使得98.7%的会话摘要无需调用Gemini即可通过筛选1。此外,决策模型也被用于路由请求至最便宜的模型以及替代过度复杂的规则引擎进行基础判断1。
为了平衡错误代价,团队建议根据具体场景调整置信度阈值策略1。若假阴性代价较高(如泄露PII),应设置较低阈值(如0.4);反之,若假阳性代价较高(如打扰用户),则需设置较高阈值(如0.85)1。值得注意的是,不同模型版本(如v1与v2)及不同任务间的阈值不可直接迁移,必须进行单独校准1。在搜索结果重排序实验中,结合嵌入向量与决策模型判断匹配度,使排名质量提升了约25%1。目前,内部已开发出名为Jeeves的混合决策模型,具备在不确定时先推理再决策的能力1。
PostHog has shared engineering insights detailing how its team utilizes structured decision models, specifically a tool called Jev, to handle routine tasks in production environments rather than relying on large language models 1. This approach demonstrates that specialized decision models can be significantly more efficient for simple operations, offering cost reductions of up to 444.6 times and speed improvements of 193.6 times compared to frontier LLMs 1. The strategy involves using these lightweight models as gateways to filter expensive LLM calls, route requests to the most cost-effective models, and replace overly complex rule engines for basic judgments 1.
In practical applications such as PostHog’s Replay Vision product, the system employs two yes/no questions to check for Personally Identifiable Information (PII), allowing 98.7% of session summaries to bypass Gemini API calls entirely 1. For search result reranking, combining embedding vectors with decision model assessments improved ranking quality by approximately 25% 1. However, the effectiveness of this method depends on careful calibration; teams must adjust confidence thresholds based on the cost of errors, setting lower thresholds like 0.4 when false negatives (such as PII leaks) are critical, and higher thresholds like 0.85 when false positives disrupt users 1. Additionally, thresholds cannot be transferred between different model versions or task types without individual recalibration 1. While an internal hybrid model named Jeeves was developed to reason before deciding when uncertain, the engineers emphasize that decision models remain unsuitable for multi-step reasoning or tasks requiring extensive context, areas where LLMs are still necessary 1.
评论
还没有评论,欢迎留下第一条。