开发者分享了一个轻量级的Jev风格包装函数，为大语言模型（包括视觉模型）提供统一接口1。该函数通过读取LLM的token对数概率来实现高效的多选问题回答，支持文本和图像输入1。
这一方法通过设置 max_completion_tokens: 1、logprobs: true 和 top_logprobs: 20 参数，强制LLM输出单个token，从而快速获取多个选项的概率分布1。在实际性能表现上，Gemma 4 12B在RTX 3090显卡上达到约1帧/秒的处理速度（每帧回答三个问题），而OpenAI的gpt-6-luna约为0.2帧/秒1。该实现支持通过attachments字段扩展Jev格式来处理base64编码的JPEG图像输入1。为了支持这一功能，开发者还为llama.cpp添加了Custom Chat Completions端点和Vision模型支持1。作者提供了完整的Python实现示例，可用于实时网络摄像头帧分析1。
A developer has shared a lightweight wrapper function designed for large language models, including vision-enabled variants, that leverages token log probabilities to efficiently answer multiple-choice questions 1. The approach works by constraining model output to a single token while retrieving probability distributions across multiple options, supporting both text and image inputs 1.
The implementation utilizes specific parameters—max_completion_tokens: 1, logprobs: true, and top_logprobs: 20—to force the language model to return only a single token, enabling rapid probability assessments 1. Performance varies by model: Gemma 4 12B achieves approximately one frame per second on an RTX 3090 GPU when processing three questions per frame, while OpenAI's gpt-6-luna operates at roughly 0.2 frames per second 1. The wrapper accepts base64-encoded JPEG images through an extended Jev format that incorporates an attachments field 1. The developer also added support for Custom Chat Completions endpoints and vision models to llama.cpp 1, providing a complete Python implementation example that can analyze real-time webcam frames 1.
评论
还没有评论，欢迎留下第一条。