Spinf Inc. 发布了一款名为 spinf-serve 的开源 AI 推理服务器，该工具专门针对批量评分和决策任务进行优化 1。与传统生成式模型不同，spinf-serve 能够在单次读取内容后并行回答多个问题，并直接返回选项的概率分布（包括 logprob、p 和 floor_p），而非生成文本，从而显著提升了处理效率 1。这一架构特性使其特别适用于市场指数构建、内容审核等需要高吞吐量分类的应用场景 1。
在技术实现上，spinf-serve 明确区分于 vLLM 等主流推理引擎，后者通过前缀缓存复用内容但仍逐 token 生成，而 spinf-serve 则将问题布局为单次读取的分支结构 1。性能基准测试显示，在配备 RTX 4090 GPU 且运行 Gemma 4 12B 模型的环境下，该服务器处理新闻文章的阅读速度可达每秒近 700 个问题 1。软件采用 Apache 2.0 许可证发布，对硬件环境有特定要求，需运行在 Linux 系统上，配备支持 fp8 tensor cores 的 24GB NVIDIA GPU（如 RTX 4090/5090、H100 等），并依赖 CUDA 13 和 Python 3.12 版本 1。
Spinf Inc. has released spinf-serve, an open-source AI inference server designed for bulk scoring and decision-making tasks 1. Unlike traditional generative models that produce text, this tool is engineered to answer multiple questions in parallel after a single content read, returning probability distributions instead of generated outputs 1. It supports high-throughput classification scenarios such as market index construction and content moderation 1.
The software requires a Linux system equipped with a 24 GB NVIDIA GPU featuring fp8 tensor cores (e.g., RTX 4090/5090, H100), along with CUDA 13 and Python 3.12 1. It provides an OpenAI-compatible API and is distributed under the Apache 2.0 license 1. Performance benchmarks indicate that when running the Gemma 4 12B model on an RTX 4090, the system can process news articles at a speed of nearly 700 questions per second 1.
Architecturally, spinf-serve differs from vLLM; while vLLM uses prefix caching but generates responses token-by-token, spinf-serve structures questions as branches within a single read pass 1. The output format consists of option probabilities represented by logprob, p, and floor_p values rather than sequential text generation 1.
评论
还没有评论，欢迎留下第一条。