Y Combinator S25批次初创公司Magnitude推出了一款自优化推理引擎，专门为本地Agent运行设计1。该引擎采用设备端编译调优、混合分页注意力等技术，在多个平台上相比llama.cpp实现了显著的性能提升1。
在Mac M4 Pro 48GB上，Magnitude的解码速度从llama.cpp的30 tok/s提升至57 tok/s，快92%；预填充速度从466 tok/s提升至507 tok/s，快9%1。在CUDA（DGX Spark）环境下，解码速度从49 tok/s提升至58 tok/s，快19%；预填充速度从2,033 tok/s提升至2,507 tok/s，快23%1。同时，该引擎还将内存使用量分别减少了28%（Mac）和27%（CUDA）1。
Magnitude采用Apache 2.0开源协议，用Rust编写，包含自定义GPU内核运行时和自动调优器1。基准测试采用Qwen 3.6 35B A3B（4 bit）模型、64k上下文长度、无推测解码配置1。该引擎支持Mac、Linux、Windows跨平台运行，并兼容Pi、OpenCode、Hermes、Codex等多个Agent框架1。
Magnitude由Anders和Tom创立，两人此前开发的开源浏览器Agent项目获得4000多个星标和100万次以上下载1。
Magnitude, a startup from Y Combinator's S25 batch, has launched a self-optimizing inference engine designed to run agents locally on consumer hardware.1 Built in Rust with custom GPU kernels and an automatic tuning system, the engine employs device-side compilation optimization and hybrid paged attention techniques to dramatically accelerate AI inference.1
Performance benchmarks demonstrate substantial improvements over existing solutions. On an Apple M4 Pro with 48GB of memory, Magnitude achieves 57 tokens per second during decoding—92% faster than llama.cpp's 30 tokens per second—and 507 tokens per second during prefill, a 9% improvement over llama.cpp's 466 tokens per second.1 On CUDA hardware, specifically a DGX Spark system, the engine delivers 58 tokens per second for decoding (19% faster than llama.cpp's 49) and 2,507 tokens per second for prefill (23% faster than llama.cpp's 2,033).1 The optimization also reduces memory consumption by 28% on Mac and 27% on CUDA systems.1 Testing was conducted on Qwen 3.6 35B A3B quantized to 4-bit precision with a 64k context window and no speculative decoding.1
The engine is released under the Apache 2.0 open-source license and supports multiple platforms including macOS, Linux, and Windows.1 It integrates with popular agent frameworks including Pi, OpenCode, Hermes, and Codex.1 Magnitude was created by Anders and Tom, who previously developed an open-source browser agent that accumulated over 4,000 GitHub stars and more than 100,000 downloads.1
评论
还没有评论，欢迎留下第一条。