开发者发布并开源了名为 LookAtMyQuant 的量化编译器,该工具通过测量压缩对输出的影响来分配比特位,显著优化了大语言模型的存储效率 1。核心演示显示,LookAtMyQuant 成功将 thinkingmachines 旗下的 Inkling-2-Mini 模型从原始的 18.89 GB 文本权重压缩至 5.38 GB,平均每个权重仅占用 1.63 比特 1。在性能对比中,这一 5.38 GB 版本的模型在 GSM8K 和 HumanEval 基准测试上的得分高于 Unsloth 推出的 9.14 GB UD-Q2_K_XL 模型,证明了其在保持性能的同时大幅降低内存占用的能力 1。
除了 Inkling-2-Mini,该工具还在其他主流模型上展现了出色的压缩效果:Qwen3.5-4B 被压缩至 1.23 GB(2.3 bits per weight),而 Qwen3.8-27B 则从 53.7 GB 大幅缩减至 7.2 GB(2.1 bits per weight) 1。技术上,LookAtMyQuant 采用了后训练量化、格状编码 (trellis codes)、基于 Fisher 信息的行比特分配以及蒸馏步骤等先进方法 1。针对 Apple Silicon 硬件的适配测试表明,在 M5 Max 芯片上运行 5.4 GB 版本时,解码速度可达串行 73.2 tok/s,中位数速度为 125.2 tok/s 1。目前,该项目的代码已遵循 MIT 许可证正式开源 1。
The developer tool LookAtMyQuant, an open-source quantization compiler, has successfully reduced the memory footprint of the thinkingmachines/Inkling-2-Mini model from 18.89 GB to 5.38 GB 1. This compression achieves a density of 1.63 bits per weight while maintaining performance metrics that surpass less compressed alternatives 1. Specifically, at this 5.38 GB size, the LookAtMyQuant version of Inkling-2-Mini outperforms Unsloth’s 9.14 GB UD-Q2_K_XL model on both GSM8K and HumanEval benchmarks 1. The underlying technology employs post-training quantization, trellis codes, Fisher information-based row bit allocation, and a distillation step to optimize storage efficiency 1.
Beyond Inkling-2-Mini, the compiler demonstrated significant reductions for other large language models, including compressing Qwen3.5-4B to 1.23 GB (2.3 bits per weight) and shrinking Qwen3.8-27B from 53.7 GB to 7.2 GB (2.1 bits per weight) 1. The tool supports Apple Silicon runtimes, delivering inference speeds of 73.2 tokens per second in serial mode and a median of 125.2 tokens per second on an M5 Max chip for the 5.4 GB model version 1. The source code for LookAtMyQuant is distributed under the MIT license 1.
评论
还没有评论,欢迎留下第一条。