WASTE是一款用C语言编写的推理引擎，能够在消费级硬件上运行超大规模语言模型1。该引擎使得拥有64GB内存的MacBook Pro M5 Pro可以运行参数量达2.78万亿的Kimi K3大模型1，推理速度达到0.49-0.54 tokens/秒1。
WASTE通过将模型主干保留在内存中、从磁盘流式加载专家权重的方法，解决了消费级设备上运行超大模型的难题1。转换后的模型容量为982 GiB1，最小内存需求约27.28GB1，推荐内存预算为46GB，其中包括17.56GB的专家缓存1。该引擎利用混合专家激活率（每token约4%）的特性进行优化1。WASTE已作为开源项目发布，采用Apache 2.0许可证1。
WASTE, a C-language inference engine, enables the operation of Kimi K3, a 2.78 trillion-parameter large language model, on consumer-grade hardware with remarkable efficiency 1. The engine achieves inference speeds of 0.49 to 0.54 tokens per second while requiring approximately 29 GB of RAM on a MacBook Pro M5 Pro with 64 GB of memory 1. This feat is accomplished through a hybrid architecture that maintains the model's core components in memory while streaming expert weights directly from disk 1.
The technical approach leverages Kimi K3's mixture-of-experts design, with approximately 4 percent of experts activated per token, resulting in a data read requirement of 17 GB per token 1. The engine recommends a memory budget of 46 GB, which includes 17.56 GB allocated for expert caching, though the absolute minimum memory requirement for running the model's backbone is approximately 27.28 GB 1. The converted model occupies 982 GiB of storage 1. WASTE is released as an open-source project under the Apache 2.0 license 1.
评论
还没有评论，欢迎留下第一条。