Deltafin项目成功在Apple M1 Max笔记本电脑上实现了参数量为2.8万亿的Kimi K3大模型的本地推理1。该项目通过专门的优化技术，证明了在消费级硬件上部署大规模混合专家模型的可行性，项目代码采用MIT许可证开源1。
运行该模型需要64GB内存的Apple Silicon Mac硬件配置1。完整安装需要1.7TB的磁盘空间，而采用流式安装模式则可将空间需求降至215GB1。模型的驻留脊柱大小约为114GB，经过量化后可缩减至60GB1。每个推理token需要读取16个专家×92层对应的25.8GB专家数据，从本地磁盘读取耗时约4秒，通过网络读取则需要数分钟1。
虽然推理速度约为每token 16秒1，但该项目提供了完整的安装指南和OpenAI API兼容接口1，为用户在资源受限的设备上本地运行大规模语言模型提供了新的解决方案。
Deltafin, a research project, has successfully demonstrated running the Kimi K3 large language model with 2.8 trillion parameters on an Apple M1 Max MacBook equipped with 64GB of memory.1 Despite inference speeds of approximately 16 seconds per token, the project showcases how specialized optimization techniques can enable large-scale mixture-of-experts model inference on resource-constrained consumer hardware.1 The project provides complete installation instructions and an OpenAI API-compatible interface.1
The implementation requires substantial storage capacity, with the full installation needing 1.7 terabytes of disk space, though a streaming installation mode reduces this requirement to 215 gigabytes.1 The resident spine—the model component kept in memory—occupies approximately 114 gigabytes unquantized, or 60 gigabytes when quantized.1 Each inference pass reads data from 16 experts across 92 layers, totaling approximately 25.8 gigabytes of expert data per token.1 When loading this data from the local disk, the process takes about 4 seconds, whereas network-based retrieval would require several minutes.1 The project's code is released under the MIT license.1
评论
还没有评论，欢迎留下第一条。