近日,一项名为“smolmaxxing”的技术方案发布,其核心目标是在电视盒子、旧手机、平板、单板计算机以及路由器等算力有限的设备上运行尽可能大的大型语言模型 1。该项目基于MIT许可证的llama.cpp代码库开发,但强调必须使用打过补丁的特定版本,因为标准版的llama.cpp无法加载经过头部裁剪的文件 1。
通过实施针对Cortex-A55/A7芯片的1-bit查找表内核优化、头部裁剪(head pruning)、专家分页(experts paged from flash)以及密集模型流式传输等关键技术,该方案实现了显著的性能突破 1。实测数据显示,在仅有约1GB内存的设备(如电视盒子和iPad)上,该系统能够成功运行8B和16B参数的MoE模型;而在拥有约3GB空闲内存的手机上,则可支持35B和80B参数的大规模模型 1。具体支持的模型示例包括大小为12.3 GB的Qwen3.6-35B-A3B以及27.6 GB的Qwen3-Next-80B-A3B 1。
A developer has launched "smolmaxxing," a technical initiative designed to run large language models (LLMs) on hardware with minimal computational power, such as TV boxes, old smartphones, tablets, single-board computers, and routers 1. The project’s primary objective is to execute the largest possible language models on the smallest available hardware by leveraging custom optimizations and streaming techniques 1. This approach allows 8B and 16B parameter Mixture of Experts (MoE) models to operate on devices with approximately 1GB of RAM, including TV boxes and iPads, while phones with around 3GB of free RAM can handle significantly larger models ranging from 35B to 80B parameters 1.
The underlying technology is built upon patches for the MIT-licensed llama.cpp codebase, requiring a specific patched version because standard llama.cpp cannot load models with pruned heads 1. Key technical innovations include specialized kernels optimized for Cortex-A55/A7 chips using 1-bit lookup tables, head pruning, and methods for streaming dense models and paging experts directly from flash storage 1. Demonstrations of feasibility include running Qwen3.6-35B-A3B (12.3 GB) and Qwen3-Next-80B-A3B (27.6 GB), with the developer emphasizing that all performance claims are backed by measured data rather than theoretical estimates 1.
评论
还没有评论,欢迎留下第一条。