AirLLM 是一个开源项目，通过创新的推理方法使大型语言模型能在配置受限的 GPU 上运行[1]。该工具无需应用量化、蒸馏或剪枝等传统优化技术，即可在单个 4GB GPU 上运行 70B 参数模型[1]。其核心方案是每次仅在 GPU 上保留一个模型层，通过流式加载的方式实现内存优化，使 VRAM 需求取决于单层大小而非总参数量[1]。
该项目支持多个大规模模型的部署[1]。405B 参数的 Llama 3.1 可在 8GB GPU 上运行，671B 参数的 DeepSeek-V3 可在约 12GB GPU 上运行，而 2.8T 参数的 Kimi K3 则可在 3.72GB VRAM 上运行[1]。此外，AirLLM v3.0 版本新增对 FP8 模型的支持，包括 DeepSeek-V3 和 Qwen3 系列[1]。该工具还支持 4bit 和 8bit 块级量化压缩功能，能将推理速度提升至 3 倍[1]。
AirLLM, an open-source tool, allows large language models to run on low-end GPUs through an innovative inference approach that eliminates the need for quantization, distillation, or pruning [1]. The project achieves this by maintaining only a single model layer in GPU memory at any given time, using streaming loading to optimize memory consumption [1].
The capability spans models of varying scales across different hardware configurations [1]. A 70-billion-parameter model can execute on a single 4GB GPU, while the 405-billion-parameter Llama 3.1 runs on 8GB GPU memory, and the 671-billion-parameter DeepSeek-V3 operates on approximately 12GB GPU [1]. AirLLM also supports the 2.8-trillion-parameter Kimi K3 model, requiring 3.72GB of VRAM [1]. Version 3.0 introduced support for FP8 models, including DeepSeek-V3 and the Qwen3 series [1]. Additionally, AirLLM supports 4-bit and 8-bit block-level quantization compression, which can enhance inference speed by up to three times [1].