Kimi团队发布了Kimi Linear混合线性注意力架构，在短上下文、长上下文和强化学习等多个场景中首次超越全注意力机制[1]。该架构的核心创新是Kimi Delta Attention（KDA）模块，采用精细的门控机制和专用的对角线加低秩转移矩阵算法，显著降低了计算成本[1]。预训练的3B激活参数模型相比全MLA架构性能更优，同时实现KV缓存减少75%、1M上下文解码吞吐量提升6倍的优化[1]。
Kimi Linear采用混合架构设计，将KDA线性层与多头隐式注意力机制相结合[1]。根据Kimi K3模型的应用，该混合架构采用3:1混合比例：3层KDA线性层穿插1层全注意力[2]。单个token的缓存大小从传统70B模型的320KB降至约24KB，尽管Kimi K3模型规模大40倍[2]。Kimi团队已开源了KDA核心代码和vLLM实现[1]。
The Kimi team has released Kimi Linear, a hybrid linear attention architecture that outperforms full attention mechanisms across short-context, long-context, and reinforcement learning scenarios for the first time.[1] The core innovation is the Kimi Delta Attention (KDA) module, which employs a more refined gating mechanism and a specialized diagonal-plus-low-rank (DPLR) state transition matrix algorithm that significantly reduces computational cost.[1]
The pretrained model with 3 billion activated parameters out of 48 billion total parameters demonstrates superior performance compared to full Multi-Head Latent Attention (MLA) architecture while achieving substantial efficiency gains.[1] KV cache usage is reduced by up to 75 percent, and decoding throughput for 1 million token contexts improves by up to 6 times.[1] The team has open-sourced both the KDA kernel and a vLLM implementation.[1]
In the Kimi K3 model deployment, the hybrid architecture employs a 3:1 mixing ratio, with three KDA linear attention layers alternating with one full attention layer, enabling single-token cache to shrink from approximately 320 kilobytes in Llama 3.1-70B to around 24 kilobytes in Kimi K3 despite the model being 40 times larger.[2]
