Kimi Linear：混合线性注意力架构的突破

Kimi团队发布了Kimi Linear混合线性注意力架构，在短上下文、长上下文和强化学习等多个场景中首次超越全注意力机制1。该架构的核心创新是Kimi Delta Attention（KDA）模块，采用精细的门控机制和专用的对角线加低秩转移矩阵算法，显著降低了计算成本1。预训练的3B激活参数模型相比全MLA架构性能更优，同时实现KV缓存减少75%、1M上下文解码吞吐量提升6倍的优化1。

Kimi Linear采用混合架构设计，将KDA线性层与多头隐式注意力机制相结合1。根据Kimi K3模型的应用，该混合架构采用3:1混合比例：3层KDA线性层穿插1层全注意力2。单个token的缓存大小从传统70B模型的320KB降至约24KB，尽管Kimi K3模型规模大40倍2。Kimi团队已开源了KDA核心代码和vLLM实现1。

The Kimi team has released Kimi Linear, a hybrid linear attention architecture that outperforms full attention mechanisms across short-context, long-context, and reinforcement learning scenarios for the first time.1 The core innovation is the Kimi Delta Attention (KDA) module, which employs a more refined gating mechanism and a specialized diagonal-plus-low-rank (DPLR) state transition matrix algorithm that significantly reduces computational cost.1

The pretrained model with 3 billion activated parameters out of 48 billion total parameters demonstrates superior performance compared to full Multi-Head Latent Attention (MLA) architecture while achieving substantial efficiency gains.1 KV cache usage is reduced by up to 75 percent, and decoding throughput for 1 million token contexts improves by up to 6 times.1 The team has open-sourced both the KDA kernel and a vLLM implementation.1

In the Kimi K3 model deployment, the hybrid architecture employs a 3:1 mixing ratio, with three KDA linear attention layers alternating with one full attention layer, enabling single-token cache to shrink from approximately 320 kilobytes in Llama 3.1-70B to around 24 kilobytes in Kimi K3 despite the model being 40 times larger.2