技术作者Jamie Dborin在Hacker News发表文章，详细阐述了Kimi模型所采用的Kimi Delta Attention（KDA）线性注意力机制的完整推导过程[1]。文章从标准softmax注意力机制出发，通过逐步去除规范化步骤得到线性注意力形式，进而引入delta规则以解决线性注意力中的干涉问题[1]。
KDA机制的核心创新在于其状态转移方程，采用对角矩阵替代标量保留门，从而为每个通道实现独立的遗忘控制[1]。这一设计基于Gated DeltaNet的基础，将其标量保留门扩展为向量形式[1]。该机制的关键数学结构采用对角加低秩（DPLR）形式，表示为Λ_t - β_t k_t k_t^T[1]。
在工程实现层面，文章介绍了Triton核心的两种执行模式：递推型适合自回归解码场景（单token处理），而分块KDA则适用于训练和长前缀填充等需要矩阵运算的场景[1]。
A technical article has outlined the derivation process of Kimi Delta Attention (KDA), a linear attention mechanism used in the Kimi model.[1] The analysis traces the development from standard softmax attention through several intermediate mechanisms, ultimately arriving at the KDA framework.[1]
The progression begins by removing normalization from softmax attention to obtain linear attention, which accumulates past key-value outer products within a fixed-size state matrix.[1] Building on this foundation, DeltaNet addresses interference issues in linear attention by applying the delta rule—writing only error values rather than complete values.[1] Gated DeltaNet subsequently introduces a scalar retention gate to manage stale information, which KDA extends into a vector form.[1]
The core innovation of KDA lies in its state transition equation, which replaces scalar retention gates with diagonal matrices to enable independent forgetting control across each channel.[1] This mechanism adopts a diagonal-plus-low-rank (DPLR) structure defined as Λ_t - β_t k_t k_t^T.[1] The implementation supports two execution modes in its Triton kernel: a recurrent form optimized for autoregressive decoding with single tokens, and a block-based variant suited for training and long prefix filling using matrix operations.[1]