技术作者Jamie Dborin在Hacker News发表文章，详细阐述了Kimi模型所采用的Kimi Delta Attention（KDA）线性注意力机制的完整推导过程1。文章从标准softmax注意力机制出发，通过逐步去除规范化步骤得到线性注意力形式，进而引入delta规则以解决线性注意力中的干涉问题1。
KDA机制的核心创新在于其状态转移方程，采用对角矩阵替代标量保留门，从而为每个通道实现独立的遗忘控制1。这一设计基于Gated DeltaNet的基础，将其标量保留门扩展为向量形式1。该机制的关键数学结构采用对角加低秩（DPLR）形式，表示为Λ_t - β_t k_t k_t^T1。
在工程实现层面，文章介绍了Triton核心的两种执行模式：递推型适合自回归解码场景（单token处理），而分块KDA则适用于训练和长前缀填充等需要矩阵运算的场景1。
A technical article has outlined the derivation process of Kimi Delta Attention (KDA), a linear attention mechanism used in the Kimi model.1 The analysis traces the development from standard softmax attention through several intermediate mechanisms, ultimately arriving at the KDA framework.1
The progression begins by removing normalization from softmax attention to obtain linear attention, which accumulates past key-value outer products within a fixed-size state matrix.1 Building on this foundation, DeltaNet addresses interference issues in linear attention by applying the delta rule—writing only error values rather than complete values.1 Gated DeltaNet subsequently introduces a scalar retention gate to manage stale information, which KDA extends into a vector form.1
The core innovation of KDA lies in its state transition equation, which replaces scalar retention gates with diagonal matrices to enable independent forgetting control across each channel.1 This mechanism adopts a diagonal-plus-low-rank (DPLR) structure defined as Λ_t - β_t k_t k_t^T.1 The implementation supports two execution modes in its Triton kernel: a recurrent form optimized for autoregressive decoding with single tokens, and a block-based variant suited for training and long prefix filling using matrix operations.1
评论
还没有评论，欢迎留下第一条。