Kimi K3是一款新发布的开源权重大模型，规模达2.8T参数，是目前最大的开源模型 [1]。该模型基于去年发布的Kimi Linear模型进行扩展，从48B参数规模扩大到2.8T [1]。
在架构设计方面，K3引入了多项创新。模型新增了LatentMoE等组件，实现方式类似于Nemotron 3 Ultra中的技术方案 [1]。同时采用NoPE（无位置嵌入）方案替代传统的RoPE位置编码方法 [1]。此外，K3新增了原生多模态支持能力 [1]。
在模型性能优化上，Attention residuals的引入虽增加了约4%的训练成本和约2%的推理成本，但带来了验证损失的一致性改进以及下游任务性能的提升 [1]。整体架构设计趋势指向更高的推理效率 [1]。
Kimi has unveiled K3, a newly released open-source large language model featuring 2.8 trillion parameters, making it the largest open-source model to date [1]. The model represents a significant expansion from the company's previously released Kimi Linear model, scaled up from 48 billion to 2.8 trillion parameters [1].
The K3 architecture incorporates several technological advances designed to improve efficiency and performance [1]. The model introduces LatentMoE, a new component similar to implementations found in Nemotron 3 Ultra, alongside the adoption of NoPE (No Positional Embeddings) position encoding in place of the previously used RoPE approach [1]. Additionally, K3 features native multimodal capabilities [1]. Attention residuals have been integrated into the design, with training costs increasing by approximately 4 percent and inference costs rising by about 2 percent, while consistently improving validation loss and downstream task performance [1].