Kimi K3是一款新发布的开源权重大模型，规模达2.8T参数，是目前最大的开源模型 1。该模型基于去年发布的Kimi Linear模型进行扩展，从48B参数规模扩大到2.8T 1。
在架构设计方面，K3引入了多项创新。模型新增了LatentMoE等组件，实现方式类似于Nemotron 3 Ultra中的技术方案 1。同时采用NoPE（无位置嵌入）方案替代传统的RoPE位置编码方法 1。此外，K3新增了原生多模态支持能力 1。
在模型性能优化上，Attention residuals的引入虽增加了约4%的训练成本和约2%的推理成本，但带来了验证损失的一致性改进以及下游任务性能的提升 1。整体架构设计趋势指向更高的推理效率 1。
Kimi has unveiled K3, a newly released open-source large language model featuring 2.8 trillion parameters, making it the largest open-source model to date 1. The model represents a significant expansion from the company's previously released Kimi Linear model, scaled up from 48 billion to 2.8 trillion parameters 1.
The K3 architecture incorporates several technological advances designed to improve efficiency and performance 1. The model introduces LatentMoE, a new component similar to implementations found in Nemotron 3 Ultra, alongside the adoption of NoPE (No Positional Embeddings) position encoding in place of the previously used RoPE approach 1. Additionally, K3 features native multimodal capabilities 1. Attention residuals have been integrated into the design, with training costs increasing by approximately 4 percent and inference costs rising by about 2 percent, while consistently improving validation loss and downstream task performance 1.
评论
还没有评论，欢迎留下第一条。