百度正式发布了基础语言模型 ERNIE 5.5 Flash Base 1。该模型专注于训练与推理的效率平衡,并已在 Hugging Face 上开源,供开发者进行下游任务微调 1。
在架构设计上,ERNIE 5.5 Flash Base 是一个稀疏混合专家(MoE)模型,拥有2800亿总参数,每次推理激活140亿参数 1。其预训练数据量达33.5万亿 tokens,并采用了混合注意力机制(MLA + HCA) 1。关于上下文长度,该模型的训练上下文最高支持 128K tokens,而在微调及推理阶段支持高达100万 tokens 1。
Baidu has officially released the foundational language model ERNIE 5.5 Flash Base, making it available on Hugging Face for developers to fine-tune for downstream tasks 1. This new model utilizes a sparse Mixture of Experts (MoE) architecture with a total of 280 billion parameters, while activating only 14 billion parameters per token during inference to balance training and computational efficiency 1. The release is associated with the year 2026, indicating it is either a future projection or the latest iteration in Baidu’s AI lineup 1.
The model was trained on a massive dataset comprising 33.5 trillion tokens and incorporates advanced architectural features such as hybrid attention mechanisms combining MLA and HCA 1. While the pre-training context window supports up to 128K tokens, the model allows for an extended context length of 1 million tokens during both fine-tuning and inference stages 1.
评论
还没有评论,欢迎留下第一条。