Aleph Alpha发布了Kolibri开源语言模型，这是一个具有78B总参数、3B活跃参数的英德双语混合专家Transformer模型1。该模型支持最长1M tokens的上下文长度1，专为主权可控和监管合规的关键任务开发，已在Hugging Face平台开源，采用Apache 2.0许可证1。
Kolibri在性能和能力上实现了多维升级。在768个B200 GPU上训练，共包含约24T tokens的数据，其中20T用于预训练、3.44T用于中期训练和200B用于长上下文适应1。预训练数据中，21.3%为德语、62%为英语、14%为代码1。与三个月前完成预训练的前代Kolibri Origin相比，新版本在数学、编码、接地和长上下文任务上的表现可匹配参数量多达四倍的模型（如Nemotron 3 Super）1。模型通过Merlin-Arthur协议训练以减少幻觉现象，其在AA-Omniscience测试中的弃权能力从Origin的15%提升至44%1。在Honeypot基准上，Kolibri超越所有对比模型，包括更大的Nemotron 3 Super和Mistral Small 41。
德语预训练数据中，80%由Aleph Alpha自主策展或生成，20%来自开源数据集1。
Aleph Alpha has released Kolibri, an open-source bilingual language model designed for sovereign and compliant critical applications 1. The model features a mixture-of-experts Transformer architecture with 78 billion total parameters and 3 billion active parameters, supporting context windows of up to 1 million tokens 1. The model is now available on Hugging Face under the Apache 2.0 license 1.
Kolibri was trained on approximately 24 trillion tokens across 768 B200 GPUs, comprising 20 trillion tokens for pretraining, 3.44 trillion for intermediate training, and 200 billion for long-context adaptation 1. The training corpus includes 21.3 percent German language data, 62 percent English, and 14 percent code 1. Of the German pretraining data, 80 percent was curated or generated by Aleph Alpha internally, while 20 percent sourced from open datasets 1.
The model represents a significant advancement over its predecessor, Kolibri Origin, which completed pretraining on June 11, with Kolibri finishing three months later on September 11 1. On mathematical, coding, grounding, and long-context tasks, Kolibri matches the performance of models with up to four times more parameters, such as Nemotron 3 Super 1. The model was trained using the Merlin-Arthur protocol to reduce hallucinations, achieving a 44 percent abstention rate on the AA-Omniscience benchmark, compared to 15 percent for Kolibri Origin 1. In the Honeypot benchmark, Kolibri outperformed all comparison models, including the larger Nemotron 3 Super and Mistral Small 4 1.
评论
还没有评论，欢迎留下第一条。