IBM Research团队(包括Abhinav Moudgil、Ningyuan Huang、Eeshan Gunesh Dhekane等)提出了一种将基于注意力机制的Transformer模型知识迁移至Mamba架构的新方法,旨在解决跨架构蒸馏过程中常见的性能损失问题。1 该研究的核心目标在于利用现有的预训练Transformer资源,同时降低模型运行时的内存消耗。1
为实现这一目标,研究者设计了一套两阶段蒸馏策略:首先通过核技巧变体将传统Transformer模型(如Pythia-1B)的知识蒸馏至线性化注意力版本,随后再将此中间模型适配到完全不含注意力块的纯Mamba架构中。1 实验数据显示,经过这种跨架构蒸馏后的Mamba模型在困惑度指标上达到了14.01,接近教师模型Pythia-1B原本的13.86,表明其基本保留了原模型的预测能力。1
尽管整体表现优异,该方法仍存在一定局限性,特别是在那些依赖上下文内召回(in-context recall)能力的任务上,蒸馏后的模型表现出轻微的性能差距。1 相关成果涉及ICML会议背景,并于2025年7月发布,文章同时也探讨了关于理解Mamba输入选择性以及蒸馏缩放定律的相关研究现状。1
Researchers from IBM Research, including Abhinav Moudgil, Ningyuan Huang, and Eeshan Gunesh Dhekane, have introduced a novel two-stage knowledge distillation method designed to transfer capabilities from attention-based Transformer models to the Mamba architecture 1. The primary objective of this initiative is to leverage existing pre-trained resources while reducing memory consumption 1. Published on July 4, 2025, in conjunction with ICML discussions, the study outlines a strategy that first converts standard Transformers into linearized attention versions before adapting them into pure Mamba models devoid of any attention blocks 1.
The proposed technique utilizes a variant of kernel tricks in its initial phase to distill knowledge from traditional Transformers, such as Pythia-1B, into linearized attention mechanisms, followed by a second stage that maps these intermediate representations directly onto the Mamba framework 1. Experimental results indicate that the distilled Mamba model achieves a perplexity score of 14.01, which is close to the teacher model’s score of 13.86, demonstrating effective preservation of core language modeling capabilities 1. While the new architecture performs well on most downstream tasks, the researchers noted a slight performance gap in tasks heavily reliant on in-context recall 1.
评论
还没有评论,欢迎留下第一条。