本文详细解析了现代大语言模型(LLM)中处理令牌位置信息的核心机制,对比了GPT-2的绝对位置嵌入、原始Transformer的正弦位置编码以及当前主流的旋转位置编码(RoPE),并探讨了新兴架构趋势 1。
GPT-2使用学习到的绝对位置向量,直接加到令牌嵌入上;其上下文长度限制为1024 tokens,扩展成本高昂,若扩至100k,位置嵌入权重占比将从0.30%升至>22% 1。相比之下,Su et al.在2021年的RoFormer论文中提出了RoPE(Rotary Position Embedding),该机制通过在注意力机制内部对Query和Key向量进行旋转来编码相对位置信息,Value向量不参与旋转 1。标准RoPE频率计算公式为 θ_i = b^{-2i/d},其中基值b通常设为10,000 1。文章通过数学推导和PyTorch代码示例深入讲解了RoPE的原理及其在注意力机制中的相对位置特性 1。
统计显示,尽管出现了混合架构和新方法,全层使用RoPE仍是2026年主流开源模型最常见的选择,但完全使用RoPE的比例呈下降趋势,混合RoPE与NoPE的模型数量增加 1。MiniMax-M3等2026年开源模型采用了部分RoPE(Partial RoPE)、无位置编码(NoPE)层及学习到的相对位置偏置等新兴架构趋势 1。具体而言,MiniMax-M3采用“部分RoPE”策略,在每个128维的头中仅旋转64个维度,从而将预计算的余弦和正弦表大小减半,节省约0.134 GB内存 1。
NoPE(No Positional Encoding)依赖因果掩码提供的隐式顺序信息;研究表明在某些任务上NoPE能超越RoPE的长距离泛化能力,SmolLM3等模型在部分层省略RoPE以提升长上下文性能 1。此外,Inkling模型结合了T5的思路,在Softmax之前向注意力分数添加基于查询隐藏状态和位置间隔的学习到的相对位置偏置 1。
Positional encoding serves as a critical mechanism for processing token order in modern large language models, with the field evolving from absolute embeddings to sophisticated rotational techniques. Early approaches included the original Transformer’s sinusoidal encodings and GPT-2’s learned absolute position vectors, which are added directly to token embeddings 1. However, GPT-2’s method faces significant scalability challenges; while it limits context length to 1,024 tokens, expanding this capacity to 100,000 tokens would cause the proportion of weight parameters dedicated to position embeddings to surge from 0.30% to over 22%, representing a prohibitive cost increase 1. To address these limitations, Rotary Position Embedding (RoPE) was introduced by Su et al. in the 2021 RoFormer paper 1. Unlike absolute methods, RoPE encodes relative positional information by rotating Query and Key vectors within the attention mechanism, leaving Value vectors unrotated, using a standard frequency formula $\theta_i = b^{-2i/d}$ where the base value $b$ is typically set to 10,000 1.
Recent architectural trends observed in 2026 open-source models indicate a shift toward hybrid and optimized variants of positional encoding. While full-layer RoPE remains the most common choice, its exclusive use is declining as developers adopt mixed strategies combining RoPE with No Positional Encoding (NoPE) 1. For instance, MiniMax-M3 employs "Partial RoPE," which rotates only 64 dimensions out of every 128-dimensional head, effectively halving the size of precomputed sine and cosine tables and saving approximately 0.134 GB of memory 1. Concurrently, some models like SmolLM3 omit RoPE in certain layers, relying instead on NoPE, which depends on implicit sequential information provided by causal masks 1. Research suggests that NoPE can surpass RoPE in long-distance generalization capabilities for specific tasks 1. Additionally, the Inkling model integrates a learned relative position bias derived from query hidden states and position gaps before the Softmax operation, reflecting ideas from T5 architecture 1.
评论
还没有评论,欢迎留下第一条。