研究人员推出了一种名为“Lattice of Latents”(潜在格)的新型语言模型架构,旨在通过增加计算容量来优化模型性能 1。该架构的核心创新在于引入了沿时间递归运行的每层水平潜在流,这一机制与通过深度的垂直残差流并行工作 1。实验数据显示,在20B FineWeb-Edu tokens上训练的406M参数模型中,采用该架构后验证集困惑度从基准Transformer的15.41降低至14.40 1。此外,该架构展现出良好的兼容性,当与超连接(hyper-connections)和输入残差技术结合时,可将基准Transformer的困惑度进一步降至13.61 1。
除了标准Transformer模型,“Lattice of Latents”机制还能显著改善Gated DeltaNet和Mamba-2等混合模型的困惑度表现 1。在这些模型中,Lattice变体在所有测试的原生状态大小下均提升了性能,部分变体的表现甚至优于原生状态大小大4倍的纯基线模型 1。在效率方面,LLLM仅使用22层网络结构,相比拥有40层的Transformer,在保持相同注意力宽度的前提下,其KV cache大小减少了45% 1。
Researchers from MIT and Adobe have proposed a new architectural framework for language models called the "Lattice of Latents" 1. This design aims to increase computational capacity by introducing horizontal latent flows within each layer, which operate recursively along time in parallel with the vertical residual stream through depth 1. The mechanism is designed to be compatible with existing techniques that enhance residual connections, such as hyper-connections and input residuals 1.
In experimental evaluations, a 406-million-parameter model trained on 20 billion FineWeb-Edu tokens demonstrated significant improvements over a parameter-matched baseline Transformer 1. The Lattice architecture reduced validation set perplexity from 15.41 to 14.40 1. When combined with additional enhancements like hyper-connections and input residuals, the perplexity dropped further to 13.61 1. These results highlight the potential of horizontal latent flows to improve model performance without solely relying on increased depth or parameters.
The study also indicates efficiency advantages in memory usage and layer count 1. The Latent Lattice Language Models (LLLM) utilize only 22 layers compared to 40 layers in standard Transformers, while maintaining a KV cache that is 45% smaller at the same attention width 1. Furthermore, the architecture benefits hybrid models such as Gated DeltaNet and Mamba-2 1. In these cases, the Lattice variants improved performance across all tested native state sizes, with some configurations outperforming pure baseline models that had four times the native state size 1.
评论
还没有评论,欢迎留下第一条。