递归循环Transformer架构面世

研究人员提出了递归循环Transformer（RLT）架构，将因果编码器与递归解码器相结合1。该设计通过在每个提示和响应令牌之间携带隐藏状态和滑动窗口注意力缓存，实现随序列增长而扩展的无界时间深度1。

该架构采用48层编码器和48层解码器的配置1，其中兼容的注意力和FFN权重在各阶段共享1。在递归路径深度上，第t个令牌后的路径遍历tL_D个解码器块，而每令牌块数保持固定。处理t个令牌后共遍历48t个解码器块，每令牌执行96个逻辑块1。滑动窗口注意力的窗口宽度为W，包含当前令牌并为下一次更新最多保留W-1个历史条目1。

该设计旨在结合潜在推理、硬件协设计和强化学习协设计1，并支持并行编码器工作、序列批处理、内存重用和检查点保存等核心特性1。

Researchers have introduced the Recurrent Looped Transformer (RLT), a novel architecture that combines a causal encoder with a recurrent decoder to achieve unbounded temporal depth that scales with sequence growth 1. The design operates by carrying hidden states and sliding window attention caches between each prompt and response token, enabling sequential processing with expanding computational paths 1.

The RLT architecture employs a configuration of 48 encoder layers and 48 decoder layers, with compatible attention and feed-forward network weights shared across stages 1. After processing t tokens, the recursive path traverses 48t decoder blocks, with each token executing 96 logical blocks in total 1. The system incorporates sliding window attention with width W, which retains the current token and preserves up to W-1 historical entries for subsequent updates 1. This design integrates parallel encoder operations, sequence batching, memory reuse, and checkpoint saving capabilities 1.

The approach aims to combine latent reasoning, hardware co-design, and reinforcement learning co-design strategies 1.