AI

Recurrent Looped Transformer

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 08/10/2026, 02:18:25 UTC.

Briefings

The Recurrent Looped Transformer (RLT) splits Transformer layers between a parallel causal encoder and a recurrent decoder, allowing computation paths to grow with sequence length at fixed per-token cost. On six algorithmic tasks, RLT generalized parity to 256 bits with 100% accuracy in every seed, whereas the baseline Transformer remained at chance. On swap-based S_5 permutation tracking, RLT reached 97% final-state accuracy versus under 1% for the Transformer, and accuracy increased with decoder depth. On modular arithmetic beyond training lengths, RLT reached up to 93% versus 33% for the Transformer. Ablations show these gains depend on feedback, with removing it dropping parity and swap-based S_5 to chance at every split.

Key points

• RLT splits Transformer layers between a parallel causal encoder and a recurrent decoder to grow computation paths with sequence length at fixed per-token cost.
• RLT generalized parity to 256 bits with 100% accuracy in every seed, while the baseline Transformer remained at chance on the same tasks.

Context

State tracking requires an update at every input, but Transformers apply a fixed depth regardless of sequence length. This study introduces RLT to solve that limitation, offering a new architecture for state tracking tasks.

Limitations

Results are limited to six algorithmic tasks and three seeds. Performance on modular arithmetic beyond training lengths is reported as up to 93% for RLT versus 33% for the Transformer, indicating potential variability in generalization beyond the training distribution.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 04/11/2026, 08:51:43 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeic66rtkyejq3w6rtzoyx4g2awl7j62gnftrlcadft4wzgycqy2yai