TBNAI
Recurrent Looped Transformer
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 05/10/2026, 20:00:00 UTC
Briefing created: 04/11/2026, 08:51:43 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 08/10/2026, 02:18:25 UTC.
Briefings
The Recurrent Looped Transformer (RLT) splits Transformer layers between a parallel causal encoder and a recurrent decoder, allowing computation paths to grow with sequence length at fixed per-token cost. On six algorithmic tasks, RLT generalized parity to 256 bits with 100% accuracy in every seed, whereas the baseline Transformer remained at chance. On swap-based S_5 permutation tracking, RLT reached 97% final-state accuracy versus under 1% for the Transformer, and accuracy increased with decoder depth. On modular arithmetic beyond training lengths, RLT reached up to 93% versus 33% for the Transformer. Ablations show these gains depend on feedback, with removing it dropping parity and swap-based S_5 to chance at every split.
Key points
- RLT splits Transformer layers between a parallel causal encoder and a recurrent decoder to grow computation paths with sequence length at fixed per-token cost.
- RLT generalized parity to 256 bits with 100% accuracy in every seed, while the baseline Transformer remained at chance on the same tasks.
Context
State tracking requires an update at every input, but Transformers apply a fixed depth regardless of sequence length. This study introduces RLT to solve that limitation, offering a new architecture for state tracking tasks.
Limitations
Results are limited to six algorithmic tasks and three seeds. Performance on modular arithmetic beyond training lengths is reported as up to 93% for RLT versus 33% for the Transformer, indicating potential variability in generalization beyond the training distribution.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 04/11/2026, 08:51:43 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeic66rtkyejq3w6rtzoyx4g2awl7j62gnftrlcadft4wzgycqy2yai