AI
Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 30/09/2026, 20:00:00 UTC
Briefing created: 05/10/2026, 04:54:49 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 05/10/2026, 04:18:24 UTC.
Briefings
Looped Transformers introduce recurrent depth as a new scaling axis for LLMs by repeatedly applying shared Transformer blocks to increase effective depth without increasing parameter count. However, benefits for large MoE LLMs under FLOPs-matched comparisons remain unclear due to diminishing gains that can turn into degradation. The study identifies two main obstacles: looping amplifies the curse of depth by increasing hidden-state variance, and looped MoE suffers from expert selection collapse where routers select the same experts across loops. The proposed LOOM method stabilizes recurrence by scaling residual updates and re-injecting input embeddings at every loop, achieving stable scaling to 9-12 loops across 100M-1.7B models.
Key points
- Looped Transformers increase effective depth by repeatedly applying shared Transformer blocks without increasing parameter count, but gains diminish quickly and can turn into degradation under FLOPs-matched comparisons.
- The LOOM method stabilizes recurrence by scaling residual updates to bound variance growth and re-injecting the input embedding at every loop, achieving stable scaling to 9-12 loops across 100M-1.7B models.
Context and limitations
Results are limited to 100M-1.7B models under near-iso-FLOP conditions; FLOPs-matched comparisons show diminishing returns beyond 5 loops for the 700M model and 9 loops for the 1.7B model.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 05/10/2026, 04:54:49 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeiawrbauhooj6lbueqv67mpvonzq5uado67chp3fr6eix55bxuutbq