AI

How to Loop MoE: Flatten the Experts, Untie the Attention

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 06/10/2026, 11:18:24 UTC.

Briefings

Foil bridges looped Transformers and sparse MoE by flattening expert layers and untying attention parameters. Experiments show Foil outperforms unflattened loops at 20B tokens, with loss improving monotonically up to 0.012 nat below baseline at 100B tokens. Routing becomes more balanced and confident when attention is untied, while widening expert layers amplifies looping benefits.

Key points

• Foil flattens MoE experts by halving layer depth and doubling expert count per layer while doubling passes, enabling routing from a larger pool.
• Untying attention parameters per pass improves routing confidence and balance compared to sharing parameters across passes.

Context

Looped Transformers reuse fixed-size blocks to utilize parameters more fully, while sparse MoE models activate only a few experts per token. Foil bridges these designs to improve MoE efficiency.

Limitations

Results are from ablation studies and experiments at 20B and 100B tokens; specific ablation details and causal mechanisms remain unverified.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 20/10/2026, 04:34:33 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeiczf32czk2qjs7kxql3eearciu7r3mfqi5krmfr2p2cmaty5dbity