AI

Towards Looped Models Done Right, Part II: Rethinking at Fixed Points

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 06/10/2026, 04:18:25 UTC.

Briefings

This article presents findings from a 2026 study on looped language models, proposing that fixed-depth training enables truncated backpropagation and orthogonal input injection. The research demonstrates that these techniques reduce training costs by 2x during RL updates and improve inference speed by 1.79x for distilled students. The study compares learned priors against existing schemes, showing significant perplexity improvements across scales from 100M to 1.6B parameters.

Key points

• Fixed-depth training enables truncated backpropagation, reducing training costs by 2x during reinforcement learning updates compared to backpropagating through replayed trajectories.
• Orthogonal input injection removes state component amplification, allowing fixed-depth training to match the downstream average of full KV cache configurations at 1.6B parameters.

Context

Looped language models introduce significant costs in training, decoding, and reinforcement learning. This study addresses these issues by rethinking fixed points, enabling truncated backpropagation and orthogonal injection to improve efficiency and accuracy.

Limitations

Evidence covers only the 100M to 1.6B parameter range. Results are based on a single 2026 study and may not generalize to other model scales or architectures.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 16/10/2026, 23:59:53 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeifaniypfwnojf2tbdc4qme2xrxacdbxv7gl5t4sfav3vk5wmujvyu