AI
WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixing
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 26/09/2026, 20:00:00 UTC
Briefing created: 30/09/2026, 04:59:28 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 29/09/2026, 23:18:24 UTC.
Briefings
WhiteMatter enables all-to-all cross-layer connections in Transformers by allowing every layer to access representations from any depth, reducing cache size while maintaining performance. The method uses a learned mixer to select optimal depths and combines them into shared KV channels. Experiments show WhiteMatter with a full-size cache performs comparably to standard Transformers with 50% more layers, and with half the cache, it outperforms matched models at 1.3B parameters. Cyclic iteration was developed to address cross-layer dependencies, achieving 12.5x faster convergence than standard Jacobi iteration on autoregressive models.
Key points
- WhiteMatter introduces all-to-all cross-layer connections where every layer can access representations from any depth, using a learned mixer to select optimal depths and combine them into shared KV channels.
- WhiteMatter with a full-size cache performs comparably to standard Transformers with 50% more layers, and with half the cache, it outperforms matched models at 1.3B parameters.
Context and limitations
Results are based on a reference model trained with exact autoregressive execution; cross-layer connections introduce dependencies that slow training and prompt processing, requiring cyclic iteration to mitigate.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 30/09/2026, 04:59:28 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeigu3h426d4wt3ougpducqraccb3z6nje4vpyrk2ljudqjcgy6qjoq