AI
CoWindow Attention: Full Causal Coverage Is a Collective Property
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 25/09/2026, 20:00:00 UTC
Briefing created: 29/09/2026, 03:27:17 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 29/09/2026, 03:18:24 UTC.
Briefings
CoWindow Attention (CoWA) addresses full causal coverage in transformer models by distributing access across KV heads rather than duplicating it. The study introduces a structured architecture where heads share near-diagonal windows and use complementary long-range windows to partition history. This design eliminates the need for learned routers while maintaining performance comparable to FullAttn across training and inference. Key findings include a 7.4x reduction in forward latency during training and 3.0x reduction in decoding latency, with 14B and 32B parameter models achieving similar knowledge scores to FullAttn. The research concludes that full causal coverage is a collective property of the head ensemble rather than a duplicated property of every individual head.
Key points
- CoWindow Attention (CoWA) distributes causal history access across KV heads using near-diagonal and complementary long-range windows, eliminating the need for learned routers.
- CoWA achieves 89.73% accuracy on a window-matched ablation at 8K tokens, outperforming FullAttn's 89.97% while reducing training FLOPs and decoding latency by 3.0x.
Context and limitations
The study covers controlled associative-recall comparisons and attention-operator benchmarks up to 128K tokens. Results are limited to 0.6B to 14B parameter scaling and do not include long-context retrieval scores beyond the 32B model tested.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 29/09/2026, 03:27:17 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeialet7vl5rb5kmv45yegt66szb2q4rqb74z4woaohsudzbikfekiu