AI

HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 07/10/2026, 07:18:24 UTC.

Briefings

Hybrid Linear Attention (HLA) improves long-context modeling in Gated DeltaNet by using query-dependent chunk-level attention to adapt historical memory access. Evaluated on Qwen3.5 models from 0.8B to 9B, HLA consistently outperforms native GDN and fixed chunk mixing, achieving gains of up to 5.57 percentage points on LongBench-V2 and 3.97 points on RULER. In a from-scratch 1.3B setting, HLA boosted RULER performance from 4K to 32K context, increasing gains from 0.83 to 4.22 points. The method uses compact per-chunk affine summaries to enable efficient long-context autoregressive decoding.

Key points

• HLA improves LongBench-V2 scores by up to 5.57 percentage points on Qwen3.5 models ranging from 0.8B to 9B parameters.
• In a from-scratch 1.3B setting, HLA increases RULER performance from 4K to 32K context, raising gains from 0.83 to 4.22 points.

Context

This research addresses limitations in existing chunk-based extensions where learned mixing coefficients may remain fixed, preventing adaptation to input content. The findings are significant for developing more efficient long-context autoregressive decoding mechanisms using compact per-chunk affine summaries.

Limitations

Results are based on preprint data from 2026; specific training details and model configurations for the 1.3B setting are not fully detailed in the provided evidence.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 25/10/2026, 19:07:00 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeiepdkvtajqvnmfng4yezcvrf556mtex26jobc4zjsfgv3ynk4jywy