AI

The Extender: A Log-Structured Transformer

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 05/10/2026, 20:18:24 UTC.

Briefings

The Extender is a log-structured Transformer variant that reduces persistent attention memory by 104 times compared to Multi-Head Attention (MHA) while matching Transformer accuracy on short-context tasks at 199M-924M parameters and exceeding it on long-context workloads at 924M parameters. The architecture introduces a concatenation channel x alongside residual h, allowing kv projections to access the full input history rather than just the residual update.

Key points

• The Extender reduces persistent attention memory footprint from 2Ld_model to sum|beta_ell|, achieving a 104x reduction in memory for a 1664-wide, 924M model.
• With beta_ell=32, the Extender matches Transformer accuracy on short-context (CORE) tasks at 199M-924M parameters and exceeds Transformer accuracy on long-context (RULER) workloads at 924M parameters.

Context and limitations

Results are limited to short-context and long-context tasks; specific memory savings grow with model width; accuracy comparisons are based on available benchmark datasets.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 15/10/2026, 00:34:29 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeifh75hcf2jc3h554tdxsmlssqaow7gxkpil54ztfhyeamek5cfjja