AI
Pretraining Transformers with Quantized Softmax in Attention
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 26/09/2026, 20:00:00 UTC
Briefing created: 30/09/2026, 05:25:20 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 30/09/2026, 05:18:24 UTC.
Briefings
This study investigates how quantized attention matrices affect gradient computation during transformer pretraining. By varying K-interval attention calibration methods and surrogate placements, researchers found that hard rounding at K=4 creates a significant loss gap compared to softmax, while fixed-window calibration with post-normalization surrogates minimizes this gap. The experiment utilized 124M parameters and 2.5B tokens, showing that specific calibration choices can reduce validation loss by up to 0.019 nats.
Key points
- Hard rounding at K=4 with min-max calibration and a pre-normalization surrogate creates a large loss gap compared to softmax.
- Fixed-window calibration with a post-normalization surrogate yields a validation loss gap of +0.019 nats relative to softmax at K=4.
Context and limitations
Results are limited to pretraining experiments with matched model, data, and optimizer configurations; findings may not generalize to inference or other training stages.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 30/09/2026, 05:25:20 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeic6wgo6mycp6zsytqodmy732euz2s4qakhzdnazmhm23y3azrievu