AI

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 08/10/2026, 03:18:24 UTC.

Briefings

STEPQuant is a post-training quantization framework for Delta-rule recurrent states in LLMs, addressing memory bottlenecks caused by persistent states. It allocates precision based on error magnitude and memory lifetime, fitting key-row and value-column scales. Experiments on Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct show STEPQuant matches FP32 accuracy at 6-bit while outperforming uniform INT8 in 4-bit configurations, achieving up to 68.7% memory reduction.

Key points

• STEPQuant allocates precision based on error magnitude and memory lifetime for Delta-rule recurrent states.
• 6-bit STEPQuant achieves over 5x recurrent-state compression and reduces total serving memory by up to 68.7%.

Context

Delta-rule recurrent states in LLMs can become memory bottlenecks under concurrent serving. Errors in long-lived memory persist across decoding steps, while errors in different key rows affect outputs differently. STEPQuant addresses these by jointly fitting key-row and value-column scales based on state distributions and key-row impact on output error.

Limitations

Evidence covers only Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct models across long- and short-generation benchmarks. Results are from a single snapshot on 2026-09-28. Code availability is confirmed but specific implementation details beyond GPU kernel optimization are not detailed.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 04/11/2026, 16:44:20 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeiffnzc6rz26emwqze2moqm3tungcmshcij3jbwzwn4q5wuort2gpa