AI
Decode-Latency Feedback Prefill: A Model-Free Controller and Its Generalization Limits
Publication: cs.AI updates on arXiv.org
Author: Gaurav Agarwal, Ashish Garg, Isha Singhal
Source published: 01/10/2026, 04:00:00 UTC
Briefing created: 01/10/2026, 04:05:55 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 01/10/2026, 04:03:25 UTC.
Briefings
This study introduces Decode-Latency Feedback Prefill (DLFP), a model-free controller for vLLM that optimizes prefill scheduling by using observed latency intervals as feedback. Evaluated on Qwen3-0.6B with Poisson arrivals, DLFP reduced P99 inter-token latency by 24.8% to 28.2% while maintaining exact output agreement and SLO compliance. However, the mechanism failed to generalize to larger models (Qwen3-8B, Qwen3-32B) or multi-GPU setups, revealing a fundamental limitation in current scheduling proxies.
Key points
- DLFP uses observed latency intervals as feedback to resize prefill chunks, reducing P99 inter-token latency by 24.8% to 28.2% on Qwen3-0.6B.
- The controller maintains exact output agreement and SLO compliance while increasing mean P99 time to first token by 34.8%.
Context and limitations
The model-free scheduling mechanism fails to generalize to larger models (Qwen3-8B, Qwen3-32B) or multi-GPU configurations, indicating a fundamental limitation in current scheduling proxies.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 01/10/2026, 04:05:55 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeigsbqbaqhd7ivtx5x7avsstf4zxkvthg3bambbppr2wddvptihcl4