AI

Cancellation Is Not Enough: What Retries Leave Behind in an Overloaded LLM Engine

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 09/11/2026, 05:03:25 UTC.

Briefings

A new arXiv paper investigates whether cancellation alone can restore service in an overloaded LLM engine when retries are enabled. The study uses vLLM with three open models and traces showing client attempts. It found that retries failed to restore service, delivering only 12-38% of calls after the overload, and that vLLM's prompt-token counter missed a third to half of computed tokens. A retry budget restored service by shedding retries but left many calls undelivered. The research concludes that overload controls should be judged by the executions that retries add and the calls that clients receive.

Key points

• vLLM retries that started prefill less than half a second before timing out failed to restore service in any test run.
• Retry budgets restored service by shedding retries but left many of the overload's calls undelivered.

Context

This research tests cancellation as a remedy for transient overload in LLM engines using vLLM with three open models and scheduler traces. It reveals that retries often start too early, causing significant token loss and failure to restore service.

Limitations

The study uses vLLM with three open models and scheduler traces of one model. It does not provide additional background on why this matters beyond the technical findings. The evidence is limited to the specific experiments conducted on vLLM and the three open models tested.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 09/11/2026, 07:26:04 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeiginjgdwphyvzutandvhix56ktxe2v7dk73uo6gfa7fgdm7am5p5u