AI
SpecRelay: Amortize or Avoid the Round Trip in Edge-Cloud Speculative Decoding
Publication: cs.AI updates on arXiv.org
Author: Jai Jain, Chihyo Ahn, Yong Liu, Charles Chen, Duc N. M Hoang, Ajay Kumar Jaiswal, Fatih Ilhan, Han-Byul Kim, Lauren Hannah, Richa Dixit, Minsik Cho, Yingyan Celine Lin, Hyesoon Kim
Source published: 17/11/2026, 05:00:00 UTC
Briefing created: 17/11/2026, 08:53:17 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 17/11/2026, 08:33:25 UTC.
Briefings
SpecRelay is an edge-cloud LLM inference system designed to reduce latency in speculative decoding by amortizing round trips when cloud verification is necessary and avoiding them when local reasoning suffices. The system uses an RTT-conditioned router to select among three on-device modes, achieving up to 50% higher throughput than a synchronous baseline across benchmarks like SPROUT and OlympiadBench. Key metrics include a 1.95x throughput improvement on SPROUT and 1.55x on OlympiadBench at matched accuracy and 250ms added latency, while sending only 47% and 39% of queries to the cloud respectively.
Key points
- SpecRelay amortizes round trips in edge-cloud speculative decoding by selecting between direct generation, Chain-of-Draft, and full reasoning modes based on RTT conditions.
- The system achieves up to 50% higher throughput than a synchronous baseline across multiple benchmarks while maintaining accuracy and minimizing cloud query volume.
Context
Edge-cloud speculative decoding moves drafting to the edge while cloud models verify in parallel, but synchronous verification waits for full round trips, limiting throughput. SpecRelay addresses this by amortizing round trips when cloud verification is necessary and avoiding them when local reasoning suffices, using an RTT-conditioned router to optimize inference without added latency.
Limitations
Performance gains are measured against a reproduced synchronous baseline across multiple benchmarks. The system operates under specific accuracy and latency constraints; exact throughput improvements vary by benchmark and model configuration. No additional background context is supplied beyond the provided evidence.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 17/11/2026, 08:53:17 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeigfe46yodpdsvycf5cidoupjjpeptjko2potx45tjrtjvfmytjwje