TBNAI
Together Compiler: Continual Compilation for AI Inference
Publication: cs.AI updates on arXiv.org
Author: Reyna Abhyankar, Dylan Lim, Drew Wadsworth, Venkata Hasith Vattikuti, YuTong Julia Zhang, Jung Hwan Heo, Taeksang Kim, Kesavan Ramakrishnan, Evangelene Stanley, Xinyu Fang, Daniel Y. Fu, Simran Arora
Source published: 13/11/2026, 05:00:00 UTC
Briefing created: 13/11/2026, 07:55:24 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 13/11/2026, 07:48:24 UTC.
Briefings
Together AI introduced the Together Compiler (TKCC), a first truly continual agent loop for AI inference optimization. The system addresses throughput-fidelity tradeoffs by decoupling engine launch overheads from forward pass computation and using a cost model to select parallelism configurations. TKCC outperforms hand-written Megakernels by up to 1.7x on Llama 1B and achieves 2.6x higher TPS on frontier models like Kimi K3 and DeepSeek v4.1 Flash, setting the state-of-the-art at batch size 1 with 1111 TPS on M3.1 Flash and 1205 TPS on DeepSeek v4.1 Flash.
Key points
- Together Compiler (TKCC) is the first truly continual agent loop designed for AI inference optimization.
- TKCC outperforms hand-written Megakernels by up to 1.7x on Llama 1B workloads and achieves 2.6x higher TPS on frontier models like Kimi K3 and DeepSeek v4.1 Flash.
Context
AI workloads are rapidly changing with new models and diverse architectures, requiring continual optimization for inference. Existing systems struggle with a tradeoff between measurement throughput and fidelity, often using agents for offline benchmarks or end-to-end optimization, but TKCC solves these issues by decoupling engine launch overheads from forward pass computation.
Limitations
Performance gains are measured on specific hardware platforms and benchmark datasets. Results are based on ablations on agent context management and production deployment insights. The system generalizes beyond frontier language model inference, serving trillions of tokens across the production fleet, but specific production metrics are not detailed in the evidence.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 13/11/2026, 07:55:24 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeig3ktxefvaw3r4j5gezhgx3h5wu6kjt2zrcprccfcba3xz75noe6a