AI
More Programs or More Rolls? Separating Coverage from Specialization in LLM Harnesses
Publication: cs.AI updates on arXiv.org
Author: Ziyang Xu, Haitian Zhong, Hao Zhou, Hao Qin, Chenhan Jin, Te Qi, Shengze Xu, Tieyong Zeng
Source published: 30/09/2026, 04:00:00 UTC
Briefing created: 30/09/2026, 04:05:13 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 30/09/2026, 04:03:24 UTC.
Briefings
This study evaluates LLM harnesses by comparing eight generated programs against nine byte-identical baselines across 386 MATH-500 tasks. Key findings show generated harnesses achieve 2.16 percentage points of repeat-averaged oracle headroom, yet most losses persist across all three executions while gains occur on only one task. Both populations reach 98.70% oracle coverage at 27 harness executions, but stable complementarity remains unresolved. The research concludes that coverage and repeatability alone cannot justify claims of useful specialization.
Key points
- Generated LLM harnesses achieve 2.16 percentage points of repeat-averaged oracle headroom on 386 MATH-500 tasks.
- Most performance losses across generated harnesses persist across all three executions, whereas gains occur on only one task.
Context and limitations
Evaluation uses three executions per harness member; coverage is limited to 98.70% at 27 harness executions; preprint status and uncertainty remain.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 30/09/2026, 04:05:13 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeico5xa7eha5kvr2jhuou5myanggf27x6vb54bbprb74cap4645u5e