AI
Fast Models, Slow Evidence: A Paired and Self-Audited Evaluation of System-1 Decision Models for LLM Agent Harnesses
Publication: cs.AI updates on arXiv.org
Author: Jiawei Li
Source published: 05/10/2026, 04:00:00 UTC
Briefing created: 05/10/2026, 04:06:45 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 05/10/2026, 04:03:24 UTC.
Briefings
A paired evaluation of Laya and Jev System-1 models on 11 agent decision points revealed Jev is significantly more accurate on 9 of 11 points, while Laya's performance degrades sharply when tool options are reordered. The study covers 7,283 base cases plus 6,640 robustness variants using byte-identical inputs across cross-hardware and cross-day checks. Key findings include a 30% answer change rate for Laya due to option ordering and a 58% gate accuracy versus 98% reported end-to-end quality. Uncertainty remains regarding the impact of pre-screen cost reporting and channel effects on injection false positives.
Key points
- Jev System-1 model achieves significantly higher accuracy on 9 of 11 agent decision points compared to Laya, with gains ranging from +10.8 to +46.0 percentage points.
- Laya's performance degrades sharply when tool options are reordered, changing 30% of its answers compared to Jev's 98% degradation rate on similar items.
Context and limitations
The evaluation covers 7,283 base cases and 6,640 robustness variants using byte-identical inputs across cross-hardware and cross-day checks. Some analysis errors and design confounds were identified, including omitted pre-screen cost reporting and gate accuracy misreporting, which may affect headline claims.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 05/10/2026, 04:06:45 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeiaclqnqiotg2tqzhwpittsye2bp6met7hf3sdov2s3pm57cmenhxy