AI
Measuring the Microtask Eligibility Gap: When Is an Off-the-Shelf SLM Enough for an Agent Harness?
Publication: cs.AI updates on arXiv.org
Author: Jundong Hu, Shekar Ramachandran
Source published: 02/10/2026, 04:00:00 UTC
Briefing created: 02/10/2026, 04:12:21 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 02/10/2026, 04:03:24 UTC.
Briefings
This article investigates whether off-the-shelf small language models (SLMs) can reliably execute microtasks defined by a large language model (LLM) planner. Using a benchmark of four microtasks with fixed prompts and automatic metrics, the study found that zero out of sixteen configurations met practitioner-defined eligibility thresholds anchored to a non-LLM baseline. The research identifies a capability deficit in some models versus failures solvable by adjusting decoding thresholds, and notes that quantization to 4-bit does not resolve the gap, which instead tracks model size rather than precision.
Key points
- A benchmark of four microtasks evaluated 16 configurations of Qwen3 models (0.6B, 1.7B, 4B, 8B) at their best performance, revealing that none of the configurations passed the eligibility threshold.
- The eligibility gap is driven primarily by model size rather than precision, as quantization to 4-bit failed to resolve the gap and replicated the ineligibility seen in Llama-3.x models.
Context and limitations
The study covers only four microtasks with fixed prompts and automatic metrics, limiting the generalizability of findings to other task types or prompt variations.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 02/10/2026, 04:12:21 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeif4b7v7hamegp2pz4scxk2r5ffegt7i2sdm5nrb7bj44ak3moyszu