AI
Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 30/09/2026, 20:00:00 UTC
Briefing created: 03/10/2026, 08:09:46 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 02/10/2026, 17:18:24 UTC.
Briefings
A 661.6M parameter Spanish security model and a 1,109M web-heavy model share identical tool-use scores despite the latter failing to emit valid tool calls on 6/6 training examples. The 1B model's failure was traced to a missing prior erased by its web-heavy training, requiring only 3.3 GPU-hours of targeted SFT to fix. On 269 corpus rows, valid emission rose from 0.100 to 0.959, and performance improved from 0.428 to 0.536 on unseen prompts (p=0.004). The repair preserved the trigger token's embedding while reducing over-triggering from 0.09 to 0.17.
Key points
- A 661.6M parameter Spanish security model and a 1,109M web-heavy model share identical tool-use scores despite the latter failing to emit valid tool calls on 6/6 training examples.
- The 1B model's failure was traced to a missing prior erased by its web-heavy training, requiring only 3.3 GPU-hours of targeted SFT to fix.
Context and limitations
Coverage is limited to a matched-architecture pair of Spanish security language models; results may not generalize to other architectures or domains.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 03/10/2026, 08:09:46 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeidsz2mnvprzb6d6oditd6q7kphdtbk5ll5tlgt5s7igyaarhbin74