AI
How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 23/09/2026, 20:00:00 UTC
Briefing created: 29/09/2026, 13:26:23 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 29/09/2026, 06:18:24 UTC.
Briefings
A self-audit of LLM evaluation conclusions reveals that ranking tables based on small prompt sets are highly unstable. Using LLM-inferred prompt structure, the study found that identical model calls do not reliably recover identical structures, with mean Jaccard similarity ranging from 0.39 to 0.96. Under cluster bootstrap auditing, only the bottom two models held their rank in 99% of replicates, while the top two held it in only 68% each. The study concludes that evaluation rankings cannot be trusted as accurate predictors of model performance and recommends reporting stability metrics alongside rankings.
Key points
- Mean Jaccard similarity between identical model calls ranged from 0.39 to 0.96, indicating high instability in LLM-inferred prompt structures.
- Under cluster bootstrap auditing, the top two models held their rank in only 68% of replicates, while the bottom two held it in 99% of replicates.
Context and limitations
The study covers eight open model variants across five families with 8B to 675B parameters. Data includes 293 raw intermediate representations but excludes caching. Four endpoints were withdrawn within ten weeks, rendering the study as specified unrunnable. Results are based on a single snapshot and preprint status.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 29/09/2026, 13:26:23 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeigjoqakjjutrjdaknnlyrklucnuof2m44lmqyc6ro4ttr5khblciy