AI
Does the Ranking Hold? A Practitioner-Facing Audit of Hallucination Detectors on Small, Quantized, On-Device LLMs
Publication: cs.AI updates on arXiv.org
Author: Shubhra Mittal, Gaurav Naresh Mittal
Source published: 16/11/2026, 05:00:00 UTC
Briefing created: 16/11/2026, 05:45:46 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 16/11/2026, 05:03:24 UTC.
Briefings
A one-GPU audit on four small on-device LLMs (Llama-3.2-3B, Qwen2.5-3B, Phi-3-mini, Gemma-2-2B) evaluated on TriviaQA found that hallucination detectors rank poorly across models, with Spearman rho at -0.80 and Kendall tau-b at -0.667. The standard TruthfulQA labeler yields near-chance AUROC (0.46-0.59), indicating the failure lies in the detector's labeler pipeline rather than the detectors themselves. This audit highlights a critical fairness-of-error-rates allocation problem for resource-constrained on-device deployments.
Key points
- A one-GPU audit on four small on-device LLMs (Llama-3.2-3B, Qwen2.5-3B, Phi-3-mini, Gemma-2-2B) evaluated on TriviaQA found that hallucination detectors rank poorly across models, with Spearman rho at -0.80 and Kendall tau-b at -0.667.
- The standard TruthfulQA labeler yields near-chance AUROC (0.46-0.59), indicating the failure lies in the detector's labeler pipeline rather than the detectors themselves.
Context
This audit addresses the gap between large model research benchmarks and small, quantized on-device models. It provides a reproducible one-GPU evaluation of four detectors on two small open instruction-tuned models, highlighting the lack of reliable detection methods for resource-constrained hardware.
Limitations
The audit is limited to a single Colab T4 GPU and closed-book TriviaQA evaluation. Results are based on a single preprint version (arXiv:2611.17339v1) and do not account for multi-GPU training or larger model variants.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 16/11/2026, 05:45:46 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeifdgb46vop46nrqukcer65y3ppesmuw2l3l6wa5cs2ncnejukiiz4