AI
It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Them
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 28/09/2026, 20:00:00 UTC
Briefing created: 01/10/2026, 11:00:20 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 01/10/2026, 10:18:24 UTC.
Briefings
Vision-language models are increasingly replacing human annotators, necessitating robust substitutability tests. Researchers introduced MIST, a stress test using 200 English sentences where image content (aligned, misleading, or absent) should not alter the label. Across thirteen VLM judges, aligned images shifted 20.5% of labels while misleading images shifted 19.4%, significantly higher than the 11.6% baseline from removing image instructions. Despite these shifts, only 37% of differing labels moved toward the image's sense, and human annotator agreement remained unchanged regardless of image presence. The effect is present in all judges but smaller in those passing an alt-test, suggesting the verdict describes a configuration rather than a model capability.
Key points
- MIST introduces 200 English sentences where image content (aligned, misleading, or absent) should not alter the label, testing if VLMs are sensitive to incidental evaluation conditions.
- Across thirteen VLM judges, aligned images shifted 20.5% of labels while misleading images shifted 19.4%, significantly higher than the 11.6% baseline from removing image instructions, indicating image presence destabilizes judgments.
Context and limitations
Evidence covers thirteen VLM judges across thirteen trials; results may vary with different models or datasets. Preprint status and uncertainty remain due to limited sample size and potential for model-specific biases.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 01/10/2026, 11:00:20 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeibriodont64blmkgodbmbnkgtqrfvewke5dil2oi76w4wrk7kqhgm