TBNAI
VagueBench: Benchmarking the Robustness of Embodied Agents on Ambient Context Illusions
Publication: cs.AI updates on arXiv.org
Author: Yang Qu, Qiang Fu, Zonghao Ying, Jiayi Zhao, Yaoyuan Zhang, Jinyang Guo, Jian Yang, Aishan Liu, Yingxun Wang, Xianglong Liu
Source published: 04/11/2026, 05:00:00 UTC
Briefing created: 04/11/2026, 07:00:51 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 04/11/2026, 05:18:24 UTC.
Briefings
A new benchmark called VagueBench was created to test how well embodied agents handle instructions without extra context. Researchers found that agents often fail when essential details are missing, even if the core task remains the same. This issue affects both simulated and real-world robots, potentially causing safety risks.
Key points
- VagueBench is a benchmark built on the AI2-THOR interactive simulation environment designed to evaluate context dependency in embodied agents.
- The benchmark contains 1,303 instruction pairs across diverse indoor environments, constructed using a human-agent co-generation framework.
Context
Embodied agents are increasingly deployed in open-world environments, but current robustness evaluations are biased toward hazardous instructions. VagueBench addresses this by testing agents under semantically sufficient but contextually minimal instructions, which are common in real-world human communication.
Limitations
The study evaluates only 6 state-of-the-art embodied agents on VagueBench. Results show significant degradation in task performance and executability, but the impact on real-world robotic systems remains unquantified.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 04/11/2026, 07:00:51 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeihpbupopzs2dgvmpoihriywx737zvzhbgc5cdfgz5acjdh2vdkmhm