AI
Video-as-Experience: Learning Computer Use via Audio-Visual Understanding
Publication: cs.AI updates on arXiv.org
Author: Guangzhi Sun, Zhonghao Zhai, Yixuan Li, Chao Zhang
Source published: 17/11/2026, 05:00:00 UTC
Briefing created: 17/11/2026, 10:48:31 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 17/11/2026, 08:33:24 UTC.
Briefings
A new framework called VEX enables computer-use agents to learn from video demonstrations by directly utilizing audio-visual information rather than converting videos into screenshots. The system uses a watch-act-rewatch paradigm where agents execute tasks and verify progress by rewatching their own execution. Instantiated with a 9B-scale model on OSWorld, the approach achieved a 22% relative increase in task success compared to its Qwen3.5 backbone, while also improving guidance quality for other models.
Key points
- The VEX framework uses a watch-act-rewatch paradigm to learn from video demonstrations by executing tasks and verifying progress through rewatching.
- Video-SALMONN-VEX achieved a 22% relative increase in task success on OSWorld compared to its Qwen3.5 backbone.
Context
This research addresses the limitation that existing approaches to learning from video demonstrations typically convert videos into screenshot-based action trajectories, rather than directly leveraging their rich audio-visual information. The findings suggest audio-visual understanding can serve as a unified interface for procedural learning and self-verification in computer-use agents.
Limitations
The study was conducted on OSWorld and a complementary subset of CUA-Gym using video-SALMONN-VEX and video-SALMONN 2 architectures. Results are based on preprint arXiv:2611.18747v1 and may not yet reflect final experimental conditions or long-term performance stability.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 17/11/2026, 10:48:31 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeicg4r6cofabguey6wok2miuf5en6llasctvuyherfkvhefnryl4te