AI
MKA: Evaluating Multimodal Knowledge Agents on end-to-end professional tasks
Publication: cs.AI updates on arXiv.org
Author: Anima Singh, Suhas Yogin, Weicheng Kuo, Ivor Rendulic, Ivan Jurin, Antoine Yang, Oran Kelly, Rohan Doshi, Hammad Ayyubi, Ioana Bica, Sergi Caelles, Matthew Conniff, Dan Graur, Anastasija Ili\'c, Ahmet
Source published: 20/11/2026, 05:00:00 UTC
Briefing created: 20/11/2026, 05:48:19 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 20/11/2026, 05:03:24 UTC.
Briefings
The MKA benchmark evaluates autonomous agents' ability to perform end-to-end professional tasks using visual data like CAD blueprints and schematics. Evaluated against expert-weighted rubrics, Claude Fable 5.2 and Gemini 4 Carbon achieved the highest scores, with Fable 5.2 scoring 56.6% Climb-WSR and 73.3% Overall WSR, while Carbon used 5.7 times fewer tokens. The study found substantial headroom for multimodal agents as even top models scored below 57% on Climb-WSR.
Key points
- The MKA benchmark measures autonomous agents' performance on end-to-end professional tasks requiring visual understanding of dense files like CAD blueprints and schematics.
- Claude Fable 5.2 and Gemini 4 Carbon achieved the highest scores in the benchmark, with Fable 5.2 scoring 56.6% Climb-WSR and 73.3% Overall WSR.
Context
This research addresses a gap where existing benchmarks leave multimodal knowledge work largely untested, as removing image-viewing tools leaves Gemini 4 Carbon's score on GDPval unchanged. The MKA benchmark contains 200 tasks across 10 economic sectors and uses a sandboxed agentic autorater to grade deliverables against expert-weighted rubrics.
Limitations
The study relies on a single preprint (arXiv:2611.25228v1) with no peer-reviewed validation. Results are based on a limited set of evaluated models and may not reflect real-world deployment conditions. Expert-graded study results show 96.2% agreement with human domain experts, but the specific methodology and sample size are not fully detailed in the provided text.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 20/11/2026, 05:48:19 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeih6c724w4s2esynhez3cii5yryeqy276vgptzkmj5qazpu7v4mhb4