Tech
The most misunderstood graph in AI just got a major update
Publication: MIT Technology Review
Author: Grace Huckins
Source published: 20/11/2026, 18:15:15 UTC
Briefing created: 20/11/2026, 18:22:40 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 20/11/2026, 18:20:24 UTC.
Briefings
A METR report reveals AI models are improving exponentially on verifiable coding tasks but lagging significantly on judgment-heavy open-ended tasks. While frontier models doubled the length of verifiable tasks every four months, they failed to design useful AI evaluations or score philosophical arguments, indicating a bottleneck in research automation. The report highlights a gap between current performance and future potential on complex, unmeasurable tasks.
Key points
- METR's time horizon plot shows AI models doubling the length of verifiable tasks every four months, indicating rapid progress on coding and math.
- Models consistently fail open-ended judgment tasks like designing AI evaluations or scoring philosophical arguments, suggesting a bottleneck in research automation.
Context
METR evaluated AI models by having them complete tasks scored by human testers, measuring success by task length. This update adds judgment-heavy tasks to the original suite, revealing that while AI excels at verifiable tasks, it struggles with open-ended research automation tasks.
Limitations
The report's original suite lacked judgment-heavy tasks, making it difficult to predict future progress. Experts note uncertainty about extrapolating current performance to open-ended tasks, with some models scoring 'borderline above the bar' on evaluation design.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 20/11/2026, 18:22:40 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeidelu5eynautvahknq6fjjp3pxk5txit4ejws2qekm4mmvl7x3wdy