AI
BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection
Publication: cs.AI updates on arXiv.org
Author: John Wu, Rushi Qiang, Yuchen Zhuang, Guowang Li, Girish Narayanswamy, Max Xu, Sam Schmidgall, Mark Malhotra, Shwetak Patel, Jimeng Sun, Daniel McDuff, Bo Dai, Xin Liu
Source published: 20/11/2026, 05:00:00 UTC
Briefing created: 20/11/2026, 05:47:17 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 20/11/2026, 05:03:24 UTC.
Briefings
Researchers introduced BuggyMLE, a framework for scaling machine learning engineering difficulty using the Code Complexity Index (CCI). By injecting controlled mutations into 222 source repositories, they synthesized over 6,600 bug-injected variants. This approach significantly improves AI agents' ability to solve complex problems, as demonstrated by a 95% to 9% drop in solve rates when scaling from easy to hard tiers across 14 models.
Key points
- The Code Complexity Index (CCI) is a metric based on ten structural features including repository scale, dependency depth, and patch size.
- BuggyMLE-Bench-Lite is the first difficulty-graded, repository-level variant of MLE-Bench Lite, showing a 95% to 9% drop in solve rates as difficulty increases.
Context
This research addresses the limitation of existing MLE benchmarks that restrict agents to single-file edits without full repository or shell access. The study provides a scalable framework to test AI agents on realistic, multi-file engineering tasks.
Limitations
The study synthesizes data from 222 source repositories and evaluates performance across 14 models. Results are based on a snapshot of the research at publication time, and the CCI metric relies on structural features that may not fully capture all engineering complexities.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 20/11/2026, 05:47:17 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeichb4yeo63gqolw6i2l7itvhwt4zsa3cgimosgcuc7vxbu5ftxr3a