AI

CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 07/10/2026, 17:18:24 UTC.

Briefings

This briefing reports on CheckerBench, a benchmark introducing 300 tasks from 297 CVEs to test if coding agents can synthesize static-analysis checkers. The study uses CheckerLab to evaluate diagnostic accuracy, patch localization, and false positives across 21 model configurations. Results indicate that while some agents achieve 45.33% Pass@1, reliable, reusable checker development remains challenging for current models.

Key points

• CheckerBench introduces 300 tasks derived from 297 CVEs across 167 repositories and 85 CWEs.
• The benchmark evaluates agents using CheckerLab to measure vulnerable-fixed diagnostic contrast and tool use.

Context

Existing coding-agent benchmarks focus on patch generation or vulnerability detection, rarely assessing whether an agent can develop a working checker from start to finish. This study addresses that gap by providing a comprehensive executable benchmark for static-analysis checker synthesis.

Limitations

Results are based on 21 model-harness configurations with three independent repeats per configuration. Mean Pass@1 is 32.30%, with best reaching 45.33%. The study does not report causal claims or opinions, and findings are subject to evaluation uncertainty.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 31/10/2026, 00:38:53 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeif52abx3ihstfunjnfbfhe6i2u2uvp7ruwbpezzuh5tczhfln6mjy