AI

UndoBench: Separating Task Competence from Recovery Capability in Tool-Using AI Agents

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 06/10/2026, 10:18:24 UTC.

Briefings

UndoBench introduces a benchmark separating task competence from recovery capability in tool-using AI agents. Evaluated on 12 held-out workflows across two open-weight models and three recovery paradigms, the study found nominal competence reached 83.54% while conditional recovery success rate (CRSR) dropped to 46.72%. Naive retry caused duplicate external effects in 53.33% of trials, revealing phase-dependent recovery vulnerabilities masked by standard evaluations.

Key points

• UndoBench evaluates 36 base workflows and 36 fault scenarios across 8 enterprise domains using counterfactual paired trials under identical seeds.
• On 12 held-out TEST workflows, conditional recovery success rate (CRSR) fell to 46.72% while nominal competence reached 83.54%.

Context

Tool-using AI agents are increasingly deployed across enterprise software systems, yet widely used benchmarks primarily evaluate nominal task completion, conflating baseline planning competence with operational fault recovery.

Limitations

Evaluation was limited to 12 held-out TEST workflows across two open-weight models and three recovery paradigms. Results are from a frozen lost-acknowledgment study with 5,760 executions and 2,880 paired trials. Findings remain preliminary as the study is in progress.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 19/10/2026, 18:58:53 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeicvuevue7fnnw5lgnzune4wufsuhkqferx7umtjgckzclnxbufaby