AI

Counting the Bandages: On measuring the time that AI saves

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 28/10/2026, 04:03:24 UTC.

Briefings

This article argues that AI assistance saves time only if the saved minutes exceed the cost of errors it lets through. The author contrasts a stopwatch view of time saved with a ledger view accounting for escaped defects. Key findings include a baseline model where checking time grows with the logarithm of the model's error rate, and a replication of a METR study showing AI helped between 3% and 82% of issues depending on the estimand. The study also tests AI reviewers checking AI drafts, finding that AI reviewers check more as draft error rates rise but switch to all-or-nothing checking near the break-even point.

Key points

• The article proposes a 'ledger view' of time saved, which accounts for escaped defects and errors, contrasting with a simple stopwatch view.
• A baseline model indicates that checking time grows with the logarithm of the model's error rate, meaning higher error rates increase the time required for checking.

Context

The study analyzes AI's impact on time saved by contrasting a stopwatch view with a ledger view that accounts for escaped defects. It replicates a METR developer study using public data and tests AI reviewers checking AI drafts of SEC filings. The findings challenge the assumption that AI always improves efficiency without introducing new errors.

Limitations

The study relies on a single preprint and limited public data from a specific developer study. Results are conditional on specific estimands and may vary depending on how errors are measured or adjusted for chance imbalance. The experiment on AI reviewers checking AI drafts is small and pre-specified, limiting generalizability.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 28/10/2026, 04:06:35 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeidufge7rdmusek2yuv76ngigl4jnv7h54yq2lar5zvjlkzkxcgn3a