AI

Proxy Confidence: Auditing Black-Box LLM Agents with a Surrogate's Log-Probabilities

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 06/10/2026, 04:18:24 UTC.

Briefings

A new method audits LLM agents by comparing their actions against a pre-trained surrogate model using log-probabilities and request-PMI. This approach detects silent errors where the agent's confidence is near chance, achieving an AUROC of 0.825 on coding tasks. It enables two deployment modes: a real-time gate for low-trustworthy calls and confidence feedback for adaptive agents, improving live-execution success by 11.9% to 13.7% compared to random baselines.

Key points

• The method uses a pre-trained surrogate model to audit LLM agents by comparing their actions against the surrogate's log-probabilities and request-PMI scores.
• The approach detects silent errors where the agent's confidence is near chance, achieving an AUROC of 0.825 on coding tasks with a generative readout gain of +0.07 to +0.28.

Context and limitations

The study relies on a pre-trained surrogate model rather than the agent's internal state, requires no access to agent internals, and uses a single prefill pass alongside the tool call, limiting scalability for high-volume deployments.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 06/10/2026, 05:09:10 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeif2eoo77et3jaqonz5o4v2lkhju5ywhbfyafll27h26dn2txg7cf4