AI
TraceDance: An Automated System for Building Agent Behavior Benchmarks from Real-World Agent Deployment Traces
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 26/09/2026, 20:00:00 UTC
Briefing created: 29/09/2026, 03:43:39 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 29/09/2026, 03:18:24 UTC.
Briefings
TraceDance is an automated system that constructs targeted benchmarks from real-world agent deployment traces to evaluate specific undesirable behaviors. It uses an Anchor-and-Confirm approach combining programmatic retrieval with a Flash LLM to generate custom behavior specifications. The system evaluates LLMs using decision-point continuation with a behavior-specific rubric, avoiding reference answers or environment replay. Experiments on 252,557 sessions produced 107 benchmarks with 4,125 instances, meeting 95.3% of build requests. Human annotators confirmed the requested behavior in 84% of sampled instances, and the automated grader's agreement matched human judgments. Nine frontier LLMs achieved a mean pass rate of 26.7%, indicating significant struggles with appropriate responses at evaluated decision points.
Key points
- TraceDance constructs targeted benchmarks from deployment traces to evaluate specific undesirable behaviors using decision-point continuation evaluation.
- The system uses an Anchor-and-Confirm approach combining programmatic retrieval with a Flash LLM to generate custom behavior specifications from 252,557 sessions.
Context and limitations
The system evaluates LLMs at recorded decision points without environment replay or reference answers. Benchmark coverage is limited to 252,557 sessions, and results may not generalize to all deployment scenarios.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 29/09/2026, 03:43:39 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeicjaojwdnfcnmrxpjveobcpylrxbkyxkfoxb4uax3phf2swxzg2wa