AI

EMBERS: Empirically Motivated Robustness Testing of Deep Research Systems via Defect-Guided Fuzzing

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 04/11/2026, 05:18:24 UTC.

Briefings

This article presents the EMBERS framework, an empirically motivated fuzzing system designed to robustly test Deep Research (DR) systems. The study analyzed 1,932 execution trajectories from three DR systems, identifying nine recurring defect categories across subquestion formulation, online search, and reasoning. EMBERS uses defect-guided mutation to generate stage-targeted test inputs, achieving 3.21 times higher failure-discovery throughput than the strongest baseline under identical budgets. Human evaluations confirm that generated inputs preserve original information needs and constraints, while fine-tuning experiments show consistent-answer performance improvements.

Key points

• The EMBERS framework achieves 3.21 times the failure-discovery throughput of the strongest baseline under identical testing budgets.
• Nine recurring defect categories were identified across subquestion formulation, online search, and final reasoning in 1,932 execution trajectories.

Context

Deep Research systems extend large language models into autonomous agents that decompose complex information needs, iteratively search the web, and synthesize evidence. Existing evaluations focus on task-solving capabilities or robustness against external evidence disturbances, leaving intrinsic weaknesses in DR execution insufficiently explored.

Limitations

The study analyzed 1,932 execution trajectories from three representative DR systems. Results are based on preprint arXiv:2611.03723v1 and may not reflect final publication. Some findings rely on human and LLM-assisted evaluations, which introduce potential bias. The framework's effectiveness depends on the specific framework-model-dataset settings tested.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 04/11/2026, 05:47:32 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeih23dtjto5fidseu754l46biq3mfivnj6ywkjbzabocygu4j7wlni