AI
EmailBench: A Benchmark for Evaluating LLM Agents on Enterprise Email and Productivity Tasks
Publication: cs.AI updates on arXiv.org
Author: Mukul Singh, Mansi Uniyal, Devin Devlin, Wen Xie, Big Thadawasin, Ritam Dutt, Vivian Lai, Hyeonsu B. Kang
Source published: 29/09/2026, 04:00:00 UTC
Briefing created: 29/09/2026, 05:24:40 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 29/09/2026, 05:18:25 UTC.
Briefings
EmailBench is a new benchmark for evaluating LLM agents on enterprise email tasks, featuring 206 scenarios across 16 categories. It uses a deterministic Enron-inspired corpus and evaluates eight LM configurations via 258 static assertions and 211 LLM rubrics. The best-performing configuration achieved a 33.5% pass rate despite 99.7% of tool calls completing successfully, indicating valid execution does not guarantee task completion.
Key points
- EmailBench introduces 206 email and productivity scenarios across 16 task categories using a deterministic Enron-inspired corpus.
- The benchmark evaluates eight LM configurations using 258 static assertions and 211 LLM rubrics, revealing a significant gap between tool execution success and task completion.
Context and limitations
The benchmark covers 16 task categories and uses a fixed single-user corpus; broader tool coverage and multi-persona testing are noted as future work areas.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 29/09/2026, 05:24:40 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeifceb3cznpknwviihr2yv4s5nip4gevbtqs6k2gnqzyrzzao23tua