AI

Benchmarking Forecasting Agents on Yesterday's Tomorrows

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 02/11/2026, 05:03:25 UTC.

Briefings

A new benchmark called FABench was released to evaluate forecasting agents using historical data from Polymarket. The system automatically generates questions and resolves outcomes for past forecast dates, allowing researchers to test agents against real-world market events. Evaluation results show that retrieval methods significantly improved the Brier score for all tested models, with the top agents scoring above market consensus on 54 questions. The study identified recurring failures in how agents weigh evidence, suggesting ample room for improvement.

Key points

• FABench is a scalable backtesting environment that automatically discovers events, generates questions, and resolves outcomes for any past forecast date.
• Retrieval methods improved the Brier score for all eight tested models and lifted the top agents to market consensus on 54 Polymarket questions.

Context

This research addresses the limitations of existing benchmarks by providing a live environment that evaluates forecasting agents immediately after events resolve, unlike fixed-corpus evaluations that wait for calendar cycles.

Limitations

The benchmark uses eight recent models and relies on an archive environment with live web search screened to prevent temporal leakage. Results are limited to the specific questions and models tested, and the study traces recurring failures rather than providing a complete causal analysis of all agent behaviors.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 02/11/2026, 06:00:04 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeicfdxl3s2oky7aofpzy4ibr4le7arlzwxaddhw622jeqtpbzqkave