AI

ListQA: A Benchmark for Evaluating List-Formatted Factual Knowledge Retrieval in Large Language Models

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 16/10/2026, 02:33:24 UTC.

Briefings

This article introduces ListQA, a benchmark designed to evaluate LLMs' ability to retrieve and organize multiple facts into structured lists rather than single answers. The benchmark consists of 9,045 human-curated questions covering 60+ countries across eight categories, featuring an evergreen design to ensure ground truth validity. Evaluation involves linear sum assignment problems with LLM-based semantic grading, benchmarking 34 models. Key findings show the best standard-inference model scored below 37% LLM-Judge-F1, while multi-faceted questions performed worse than single-answer tasks, and providing source Wikipedia pages significantly improved scores.

Key points

• The benchmark ListQA contains 9,045 human-curated questions requiring LLMs to recall multiple facts and compose them into structured lists of 3-10 elements.
• Providing the source Wikipedia page for questions significantly improved model scores, confirming that models can organize facts but struggle to retrieve them from parameters alone.

Context

Existing factual knowledge benchmarks focus on single answer retrieval, whereas real-world questions often require organized collections of facts. ListQA addresses this gap by testing multi-fact retrieval and composition capabilities across diverse global topics.

Limitations

The benchmark relies on human-curated data with an audited error rate of 1.7%, and evaluation metrics are based on linear sum assignment problems rather than direct model performance measurements.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 25/10/2026, 17:27:04 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeiaczssxe7ctez6zdk2rcshc6c3fu7n6zcvcwu6bv5agsd3w3k2nbi