AI

AGORA: Reliability-Aware Aggregation of LLM Judges via Item-Response Theory

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 05/11/2026, 05:03:25 UTC.

Briefings

AGORA is a new item-response-theory model that replaces human labels with a latent quality scale to aggregate LLM judges. It learns judge reliability from all items jointly, matching or exceeding Dawid-Skene on large evaluation sets while improving accuracy as the evaluation set shrinks to a few hundred items. The model uses ordered rating thresholds and sensitivity parameters to describe judges without human labels, with trust weights tracking per-judge human validity.

Key points

• AGORA uses an item-response-theory model to aggregate LLM judges by placing each item on a single latent quality scale.
• AGORA learns judge reliability from all items jointly without requiring human labels, matching or exceeding Dawid-Skene on large evaluation sets.

Context

LLM-as-a-judge is a common substitute for costly human evaluation, and aggregating several judges is the standard remedy for any single judge's bias. The human-labeled sets that validate them hold a few hundred items or fewer, creating a regime where per-item mean weights weak judges as heavily as strong ones.

Limitations

AGORA's performance is evaluated across three benchmarks, with specific results for large evaluation sets and sets shrinking toward a few hundred items. The model's accuracy is confirmed as pools grow more uneven, but the specific conditions under which it remains accurate as pools grow more uneven are not fully detailed in the evidence.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 05/11/2026, 05:17:05 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeifubckqaejpybaot4comgdms4kxhnzmgyku76jshinjwh6seoksra