AI
Rubric Rewards from Item Response Theory
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 27/09/2026, 20:00:00 UTC
Briefing created: 02/10/2026, 08:56:06 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 01/10/2026, 17:18:24 UTC.
Briefings
Rubric Response Theory (RRT) addresses the aggregation problem in language evaluation by using an item response model to predict criterion difficulty and discrimination. Unlike standard rubric scoring, RRT treats verdict patterns as evidence for scalar quality rather than summing fixed points. The system uses an online expectation maximization algorithm to update a Response Parameter Network (RPN) during training. On the Qwen3.5-4B policy, RRT achieved a macro criterion score 1.7 points higher than GRPO across Medical, Science, and RubricBench datasets. Performance gains of 2.8 to 5.6 points were observed on hard criteria in Medical and Science. Adaptive Fisher selection maintained scores within 0.1 points of GRPO at half the criterion budget.
Key points
- Rubric Response Theory (RRT) uses an item response model to predict criterion difficulty and discrimination rather than summing fixed points for aggregated rewards.
- The RPN predicts criterion difficulty and discrimination from prompt and criterion text, while online expectation maximization updates parameters during training.
Context and limitations
Results are limited to four datasets (Medical, Science, Rubrics as Rewards, RubricBench) and rely on the Qwen3.5-4B policy. Performance comparisons are against GRPO, and specific judge request counts are not provided.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 02/10/2026, 08:56:06 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeiahrrza36inbb72l4cx4r6umzl45azwlpq5rthdbl5fnz3obk6aqy