AI

Equal Ranking Quality, Different Decisions: Measuring and Reducing Order Dependence in LLM Scorers

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 05/10/2026, 14:18:24 UTC.

Briefings

Research reveals that LLM scorers in reranking tasks exhibit order dependence despite equal ranking quality, meaning identical scores do not guarantee consistent decisions when candidate order changes. The study introduces OC-SFT, a method that penalizes score disagreement across orderings to stabilize decisions. Results show OC-SFT improves ranking quality while reducing decision overlap between scorers, outperforming order-averaged distillation on 12 base models.

Key points

• Five trained scorers within 0.010 nDCG@10 retain overlapping sets of 0.66-0.84 when reordered, demonstrating that equal ranking quality does not imply equal decisions.
• OC-SFT attenuates order dependence by penalizing disagreement between a candidate's scores across orderings, leading to more stable decisions than order-averaged distillation.

Context and limitations

Results are limited to passage reranking, response ranking, and multi-document question answering tasks; no prompt-time changes were tested to resolve the dependence.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 08/10/2026, 02:49:48 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeifgcbt7ignctph3tzd72e3q47wlxmx4bzlqyguudyg4fokxgyqeh4