AI

Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 13/11/2026, 07:48:24 UTC.

Briefings

A new arXiv preprint reveals that public language model leaderboards using the Bradley-Terry voting model are highly vulnerable to strategic manipulation. Researchers found that individual models can be boosted by up to 15-20 ranks via strategic voting, strategic prompting, or strategic nomination, with combined attacks potentially pushing a model up to 45 places higher. This vulnerability affects any open preference-based evaluation system.

Key points

• Leaderboards using the Bradley-Terry model allow models to be boosted by up to 15-20 places through strategic voting.
• Combined strategic manipulation strategies can push a model up to 45 places higher in a ranking.

Context

Arena-style rankings of language models are a widely used evaluation framework in online leaderboards, research papers, and model evaluation in production. These rankings aim to realistically assess model performance using pairwise comparisons of anonymous models on user prompts, aggregating votes via the Bradley-Terry model to produce the final ranking.

Limitations

The evidence is limited to a single arXiv preprint (arXiv:2611.14769v1) from November 2026. The study analyzes public leaderboard data but does not specify the exact dataset size, number of models tested, or the specific platform hosting the evaluations. The findings are based on observed effects rather than controlled experiments, and the authors note these issues hold for any preference-based evaluation using the Bradley-Terry model that allows open prompting, voting, or model submission.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 13/11/2026, 08:01:48 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeidwomjdsbc5cvkjdtxqn4o2j2x5cl7jtzfckn5huen3kmnae26lga