AI
Smaller Models, Better Rejects: Preference Distillation Scaling
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 29/09/2026, 20:00:00 UTC
Briefing created: 03/10/2026, 02:14:26 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 02/10/2026, 06:18:24 UTC.
Briefings
A 2026 study challenges the assumption that larger models are required for effective rejection sampling in preference distillation. The research demonstrates that smaller frozen models (7B to 72B) generate high-quality rejects with significantly lower inference costs than self-generated responses, enabling stronger student models to learn effectively. By deriving a utility bound for Direct Preference Optimization, the authors identify three key interventions: mixing rejects from smaller and larger models, reassigning rejects to other prompts, and selecting candidates with lower likelihood under the reference policy. These findings suggest that preserving task structure while limiting coupling to the reference policy allows smaller models to provide effective rejects at low cost.
Key points
- Smaller frozen models (7B to 72B) generate rejects with less inference compute than self-generated responses, yet train stronger students than self-generated rejects.
- Mixing rejects from smaller and student-scale models improves performance as the smaller model's share increases.
Context and limitations
Results are based on code generation and mathematical reasoning tasks. Findings apply to preference distillation in linearized feature models and may not generalize to all downstream applications or model architectures.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 03/10/2026, 02:14:26 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeigai47bdn4xchwboblpxl7lb6ygf6k3h44rjegu26p6vqetidcn2e