TBNAI
Sharpen Without Search: On-Policy Distillation of Sequence-Level Power Distribution
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 04/10/2026, 20:00:00 UTC
Briefing created: 23/10/2026, 08:55:53 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 06/10/2026, 19:18:24 UTC.
Briefings
A new method called On-Policy Power Distillation (OPPD) enables language models to generate high-quality answers in a single generation by training them to mimic a teacher's power distribution. This approach significantly improves reasoning accuracy on math benchmarks like MATH500 and GSM8K compared to standard on-policy distillation and even outperforms GRPO methods without reference answers. The technique uses a sequential Monte Carlo sampler where the model's own answers are weighted by a frozen teacher's power distribution, shifting probability toward correct solutions. Training on mathematics alone yields up to 5.3 points higher accuracy on HumanEval, and gains persist across model families and sizes.
Key points
- OPPD uses a sequential Monte Carlo sampler where the model generates candidates and a frozen teacher's power distribution weights them, shifting probability toward correct solutions.
- Training on mathematics alone raises HumanEval accuracy by up to 5.3 points, while gains persist across model families and sizes including those trained with verified rewards.
Context
OPPD improves reasoning accuracy on math benchmarks by training models to mimic a teacher's power distribution, shifting probability toward correct solutions. It outperforms GRPO methods without reference answers and adds up to 9.3 points when applied after GRPO training.
Limitations
Training on mathematics alone yields up to 5.3 points higher accuracy on HumanEval; gains persist across model families and sizes including those trained with verified rewards. Code is available via GitHub but specific implementation details remain unverified.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 23/10/2026, 08:55:53 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeiht62dy3bloou2yvuw53l7hpbdvkn7r3i52nsu7mw3c7rlximvwhi