AI

On-Policy Distillation with Negative-Policy Rollouts

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 08/10/2026, 03:18:24 UTC.

Briefings

This article presents Negative-Policy OPD (NP-OPD), a method that enhances on-policy distillation by adding a negative reference policy to the rollout stage. Unlike prior approaches that rely solely on teacher supervision, NP-OPD uses rollouts from a lower-capability negative policy to provide explicit negative signals, helping the student model avoid harmful outputs. Experiments demonstrate that NP-OPD improves performance across various model scales, generation modes, and reasoning domains compared to standard OPD.

Key points

• Negative-Policy OPD (NP-OPD) introduces a negative reference policy that generates tokens preferred by the negative policy over the teacher, providing explicit negative signals during training.
• The method improves on-policy distillation performance across multiple model scales, generation modes, reasoning domains, and different OPD variants compared to standard approaches.

Context

On-policy distillation (OPD) uses a student model to learn from a stronger teacher via rollouts, but limited distributional overlap between teacher and student can reduce learning signals. This research addresses that gap by incorporating a negative policy reference.

Limitations

Results are based on extensive experiments with limited model scales, generation modes, and reasoning domains. The study does not report specific numerical improvements or causal claims beyond observed performance gains.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 04/11/2026, 14:23:45 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeihf74bigfavsdwfop6ddfuuuvy7nui3v5r7cafj44aw3ikx4gg6la