AI
On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 27/09/2026, 20:00:00 UTC
Briefing created: 03/10/2026, 08:23:17 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 02/10/2026, 16:18:24 UTC.
Briefings
A systematic study investigates distillation dynamics by varying rollout policy, token-level KL direction, and learning rate across Llama3 and Qwen2.5 models on scientific, medical, and arithmetic tasks. Results show rollout policy has limited impact, while token-level KL direction significantly shapes performance and output coverage, and learning rate controls forgetting and update sparsity. Forward KL gradients remain robust to policy changes, whereas reverse KL favors student-generated rollouts. On-policy data improves generalization on harder arithmetic variants but this advantage is not sustained after RLVR.
Key points
- Token-level KL direction significantly shapes task performance and output coverage, whereas rollout policy has limited impact on distillation dynamics.
- Forward KL gradients are remarkably robust to changes in rollout policy, while reverse KL is substantially more sensitive and favors student-generated rollouts.
Context and limitations
Analysis covers only Llama3 and Qwen2.5 model families across three reasoning domains. Findings are limited to the specific strong-to-weak distillation setting and do not account for gradient clipping, sampled KL estimators, or longer reasoning chains.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 03/10/2026, 08:23:17 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeidrh7id3vjsdcyqo7ntzwhjc6vxuxxmiu3vkslukmcya4p5gs7x3a