AI
An RL View of OPD: Least Square Policy Distillation for Sample-Efficient LLM Reasoning
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 27/09/2026, 20:00:00 UTC
Briefing created: 29/09/2026, 18:24:56 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 29/09/2026, 08:18:24 UTC.
Briefings
This article proposes Least-Square Policy Distillation (LSPD), a reinforcement learning framework that improves language model reasoning by connecting reverse-KL objectives to KL-regularized policy optimization. LSPD combines optimistic exploration with off-policy data reuse, achieving a sharp O(log K) regret bound under online exploration. Empirically, LSPD outperforms existing distillation baselines by +1.59 points on average in Avg@16 across six mathematical reasoning benchmarks, while Pass@k evaluations up to k=64 show stronger performance as k grows, indicating better policy diversity preservation.
Key points
- LSPD introduces an RL-inspired framework combining optimistic exploration and off-policy data reuse to improve language model reasoning.
- The framework achieves an O(log K) regret bound under online exploration, outperforming baselines by +1.59 points on average in Avg@16.
Context and limitations
Results are limited to six mathematical reasoning benchmarks and diverse teacher-student settings; Pass@k evaluations up to k=64 show stronger performance as k grows, indicating better policy diversity preservation.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 29/09/2026, 18:24:56 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeici7x5erablwlcldlbe3myizkhvainvrdko5uttzycc3woq6scvp4