AI
Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 27/09/2026, 20:00:00 UTC
Briefing created: 29/09/2026, 11:10:52 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 29/09/2026, 06:18:25 UTC.
Briefings
Research on multi-teacher on-policy distillation (MOPD) for language models reveals that routing specialists to specific domains does not automatically improve performance. In Qwen3.5 models, MOPD failed to outperform a single best specialist, with feedback from the mathematics domain being significantly less effective than instruction-following feedback. The authors propose Domain-Normalized MOPD (DN-MOPD), which rescales domain-specific feedback by its measured spread. Experiments on six benchmarks show DN-MOPD improves average scores across all model sizes, but gains are primarily driven by reducing instruction-following feedback rather than enhancing mathematics feedback alone.
Key points
- Multi-teacher on-policy distillation (MOPD) fails to outperform a single best specialist in Qwen3.5 models, as routing specialists to specific domains does not yield performance gains.
- Feedback from the mathematics specialist is significantly less effective than instruction-following feedback, which dominates the student's updates in MOPD.
Context and limitations
Results are limited to six public benchmarks, three random seeds, and two different answer-length limits. Findings apply to Qwen3.5 models at three sizes and remain unverified for other architectures or larger model sizes.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 29/09/2026, 11:10:52 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeicxumv47u72xedvh6c2hwrsdmbnxkxzx73k4h4n46uj2kf577a6le