AI

Composing What Each Teacher Learned: Multi-Teacher On-Policy Distillation through Teacher-Relative Shifts

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 08/10/2026, 16:18:24 UTC.

Briefings

This article presents a method called Delta-MOPD for improving teacher composition in large language models. The study compares Delta-MOPD against endpoint supervision across two settings: common-domain composition and routed-domain distillation. Key findings show Delta-MOPD outperforms endpoint methods by 4.11 Math and 1.95 five-benchmark points when using three composed teachers, while matching endpoint accuracy with two teachers. In routed-domain distillation, Delta-MOPD achieves higher mean performance in both phase orders, reducing the order gap from 10.50 to 6.42 points. The research concludes that shift targets are particularly useful when teacher signals are combined at a state, and that target construction is an independent design axis in MOPD.

Key points

• Delta-MOPD transfers each teacher's teacher-minus-base logit shift re-anchored at the student's frozen initialization.
• Delta-MOPD exceeds endpoint composition by 4.11 Math and 1.95 five-benchmark points with three composed teachers.

Context

The study investigates multi-teacher on-policy distillation (MOPD) in two settings: common-domain composition and routed-domain distillation. It compares Delta-MOPD against endpoint supervision while holding teacher selection fixed. The findings suggest shift targets are useful when teacher signals are combined at a state, and that target construction is an independent design axis in MOPD.

Limitations

Evidence is limited to three composed teachers and two settings. Results are based on experiments rather than forecasts. Specific city, region, or country locations are not supported by the source evidence.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 09/11/2026, 15:43:40 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeig5kqvkpqbxkxj7qkljpizquqbhtxm3rnhq6heszhimbmb6drjuay