AI

From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 05/10/2026, 17:18:25 UTC.

Briefings

This study investigates how teacher signals influence parameter changes in Qwen3-1.7B during multi-teacher on-policy distillation. Using four domain teachers trained from the same initialization, the research compares gradient magnitudes, optimizer updates, and task learning curves. Key findings reveal that loss averaging implicitly weights responses, with token averaging favoring longer outputs and domain contribution equalization preserving this weighting. Adam's first moment reduces parameter update differences, achieving 0.83 cosine similarity between teachers. BF16 rounding significantly masks weight changes, reducing differences from 97% to 7-11% of FP32 master weights. The top-64 intersection KL gradient closely matches Qwen's full-vocabulary gradient, though task performance varies with averaging strategies.

Key points

• Loss averaging implicitly weights responses, with token averaging favoring longer responses and domain contribution equalization preserving this weighting within domains.
• Adam's first moment reduces differences in parameter updates, achieving 0.83 cosine similarity between teachers despite raw gradient differences.

Context and limitations

Coverage is limited to Qwen3-1.7B with four domain teachers trained from the same initialization. Results are based on gradient analysis and optimizer updates, with task learning curves and SmolLM3-3B diagnostics included but not fully detailed. Findings are preliminary and subject to further validation.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 09/10/2026, 15:18:07 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeiaqoacge75julnrhjibor6nqchnvp7n2xk5dzqjq52xpzjgucwoqu