AI
PMOPD: Task Ordering, Cycling, and Parameter-Update Subspace Protection in Multi-Teacher On-Policy Distillation
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 27/09/2026, 20:00:00 UTC
Briefing created: 30/09/2026, 14:14:10 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 30/09/2026, 13:18:24 UTC.
Briefings
Multi-teacher on-policy distillation (MOPD) aggregates specialized capabilities in shared parameters, but capability seesaw effects can suppress gains. The proposed PMOPD method addresses this by using projection-based subspace memories to isolate task-specific updates. Experiments on Code, Reason, and Math tasks demonstrate that PMOPD consistently outperforms MOPD, raising average scores by 2.54 points on Qwen2.5-7B and 2.09 points on Llama-3.1-8B. The approach utilizes a lightweight conflict probe and cycling strategy to balance subspace estimation and task revisitation.
Key points
- PMOPD constructs subspace memories from cumulative parameter displacements of different tasks to isolate interference.
- The method improves every evaluated capability on Code, Reason, and Math tasks, raising average scores by 2.54 points on Qwen2.5-7B and 2.09 points on Llama-3.1-8B.
Context and limitations
Results are limited to three representative tasks (Code, Reason, Math) and apply to Qwen2.5-7B and Llama-3.1-8B models; no generalizability to other architectures or tasks is confirmed.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 30/09/2026, 14:14:10 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeieutkth7sdqpnv2iv5rdfdagfkcym7i5g5boq2uz3y2wytwzj3z5i