AI
Fewer Tokens, More Self-Teaching: On-Policy Self-Distillation for Extreme Visual Token Reduction
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 25/09/2026, 20:00:00 UTC
Briefing created: 29/09/2026, 02:31:14 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 29/09/2026, 02:18:24 UTC.
Briefings
A 2026 study introduces LT-OPD, a training framework for extreme visual-token reduction in MLLMs. By having a frozen full-token model supervise a student using only a small fraction of visual tokens, the method stabilizes on-policy learning. Across nine benchmarks on Qwen3.5-4B, LT-OPD maintained 82.3% average performance despite 5% visual-token retention, significantly outperforming baselines. The approach also reduced KV-cache usage by 85.2% and prefill FLOPs by 85.4% without extra inference overhead.
Key points
- LT-OPD uses a frozen full-token MLLM to provide distributional supervision to a student model generating responses with only a small fraction of visual tokens.
- The framework includes a budget-level curriculum that progressively decreases the token budget during training to stabilize on-policy learning.
Context and limitations
Results are limited to nine benchmarks on Qwen3.5-4B; performance gains were observed across Qwen3.5-9B, GLM-4.6V-9B, and LLaVA-OV-1.5-4B.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 29/09/2026, 02:31:14 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeih6rqm24x7bqo6hhhon4lade6teuiw7bhnng5hyllqdngs67m4z7u