AI
Model Annealing: Data-Efficient Learning through Review and Consolidation
Publication: cs.AI updates on arXiv.org
Author: Ali Mohammadi Ruzbahani, Rezvan Abouzadeh
Source published: 16/11/2026, 05:00:00 UTC
Briefing created: 16/11/2026, 05:32:56 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 16/11/2026, 05:03:25 UTC.
Briefings
A new method called Model Annealing uses a small labeled dataset to memorize the network before training, then uses the memory of past self-predictions to relearn from the data. This approach improves accuracy and robustness across several architectures on CIFAR-100, Tiny-ImageNet, and Imagewoof, matching or exceeding snapshot distillation while reducing inference costs. The method separates the reheating schedule from actual relearning, showing that relearning from the memory is the key benefit.
Key points
- Model annealing divides training into episodes where the learning rate is raised and cooled, with each cooled state joining a memory of past self-predictions.
- The memory averages predictions rather than weights, and the deployed network is the last student, so inference costs nothing extra.
Context
This research proposes a data-efficient learning technique that leverages a small labeled dataset to pre-memorize a model before training begins, using the memory of past self-predictions to guide relearning from the data.
Limitations
Results are limited to four architecture families and 5% to 100% of labels on specific datasets; accuracy gains are not predicted by how early cross-entropy accuracy saturates.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 16/11/2026, 05:32:56 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeiesdr56p3srqr5nufltsdgiwvacrqbaynhixjy4vc5mauhro3mx2e