AI

PaLoRA: Paced Low-Rank Adaptation for Continual Learning

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 06/10/2026, 03:18:24 UTC.

Briefings

PaLoRA addresses catastrophic forgetting in continual learning by adapting gradient step magnitude based on the effective rank of historical knowledge. Unlike fixed small learning rates, PaLoRA uses an anisotropic pacing law s* = R/c to balance stability and plasticity. Experiments show PaLoRA achieves 4% accuracy gains on challenging 50-task ImageNet-A benchmarks compared to prior methods.

Key points

• PaLoRA proposes PaLoRA, which compresses historical knowledge via adaptive SVD truncation, projects gradients onto the nullspace of prior tasks, and applies rank-aware adaptive pacing.
• PaLoRA achieves substantial gains of 4% accuracy on challenging 50-task ImageNet-A benchmarks compared to prior methods.

Context

Continual learning methods mitigate catastrophic forgetting through various mechanisms, yet nearly all complement these with small learning rates as a heuristic to restrict gradient scaling magnitude. Such fixed heuristics lack theoretical guidance on how the strength of this restriction should evolve as tasks accumulate. PaLoRA addresses this by deriving a pacing law s* = R/c that adapts to the effective rank of past updates.

Limitations

The study relies on an anisotropic leakage model and finite-precision updates, which may not fully capture all sources of forgetting. Results are from a single snapshot on ImageNet-A and ImageNet-R benchmarks, and the 4% gain represents a specific long-horizon setting rather than a general rule for all tasks.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 16/10/2026, 14:09:31 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeif3bedaubvijfw5yraee6k2bfhboigawrwuvdr6siz33xyyxdpxga