AI

Learning Functional Subspaces for Neural Network Compression

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 07/10/2026, 16:18:24 UTC.

Briefings

A new method called Learnable Subspace Projections (LSP) reduces neural network compression by learning which subspaces to discard rather than using fixed criteria. This approach outperforms existing baselines across various large language models and vision transformers, achieving up to 1.6x faster decoding at low compression levels and shrinking memory usage by 13.5x compared to standard factorizations.

Key points

• Learnable Subspace Projections (LSP) optimizes discarded subspaces end-to-end using a global objective like KL divergence to the original model's output distribution.
• At -70% compression, LSP achieves 10.9 WikiText-2 perplexity and 42.2% mean zero-shot accuracy on Llama-2-7B, significantly outperforming baselines at 13.3 perplexity and 36.0% accuracy.

Context

Modern transformers require substantial memory and compute, and existing compression methods use local criteria like activation energy that ignore error propagation. LSP learns optimal subspaces to discard, improving efficiency and accuracy.

Limitations

Evidence covers LLMs and ViT-B/16 only; performance at higher compression levels remains untested. Results are from a preprint with no author verification or peer review.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 30/10/2026, 06:01:27 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeic4tv4kprqlklmsax56r57tlxgjqdfxttln5uwcagdugx2pxbhlye