TBNAI
LRMoE: Demand-Side Load Re-Calibration for Efficient MoE Training on Memory-Constrained Nodes
Publication: cs.AI updates on arXiv.org
Author: Jiaming Hu, Zhida Jiang, Zhaolong Xing, Yu Li, Zhen Chen, Ke Zhang
Source published: 30/10/2026, 04:00:00 UTC
Briefing created: 30/10/2026, 10:03:42 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 30/10/2026, 06:48:24 UTC.
Briefings
LRMoE is a memory-aware MoE training system that jointly recalibrates backward demand and balances workload on constrained nodes. Experiments on Qwen3-30B-A3B and GLM4.5-106B-A12B show it reduces cumulative MoE-layer training time by 21.2-44.8% relative to SYMI. The system plans retention quotas during forward execution and uses a load recalibration metric to select token-expert pairs during backpropagation.
Key points
- LRMoE reduces cumulative MoE-layer training time by 21.2-44.8% relative to SYMI on Qwen3-30B-A3B and GLM4.5-106B-A12B.
- LRMoE uses a capacity-aware routing algorithm to distribute retained token-expert pairs among eligible instances of their original experts.
Context
Sparse Mixture-of-Experts (MoE) models offer scalable capacity for Web applications but face uneven routing creating computation and communication bottlenecks. Existing methods redistribute workload, yet additional expert copies compete with token-related buffers for limited GPU memory. LRMoE addresses this by jointly recalibrating backward demand and balancing retained workload during forward and backpropagation phases.
Limitations
Results are limited to two specific models (Qwen3-30B-A3B and GLM4.5-106B-A12B) and a single training run. The 21.2-44.8% reduction represents relative improvement over SYMI, not absolute time savings. Physical compaction and variable-size grouped GEMMs are implemented but their specific impact on execution buffer demand remains unmeasured in this snapshot.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 30/10/2026, 10:03:42 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeifrpqhw7dinkxsqfkbugfu5wf6z7qmj5djlztuvkh6u53euyjo4de