LRMoE Introduces Demand-Side Load Re-Calibration for Efficient MoE Training on Memory-Constrained Nodes — Artificial Intelligence

Researchers have unveiled LRMoE, a new memory-aware training system designed to address computation and communication bottlenecks in Sparse Mixture-of-Experts (MoE) models. By jointly recalibrating backward demand and balancing workloads under strict GPU memory constraints, the system achieves significant reductions in training time compared to existing baselines.

Sparse Mixture-of-Experts (MoE) models provide scalable capacity for web applications but often suffer from uneven routing that creates bottlenecks in computation and communication. While existing methods attempt to redistribute workloads through placement and replication, these approaches frequently cause additional expert copies to compete with token-related buffers for limited GPU memory. To address this, researchers have introduced LRMoE, a system that jointly recalibrates backward demand and balances the retained workload within resource constraints.

The LRMoE architecture operates through a two-phase design that separates forward execution planning from backward contribution selection. During the forward pass, the system plans retention quotas and expert placement while preserving all routed computations to maintain necessary context. In the backpropagation phase, a load recalibration metric estimates the contribution of each token-expert pair to the expert update. The system then selects only those pairs that fall within the pre-determined quotas, ensuring that backward work aligns with available capacity.

A capacity-aware routing algorithm subsequently distributes these retained pairs among eligible instances of their original experts. This process is followed by physical compaction, which forms smaller communication payloads and variable-size grouped GEMMs. These optimizations collectively reduce backward traffic, computation requirements, and the demand on execution buffers. The approach treats backward demand as a contribution-aware scheduling variable rather than a fixed consequence of forward routing, allowing the system to prioritize high-value updates under constrained budgets.

Experiments conducted on Qwen3-30B-A3B and GLM4.5-106B-A12B models demonstrate the efficacy of this method. LRMoE reduces cumulative MoE-layer training time to fixed-loss targets by 21.2% to 44.8% relative to the SYMI baseline. Furthermore, a separate GPU replay analysis indicates a 57.8% reduction in the subtotal memory required for expert placement, replicas, and token buffers compared to fixed ownership strategies. The system effectively coordinates expert placement, communication, and backward workload management without requiring improvements to the underlying search or recommendation models themselves.

Sources: arXiv cs.AI.