AI & ML
ISER-MoE: Dynamic In-Switch Replication of Hot Experts for Load Balancing in Expert-Parallel
arXiv cs.DC0 views
arXiv:2611.09008v1 Announce Type: new Abstract: Expert-parallel mixture-of-experts (MoE) layers run at the pace of the most loaded GPU, and a few hot experts carry most of the excess load. Replicating them at every MoE layer balances the load only if their weights arrive within the layer, and broadcasting from the owner is the bottleneck: point-to-point copies overload it, and baseline broadcasts pay latency or bandwidth overheads. We present ISER-MoE (In-Switch Expert Replication for MoE), built in TensorRT-LLM, which multicasts hot experts through the NVLink switch to fixed GPU groups, so each byte leaves its owner once. A GPU scheduler assigns hot experts to groups per layer, and a multi-port broadcast spreads the injection over the group's ports, both overlapped with the shared expert. In trace replay, the scheduler removed 98-99% of the excess load; in prefill serving of DeepSeek-V4-Pro, throughput rose 17.4% on a math benchmark and 7.2% on agentic prompts.
Source:https://arxiv.org/abs/2611.09008PreviousCapital Efficiency, Isolation, Risk, and Ruin in Generative AI Inference Serving InfrastructureNextDARIC: A Position-Independent KV Cache Reuse System with Distribution-Aware Recomputation
Related posts
A Straight Line Is Not a Camera Pose: Auditing Short-Clip Pose Benchmarks
0arXiv cs.RO
Mechanisms of Misgeneralization in Physical Sequence Modeling
0arXiv cs.RO
Behavior Graphs: Unsupervised Graph Structures for Interpreting Robot Deployment Data
0arXiv cs.RO
RoboFQE: Rollout-Free Evaluation of Visual Robot Policies from Offline Trajectories
0arXiv cs.RO
Improving Imitation Learning Efficiency for Manipulation through Geometric Prior Pretraining
0arXiv cs.RO
MATE: Asynchronous Multi-Rate Diffusion Policy Composition
0arXiv cs.RO