AI & ML
ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
arXiv cs.DC0 views
arXiv:2607.00466v3 Announce Type: replace Abstract: Equal load does not mean equal decode latency in prefill-decode (PD) disaggregated mixture-of-experts (MoE) serving. In bandwidth-bound decode, more distinct experts mean more weight loads and higher latency. We present ELDR, a lossless, expert-locality-aware decode router. Prefill signatures predict decode expert usage. Balanced $K$-means fits per-decoder centroids with background refitting. Locality-band routing selects the least-loaded worker among close matches. A block-granular cache keeps signatures exact under prefix caching. We evaluate ELDR in vLLM on up to 40 GPUs. Across three MoE models on 12 public task benchmarks and multilingual WildChat, it reduces median time-per-output-token (TPOT) by 5.4-14.5% on average across request rates and up to 17.8% over the best of four load-balancing baselines, preserving model outputs and token-to-expert assignments.
Source:https://arxiv.org/abs/2607.00466PreviousOnline Graph Exploration under T-Interval ConnectivityNextGreedy-Like Defective Coloring: Distributed Algorithms and Applications
Related posts
![AMR AI and Embodied Intelligence — Ch.11: Self Supervised Robot Learning [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fh8y4egw0kdqtzf6m3hzx.png)
AMR AI and Embodied Intelligence — Ch.11: Self Supervised Robot Learning [EN]
0dev.to
![AMR AI and Embodied Intelligence — Ch.13: Human Robot Interaction [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjm36fpwnygomah6qsu4n.png)
AMR AI and Embodied Intelligence — Ch.13: Human Robot Interaction [EN]
0dev.to
![AMR AI and Embodied Intelligence — Ch.14: VLM and VLA [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fo06nxmfdm3t6o1oznau9.png)
AMR AI and Embodied Intelligence — Ch.14: VLM and VLA [EN]
0dev.to
370 Books in 20 months: South African author defends her use of AI
0Hacker News
The climate case for AI has not been made
0MIT Technology Review
I Let Pangram Loose on My Gmail Inbox. Here’s How Much AI It Found
0Wired