AI & ML
Communicate the Change: Efficient Expert Parallelism via Activation Compression
arXiv cs.DC0 views
arXiv:2611.22215v1 Announce Type: cross Abstract: Scaling has driven the success of large language models (LLM), with sparse mixture-of-experts (MoE) architectures enabling greater parameter capacity without a proportional increase in computation. At large scales, MoE models rely on expert parallelism to distribute experts across devices. Each MoE layer requires two synchronous collective communication steps during inference: dispatching hidden states to the activated experts and combining the resulting expert outputs. As compute throughput has outpaced interconnect bandwidth, these synchronous collectives have become a major bottleneck to MoE scaling and efficiency. We address this challenge with EchoMoE, a lightweight architecture modification that retrofits existing MoE models, requiring approximately 8,500x fewer training tokens than full pretraining. Motivated by the observation that transformer blocks update the residual stream incrementally, we use a hidden state from an earlier layer as a reference and communicate only the compressed difference between it and the current state, rather than communicating each hidden state in full. EchoMoE periodically refreshes these reference states and transmits only the compressed differences synchronously, reducing the traffic on latency-critical paths without significant computational overhead. EchoMoE reduces inter-node communication volume by 58% and improves throughput by 22.6%, while preserving average performance within 3.8 percentage points across 18 benchmarks. EchoMoE further improves inference throughput by up to 2.38x over bandwidth-constrained Ethernet, paving the way for wide-area distributed inference. EchoMoE also tolerates quantization of the communicated representations without measurable quality loss. We release an efficient implementation for applying EchoMoE to pretrained MoE models, together with an inference integration for vLLM.
Source:https://arxiv.org/abs/2611.22215PreviousSLED: Sensor SLicing from Edge to DatacenterNextSpecSearch: Towards Hardware-Aware Efficient Speculative Decoding
Related posts
Monitoring SPF, DKIM and DMARC Drift in CI
0dev.to DevOps
Software Could Turn Edge AI Chips Into Shared Workstations
0dev.to Docker

Tencent reports Q3 revenue up 8% YoY to ~$31.1B, below est., and operating profit down 7% in its first earnings decline in years, as AI capex surges 5x to ~$12B (Bloomberg)
0Techmeme

KPMG uses 100+ AI agents to conduct a parallel audit of a FTSE 250 company in a pilot, saying the AI asked "better questions" and dug deeper than human staff (Financial Times)
0Techmeme

GTA VI review: an unparalleled technical and comedic masterpiece, with elaborate depictions of a heightened reality and an unpredictable environment by design (Jason Schreier/Bloomberg)
0Techmeme

Sony and Rockstar say a new PS5 firmware update is mandatory to play GTA VI ahead of its November 19 launch; the update aims to combat rapid homebrew exploits (Wesley Yin-Poole/IGN)
0Techmeme