AI & ML
LEDGE: An Efficient Simulator for Distributed LLM Serving on Heterogeneous Edge Devices
arXiv cs.DC0 views
arXiv:2611.24085v1 Announce Type: new Abstract: Evaluating distributed large language model (LLM) serving on heterogeneous edge devices requires exploring diverse model partitions, device placements, and network conditions, making repeated physical experiments costly. An efficient simulator for edge LLM serving must support diverse, fine-grained placement configurations, capture runtime behavior, and accurately predict execution times while keeping simulation overhead low. We present LEDGE, a lightweight simulator for fast, fine-grained evaluation of distributed LLM serving on heterogeneous edge devices without executing tensor operations. LEDGE constructs event graphs supporting non-uniform tensor and pipeline parallelism, estimates execution costs using reusable performance models calibrated from end-to-end measurements, and simulates shared-link contention. We evaluate LEDGE on a heterogeneous testbed of 10 Jetson and Raspberry Pi devices, covering 17 model--precision combinations across 4 LLM families. On Qwen deployments, LEDGE achieves request-level mean absolute percentage errors below 3.6% and a 135.3 times aggregate speedup over physical inference. Performance model calibrated on one LLM family yield 7.44% zero-shot end-to-end latency MAPE across three other families. Further simulations show how shared-link contention affects placement decisions and network topology constrains scaling. Across four edge environments, LEDGE completes 26,773 simulation runs in 83.8 minutes, enabling efficient exploration of deployment strategies under diverse resource conditions and serving objectives. The source code is publicly available at https://github.com/gdfwj/LEDGE.
Source:https://arxiv.org/abs/2611.24085PreviousLocality-Aware Scheduling for Stream Processing Applications in Serverless EnvironmentsNexteFAIR: Fair LLM Serving under Hourly Electricity Commitments
Related posts
How Music LLMs Actually Work: From Tokens to Waves (MusicGen, DiT, and Building Your Own Engine)
0dev.to Machine Learning
蒸馏 Lora RHLF 都是什么意思
0dev.to Machine Learning

Teaching an offline plant doctor to spot rust
0dev.to Machine Learning
Is Your Model Good Enough? Precision vs Recall
0dev.to Machine Learning
How to Trim Audio for a Starlit Night Video — Free, in Your Browser
0dev.to Tutorial
How to Write Numbers in Words on Birth Certificates and Identity Documents
0dev.to Tutorial