AI & ML
Speculative Decoding for Mixture-of-Experts Models: Expert Bytes, Kernel Bandwidth and the Best Draft Length
arXiv cs.DC0 views
arXiv:2610.34440v1 Announce Type: cross Abstract: Speculative decoding checks several drafted tokens in one verification pass. On a mixture-of-experts (MoE) model, consecutive tokens often select the same experts. A deduplicating engine reads each selected expert once per pass, so it requests fewer bytes than a per-slot engine, which reads it once per token. We write a step's time as the bytes each kernel requests over its effective bandwidth, plus fixed time. Under geometric acceptance, we derive from this step time when speculation is faster than plain decoding. The best draft length is a monotone staircase in the acceptance rate. We bound how a bandwidth error moves each result, and we check the main steps in Lean. Deduplication is faster exactly where its share $\bar u(n)$ of the per-slot bytes is below its ratio $r(n)$ of the per-slot bandwidth. On llama.cpp Metal, MLX-LM and llama.cpp CUDA, which we modified to deduplicate, the per-slot kernel has the higher effective bandwidth (r = 0.32 to 0.75). Thus deduplication is faster only from 5 to 8 tokens per pass, or never. We fit the bandwidths on one layer's expert operation, run alone, in under an hour per model. Bytes from routing traces over these bandwidths predict the time that deduplication saves. The measured saving lies inside its error bound at every point on CUDA and MLX-LM and at 36 of 48 points on Metal. The crossover is the first pass size at which deduplication is faster. The prediction places it within one pass size in 10 of 12 configurations of engine, model and depth. We computed these predictions after the measurements. On DeepSeek-V2-Lite, operation times measured before its speculative runs give the best draft length on CUDA within one at all 22 acceptance values.
Source:https://arxiv.org/abs/2610.34440PreviousScaleRLT: Exact Log-Probabilities and Fast Deferred Rollouts for Reinforcement Learning on Looped TransformersNextAccelerating SO(2)-Equivariant Materials Models through Low-Precision Emulation
Related posts
Cloudflare DNS Zone Builder cuts peak memory 85% and halves build delay
0dev.to News
O dia em que a cliente pediu desconto num bolo de pistache (e pagou R$ 30 num cafe)
0dev.to Design
O telefone tocou as 19h: Por que cancelam seu contrato primeiro na crise?
0dev.to Design
Which Cursor model is best for everyday work?
0dev.to Productivity
Is AI going to get cheaper or more expensive?
0dev.to Machine Learning
Điểm tin Trí tuệ Nhân tạo & Mô hình Ngôn ngữ lớn (LLMs) tại Việt Nam (03/11/2026)
0dev.to Machine Learning