AI & ML
COPE: Composable Performance Emulation for Mixture-of-Experts Offloading Policies
arXiv cs.DC0 views
arXiv:2610.33404v1 Announce Type: cross Abstract: As LLMs scale, mixture-of-experts (MoE) architectures increase model capacity while keeping per-token computation manageable through sparse expert activation. This sparsity also enables memory-efficient deployment by retaining most expert weights in CPU memory and transferring or executing only the routed experts on demand. Existing MoE offloading systems typically target specific model-platform configurations, yet differences in model sparsity, CPU and GPU throughput, VRAM capacity, and host-device bandwidth can change the policy ranking. Because implementing and exhaustively evaluating all candidate policies incurs substantial engineering cost, efficiently selecting the best-performing policy for each model-platform configuration remains a key challenge for practical MoE offloading. We present COPE (Composable Offloading Policy Emulator), which selects the best-performing MoE offloading policy for a target model-platform configuration before runtime integration. COPE comprises two components. Its Policy Matrix factors the design space into two independent axes, expert residency and MoE execution strategy, enabling mechanism reuse across candidate policies and exposing four new candidates. Its Virtual Heterogeneous Machine profiles combined-operator schedules on the target platform and combines these measurements with offline routing traces to estimate candidate performance. Compared with the evaluated baselines, COPE improves fastest-policy selection accuracy by 24%-100% and reduces performance-prediction error by $1.7\times$ to $6.8\times$. By identifying the preferred policy before integration, COPE further reduces policy-specific implementation effort by up to 74%.
Source:https://arxiv.org/abs/2610.33404PreviousOperating System Support for Accelerators as First-Class Compute Elements in Heterogeneous SystemsNextZipFusion: Fusing Lossless KV Decompression with Attention for Efficient LLM Decoding
Related posts

10 Open-Source Robotics Repositories for 3D Perception, VLA Policies, and Sensor Calibration
0dev.to
My self-modifying system tried to wire a gate-passing module into its own protection enforcer. It was refused three times.
0dev.to
Last night an AI agent performed brain surgery on a mouse
0Hacker News
The Card-Update Page Gets One Visit. Design for That Visit.
0dev.to UX
SaaS Partner Scout: A Script for Identifying GitHub SaaS Project Contributors
0dev.to GitHub
WARN Act and Retaliation — Can Your Employer Punish You for Asking About a Layoff Notice?
0dev.to Data