AI & ML
Beyond Expert Rankings: Contrast Calibration for Mixed-Precision MoE Quantization
arXiv stat.ML0 views
arXiv:2610.20353v1 Announce Type: cross Abstract: Mixture-of-experts (MoE) language models activate only a few experts per token, yet all expert weights must be stored. Mixed-precision quantization reduces this storage cost by assigning higher bit-widths to sensitive experts and lower bit-widths to resilient ones, but expert rankings alone do not determine how strongly to weight their errors. To address this gap, we propose NatScale, a contrast-calibrated allocation framework for mixed-precision MoE quantization. It uses a model-specific power transformation to calibrate gradient-based expert weights to an information-theoretic contrast target while preserving rankings. We jointly evaluate each expert's gate, up and down configurations to estimate its output error. We then combine these errors and calibrated weights with an existing prior based on weight spectra, variance and precision decay to select configurations under a storage budget. Across three models, NatScale achieves the lowest native-to-quantized KL divergence and the highest top-token agreement in all nine settings against five allocation baselines. It also yields the lowest perplexity in seven settings. These findings highlight the importance of objective-aware precision allocation.
Source:https://arxiv.org/abs/2610.20353PreviousHierarchical Gaussian Processes for Survey Weighting: Local Equivalent Weights in Structured PopulationsNextFractional is Better: Learnable Derivative Orders in Neural Operator Learning
Related posts
How I Diagnose a Slow, Loud, or Overheating PC: A Practical Checklist
0dev.to Beginners
You Did Great for Three Days. Then Life Happened. Here Is a Habit System That Survives That.
0dev.to Beginners
Building an Agent Evaluation CI/CD Pipeline: Architecture, Budgets, and Regression Gates
0dev.to Testing
Truncado silencioso: cuando un verificador de LLM da el mismo puntaje a una respuesta correcta y a una incorrecta
0dev.to Testing
Your CI is red because of code you didn't write
0dev.to OpenSource
Summer 2026 Internships: 8 New Openings This Week (Nov 2)
0dev.to Programming