MuonGS: Squeezing 15% More Speed Out of LLM Pretraining via Relaxed Weight-Norm Control

In the high-stakes world of Large Language Model (LLM) pretraining, efficiency is everything. When you are training a model on trillions of tokens, a 5% improvement in optimizer efficiency can save millions of dollars in compute costs. Today, a new contender has emerged in the "Muon" family of optimizers: MuonGS.

Published on October 27, 2026, by researchers Athanasios Glentis, Dawei Li, and Mingyi Hong, MuonGS (Gram-Schmidt) introduces a sophisticated way to manage weight-norm growth. By relaxing the strict constraints found in previous versions of the optimizer, MuonGS delivers an estimated 15% token-equivalent speedup on dense models and shows strong results on Mixture-of-Experts (MoE) architectures.

The Problem: The Effective Learning Rate Decay

To understand why MuonGS is such a significant step forward, we first need to look at the "effective learning rate" problem. In deep learning, the actual progress a model makes isn't just determined by the nominal learning rate ($\eta$) you set in your config file. It is determined by the ratio of that learning rate to the magnitude (Frobenius norm) of the weight matrix:

$$\text{Effective LR} = \frac{\eta_t}{|W_t|_F}$$

As an LLM trains, its weight matrices naturally tend to grow in magnitude. In the vanilla Muon optimizer—which uses matrix-level Newton-Schulz orthogonalization to coordinate updates—this growth is linear. As the weights ($W$) get larger, the effective learning rate shrinks. Over a long training run, the model’s ability to explore the loss landscape stalls because the updates become too small relative to the weights they are trying to change.

The Muon Family: From Weight Decay to Hard Clamping

Researchers have tried several ways to stop this "norm explosion." The two most common variants are:

  1. MuonW: Uses traditional weight decay to pull the weights back toward zero.
  2. MuonH: Uses a "hard-radius" constraint (Hyperball) to fix the weight norm to a specific value.

While these methods work, they are somewhat heavy-handed. By forcing the weights into a specific "box" or radius, they can over-constrain the model's trajectory, potentially preventing it from reaching the most optimal minima in the loss landscape.

How MuonGS Works: The Gram-Schmidt Projection

MuonGS takes a more elegant, "relaxed" approach. Instead of forcing the weight norm to be a specific number, it focuses on the direction of the growth.

The core innovation of MuonGS is the use of Gram-Schmidt (GS) projection to remove the radial component of every update.

1. Identifying the Radial Component

Every update produced by the Muon optimizer can be decomposed into two parts: a radial part (pointing directly away from the origin, increasing the norm) and a tangential part (rotating the weight vector without changing its length).

2. The Projection Step

MuonGS identifies the radial part of the update—the portion that pushes the weight matrix directly outward—and removes it using the Gram-Schmidt process. This ensures that the update applied to the weights is always orthogonal to the current weights in the Frobenius inner product space.

3. Square-Root Growth Dynamics

Because the radial component is removed, the first-order change to the weight norm becomes zero. The norm only increases due to second-order effects. Mathematically, this means the weight norm grows at a rate proportional to the square root of the training steps ($\sqrt{t}$), rather than growing linearly ($t$) like in vanilla Muon.

This "sweet spot" allows the accumulated effective learning rate to grow at a rate of $\Omega(\sqrt{T})$. This is significantly more robust than the logarithmic accumulation seen in vanilla Muon, providing the model with sustained "exploration energy" throughout the entire training run without the rigid boundaries of MuonH.

Why It Matters: 15% More Efficient Pretraining

The theoretical elegance of MuonGS is backed by impressive empirical data. The researchers tested the optimizer across several architectures:

  • Dense Models: On 160M and 400M parameter models, MuonGS achieved lower validation perplexity than MuonH at long training horizons. This translated to a 15% token-equivalent speedup—meaning you can reach the same level of model quality using 15% fewer tokens (and 15% less compute).
  • Mixture-of-Experts (MoE): The optimizer was also validated on an MoE model with 135M active parameters and 1B total parameters. It successfully outperformed MuonH in this setting, proving that it is ready for the complex architectures used in modern production models.

What to Watch: The Future of "Relaxed" Optimization

MuonGS represents a shift in how we think about optimizer constraints. As we move toward training runs that span trillions of tokens, "hard" constraints like those in MuonH may become too restrictive.

By providing a formal stationarity guarantee of $O(T^{-1/2})$ and demonstrating that "relaxed" control is superior for long-term progress, MuonGS opens the door for a new generation of optimizers that are stable enough for massive scale but flexible enough for deep exploration. For researchers and engineers looking to squeeze every drop of performance out of their GPU clusters, MuonGS is a technique that demands attention.

Sources