Draft for blog.localai.hex7.com and Ghost · Private Local AI · 2026-11-10

A GGUF file used to tell you one thing about its precision: the type in its name. Q4_K_M meant most
weights were 4-bit with a medium-quality recipe, and that was the whole story. Recent repositories break
that convention. Files named Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf carry a recipe
label rather than a bit-width, and they are worth understanding, because the trick they encode — one
quantization type per tensor instead of one for the whole model — is how a 354 GB model becomes 75.8 GB
without a noticeable fight about it.

The two methods are called GSQ (Gumbel-Softmax Quantization) and RCO (Riemannian Constrained
Optimization), both from the Deep Algorithms and Systems Lab at the Institute of Science and Technology
Austria. Their model cards describe them plainly: "GSQ provides accurate low-bit scalar quantization of
each tensor at a given quantization type; RCO assigns the per-tensor quantization types under a size
budget." That division of labour is the whole design.

GSQ: making 2 and 3 bits behave like vector quantization

Post-training quantization has a long-standing split. Scalar quantization rounds each weight to the nearest
value on a grid — fast, and it is what GGUF formats actually store. Vector quantization (codebooks, k-means)
is usually more accurate per bit but needs extra decode machinery, which is why formats like IQ types need
lookup tables.

GSQ sits in between by learning the grid instead of using a fixed one. Concretely: it jointly learns the
per-coordinate assignments — which weight goes to which codebook entry — and the per-group scales that
decode them, optimising both at once through a Gumbel-Softmax relaxation, a differentiable stand-in for
the discrete choice you cannot back-propagate through. The result the authors claim is that scalar formats at
2–3 bits get close to what vector quantization would achieve, and stay in standard GGUF form so existing
engines can run them unchanged.

That last clause matters more than it sounds. A method that only works inside a research runtime is not a
practical deployment story; a method that produces ordinary GGUF blocks and scales is.

RCO: choosing where precision goes, with the budget actually enforced

RCO is the allocation problem. Given N tensors and K candidate quantization types (Q4_K, Q5_K, IQ4_NL,
IQ3_S, …), which type should each tensor get so that the total file size hits your target — say 4.0 bits per
weight average — while task loss is as low as possible?

The usual approach is a heuristic or a knapsack after thought: pick sizes, then hope the model still works.
RCO's move is to make the constraint itself geometrically tractable. It reformulates the exact size budget as
a smooth Riemannian manifold in logit space, then optimises directly on that manifold, so every step it
takes is a step that satisfies the budget exactly. No penalty weights to tune, no "close enough on size"
slop. Gradients come from the task loss, which means the search is sensitive to what the model actually does
badly at low precision rather than to a proxy like weight-space error.

Practically, this changes what you can ask for: build me a 75 GB version of this model, and get one.

Put together, the procedure in the model cards is three steps:

  1. Per-tensor database. Every weight tensor is quantized at every candidate type with GSQ. This is the search space, and it costs GPU time to build — for a Qwen3.8-Flash-Next build the card says 352 tensors (304 dense plus 48 fused routed-expert matrices).
  2. RCO search. Assign one type per tensor on the manifold, so the whole-file average bit-width meets the target exactly.
  3. Assembly. Stitch the selected per-tensor variants into one standard GGUF.

The released files ship an audit artifact for this: tensor-allocation/<model>.rco-allocation.txt lists the
type assigned to every tensor, with a histogram of types and the target bit-width — so you can inspect what
the search decided without loading the model at all. That is unusually honest packaging.

What the numbers actually look like

The Qwen3.8-Flash-Next card is the clearest worked example. It is a sparse mixture-of-experts model: 512
routed experts per layer across 48 layers, ten active per token, and the expert matrices dominate both the
parameter count and therefore the size budget — "95% of the searchable weights live in the experts." The
search assigns a type per layer, not per expert, because GGUF cannot express per-expert types within a
fused matrix.

The shipped variants, with average bit-width over transformer weights:

build bpw shard 1 (resident) shard 2 (n-gram table) total note
Q2_0 2.40 37.6 GB 28.8 GB 66.4 GB built for throughput, avoids lookup-table formats
IQ2_XS 2.50 39.2 GB 28.8 GB 68.0 GB smallest at equal quality
IQ3_XXS 3.00 47.0 GB 28.8 GB 75.8 GB matches the base on AIME25
IQ3_S 3.50 54.8 GB 28.8 GB 83.6 GB recommended
BF16 base 16.00 — — 354 GB reference

Quality numbers from the same card, evaluated against the BF16 base (task average is the mean of AIME25,
GPQA-Diamond and LiveCodeBench v6): at 3.00 bpw, IQ3_XXS reaches 92.57 against 93.12 — 4.7× smaller —
with AIME25 exactly equal to the base. At 2.50 bpw, IQ2_XS keeps 89.16 of 93.12 at 68 GB. And note what the
card says about recoveries above 100% on zero-shot averages: they are "within roughly one standard error"
and should be read as parity, not improvement. That caveat is worth quoting, because marketing copy does not
usually include it.

The part that matters for someone running this locally

Two things in these files are easy to miss and both are deployment advice.

The second shard is a lookup table, not weights. Every build carries the same 28.8 GB shard: a
per-layer n-gram embedding table (51.2B parameters) held at IQ4_NL, fixed at 4.5 bpw in every build and
excluded from the search. It is read sparsely — one row per token — so it does not have to be resident. The
card's instruction is -lm mmap --lazy-mode on, plan for shard 1 in VRAM, and keep shard 2 on an SSD. That
is how a 75.8 GB model becomes something you can actually run on a big card rather than something you cannot
run at all.

Format choice is a speed decision, not only a quality one. The card measures Q2_0 against IQ2_XS on the
same 55-prompt suite: 367.49 prompt t/s and 93.79 decode t/s versus 108.19 and 70.30, average latency
6.70 s versus 12.68 s — with Q2_0 being the smaller file by 1.6 GB. The difference is decoding cost: IQ
formats pack more accuracy per bit but decode through lookup tables, and on this model that cost dominates.
Q2_0's decode rate stays within a 2.0 t/s spread across categories; IQ2_XS swings 49.4 to 94.5 t/s, a 45.1
t/s spread, because how much of each layer a request touches varies with the prompt. Long-context work is
where the gap widens: 9.6× faster prefill on RAG, 6.8× on writing. Quality-wise the two are 0.09 apart on
task average. If you want throughput rather than bragging rights, Q2_0 is the honest pick at that size.

Two more practical notes from the cards. The DeepSeek-V4.1-Flash build needed "llama.cpp b11870 or newer"
to load its layout — these builds move faster than the engines that run them, so check your engine version
before assuming a GGUF will work. And ffn_down_exps has its own constraint: 640 rows, not divisible by
256, which rules out every block-256 K and I format and leaves Q2_0 (block 64) and IQ4_NL (block 32). In
the smaller builds the search puts every such layer on Q2_0; in IQ3_XXS it lifts 18 of 48 layers to IQ4_NL.
Constraint like that is what "quantizing a model" really looks like in practice: not a formula, a set of hard
edges the optimiser works around.

Why this is worth your attention

Per-tensor mixed precision has been available in inference engines for a while and invisible in GGUF. GSQ-RCO
moves it into the file format itself: ordinary blocks, ordinary scales, ordinary engine support, plus a
sizeable allocation that a gradient search picked. The naming change is the tell — UD-Q4_K_XL was always a
hand-tuned recipe; GSQ-RCO-IQ3_XXS is the output of an optimiser told exactly how big to be.

For local use, the useful consequence is that a memory budget stops being a guess. You say 8 GB, 12 GB, 24
GB, and something decides which weights keep precision. The model cards claim Strata runs these GGUFs on a
single NVIDIA GPU from 8 GB of VRAM — on our machines that is where these files start to matter, and it is
also where I would urge you to measure rather than believe: the two builds at the same size differ by 3× in
prompt throughput.

The paper references are GSQ (arXiv:2604.18556) and RCO (arXiv:2605.00649); reference implementations are
at IST-DASLab/GSQ and IST-DASLab/RCO, and tooling is picking it up — GPTQModel's changelog mentions a
GSQ/RCO teacher-KL optimisation path. The model cards list the per-build benchmark tables if you want to
check the numbers rather than take them as given.

Sources