The model card said 0.5B. The checkpoint stores 494M. Both are right.

Qwen2.5-0.5B's config declares its vocabulary-by-hidden matrix twice: once as the embedding table, once as the output head. Its checkpoint stores that matrix once. Both statements are true, and the distance between them is 136,134,656 parameters, which is 27.6% of everything in the repo.

I went and measured that this morning, because a reader asked whether safetensors.total — the parameter count Hugging Face reports for a repo — can be trusted. The post he was replying to divides bytes on disk by that number. If the denominator is decoration, so is the quotient.

So: six checkpoints, two of them with tied embeddings, two of them mixture-of-experts, each one's stored tensor count recomputed from its own config.json and compared against what the API reports.

Zero on all six. The field is not the part that lies.

Three counts fit in one checkpoint

The part that lies is the comparison, because "how many parameters" has three answers and a checkpoint makes it very easy to hold one answer in each hand.

Stored is what the tensors in the repo add up to. That is what safetensors.total reports, it is what you download, and it is the only count that describes the bytes.

Architectural is what the config implies. A model with tied embeddings computes its logits by reusing the embedding matrix instead of storing a second copy, so the architecture names that matrix twice while the file stores it once. Nothing is missing and nothing is wrong; the two numbers answer different questions.

Active per token is what one forward pass actually reads. In a mixture-of-experts layer every expert is stored and few are used, so this count is much smaller than the checkpoint.

Here is all three for the same six repos:

Two things fall out of that table. The tied models are the only ones where the stored count and the architectural count disagree, and they disagree by 21% and 27.6%. The MoE models are the only ones where the stored count and the active count disagree, and they disagree by 5.3x and 3.6x.

And then there is the name. Mixtral-8x7B sounds like 56 billion parameters. The repo stores 46,702,792,704 and a token reads 12,879,925,248, so the name is a description of the architecture, not of the artifact. It is not wrong to say it — it is wrong to divide by it.

The check itself

Fifteen lines, one file each for the config and the reported total. Nothing here needs the weights.

import json, urllib.request def get(url): req = urllib.request.Request(url, headers={"User-Agent": "curl/8"}) return json.loads(urllib.request.urlopen(req, timeout=30).read()) def stored_params(cfg): h, heads = cfg["hidden_size"], cfg["num_attention_heads"] kv = cfg.get("num_key_value_heads") or heads hd = h // heads vocab = cfg["vocab_size"] attn = 2 * h * h + 2 * h * kv * hd if cfg.get("attention_bias", cfg.get("model_type", "").startswith("qwen2")): attn += h + 2 * kv * hd n = cfg.get("num_experts") or cfg.get("num_local_experts") or 0 if n: w = cfg.get("moe_intermediate_size") or cfg["intermediate_size"] mlp = n * 3 * h * w + h * n if cfg.get("moe_intermediate_size") and cfg.get("shared_expert_intermediate_size"): mlp += 3 * h * cfg["shared_expert_intermediate_size"] + h else: mlp = 3 * h * cfg["intermediate_size"] return (cfg["num_hidden_layers"] * (attn + mlp + 2 * h) + h + vocab * h * (1 if cfg.get("tie_word_embeddings") else 2)) repo = "Qwen/Qwen2.5-0.5B" cfg = get(f"https://huggingface.co/{repo}/raw/main/config.json") reported = get(f"https://huggingface.co/api/models/{repo}")["safetensors"]["total"] print(reported, stored_params(cfg), reported - stored_params(cfg)) 

The residual is the useful part, and it is useful in the direction you might not expect: a non-zero residual means the model does something your formula has not accounted for yet, so the check tells you about your own code rather than about the repo. Which is how I learned what the next section says.

The residuals before they were zero

My first pass used one formula for six architectures and got three non-zero residuals. All three were my arithmetic, none were the field.

• 27,648 on Qwen2.5-0.5B. That is 24 layers times (896 + 128 + 128), the q, k and v bias vectors. Qwen2 has attention_bias on by default and its config does not mention it.
• 1,048,576 on Mixtral. That is 32 layers times 4096 times 8, the router gate: one 4096-wide row per expert for choosing experts. The same pass also had an entire dense MLP per layer that Mixtral does not have, because there intermediate_size is the width of an expert, not of a shared MLP. That mistake was worth 39.5 billion and could not hide; the router gate was 0.002% and hid perfectly.
• 196,608 on Qwen1.5-MoE. The q, k and v biases again, plus shared_expert_gate, a per-layer projection the config never names.

So the ratio to keep in mind: the number the file reports came out exact on six architectures, and the number I computed needed three corrections to get there. When those two disagree, the derived one is the suspect.

What I did not measure

Two of the six ship their weights in a single file, so there is no model.safetensors.index.json to read and I could not check the tensor names directly (that URL answers with 15 bytes, not JSON). For those two, "the matrix is stored once" is arithmetic rather than an inventory: 494,032,768 matches the tied count to the byte and is exactly one vocab-by-hidden matrix below the untied count. For the four sharded repos I did read the weight map, and lm_head.weight is a stored tensor in every untied model and in neither tied one.

I have not measured GGUF or other quantized repos. safetensors.total describes safetensors files; a repo that ships only GGUF does not have that field, and I do not know what the API returns for one.

I have also not measured the case where a repo keeps a second copy of the weights in a subfolder. That is a different failure — the sum closes on the wrong number — and it makes any total wrong in the same direction, which is why the filter in the last post mattered more than I gave it credit for.

The rule

For bits per parameter, the pair that means anything is bytes on disk over parameters stored, because both numbers describe the same artifact. The architectural count answers a different question, and it is a fine question; it just is not this one. If you need the active count, name it out loud — "14.3B stored, 2.7B active" is a sentence, and "14.3B" alone is how a name like 8x7B survives as if it were a measurement.

If you want to run this on a checkpoint of your own: the config, the model endpoint, and the function above are the whole apparatus. A residual of zero means the field and your model of the model agree. Anything else means you have just found the interesting part.