Eight model repos, added up: the file list and the model disagree

I wanted one number: how big is this model. Not a parameter count, not a formula — the bytes that land on disk when you pull it down.

So I did the dumbest possible thing. I asked eight model repos on Hugging Face for their file list, added up every file that looks like weights, and compared that sum with the single file I would actually load. One request each, no token:

import json, urllib.request url = "https://huggingface.co/api/models/openai-community/gpt2?blobs=true" d = json.load(urllib.request.urlopen(url)) for s in d["siblings"]: print(s["rfilename"], s["size"], (s.get("lfs") or {}).get("sha256")) 

Seven of the eight repos came back with a number between 2.0× and 9.7× the bytes I would download.

That last row is the control, and it is the one that tells you what the other seven rows are saying. Qwen2.5-0.5B-Instruct ships exactly one weight file, so its file list and its model are the same object and the ratio is 1. The others are not wrong about their own contents; they just contain more than one copy of the same weights, in more than one container.

One model, several containers

gpt2 is 137,022,720 parameters, which the Hub reports as safetensors.total. Its repo holds seven weight files:

onnx/decoder_model_merged.onnx 655,189,339 onnx/decoder_with_past_model.onnx 653,672,649 onnx/decoder_model.onnx 653,665,842 pytorch_model.bin 548,118,077 model.safetensors 548,105,171 tf_model.h5 497,933,648 flax_model.msgpack 497,764,120 

Four of them are the same 137M weights written by four different libraries — PyTorch, safetensors, TensorFlow, Flax. The three .onnx files are graph exports for different decoding paths. Load one, and you download 523 MiB. Sum the list, and you get 3.78 GiB, which is 7.4× the file you would load, and none of that extra is more model.

Nobody would do that on purpose. But "sum the files that look like weights" is exactly what a crawler, a cost estimate, or a "repo size" badge does, and the result is internally consistent — it adds up, it is just not the model.

One blob, several names

Two of the eight repos list the same bytes under more than one name. Not similar bytes — the same lfs.sha256:

sentence-transformers/all-MiniLM-L6-v2 23,026,053 bytes -> onnx/model_qint8_arm64.onnx onnx/model_qint8_avx512.onnx onnx/model_qint8_avx512_vnni.onnx HuggingFaceTB/SmolLM2-135M-Instruct 137,147,981 bytes -> onnx/model_quantized.onnx onnx/model_uint8.onnx 

Three names, one object. Summing the file list counts that 23 MB three times; deduplicating by hash brings all-MiniLM-L6-v2 from 884,718,327 down to 838,666,221, and the difference is 46,052,106 — exactly two extra copies. Same arithmetic on SmolLM2-135M-Instruct: 1,966,318,064 minus 1,829,170,083 is 137,147,981, exactly the one duplicated pair.

So a file list is not a set. If you are counting storage, it is a multiset and you have to collapse it by hash first. If you are counting what a user downloads, it is not even that, because they will only ever take one of those formats.

No safetensors at all: null is not zero

facebook/opt-125m contains no .safetensors file. Three weight containers, all of them other formats. Ask the API for its parameter count and you get:

safetensors: null safetensors.total: None safetensors.parameters: {} 

Not zero — absent. Which means a pipeline that does total = d["safetensors"]["total"] or 0 will record a 125-million-parameter model as having zero parameters, and nothing downstream will complain. That is not a hypothetical formatting problem; it is the same shape as reading a missing field as an empty result, and the number it produces is a perfectly ordinary-looking 0.

If you need a parameter count for a repo that ships only .bin, .h5 or .msgpack, you have to open the files. No field will answer for you.

Why my first number was wrong

The duplicate-blob section above is the second version of this article. The first version said something else: that none of the eight repos stored the same bytes twice.

I wrote the check to read siblings[i]["lfs"]["oid"]. There is no oid key there. The real field is lfs.sha256. My extractor returned None for every file, the deduplication map came back empty, and the script printed:

oids_in_weight_files=0 oids_under_two_or_more_names=0 

for all eight repos. Read on its own, that line is a clean bill of health. What caught it was the line directly above it, which said every one of those files was LFS-backed:

onnx/decoder_model_merged.onnx 655189339 lfs=True 

You cannot have seven LFS files and zero content hashes. The two readings contradicted each other, and the contradiction was the bug report.

The general shape is worth more than the field name: a lookup into a structure you have not actually printed will return "nothing" for every input, and "nothing" is the same answer you would get if the thing genuinely was not there. I got lucky because a second column disagreed. Usually nothing disagrees, and you publish the zero.

What I would do instead of adding up files

• For "what will this cost me to download" — pick the one file you would load, and quote its size. model.safetensors if it exists, otherwise the largest single .bin. The file list is not the answer to this question.
• For "how many parameters" — safetensors.total is the Hub's own count over the safetensors files, and where it exists it agreed with the per-format arithmetic I checked it against. Where it does not exist it is null, and null is a different word than 0.
• For "how much space does this repo occupy" — collapse by lfs.sha256 first. Two names, one blob, and the blob is charged once.
• If you filter the file list by anything, filter by both name and location. all-MiniLM-L6-v2 keeps 11 of its 14 weight files in subdirectories; a filter that only looks at root-level filenames sees 3 of 14 and reports a repo that is three times smaller than itself.

None of this needs a token, a script library, or a download. It is one request per repo and some arithmetic, and the arithmetic is the part that catches you.

What I did not measure

Whether a second copy of the weights in a subdirectory inflates safetensors.total the way it inflates a naive file sum. None of the eight repos I opened keeps two .safetensors files, so I had nothing to test it on, and I am not going to guess at the Hub's behaviour from its output on a different shape. If you know a repo that ships model.safetensors and a copy of it under a subdirectory, that is the experiment, and it is one request.

Eight repos is also not a survey. It is eight repos, chosen because I already had them open, and the ratios above are what those eight contain.