Open‑source audits reveal that a non‑trivial slice of publicly released low‑bit LLM checkpoints cannot reliably generate code. Quantcheck automates the detection of these regressions by analysing GGUF quantised files before they ever reach a developer’s IDE.
Before this work, practitioners treated GGUF tags as drop‑in replacements for their full‑precision counterparts, assuming that the quantisation pipeline preserved functional behaviour. The community relied on informal sanity checks or downstream testing only after integration, leaving a blind spot for safety‑critical code generation tasks.
Only 2.3 % of the 305 official GGUF artifacts examined fail to solve more than six HumanEval+ tasks – “As shares, 7 of the 305 official artifacts (2.3%) and 3 of the 35 model variants (8.6%) are affected” [1]. All seven failures belong to 4‑bit or 8‑bit tags (q2_K–q3_K_L) from model families under four billion parameters, confirming that low‑precision quantisation can silently cripple coding ability.
A simple importance‑matrix calibration can revive three low‑bit Qwen2.5‑Coder‑3B builds, boosting solved tasks from zero to between 76 and 113 out of 164 – “With the weights, toolchain and tensor mix held fixed, adding an importance matrix raised Qwen2.5‑Coder‑3B scores at three quantization levels from zero to 76–113 of 164 tasks” [1]. The same uncalibrated recipe leaves smaller variants functional, showing that the failure mode is not solely tied to model size.
The study only covers GGUF checkpoints and a single suite of code‑generation benchmarks; it does not address other formats, multilingual prompts, or real‑world latency constraints. Moreover, Quantcheck flags artifacts but does not guarantee that calibration will succeed for every broken tag, leaving open the question of how many failures can be repaired versus must be discarded.
If low‑bit models are to power developer tools, every new GGUF release should be screened with Quantcheck and, when flagged, calibrated before publication. This systematic checkpoint replaces ad‑hoc testing and prevents unsafe artefacts from propagating into production code assistants.
