Classifying a genetic variant as pathogenic (disease-causing) or benign is a classic machine-learning task in genomics. Our XGBoost model reached an ROC-AUC of 0.995 on genes it had never seen. That sounds almost perfect, but most of that score comes from variants that are easy to call. This post looks at where the difficulty actually lies.
The data: real ClinVar labels, real population frequencies
- Labels come from ClinVar (GRCh38 VCF, release 19 October 2026, 4.6 million records). We kept germline variants of up to 50 bp, in a single gene, with at least one review star. From these we drew a seeded random sample of 25,000 pathogenic/likely pathogenic and 25,000 benign/likely benign variants across 6,531 genes.
- Population allele frequencies (gnomAD, TOPMed, ALFA, 1000 Genomes) come from dbSNP via NCBI E-utilities. Gene constraint scores (pLI, LOEUF) come from gnomAD.
- We built 58 features in total: allele frequencies, molecular consequence, variant type and length, intron offset, protein position, and the amino-acid change (Grantham distance, BLOSUM62, hydropathy, volume, charge).
We deliberately left out ClinVar review status, submitter count and dbSNP clinical significance. Each of these partly is the label, so including them would leak the answer into the model.
Split by gene, not by variant
If variants are split at random, the model sees other variants of the same gene during training. It can then learn "variants in gene X are usually pathogenic" instead of learning anything about the variant itself. Our main test therefore uses 673 genes that never appear in training (a 70/15/15 split by gene). We also ran a random split and a temporal split that tests on the newest 15% of ClinVar submissions.
Headline results on unseen genes
| Model | ROC-AUC | MCC |
|---|---|---|
| ACMG-style rule score | 0.975 | 0.44 |
| Logistic regression | 0.993 | 0.93 |
| SVM (RBF) | 0.991 | 0.93 |
| XGBoost | 0.995 | 0.94 |
| XGBoost + SVM stacked | 0.995 | 0.94 |
The stacked ensemble did not beat XGBoost alone: the difference was +0.0001, with a 95% CI of [−0.0000, +0.0002]. We kept the simpler model.
Look at the first row. A transparent rule score already reaches an AUC of 0.975. That is the first hint that the overall number is inflated by easy cases.
Where it is easy, and where it is hard

Test ROC-AUC by subgroup — the overall 0.995 hides much weaker groups
- Null variants (nonsense, frameshift, canonical splice) are 99.7% pathogenic in the test set.
- Synonymous variants are 99.6% benign.
- Common variants are almost always benign because of their frequency.
A model scores well on these largely by recognising the type of variant. The clinically difficult group is missense variants, where one amino acid changes:
| Subgroup | n | XGBoost AUC |
|---|---|---|
| All test variants | 7,513 | 0.995 |
| Missense | 1,065 | 0.930 |
| Rare missense (AF < 0.1% or absent) | 938 | 0.909 |
| Synonymous (almost all benign) | 1,770 | 0.749 |
The low AUC for synonymous variants mostly reflects how few positives there are (7 out of 1,770). It doesn't make the classifier unsafe, but it shows that a single overall AUC hides very different sub-problems.
The feature ablation tells the same story. Removing population frequency costs the most (AUC 0.995 → 0.991). Removing amino-acid or consequence features barely changes the overall AUC, because the overall number is dominated by the easy groups.
Three zones instead of one yes/no
A clinician needs a model that says when it is unsure. We calibrated every model on the validation genes (Platt scaling). We then chose two thresholds that give at least 99% precision on the pathogenic side and 99% NPV on the benign side, which splits the scores into three zones:
- Likely pathogenic, uncertain and likely benign
- On the test genes, XGBoost puts 94.8% of variants in a confident zone, with 98.8% precision on the pathogenic side and 98.6% NPV on the benign side. The rest are marked uncertain.

Calibrated scores and the three decision zones
We also scored 4,000 ClinVar variants of uncertain significance (VUS). The model left 2,310 of them in its own uncertain zone, called 1,116 likely pathogenic and 574 likely benign. These are hypotheses for expert review, not reclassifications.
Explaining every score
Each prediction comes with exact TreeSHAP values for XGBoost and sampled Shapley values for the SVM. Where the data allow, it also gives ACMG/AMP-style hints: PVS1 for null variants, PM2/BA1/BS1 from allele frequency, PM4 for in-frame length changes, BP7 for synonymous variants, and PP3/BP4 from the model score itself.

Which features drive the XGBoost scores
Limitations
- This is a research and teaching tool, not a clinical classifier.
- ClinVar labels are themselves partly based on frequency and variant type, so some of the "easy" cases are easy by construction.
- The temporal split shows a small drop (AUC 0.993), as expected for newer submissions.
Takeaways for any ML-in-biology project
- Split by the unit that leaks. Here that is the gene.
- Remove features that are the label in disguise.
- Report subgroup results. An overall AUC can hide the case that matters, such as rare missense variants.
- Give the model an "I don't know" zone and report its coverage.
Full project (report, Python and MATLAB versions, web application with VCF batch scoring): Pathogenic genetic variant classification — ProTechSC Academy. Video walkthrough: youtu.be/NS2jAmZVkAM.
We guide final-year and PhD research projects in engineering, AI and bioinformatics. WhatsApp +91 97907 09424.
