In short

  • When 1 part in 30 is defective, a model that calls every part good scores 96.7% accuracy and catches no defect at all.
  • On synthetic washer images, a trained model scored lower, 96.0%, yet caught 4 of 10 corroded washers. Accuracy ranks the useless model first.
  • Count defects caught and good parts flagged instead, and set the decision threshold from what a missed defect costs. The same model then caught 6 to 9 of 10, while flagging 27 to 42 of 290 good washers instead of 6.

The problem

A classifier sorts each image into a class, here good or corroded. The usual score is accuracy: the share of images it sorts correctly.

In inspection, defects are rare. If 1 part in 30 is defective, 29 of every 30 images are good ones. Accuracy then mostly measures how well the model recognizes good parts. A model can score high while it lets almost every defect through.

Two numbers tell you what you need to know:

  • Defects caught: of all defective parts, how many the model flags. Also called recall.
  • False alarms: of all good parts, how many the model flags by mistake.

Balanced accuracy sums it up in one number: the average of the catch rate on defects and the correct rate on good parts [4]. Always saying "good" scores exactly 50% on it, however rare the defects are.

Why it happens

The first reason is plain arithmetic. With 290 good and 10 corroded washers, answering "good" every time is right 290 times out of 300.

300 test washers, 10 of them corroded. A model that answers "good" for every washer scores 96.7% accuracy and catches none of the corroded ones.

The second reason is how a classifier decides. It gives each image a probability for each class and answers with the most likely one. With two classes, that means it flags a defect only when the defect probability is at least 0.5.

When defects are rare, a borderline mark is more often harmless than a defect, so the defect probability seldom reaches 0.5. As a statement about probabilities, that is sensible. As a decision rule for your line, it is wrong: a missed defect usually costs far more than a second look at a good part. Most learners lean toward the common class this way, and on heavily imbalanced data they can ignore the rare class altogether [3].

The experiment

We made synthetic washer images with a script: 256 × 256 pixels, gray metal on a gray background, with random position, size, lighting and the edge of a fixture crossing some frames.

  • Corroded washers carry one or two small orange-brown spots. A third of them are faint.
  • Good washers are not all clean. 60% carry a light, harmless tarnish that looks like faint corrosion, as real parts do.

Top: good washers. Bottom: corroded washers, each spot circled and shown 3 times larger. The images are enhanced for display.

The sets keep the same defect rate, 1 corroded washer in 30:

  • Training: 406 good, 14 corroded
  • Validation: 58 good, 2 corroded (scored during training, then added to the training images for the final fit)
  • Test: 290 good, 10 corroded (used only for the final score)

About 1 in 5 held-out washers came out too similar to a training washer, so we drew them again, using the check from Your test score may be lying. No test washer is a near-copy of a training one.

We trained a fast image classifier. It describes each image with a tiny thumbnail and simple brightness, edge, color and texture statistics, then fits a small neural network in the browser. For the main run we switched class weighting off, so the model learned from the data as it is.

As a check that the model can see the spots at all, we also trained it on a balanced set of 210 good and 210 corroded washers. It caught 8 of the 10 corroded test washers.

What we found

To compare the rows on one scale, we also count a total cost: each false alarm costs 1, each missed corroded washer costs 20.

Model Accuracy Balanced accuracy Corroded caught Good flagged Total cost
Answer "good" for every washer 96.7% 50.0% 0 of 10 0 of 290 200
Trained model, standard rule (0.5) 96.0% 69.0% 4 of 10 6 of 290 126
Same model, threshold for a miss worth 20 false alarms 89.7% 75.3% 6 of 10 27 of 290 107
Same model, threshold for a miss worth 50 false alarms 85.7% 87.8% 9 of 10 42 of 290 62

The trained model is useful: it catches 4 corroded washers that "always good" lets through, and its total cost drops from 200 to 126. Its accuracy is still lower than doing nothing, because it also flags 6 good washers. If you compared the two by accuracy, you would pick the model that never finds a defect.

The last two rows use the same model and change only the threshold. Here, each step lowers both the accuracy and the total cost.

Accuracy and corroded washers caught for "always good" and for the same trained model at three thresholds. Accuracy falls while catches rise and the total cost falls.

The scores show why the threshold matters so much. Most corroded washers score far above most good ones, but 6 of the 10 corroded washers score below the standard line at 0.5.

The model's corrosion score for each of the 300 test washers, on a log scale, with the standard threshold of 0.5 and two thresholds set by cost.

The fix

1. Measure catches and false alarms, not accuracy alone. Put the two counts next to every accuracy figure, per class, and compare against "always good". Judge a model at more than one threshold. One that stays at 50% balanced accuracy whatever the threshold has learned nothing useful; one that fails only at 0.5 may just need a different threshold.

2. Set the threshold from costs. Decide how many false alarms one missed defect is worth. If a miss costs as much as c false alarms, and a correct answer costs nothing, flag a part when its defect probability is at least

threshold = 1 / (1 + c)

This is the decision rule from cost-sensitive learning [1]. A miss worth 20 false alarms gives 1/21 ≈ 0.048; worth 50, it gives about 0.020.

The rule trusts the model's probabilities. Ours were not calibrated: the threshold set for a miss worth 50 is even cheaper at a miss cost of 20 (62 against 107). When the probabilities are off, Elkan suggests adjusting the threshold on data [1]: use a held-out set with enough defects, never the final test set. Our validation set had only 2 corroded washers, so we used the rule as it is. It also assumes the model learned from the real defect rate; class weights or a rebalanced training set raise the defect probabilities, which must be converted back first [1].

3. Collect more defect images, without dropping good ones. We trained the same model with 54 corroded training washers instead of 14, same total. With each model's threshold picked afterwards on the test set to allow at most 30 false alarms (a fair way to compare models, not a threshold you could deploy), it caught 9 of 10 instead of 7. That is 2 washers on one draw of data: a hint, not proof. With 210 corroded and only 210 good washers, it caught 7 at the same budget, and flagged 52 good ones at the standard rule.

4. Treat class weighting and resampling as a trade-off, not a cure. They move where the model draws the line, and they also change what it learns. Class weighting gave nearly the same answers at 0.5 here (4 of 10 caught, 5 false alarms) but ranked the washers worse: allowing 30 false alarms, it caught 4, against 7 without weighting. Elkan argues that rebalancing the training data has little effect on standard Bayesian and decision-tree learners and recommends deciding from the probabilities and the costs instead [1]. For convolutional networks, Buda and colleagues found oversampling the rare class the most reliable fix in almost all the cases they tested [2]. Try each on a held-out set and keep what catches more at the false-alarm rate you can afford.

The code

This script checks any model's output the way rare defects need. It reads a CSV with one row per image: the true label and the model's probability for the defect class.

"""Check a defect classifier the way a rare defect needs: catches, false alarms and a threshold set by cost. Input: a CSV with one row per image and two columns, `label` (the true class) and `score` (the model's probability for the defect class, 0 to 1). Works with any model's output. Usage: python check_imbalance.py predictions.csv --defect corroded --miss-cost 20 --miss-cost: what one missed defect costs, counted in false alarms. The threshold 1 / (1 + miss cost) follows from the costs alone (Elkan 2001) if the scores are calibrated probabilities from a model trained on the real defect rate. Every class other than --defect counts as "other". MIT License. No warranty. """ import argparse import csv def counts(rows, defect, t): """Defects caught and good images flagged when every score >= t counts as a defect.""" tp = sum(1 for lab, s in rows if lab == defect and s >= t) fp = sum(1 for lab, s in rows if lab != defect and s >= t) return tp, fp def report(name, rows, defect, t): pos = sum(1 for lab, _ in rows if lab == defect) neg = len(rows) - pos tp, fp = counts(rows, defect, t) acc = (tp + neg - fp) / len(rows) balanced = (tp / pos + (neg - fp) / neg) / 2 print(f"{name:<34} accuracy {acc:6.1%} balanced accuracy {balanced:6.1%} " f"caught {tp} of {pos} false alarms {fp} of {neg}") def main(): ap = argparse.ArgumentParser() ap.add_argument("csv") ap.add_argument("--defect", required=True, help="name of the defect class in the label column") ap.add_argument("--miss-cost", type=float, default=10.0, help="cost of one missed defect, in false alarms") ap.add_argument("--threshold", type=float, default=0.5) args = ap.parse_args() with open(args.csv, newline="") as f: rows = [(r["label"], float(r["score"])) for r in csv.DictReader(f)] pos = sum(1 for lab, _ in rows if lab == args.defect) if pos == 0 or pos == len(rows): raise SystemExit("Need images of both the defect class and the other class(es).") print(f"{len(rows)} images, {pos} {args.defect}, {len(rows) - pos} other\n") report(f'Always "not {args.defect}"', rows, args.defect, float("inf")) report(f"Model at threshold {args.threshold:.2f}", rows, args.defect, args.threshold) # Flag when the expected cost of a miss outweighs that of a false alarm: p * miss_cost >= (1 - p). t = 1 / (1 + args.miss_cost) report(f"Threshold from costs {t:.3f}", rows, args.defect, t) print(f"\n(a missed {args.defect} costs as much as {args.miss_cost:g} false alarms)") if __name__ == "__main__": main() 

On the 300 test washers of our example it prints:

300 images, 10 corroded, 290 other Always "not corroded" accuracy 96.7% balanced accuracy 50.0% caught 0 of 10 false alarms 0 of 290 Model at threshold 0.50 accuracy 96.0% balanced accuracy 69.0% caught 4 of 10 false alarms 6 of 290 Threshold from costs 0.048 accuracy 89.7% balanced accuracy 75.3% caught 6 of 10 false alarms 27 of 290 (a missed corroded costs as much as 20 false alarms) 

For two classes, scikit-learn's balanced_accuracy_score [5] gives the same balanced accuracy, and many of its classifiers take class_weight="balanced".

What this does not show

  • The images are synthetic. Real corrosion, tarnish and lighting vary in ways a script does not capture. Treat the numbers as an illustration, not a benchmark.
  • The test set is small. With 10 corroded washers, each one is 10 points of catch rate: 4 of 10 fits a true rate anywhere from about 12% to 74% (95% interval). The 9 of 10 at a miss cost of 50 rests on one washer that scored just above the threshold. One run per setting; a different random draw of washers would move the numbers.
  • One model. We used a fast model that reads a thumbnail and simple image statistics. A deep model sees spots differently and may separate them better, but the arithmetic of accuracy and the role of the threshold stay the same.
  • The cost rule assumes calibrated probabilities. Ours were not, so the rule was a sound start but not the cheapest threshold. Check it on a held-out set with enough defects before you rely on it.

Checklist

  • [ ] Report defects caught and good parts flagged next to accuracy, per class.
  • [ ] Compare every model with "always good", and judge it at more than one threshold.
  • [ ] Write down what a miss and a false alarm cost, and set the threshold from them.
  • [ ] Keep enough defects in your test set to measure catches. With 10, each one moves the catch rate by 10 points; aim for 30 or more before you decide.
  • [ ] Collect more defect images before tuning the model.
  • [ ] Keep test images distinct from training images.

Sources

  1. C. Elkan, "The Foundations of Cost-Sensitive Learning," Proceedings of the 17th International Joint Conference on Artificial Intelligence (IJCAI), pp. 973–978, 2001. ijcai.org/proceedings/2001/0133.pdf
  2. M. Buda, A. Maki and M. A. Mazurowski, "A systematic study of the class imbalance problem in convolutional neural networks," Neural Networks, vol. 106, pp. 249–259, 2018. doi.org/10.1016/j.neunet.2018.07.011 (preprint: arXiv:1710.05381)
  3. J. M. Johnson and T. M. Khoshgoftaar, "Survey on deep learning with class imbalance," Journal of Big Data, vol. 6, article 27, 2019. doi.org/10.1186/s40537-019-0192-5 (open access)
  4. K. H. Brodersen, C. S. Ong, K. E. Stephan and J. M. Buhmann, "The Balanced Accuracy and Its Posterior Distribution," 20th International Conference on Pattern Recognition (ICPR), pp. 3121–3124, 2010. doi.org/10.1109/ICPR.2010.764
  5. F. Pedregosa et al., "Scikit-learn: Machine Learning in Python," Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011. jmlr.org/papers/v12/pedregosa11a.html · Documentation: balanced_accuracy_score, compute_class_weight
  6. The MIT License. opensource.org/license/mit

All links were checked on 21 Nov 2026. The figures and the synthetic example images were made with our own scripts.

The results come from training and testing on the synthetic example images. The full scripts are make_images.py (example data), analyze.py (the numbers), check_imbalance.py (the check) and make_figures.py (the figures). You may use the code freely under the MIT License [6]; it comes without any warranty. This article was drafted using an AI assistant and reviewed by Sathvik Dembale Krishnappagowda.

Originally published at https://perceora.ai/articles/accuracy-misses-defects/.