Every plant that has run SAP for more than a few years has the same part under several codes.
A taper roller bearing lives once as BRG TPR RLLR DBL ROW 510X950. It lives again as BEARING, TAPER ROLLER, ID 510MM. A third record holds only the manufacturer's part number. Each has its own stock, its own reorder point and its own purchase history.
The cost shows up as stock bought twice, and as a part reported out of stock while forty of them sat under another code.
Why the obvious fix fails
The obvious fix is fuzzy text search: find descriptions that look alike. It catches the easy cases and misses the ones that matter.
Two descriptions of the same bearing can share no words at all. Two descriptions that are nearly identical can be different parts, with one digit changed in the bore diameter.
Text similarity measures how a description was written. Duplicate detection has to compare what the part is.
A method that holds up
Turn each description into a structured record first. Then compare the records.
- Normalise the text. Expand abbreviations, convert every unit to one system, and separate the manufacturer from the description.
- Classify each record. Bearing, motor, valve, pump. The class decides which specifications matter.
- Extract the critical specifications into named fields, each with a source and a confidence. A value that cannot be read with confidence is left blank and flagged, never guessed.
- Build a canonical key from the critical specifications only. Two records with the same key are exact duplicates, however differently they were written.
- Find near-duplicates by comparing specifications within engineering tolerances.
- Review with evidence. Show a person both records, the specifications that matched and the rule that matched them. The system proposes merges. People approve them.
- Stop new duplicates at the door. Check every new code request against the cleaned master by specification before it is created.
Three kinds of duplicate
- Exact duplicates are the same part under different codes. Merge them.
- Functional duplicates are different parts that do the same job. Link them, so a buyer sees the alternative.
- Conditional substitutes would serve if a stated condition holds. Show the condition. Never substitute silently.
Treating all three the same way is how clean-ups cause damage.
Measure before you trust
A classifier that claims to be 90% sure should be right nine times in ten on a hand-checked sample of your own data. If nobody has checked, the number means nothing.
The whole pipeline can run on your own infrastructure with local models. Material data does not have to leave your network to be cleaned.
The full guide, with the traps to avoid, is at https://quantumbeetle.ai/guides/duplicate-material-codes-sap. It is the method behind MIDAS, the material intelligence system built by Quantum Beetle.
