A source separation model learns one thing: this mix is made of these parts. If the parts in your dataset do not add up to the mix, the model learns something slightly wrong on every song, and no loss curve will tell you.

It happens more often than you would think. Stems get exported with a limiter still on the master. A reverb bus is left out. One stem is at 48 kHz in a 44.1 kHz session. A producer uploads an empty track. Some "stems" on the web were never stems at all: somebody ran a finished song through a separation model and uploaded the output.

Here is a small script that checks for all of this, and how to read what it prints. It needs numpy and soundfile.

pip install numpy soundfile git clone https://github.com/ximaudio/datasets && cd datasets

The one number that matters: the sum residual

Add up all the stems of a song, sample by sample. Subtract the result from the mix. What is left is the residual. The script reports it in dB relative to the mix:

residual = 20 * log10( rms(mix - sum of stems) / rms(mix) )
  • Under about -60 dB: the stems are the mix. What is left is rounding from the bit depth.
  • Around -40 to -20 dB: something small differs. Often dither, a bus effect, or a master chain with light processing.
  • Above -20 dB: a part is missing, the mix was processed after the stems were printed, or the stems were not printed from the session at all.

Separated stems sometimes show up this way, because many separation models do not conserve the signal and their outputs do not add back up to the song. Not always: some tools compute the last stem as the mix minus the others, and those sum perfectly. So a high residual on files sold as session exports is a reason to ask questions, and a low one is not proof of anything by itself.

Try it without downloading anything

The repository has a script that writes six four-second test songs made of noise and a sine: one clean, five broken on purpose.

python3 tools/make_test_songs.py test-songs python3 tools/check_stems.py test-songs --all

This is the real output:

ok test-songs/clean stems 3 residual -119.7 dB FAIL test-songs/clip stems 4 residual 12.6 dB - 1 clipped stem(s): bass/04 - hot.wav - stems do not sum to the mix: residual 12.6 dB (best-fit gain -17.28 dB) FAIL test-songs/gain stems 3 residual -0.0 dB - stems do not sum to the mix: residual -0.0 dB (best-fit gain -6.02 dB) FAIL test-songs/missing stems 3 residual -11.5 dB - stems do not sum to the mix: residual -11.5 dB (best-fit gain -0.01 dB) FAIL test-songs/rate stems 4 residual -44.7 dB - sample rates differ: [44100, 48000] - length differs from the mix by +0.35 s: bass/04 - other rate.wav FAIL test-songs/silent stems 4 residual -119.6 dB - 1 silent stem(s): bass/04 - empty.wav 6 song(s), 5 with problems

Reading the failures

gain. The residual is 0 dB, which looks like a disaster, but the best-fit gain says -6.02 dB. The stems are right; the mix was turned down by half after they were printed. You can fix this one yourself: rebuild the mix as the sum of the stems.

missing. The best-fit gain is 0 dB and the residual is -11.5 dB. No gain change explains it, so there is audio in the mix that is in none of the stems. Usually a return bus or a track the producer forgot to export. You cannot fix this; drop the song or use the sum of the stems as the mix and accept that it is not the released song.

clip. A stem with runs of full-scale samples was clipped on export. The model will learn the distortion as part of the instrument.

rate. One file at another sample rate. If your loader does not resample per file, that stem plays at the wrong speed and pitch inside every training example.

silent. A stem with nothing in it. Harmless for the sum, but it will teach a model that "bass" can be silence for a whole song if you train with the labels.

Run it on a real set

Point it at any folder that holds a mix file and its stems, at any depth. The mix is mix.wav or mixture.wav unless you pass --mix:

python3 tools/check_stems.py /data/musdb18hq/train --all python3 tools/check_stems.py /data/my-set --all --json > report.json

The exit code is 1 when any song fails, so it can sit in a data pipeline. It reads the audio in blocks, so a song with sixty stems does not need sixty songs' worth of memory.

Two cautions. Some good datasets ship a mastered mix on purpose, and then a high residual is expected: check what the datasheet says the mix is. And a low residual does not prove the stems are clean recordings, only that they add up. Listen to a few.

Why we wrote it

I am the founder of XIM Audio. We license stems that are printed from the producer's own session, one WAV per channel, and every song ships with the sum of its stems as the mix. Our delivery build measures this residual for every song and writes it into the manifest, and we thought the check was worth more in the open than in our pipeline. The script is MIT licensed: github.com/ximaudio/datasets.

If you are choosing between open stem sets, the same repository has a table of what each one's licence says about commercial use.