By Tania Alwan

"Vocal to MIDI" means estimating the pitch of a sung (or hummed) melody and writing it out as MIDI notes a DAW or soft synth can play. It's useful for rewriting a topline, building harmony parts, or feeding a melody into a plugin when you don't play keyboard fluently.

This post explains the pipeline — pitch detection, smoothing, note segmentation, quantisation — and walks through a small Python sketch with librosa and pretty_midi. No invented benchmarks; just the mechanics and the failure modes you'll hear.

The pipeline at a glance 

audio → (optional) vocal isolation → pitch track (f0 over time) → confidence / voicing mask → segment into notes → map Hz → MIDI note numbers → quantise timing (optional) → write .mid

Each stage makes assumptions. Most "bad MIDI" results come from a weak stage earlier in the chain, not from the MIDI file format itself.

Step 1: Start from a clean vocal when you can

Pitch trackers hate accompaniment. Drums, bass and dense chords create competing periodicities, so the estimated fundamental frequency (f0) jumps around.

Practical options:

  • Record a dry hum or guide vocal on its own track.
  • Isolate vocals first with Demucs / Ultimate Vocal Remover, LALAL.AI, Moises, or a hosted stems splitter if you're already in a browser workflow.
  • High-pass gently to reduce low rumble that confuses some trackers.

Isolation artefacts can still confuse pitch detection, so a purpose-recorded dry take is ideal when you have the choice.

Step 2: Pitch detection (f0 tracking)

Common algorithms you'll meet in libraries and plugins:

  • YIN / pYIN: classic time-domain / probabilistic methods; pYIN (in librosa) also estimates voicing probability.
  • CREPE: a neural pitch tracker that often handles wider pitch ranges and noisier signals, at higher compute cost.
  • DAW built-ins: Melodyne, Logic's Flex Pitch, and similar editors run their own trackers with a visual note editor on top.

Conceptually, for each analysis frame the tracker returns:

  • f0 in Hz (or an unvoiced flag)
  • sometimes a confidence score 
import librosa import numpy as np y, sr = librosa.load("vocal.wav", sr=22050, mono=True) f0, voiced_flag, voiced_probs = librosa.pyin( y, fmin=librosa.note_to_hz("C2"), fmax=librosa.note_to_hz("C6"), sr=sr, ) times = librosa.times_like(f0, sr=sr)

f0 will contain np.nan where the frame is unvoiced (breaths, consonants, silence). That is a feature, not a bug.

Step 3: Hz to MIDI, with smoothing

MIDI note numbers are logarithmic in frequency:

midi = 69 + 12 * log2(f0 / 440)

Middle A (440 Hz) is MIDI 69. Because raw f0 wiggles with vibrato and tracker noise, convert after a light smooth — or smooth in MIDI space.

def hz_to_midi(f0): out = np.full_like(f0, np.nan, dtype=float) valid = ~np.isnan(f0) & (f0 > 0) out[valid] = 69.0 + 12.0 * np.log2(f0[valid] / 440.0) return out midi_contour = hz_to_midi(f0) # simple median filter in MIDI space (ignore NaNs) def nan_median_filter(x, width=5): y = x.copy() half = width // 2 for i in range(len(x)): lo, hi = max(0, i - half), min(len(x), i + half + 1) window = x[lo:hi] window = window[~np.isnan(window)] if len(window): y[i] = np.median(window) return y midi_smooth = nan_median_filter(midi_contour, width=5)

Vibrato will still show up as a ripple around the intended note. Note segmentation has to decide whether that ripple is one note or many.

Step 4: Segment the contour into notes

A minimal rule-based segmenter:

  1. Treat frames with low voicing probability as gaps.
  2. Start a new note when the rounded MIDI value changes and stays changed.
  3. Drop notes shorter than a threshold (for example 60–80 ms) so consonants don't become MIDI chatter. 
def contour_to_notes(times, midi_smooth, voiced_probs, min_dur=0.07): notes = [] i, n = 0, len(midi_smooth) while i < n: if np.isnan(midi_smooth[i]) or voiced_probs[i] < 0.5: i += 1 continue pitch = int(np.round(midi_smooth[i])) start = times[i] j = i + 1 while j < n and not np.isnan(midi_smooth[j]) and voiced_probs[j] >= 0.5: if int(np.round(midi_smooth[j])) != pitch: break j += 1 end = times[min(j, n - 1)] if end - start >= min_dur: notes.append((start, end, pitch)) i = max(j, i + 1) return notes notes = contour_to_notes(times, midi_smooth, voiced_probs)

This is deliberately simple. Production tools add energy envelopes, onset detection and user-editable blobs (Melodyne-style) because real singing rarely sits on fixed integers.

Step 5: Quantise timing (optional) and write MIDI

Quantisation snaps note starts (and sometimes ends) to a grid. For songwriting you may want light quantisation; for a human feel, leave timing alone or quantise only to eighth notes.

import pretty_midi pm = pretty_midi.PrettyMIDI() inst = pretty_midi.Instrument(program=0) # acoustic grand  for start, end, pitch in notes: inst.notes.append( pretty_midi.Note( velocity=80, pitch=int(pitch), start=float(start), end=float(end), ) ) pm.instruments.append(inst) pm.write("vocal_melody.mid")

Load the .mid beside the original vocal in your DAW and compare. That listening check matters more than any single parameter.

Failure modes (and what to do)

  • Octave errors: trackers jump an octave on strong harmonics. Narrow fmin/fmax to the singer's range.
  • Short chatter on consonants: raise min_dur or require higher voicing probability.
  • Missing scoops and bends: hard quantisation to integers destroys expressive slides. Keep a continuous pitch lane if your DAW supports it, or edit in Melodyne instead of exporting MIDI.
  • Dense mix bleed: isolate the vocal first; don't expect miracles from a full stereo master.
  • Choir / stacked doubles: multiple pitches at once break monophonic trackers. Separate takes before converting.

Putting the MIDI to work in a DAW

Once you have vocal_melody.mid:

  • Layer, don't replace. Keep the original vocal muted under the synth while you edit. Hearing both makes octave errors obvious.
  • Velocity is flat in this sketch. Draw a simple velocity arc, or map audio RMS to velocity in a second pass.
  • Optional scale snap for pop writing — then re-listen for blue notes you meant to keep.
  • For harmony, duplicate the MIDI, transpose by 3–4 semitones, and delete clashes.

If the MIDI is only a sketch, finishing pitch in a Melodyne-style editor is often faster than fighting a brittle segmenter.

How this relates to AI music tools

Vocal-to-MIDI is analysis, not generation. It does not create a new singer; it estimates notes that already happened. That makes it a good neighbour to AI cover and stem tools rather than a replacement:

  • Isolate → pitch-track → MIDI for arrangement work.
  • Or isolate → convert timbre with an AI voice for covers, without needing MIDI at all.

Hosted platforms differ in focus. Melodyne remains the detailed editorial reference for pitch-to-notes. Browser ecosystems such as Kits.ai, Musicfy or Voicestars lean toward covers, voice libraries and stems; according to Voicestars' public pages they advertise large AI voice catalogues and credit-based plans (features and pricing may change). Use vocal-to-MIDI when you need editable notes; use voice conversion when you need a different timbre on the same performance.

Try the sketch on your own take

Record a 10-second dry melody, run the pyin → segment → pretty_midi path, and drag the MIDI onto a soft synth under the vocal. Adjust min_dur and the voicing threshold until chatter disappears. If your source is buried in a mix, isolate it first — locally with Demucs, or in the browser with a stems splitter — then pitch-track the vocal stem alone.

Disclosure: this article was written in partnership with Voicestars.