I build CutVoke, an open-source (MIT) local video editor where Claude Code / Codex edit the same timeline you do, over MCP. This post is about one small piece of it: multicam audio sync, and why it refuses instead of guessing.

The boring part of a multi-angle edit

Podcast and interview editors keep describing the same week: the edit itself is fine, but lining up the camera angles, switching between them and levelling the audio eat the time. Lining up angles is mechanical work, so it is a good job for software, or for an agent driving the editor. It is also a job where a confident wrong answer is worse than no answer: a camera that is 0.3 s off looks like a dubbed film, and you may only notice it at export.

So the rule for multicam.sync in CutVoke is: align when the audio clearly agrees, and refuse to touch the timeline when it doesn't.

What it actually does

multicam.sync {referenceTrackId, targetTrackId, maxOffsetSeconds?} (source: src/cutvoke/rendering/multicam_sync.py):

  1. Reads each camera's real source audio with ffmpeg: mono, 8 kHz, up to the first 60 seconds.
  2. Turns it into an RMS energy envelope in 10 ms windows, so the time resolution is 10 ms.
  3. Runs normalized cross-correlation over lags within ±10 s by default (you can raise it to ±30 s).
  4. Takes the best score as confidence, and the best score at least 150 ms away from it as the runner-up. ambiguityMargin = best − runner-up.
  5. If confidence < 0.65 or ambiguityMargin < 0.05, it raises "Ambiguous waveform sync … align manually" and changes nothing.
  6. Otherwise it moves the target clip. If the shift would push it before timeline zero, it trims that prefix and moves the source in-point to match.

The result is returned to the caller and stored in the project: method, lagSeconds, confidence, ambiguityMargin, appliedDeltaSeconds, clippedPrefixSeconds. The sync can be undone like any other edit.

Why a margin, not just a threshold

A high peak alone isn't enough. Repetitive audio (a steady beat, a clap track, room hum) can produce several peaks of almost the same height a fraction of a second apart. Picking the tallest one is a coin toss that looks like a precise number. The margin check turns "two answers are almost equally good" into an explicit refusal, which an agent can report back ("I couldn't sync camera B, please align it by hand") instead of quietly shipping an offset.

The guard rails (stated limits)

  • One continuous, ordinary 1x clip per camera. Speed changes, freeze frames and nested clips are rejected.
  • At least two seconds of non-silent audio.
  • Sync the target camera before adding effects or keyframes to it, or the command refuses.
  • It does not fix clock drift on long recordings. It finds one offset from the first minute. Check the head and the tail after syncing.

After sync: switching and audio

  • multicam.switch {trackId, time} stores explicit cut points at timeline times (as rationals). Switching again at the same time replaces that cut. Camera source clips are not chopped up, and preview and export use the same selection.
  • Program audio stays on the initial camera and doesn't jump with the picture cuts.
  • audio.normalize runs ffmpeg loudnorm in two passes (measure, then apply with the measured values) to a file, default target −16 LUFS.

Doing it from an agent

Because these are ordinary commands on the shared timeline, Claude Code or Codex can call them over MCP and you can watch and take over in the UI. A reasonable agent loop is: create the multicam group, sync each angle, report the confidence / ambiguityMargin for each, and stop at the first refusal for a human to look at. Picking which angle to show when is still your call; CutVoke doesn't choose shots for you.

Repo: https://github.com/frankShan001/CutVoke (pre-release, desktop, MIT). If you sync multicam footage another way (timecode, slate claps, PluralEyes-style tools), I'd like to hear where audio-only sync breaks for you.