How to find scene cuts with ffmpeg and split the video exactly on them

ffmpeg can find the cuts in an edited video with one filter, and most guides stop there: select='gt(scene,0.4)',showinfo, read the timestamps, done. On a real edit two things are still open afterwards. Which threshold separates cuts from camera shake, and how do you turn the timestamps into files that actually start on the cut? The video editor in the osFoundry desktop app does the first half in Rust (canvas_scene_split in canvas_operations.rs) and avoids the second half entirely. Below, both halves are rebuilt in Python and measured on one video, with the numbers that decide each choice.

The test file is NASA's public-domain launch highlights from the Artemis II test flight: 92 seconds of H.264 at 640×360, 30 fps, 2,760 frames, cut from 19 shots. All timings are from an Apple M1 with ffmpeg 7.1 and Python 3.9.

Detecting the cuts

The command is the one every guide shows. The detail that costs people an afternoon is where the output goes: showinfo writes to ffmpeg's log, which is stderr. Read stdout and you get an empty list from a run that worked.

import re, subprocess def detect(video, threshold): vf = f"select='gt(scene,{threshold})',showinfo" p = subprocess.run(["ffmpeg", "-hide_banner", "-i", video, "-an", "-filter_complex", vf, "-f", "null", "-"], capture_output=True, text=True) return [round(float(t) * 1000) for t in re.findall(r"pts_time:([0-9.]+)", p.stderr)] 

At 0.27, the app's default, this returns 18 cut times in milliseconds, from 3,133 to 73,033. The pass took 0.80 s (median of 5 runs), so detection is cheap enough to rerun with a different threshold whenever you like.

What the scene score measures

The threshold is easier to pick once you know what the number is. In libavfilter/f_select.c, get_scene_score computes the mean absolute difference between this frame and the previous one (mafd), then subtracts the previous frame's mafd:

score = min(mafd, |mafd - prev_mafd|) / 100 

For YUV input only the luma plane is compared. That makes the score easy to rebuild in a few lines of Python, which is the quickest way to understand a surprising value:

def luma(video, n): p = subprocess.run(["ffmpeg", "-v", "error", "-i", video, "-vf", f"select='eq(n,{n})'", "-frames:v", "1", "-f", "rawvideo", "-pix_fmt", "yuv420p", "-"], capture_output=True) return p.stdout[:640 * 360] # Y plane only  def mafd(a, b): return sum(abs(x - y) for x, y in zip(a, b)) / len(a) f0, f1, f2 = (luma(VIDEO, k) for k in (n - 2, n - 1, n)) prev, cur = mafd(f0, f1), mafd(f1, f2) score = min(cur, abs(cur - prev)) / 100 

Taking the Y plane from yuv420p rather than asking for -pix_fmt gray matters: the gray conversion stretches the value range and the rebuilt scores came out about 16 % too high. With the raw Y plane, four frames from this video match ffmpeg's own scores:

The subtraction is what keeps moving shots from firing. The ignition frame differs from the one before it by 27 brightness levels on average, twice as much as the frame after the crowd cut. But the frame before it had already changed by 10.9, so only the increase counts, and it scores 0.16. The cut from the pad to the sky changes less than many cuts do, but it follows a nearly still frame, so it scores 0.34. A cut is a jump in change, not just a lot of change.

Picking the threshold from the data

Instead of guessing, dump the score of every frame once and look at where the gap is:

ffmpeg -i launch.mp4 -an -vf "select='gte(scene,0)',metadata=print:file=-" -f null - \ | grep -o 'scene_score=[0-9.]*' 

The median frame of this video scores 0.0015. Seventeen frames score above 0.5, one scores 0.337, and the next highest is the ignition frame at 0.161. Any threshold between 0.162 and 0.337 returns the same 18 cuts. The sweep, with and without a 500 ms minimum gap between cuts:

Most of the 0.3–0.5 range that the ffmpeg mailing list and many tutorials suggest is too high for this video: at 0.35 and above, the pad-to-sky cut is gone. At 0.15 and below, the ignition starts producing cuts. On footage with a lot of fast motion the safe band can be narrow, so it's worth one look at the distribution before trusting a number from someone else's video.

The minimum gap

The rows at 0.08 to 0.12 show why the app drops hits that come too soon after the previous one (parse_scene_times, 500 ms by default). At 0.12, the cut at 33.300 s fires again on the next frame at 33.333 s, because that frame still differs from its neighbour by 13.75 while the jump before it was huge. At 0.08, the ignition produces eight hits between 57.7 s and 59.0 s. A gap of half a second collapses each burst into one marker:

def min_gap(hits, gap_ms=500): out = [] for ms in hits: if not out or ms - out[-1] >= gap_ms: out.append(ms) return out 

It doesn't fix a low threshold, only the duplicates. At 0.08 the gap still leaves two false cuts in the ignition.

Splitting with -c copy doesn't land on the cuts

The obvious next step is ffmpeg's segment muxer with stream copy:

ffmpeg -i launch.mp4 -c copy -f segment -segment_times 3.133,6.333,... \ -reset_timestamps 1 scene%02d.mp4 

It took 0.05 s and produced 18 files, not 19, all between 3.3 and 6.0 seconds long. The scenes themselves run from 1.5 to 19.0 seconds. A stream copy can only start a file on a keyframe, and this encode has 18 of them, at most 6 seconds apart. ffprobe -skip_frame nokey lists them:

ffprobe -v error -select_streams v -skip_frame nokey \ -show_entries frame=pts_time -of csv=p=0 launch.mp4 

Only 5 of the 18 cuts fall on a keyframe, the ones where the original encoder happened to place one. Every other file boundary comes late, the worst by about 4 seconds, and where two cuts share the same next keyframe they collapse into one boundary.

Re-encoding, and the half-frame trap

Re-encoding with a keyframe forced at each cut makes the boundaries free to land anywhere:

ffmpeg -i launch.mp4 -c:v libx264 -crf 18 -preset fast -c:a aac \ -force_key_frames "$T" -f segment -segment_times "$T" \ -reset_timestamps 1 scene%02d.mp4 

With $T set to the detected times rounded to milliseconds, this produced 19 files, and 4 of the 18 cuts were still one frame late. One cut was missed entirely, merging two scenes into a 12.3-second file, and a stray file held a single frame. The cause is rounding. A cut at frame 269 is at 8.9666… s, and "8.967" lies just after it, so the comparison picks the next frame. All four late cuts sit at times ending in a repeating 6, and so does the stray frame at 44.067 s.

Moving every time half a frame earlier (1/60 s at 30 fps) puts it unambiguously before the cut frame:

T = ",".join(f"{ms / 1000 - 0.5 / 30:.4f}" for ms in cuts) 

With that change, all 18 boundaries landed on the cut frame and the 19 file lengths matched the scene lengths to the frame. The re-encode took 6.9 s for the 92-second clip and wrote 15.2 MB, against 18.2 MB for the source at CRF 18.

Or don't write files at all

The app takes the third route. Its Split at buttons call canvas_split_clip in canvas_layers.rs, which shortens the selected clip and inserts a new clip on the same asset with a later in-point. Nothing is decoded or written, so there is no keyframe problem and no generation loss, and the encode happens once, when the finished edit is exported. If the goal is to recut the video rather than to hand someone 19 files, that's the version to copy: keep the cut list as in/out points and let the final render do the frame-accurate work.

The full script, with every measurement above, is about 120 lines of standard-library Python around ffmpeg and ffprobe. In the osFoundry desktop app the same detection is the Find the cuts preset in the video editor, and it runs on your own machine as described on the local-first page. The app is free from the osFoundry download page.