A three-line script that prints the oldest clip in a folder passed every test I gave it with a small folder. With 1000 clips it failed 22 times out of 100. With 2000 clips it failed every time. Each failure was exit status 141, with nothing on stderr and nothing on stdout.

It ran under set -euo pipefail, a common first line for scripts like this, and its pipeline ended in head -n 1. This post is about what those two do together, why the failure comes and goes with folder size, and how to keep pipefail anyway.

Disclosure: I make an Android camera app. Nothing below uses its code or describes how it stores files. Every result comes from the scripts shown, run on this machine:

GNU bash, version 5.1.16(1)-release (aarch64-unknown-linux-gnu) sort (GNU coreutils) 8.32 head (GNU coreutils) 8.32 find (GNU findutils) 4.8.0 CPUs: 2

The script

The folder is fake: empty files, each touched one minute after the last, so the oldest is cam1_00001.mp4.

#!/bin/bash # mk.sh N : make N empty clip files, one minute apart, oldest first. n=$1 rm -rf clips && mkdir clips for i in $(seq 1 "$n"); do f=$(printf 'clips/cam1_%05d.mp4' "$i") : > "$f" touch -d "@$((1761000000 + i * 60))" "$f" done

The script asks find for each clip's modification time and path, sorts by time, and keeps the first line:

#!/bin/bash set -euo pipefail # Print the oldest clip, so a cleanup job knows where to start. oldest=$(find clips -name '*.mp4' -printf '%T@ %p\n' | sort -n | head -n 1) echo "oldest: ${oldest#* }"

How often it fails

This runs the script 100 times per folder size and counts non-zero exits. list_bytes is what sort tries to hand to head.

#!/bin/bash # Run oldest.sh 100 times per folder size and count the non-zero exits. for n in 50 90 100 200 500 1000 1500 2000 3000; do ./mk.sh "$n" bytes=$(find clips -name '*.mp4' -printf '%T@ %p\n' | wc -c) fails=0 for k in $(seq 1 100); do ./oldest.sh > /dev/null 2>&1 || fails=$((fails + 1)) done echo "clips=$n list_bytes=$bytes failed=$fails/100" done

Output:

clips=50 list_bytes=2150 failed=0/100 clips=90 list_bytes=3870 failed=0/100 clips=100 list_bytes=4300 failed=3/100 clips=200 list_bytes=8600 failed=6/100 clips=500 list_bytes=21500 failed=4/100 clips=1000 list_bytes=43000 failed=22/100 clips=1500 list_bytes=64500 failed=30/100 clips=2000 list_bytes=86000 failed=100/100 clips=3000 list_bytes=129000 failed=100/100

Three bands. Up to 90 clips: 0 failures. From 100 to 1500 clips, it failed some of the time, and an earlier run gave different counts in that band. From 2000 clips up, it failed every time.

The middle band is the bad one: a job tested on a few dozen clips works, gets scheduled, and then fails on some nights and not others once the folder fills up.

What 141 is

The Bash manual: "When a command terminates on a fatal signal whose number is N, Bash uses the value 128+N as the exit status." 141 minus 128 is 13, and signal 13 is:

#!/bin/bash ./mk.sh 3000 ./oldest.sh echo "oldest.sh exit status: $?" kill -l 13 set -o pipefail find clips -name '*.mp4' -printf '%T@ %p\n' | sort -n | head -n 1 > /dev/null echo "PIPESTATUS: ${PIPESTATUS[*]}"

Output:

oldest.sh exit status: 141 PIPE PIPESTATUS: 0 141 0

The first line is all oldest.sh printed: its echo did not run, so no partial result and no message. PIPESTATUS shows which part died: find 0, sort killed by SIGPIPE, head 0. head had already printed the wanted line.

head -n 1 reads until it has one line, prints it, and exits. Its end of the pipe closes. The Linux pipe man page says what happens to the process still writing: "If all file descriptors referring to the read end of a pipe have been closed, then a write(2) will cause a SIGPIPE signal to be generated for the calling process." Its default action ends the process. Nothing was wrong with the data; sort was still talking when head hung up.

Without pipefail nobody notices: the pipeline takes head's status, 0. With it, per the manual, "the return value of a pipeline is the value of the last (rightmost) command to exit with a non-zero status". That is sort's 141, and set -e ends the script.

Why the folder size decides it

Whether sort gets the signal depends on whether it still has a write to make after head has gone. Tracing sort's writes on a 1000-clip list:

#!/bin/bash ./mk.sh 1000 find clips -name '*.mp4' -printf '%T@ %p\n' > list.txt strace -e trace=write -o sort.trace sort -n list.txt | head -n 1 > /dev/null cut -c1-72 sort.trace

Output:

write(1, "1761000060.0000000000 clips/cam1"..., 4096) = 4096 write(1, "0000000000 clips/cam1_00096.mp4\n"..., 4096) = -1 EPIPE (Broke --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=8043, si_uid=5924 --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_TKILL, si_pid=8043, si_uid=592 +++ killed by SIGPIPE +++

sort writes in 4096-byte blocks. Slowed by strace, its second write found head gone. Without the tracer it is a race:

  • At most one block (the 50 and 90 clip lists are 2150 and 3870 bytes). sort makes one write and is done before head can close anything. 0 failures.
  • More than one block, but less than the pipe holds. The pipe man page gives its capacity: "Since Linux 2.6.11, the pipe capacity is 16 pages (i.e., 65,536 bytes in a system with a page size of 4096 bytes)." Every block fits, so sort can usually write them all before head exits. Usually. When the scheduler runs head first, the next write hits a closed pipe.
  • More than the pipe holds (the 2000 clip list is 86000 bytes). sort has to wait for head to read before it can write the rest. head reads the first line and leaves. The next write hits a closed pipe, every run.

The man page adds "Applications should not rely on a particular capacity". Your thresholds will differ with CPU count, load and versions. The three bands will not.

The quieter version: grep -q in an if

grep -q stops reading at the first match, for the same reason head stops at one line. Put it in an if:

#!/bin/bash set -euo pipefail # Is there at least one clip from camera 1 in the folder? if find clips -name '*.mp4' | grep -q 'cam1_'; then echo "cam1 clips found" else echo "no cam1 clips" fi

Run 100 times on a folder of 10 clips, then on a folder of 3000, all of them from camera 1:

#!/bin/bash for n in 10 3000; do ./mk.sh "$n" wrong=0 for k in $(seq 1 100); do ./grepq.sh | grep -q 'no cam1' && wrong=$((wrong + 1)); done echo "clips=$n said 'no cam1 clips' $wrong/100 times" done

Output:

clips=10 said 'no cam1 clips' 0/100 times clips=3000 said 'no cam1 clips' 100/100 times

With 3000 matching clips it said there were none, every time. grep -q matched and exited 0; find, still listing, was killed by SIGPIPE; pipefail made the status 141; if read that as false.

set -e does not save you here. The manual lists the cases where it does not exit, and one of them is a command that is "part of the test in an if statement". So the script runs the wrong branch and exits 0. If the else branch sends an alert that camera 1 stopped recording, it sends it every time the folder is large.

Same script, different status: when the parent ignores SIGPIPE

A process can be set to ignore SIGPIPE, and child processes inherit that. Then, per the pipe man page, the write "fails with the error EPIPE" instead, and the program decides what to do. Python ignores SIGPIPE itself. Its subprocess documentation says: "If restore_signals is true (the default) all signals that Python has set to SIG_IGN are restored to SIG_DFL in the child process before the exec." Run the script both ways from Python:

import subprocess # Python ignores SIGPIPE itself. restore_signals decides whether the child gets it back. subprocess.run(["./mk.sh", "3000"], check=True) for restore in (True, False): p = subprocess.run(["./oldest.sh"], capture_output=True, text=True, restore_signals=restore) print("restore_signals=%s exit=%d stdout=%r" % (restore, p.returncode, p.stdout)) print(" stderr=%r" % p.stderr)

Output:

restore_signals=True exit=141 stdout='' stderr='' restore_signals=False exit=2 stdout='' stderr="sort: write failed: 'standard output': Broken pipe\nsort: write error\n"

With the default, the same silent 141. With the signal left ignored, sort gets the error, prints Broken pipe, and exits 2. One script, one folder: 141 and silence from a terminal, 2 and a write error from a parent that leaves SIGPIPE ignored. I only tested Python as the parent; check other launchers before reading a 2 as a disk problem.

Fix one: a last command that reads everything

A reader that reads to the end of its input cannot cut anything off. Here awk keeps the smallest timestamp it has seen and prints that line at the end, so no sort is needed:

#!/bin/bash set -euo pipefail dir="${1:-clips}" # awk reads every line before it prints, so nothing upstream is cut off. oldest=$(find "$dir" -name '*.mp4' -printf '%T@ %p\n' | awk 'NR == 1 || $1 < min { min = $1; line = $0 } END { print line }') echo "oldest: ${oldest#* }"

The tempting fix is to delete pipefail. Here is the same pipeline as oldest.sh without it, run against the 3000-clip folder and against a folder that does not exist:

#!/bin/bash set -eu dir="${1:-clips}" # The same pipeline as oldest.sh, with pipefail removed. oldest=$(find "$dir" -name '*.mp4' -printf '%T@ %p\n' | sort -n | head -n 1) echo "oldest: ${oldest#* }" 
#!/bin/bash ./mk.sh 3000 for s in oldest_awk.sh oldest_nopf.sh; do fails=0 for k in $(seq 1 100); do ./$s > /dev/null 2>&1 || fails=$((fails + 1)); done echo "$s on 3000 clips: failed=$fails/100, prints: $(./$s)" done echo "--- a folder that does not exist ---" ./oldest_awk.sh no_such_dir; echo "oldest_awk.sh exit status: $?" ./oldest_nopf.sh no_such_dir; echo "oldest_nopf.sh exit status: $?"

Output:

oldest_awk.sh on 3000 clips: failed=0/100, prints: oldest: clips/cam1_00001.mp4 oldest_nopf.sh on 3000 clips: failed=0/100, prints: oldest: clips/cam1_00001.mp4 --- a folder that does not exist --- find: ‘no_such_dir’: No such file or directory oldest_awk.sh exit status: 1 find: ‘no_such_dir’: No such file or directory oldest: oldest_nopf.sh exit status: 0

Both pass on the big folder, 0 failures in 100. On the missing folder, the awk version stops with status 1, because pipefail still counts find's failure. The other prints oldest: with nothing after it and exits 0, and a cleanup job reading that line carries on with an empty path. Removing pipefail trades a false failure for a false success, and the false success is the worse of the two.

Fix two: let the first command stop itself

To ask whether there is at least one match, tell find to stop at the first match with -quit. There is no pipe, so there is nothing left writing into a closed one, and the assignment keeps find's own status where set -e can see it:

#!/bin/bash set -euo pipefail # find stops at the first match itself, so nothing is left writing into a closed pipe. first=$(find clips -name 'cam1_*.mp4' -print -quit) if [ -n "$first" ]; then echo "cam1 clips found, e.g. $first" else echo "no cam1 clips" fi 
#!/bin/bash for n in 10 3000; do ./mk.sh "$n" wrong=0 for k in $(seq 1 100); do ./grepq_fix.sh | grep -q 'no cam1' && wrong=$((wrong + 1)); done echo "clips=$n said 'no cam1 clips' $wrong/100 times" done

Output:

clips=10 said 'no cam1 clips' 0/100 times clips=3000 said 'no cam1 clips' 0/100 times

Right answer on both folders, every time.

Checklist

  • Search for pipefail scripts whose pipelines end in head, grep -q, grep -m or anything else that stops reading early.
  • Read 141 in a log as an upstream command still writing when the reader left, not as a crash.
  • Test with more output than one pipe holds, not a handful of files.
  • Prefer a last command that reads to the end, or a first command that stops itself, over removing pipefail.
  • Check what an if around such a pipeline does; set -e will not stop it.
  • If a job acts differently under a supervisor, check whether it starts with SIGPIPE ignored.

Things I did not test

  • Other shells. This is bash 5.1; other shells may lack pipefail or report signals differently.
  • Other sort builds. The 4096-byte block is GNU coreutils 8.32 here. Other builds may write other sizes, which moves the bands.
  • Real clips. The files are empty; only the length of the list matters here.
  • cron or systemd. I did not check whether either starts jobs with SIGPIPE ignored.
  • A loaded machine. I only ran on an idle one.

Sources. Quotes are from the Bash manual (Exit Status section and The Set Builtin), the Linux pipe(7) man page, and Python's subprocess documentation. Outputs are pasted unedited.

If you want an old phone recording clips in the background so there is a folder like this to clean up, the app I make is Background Camera RemoteStream on Google Play. Nothing above depends on it.

Exit 141 under pipefail means the reader left early, not that the writer broke. Fix the reader.