wc -l says a list of camera clips has two lines. grep, awk and sed say three. The file holds three clip names, and the while read loop that uploads them handles two.

All three answers come from one missing byte: the list does not end with a newline. A short Python script writes it for the shell loop to read:

names = ["clip_0700.mp4", "clip_0715.mp4", "clip_0730.mp4"] with open("upload.txt", "w") as f: f.write("\n".join(names)) print("bytes written:", len("\n".join(names))) 
bytes written: 41

join() puts its separator between names, not after the last one. Three names, two newlines.

Disclosure: I make an Android camera app. Nothing below uses its code or describes how it records, names or uploads anything. The clip names are made up, and every output is pasted from the scripts shown. Versions:

GNU bash, version 5.1.16(1)-release (aarch64-unknown-linux-gnu) wc (GNU coreutils) 8.32 grep (GNU grep) 3.7 GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.1.0, GNU MP 6.2.1) sed (GNU sed) 4.8 Python 3.10.12

Four tools, two answers

First, ask a few common tools how many lines the file has:

#!/bin/bash echo "wc -l: $(wc -l < upload.txt)" echo "grep -c '': $(grep -c '' upload.txt)" echo "awk END{NR}: $(awk 'END { print NR }' upload.txt)" echo "sed -n '\$=': $(sed -n '$=' upload.txt)" echo "last byte: $(tail -c 1 upload.txt | od -An -c)" 
wc -l: 2 grep -c '': 3 awk END{NR}: 3 sed -n '$=': 3 last byte: 4

wc -l says 2. grep, awk and sed say 3. The last check shows why they disagree: the final byte of the file is the character 4 from clip_0730.mp4, not a newline.

None of these tools is broken. They are counting different things, and only one of them says so in its name.

What a line is, according to POSIX

POSIX defines a line as "A sequence of zero or more non- characters plus a terminating character." The newline is part of the line, not a separator between lines.

It also has a name for what is left over at the end of this file. An incomplete line is "A sequence of one or more non- characters at the end of the file." So by the standard's own words, upload.txt holds two lines and one incomplete line.

The GNU manual for wc is just as plain about what -l does: "Print only the newline character counts. If a file ends in a non-newline character, its trailing partial line is not counted."

So wc -l is right, and it was answering the question its manual says it answers. It counts newline characters. A script that treats that number as the count of clips in the list is the thing that is wrong.

grep, awk and sed are more forgiving and treat the trailing text as a last line. That is useful, but it means a quick sanity check with one of them will agree with the eye and disagree with wc -l, and the check that happens to be in the script decides which answer you get.

The loop that skips the last clip

The upload loop is the usual safe form, with IFS= and -r:

#!/bin/bash n=0 while IFS= read -r f; do n=$((n + 1)); echo "upload $f" done < upload.txt echo "loop body ran $n times; leftover f='$f'" 
upload clip_0700.mp4 upload clip_0715.mp4 loop body ran 2 times; leftover f='clip_0730.mp4'

The body ran twice. After the loop, f still holds clip_0730.mp4, so the name was read. The body just did not get it.

The Bash manual explains this in the read entry. read does assign what it got, and then: "The exit status is zero, unless end-of-file is encountered, read times out (in which case the status is greater than 128), a variable assignment error (such as assigning to a readonly variable) occurs, or an invalid file descriptor is supplied as the argument to -u."

Reading the last name hits end-of-file before a newline, so read returns non-zero, and while takes any non-zero status as the signal to stop. Here it is with the loop removed:

#!/bin/bash printf 'clip_0730.mp4' | { IFS= read -r f; echo "no newline: status=$? f='$f'"; } printf 'clip_0730.mp4\n' | { IFS= read -r f; echo "with newline: status=$? f='$f'"; } 
no newline: status=1 f='clip_0730.mp4' with newline: status=0 f='clip_0730.mp4'

Same name, same variable, different status. The loop looks at the status and nothing else.

A non-zero status at end-of-file is how every while read loop is meant to finish, so the short pass and a complete one end the same way. There is no error to catch, which is why this kind of bug tends to be found by someone counting files by hand.

Two fixes on the reading side

If you cannot control the writer, make the reader tolerate the missing newline:

#!/bin/bash n=0 while IFS= read -r f || [ -n "$f" ]; do n=$((n + 1)); echo "upload $f" done < upload.txt echo "loop body ran $n times" mapfile -t clips < upload.txt echo "mapfile got ${#clips[@]} entries: ${clips[*]}" 
upload clip_0700.mp4 upload clip_0715.mp4 upload clip_0730.mp4 loop body ran 3 times mapfile got 3 entries: clip_0700.mp4 clip_0715.mp4 clip_0730.mp4

The -n test after || keeps the loop going for one more pass when read hit end-of-file but still put text in f. On the next pass read gets nothing, f is empty, and the loop stops. The order matters: the test has to come after read in the same condition, because it is checking what that read just assigned.

mapfile -t reads the whole file into an array and kept all three names here. The manual describes -t as "Remove a trailing delim (default newline) from each line read." It is a good fit when the list is small enough to hold in memory and the body does not need to run while the file is still being read.

The same byte breaks concatenation

Lists like this often get joined, for example a day's file appended to a weekly one. cat copies bytes and adds nothing between files:

#!/bin/bash printf 'clip_0800.mp4\nclip_0815.mp4\n' > tuesday.txt cat upload.txt tuesday.txt > week.txt while IFS= read -r f; do echo "[$f]"; done < week.txt 
[clip_0700.mp4] [clip_0715.mp4] [clip_0730.mp4clip_0800.mp4] [clip_0815.mp4]

There were four clips across the two files. The loop sees three names, and one of them is two clips glued together. An upload step would try to open a file called clip_0730.mp4clip_0800.mp4, fail on it, and if that failure is handled per file, carry on. This is worse than the skipped clip because two clips are lost from one missing byte, and the error that does appear names a file nobody made.

A check for the end of a file

tail -c 1 prints the last byte. Command substitution strips trailing newlines from what it captures, so if that byte is a newline the result is empty, and if it is anything else the result is not:

#!/bin/bash for file in upload.txt tuesday.txt; do if [ -s "$file" ] && [ -n "$(tail -c 1 "$file")" ]; then echo "$file: last line has no newline" else echo "$file: ends cleanly" fi done 
upload.txt: last line has no newline tuesday.txt: ends cleanly

The -s test skips empty files, which have no last byte to check. This is cheap enough to run before every loop over a list you did not write.

The fix on the writing side

The real fix is to have the writer end every line with a newline, including the last one:

names = ["clip_0700.mp4", "clip_0715.mp4", "clip_0730.mp4"] with open("upload.txt", "w") as f: for name in names: f.write(name + "\n")

After running that, the same counting and loop scripts agree:

wc -l: 3 grep -c '': 3 awk END{NR}: 3 sed -n '$=': 3 last byte: \n upload clip_0700.mp4 upload clip_0715.mp4 upload clip_0730.mp4 loop body ran 3 times; leftover f=''

Every tool says 3, the loop uploads all three clips, and f is empty after the loop because the last read found nothing left.

If you keep the join, write one more newline after it. The point is the same either way: in a list of lines, the newline belongs to each line, not between them.

What to change in a script like this

  • Write every line with its own newline, the last one included. In Python, add the newline to each name as you write it, or write one more after a join.
  • In Bash, read lists with the read -r loop plus its -n test, or with mapfile -t, so a list from somewhere else does not lose its last entry.
  • Do not use wc -l as a count of records unless you know the file ends with a newline. It counts newline characters, as its manual says.
  • Check the last byte before concatenating lists, or join them with a tool that adds the missing newline, rather than plain cat.
  • Compare the number of items processed with the number expected and log both. A loop that ends cleanly one item short gives no other sign.

Things I did not test

  • Other shells. I ran Bash 5.1. dash, zsh and others have their own read. The quote above is from the Bash manual, and I did not check another shell's.
  • Other versions of the tools. The wc quote is from the current GNU manual, and I ran coreutils 8.32. The output matched the quote, but I did not run another version. I did not read POSIX's page for wc itself, so I make no claim about non-GNU wc here.
  • Other awk and grep builds. I ran GNU awk and GNU grep. Other builds may treat a trailing incomplete line differently, and I did not test any.
  • Windows line endings. A list written with carriage returns has a separate problem, a stray \r on every name. I did not mix that in here.

Sources. Definitions are from POSIX.1-2024 Base Definitions, chapter 3: Definitions (3.172 Incomplete Line, 3.185 Line). The wc quote is from the GNU coreutils manual page wc invocation. The read and mapfile quotes are from the GNU Bash Reference Manual page Bash Builtins. Outputs are pasted unedited from the scripts above.

If you want an old Android phone recording in the background so there are clips for a list like this to name, the app I make is Background Camera RemoteStream on Google Play. Nothing above depends on it.

Before blaming the upload loop, look at the last byte of the list it reads: join() puts the newline between names, so the last name it writes is, to wc -l and while read, not yet a line.