A script that records or encodes camera clips all night usually logs as it goes, often with nothing fancier than >> rec.log on the loop. Sooner or later that log gets big enough that someone rotates it while the script is still running, by hand with mv or with a logrotate rule. Whether the next line lands in the new log, the old one, or nowhere depends on how the writer opened the file, not on how carefully the rotation was done.

I ran one stand-in writer three ways and rotated it four ways. Renaming the log, even with a fresh empty file put in its place, sent every later line into the renamed file. Copy-then-truncate kept every line in my fast runs and lost 17 of them once the copy was slowed a little. Copy-then-truncate under a writer opened with > instead of >> produced a log whose first 2580 bytes were zeros.

Disclosure: I make an Android camera app. Nothing below uses its code or describes how it works. The writer is a shell loop that prints numbered lines, and every output is pasted from the script shown. Versions:

GNU bash, version 5.1.16(1)-release (aarch64-unknown-linux-gnu) cp (GNU coreutils) 8.32 Linux 6.8.0-138-generic ext2/ext3 

The writer, the rotations, and the count

The writer prints 600 numbered lines, one every 10 ms, so a run takes about six seconds. About three seconds in, the script rotates the log, then waits for the writer to finish and counts what ended up in rec.log.1 (the rotated file) and rec.log (the name the writer was told to use). Every line is numbered, so the last check can say exactly how many reached neither file.

The three writers differ only in where the redirect sits:

  • append: the whole loop is redirected once with >>, so the file is opened once and held for the whole run. This is the common way to log a loop.
  • trunc: the same, but with >.
  • perline: the >> is on the printf inside the loop, so the file is opened, written and closed again for every line.

The four rotations:

  • mv: rename the log and do nothing else.
  • mv+create: rename it and create a new empty file under the old name, which is what the create directive in logrotate does.
  • copytruncate: copy the log to the rotated name, then truncate the original to zero, which is what the copytruncate directive does.
  • copytruncate-slow: the same with a 0.2 second pause between the copy and the truncate, standing in for a copy that takes a moment because the log is large or the disk is busy.
#!/usr/bin/env bash # Rotate a log while the script writing it is still running, six ways. # The writer prints 600 numbered lines, one every 10 ms. Rotation happens # about 3 seconds in. Afterwards, count what ended up where. set -u writer() { # $1: append | trunc | perline case $1 in append) ( for i in $(seq 1 600); do printf 'line %04d\n' "$i"; sleep 0.01; done ) >> rec.log ;; trunc) ( for i in $(seq 1 600); do printf 'line %04d\n' "$i"; sleep 0.01; done ) > rec.log ;; perline) for i in $(seq 1 600); do printf 'line %04d\n' "$i" >> rec.log; sleep 0.01; done ;; esac } rotate() { # $1: mv | mv+create | copytruncate | copytruncate-slow case $1 in mv) mv rec.log rec.log.1 ;; mv+create) mv rec.log rec.log.1; : > rec.log ;; copytruncate) cp rec.log rec.log.1; : > rec.log ;; copytruncate-slow) cp rec.log rec.log.1; sleep 0.2; : > rec.log ;; esac } report() { local f n nul for f in rec.log.1 rec.log; do if [ -e "$f" ]; then n=$(grep -ac '^line' "$f"); nul=$(tr -dc '\0' < "$f" | wc -c) printf ' %-9s %6d bytes %3d lines %5d NUL bytes\n' "$f" "$(stat -c %s "$f")" "$n" "$nul" else printf ' %-9s (does not exist)\n' "$f" fi done missing=$(comm -23 <(seq -f 'line %04g' 1 600) \ <(cat rec.log.1 rec.log 2>/dev/null | tr -d '\0' | sort -u) | wc -l) echo " lines missing from both files: $missing" } run() { # $1 writer style, $2 rotation local d; d=$(mktemp -d); cd "$d" writer "$1" & local w=$! sleep 3; rotate "$2"; wait "$w" echo "writer=$1 rotation=$2"; report; echo cd /; rm -rf "$d" } run append mv run append mv+create run perline mv+create run append copytruncate run append copytruncate-slow run trunc copytruncate 

Output:

writer=append rotation=mv rec.log.1 6000 bytes 600 lines 0 NUL bytes rec.log (does not exist) lines missing from both files: 0 writer=append rotation=mv+create rec.log.1 6000 bytes 600 lines 0 NUL bytes rec.log 0 bytes 0 lines 0 NUL bytes lines missing from both files: 0 writer=perline rotation=mv+create rec.log.1 2510 bytes 251 lines 0 NUL bytes rec.log 3490 bytes 349 lines 0 NUL bytes lines missing from both files: 0 writer=append rotation=copytruncate rec.log.1 2520 bytes 252 lines 0 NUL bytes rec.log 3480 bytes 348 lines 0 NUL bytes lines missing from both files: 0 writer=append rotation=copytruncate-slow rec.log.1 2600 bytes 260 lines 0 NUL bytes rec.log 3230 bytes 323 lines 0 NUL bytes lines missing from both files: 17 writer=trunc rotation=copytruncate rec.log.1 2580 bytes 258 lines 0 NUL bytes rec.log 6000 bytes 341 lines 2580 NUL bytes lines missing from both files: 0 

Renaming: the writer follows the file, not the name

With mv alone, all 600 lines are in rec.log.1 and rec.log does not exist. With mv+create, all 600 are still in rec.log.1, and the new rec.log sits at 0 bytes for the rest of the run.

That is how renaming works on Linux. The writer opened the file once, when the loop started, and from then on it writes to that open file, whatever it is called. Renaming changes the name in the directory, not the file the writer holds. Creating a new file under the old name makes a second, unrelated file that nothing is writing to. The logrotate manual expects this: its delaycompress directive exists because a program "might continue writing to the previous log file for some time."

The usual fix is to tell the program to reopen its log after the rename, which is what the postrotate scripts in the logrotate examples do by sending a signal. A plain shell loop has no handler for that, so for a shell writer the useful fix is in the next row.

Reopening on every line

The perline writer survived the same mv+create rotation: 251 lines before the rename went into rec.log.1, the other 349 went into the new rec.log, and nothing was lost. Each printf ... >> rec.log opens the file by name, so the line after the rename finds the new file.

The cost is an open and close for every line, which for a loop that logs a few lines a minute is nothing. For a writer that logs thousands of lines a second it would matter, and I did not measure that.

The same idea works without moving the redirect onto every command: group the work for one clip in { ...; } >> rec.log, so the file is reopened once per clip instead of once per line. That version is not in the measured runs above.

copytruncate: the window is real

With copytruncate and the >> writer, the rotated file got 252 lines, the original 348, and no line was missing. That fits what open(2) says about append mode: "Before each write(2), the file offset is positioned at the end of the file, as if with lseek(2)." After the truncate, the end of the file is position zero, so the next line starts the new log cleanly.

It looks safe, and in these runs the window between cp finishing and the truncate was too short for a line to slip through. The logrotate manual does not promise that: "Note that there is a very small time slice between copying the file and truncating it, so some logging data might be lost."

Widening that window to 0.2 seconds shows what lands in it. The writer kept appending after cp had finished reading, and the truncate then threw those lines away. 17 lines are in neither file. A second full run of the same script lost 16. The exact count depends on timing; a line written between the copy and the truncate is lost in every run.

copytruncate under a > writer: zeros at the top

The trunc writer opened its file with >, which the Bash manual describes as opening it "for writing" and truncating it if it exists. It is not append mode, so the writer keeps its own position in the file and does not move to the end before each write.

After copytruncate, the original was truncated to zero, but the writer's position was still 2580 bytes in. Its next line went there. The result is a rec.log of 6000 bytes, the size of a full uninterrupted run, starting with 2580 NUL bytes, exactly the size of the rotated copy. grep -c counts 341 lines in it, but 342 were written there: the first one after the zeros has no newline before it, so it does not start a line, and grep misses it. The missing-lines check strips the zeros first, which is why it reports nothing lost.

The second run printed 2560 for both numbers, and a third printed 2530. In every run the zeros matched the size of the copy, wherever the rotation landed.

An earlier test in this series, rm Freed Nothing and df Stayed Full, truncated a deleted clip through /proc under a writer in plain write mode and got the same kind of hole, and it left append mode untested. The rows above fill that gap: under >> the truncate left no zeros in any run, and under > the zeros were exactly as long as what the writer had written before the truncate.

The zeros also change how tools read the file. In a separate quick check on a log rotated the same way, GNU grep 3.7 without -a printed only grep: rec.log: binary file matches instead of the lines. Before trusting a log that was rotated with copytruncate, check which redirect its writer used.

What I would do with a long-running script

  • Shell loop logging to a file: use >> rather than >, so that a truncate cannot leave a hole.
  • Rotating it by renaming: move the redirect onto the commands inside the loop, or onto a per-clip group, so the writer reopens the name.
  • Rotating with copytruncate: acceptable for a log that can afford to lose a few lines, and only with a >> writer.
  • A program that can reopen its log on a signal: rename and signal. Nothing is lost and nothing is truncated under it.
  • Check the result once: after the first rotation, look at whether the new file grows and whether the old one stopped. In the rename runs, the visible sign of trouble was a rotated file that kept growing next to a new one stuck at 0 bytes.

Things I did not test

  • logrotate itself. The rotations here are the steps its create and copytruncate directives describe, done by hand with mv, cp and a truncate. I did not run logrotate against the writer.
  • Programs with their own log handling. Many daemons and some encoders open, buffer and reopen their logs on their own terms. The writers here are plain shell redirects with no buffering of their own.
  • Large logs. The logs here are a few kilobytes. The slow variant only imitates a long copy with a pause.
  • Other filesystems. stat -f reported the test directory as ext2/ext3. Network filesystems treat append mode differently, and the open(2) page has a warning about that which I did not test.

Sources. logrotate quotes are from the Debian manual page logrotate(8) for logrotate 3.22.0. Redirection quotes are from the GNU Bash manual page Bash Reference Manual: Redirections. The append-mode quote is from the Linux open(2) manual page on man7.org. Outputs are pasted unedited from the script above.

If an old Android phone is the camera behind the clips that an overnight script works through, the app I make for keeping it recording in the background is Background Camera RemoteStream on Google Play. Nothing above depends on it.

A writer that holds its log open keeps writing to that file after a rename, so rotate a shell loop's log by making the loop reopen the name, and do not copytruncate a log the writer opened with >.