A long job started by hand over a remote shell, such as an encode of a night's camera clips or a loop that pulls frames off a phone, usually gets the same ending: type exit, close the laptop, come back to the output. Whether it is still running depends less on how the job was started than on how the terminal went away.
I ran one stand-in loop five ways, then ended the session three ways. A plain cmd & job survived when I typed exit and died when the terminal was closed under the shell. Typing exit in a login shell with one option set killed it too. The other four ways of starting it, nohup, disown, disown -h and setsid, came through all three endings.
Disclosure: I make an Android camera app. Nothing below uses its code or describes how it works. The loop writes one timestamp per second to a log file, and every output is pasted from the scripts shown. Versions:
GNU bash, version 5.1.16(1)-release (aarch64-unknown-linux-gnu) nohup (GNU coreutils) 8.32 setsid from util-linux 2.37.2 Linux 6.8.0-138-generic The loop and the five ways to start it
The stand-in job. A real encode would hold a file open for hours; this one appends a line a second, so a dead job shows up as a log that stops growing.
#!/bin/bash # stand-in for a long recording or encode loop: one line per second while :; do date +%s >> "$1.log"; sleep 1; done To control exactly how the terminal ends, the test opens a fresh pseudo-terminal from Python, runs an interactive bash in it, and types into it. Closing the Python end of that terminal is the same event a terminal window or sshd produces when it goes away: the shell is left holding a terminal with nothing on the other side.
# usage: python3 hangup.py close|exit [login] import os, pty, sys, time mode, login = sys.argv[1], len(sys.argv) > 2 pid, fd = pty.fork() # a fresh terminal with bash in it if pid == 0: os.execvp("bash", ["bash", "--norc", "-i"] + (["-l"] if login else [])) def type_line(s): os.write(fd, (s + "\n").encode()); time.sleep(0.6) try: os.read(fd, 65536) except OSError: pass type_line("cd /tmp/hup") if login: type_line("shopt -s huponexit") type_line("./rec.sh plain &") type_line("nohup ./rec.sh nohup &") type_line("./rec.sh disowned & disown") type_line("./rec.sh marked & disown -h") type_line("setsid ./rec.sh setsid &") time.sleep(2) if mode == "exit": type_line("exit") os.close(fd) # the terminal goes away time.sleep(1); os.waitpid(pid, 0) Each of the five jobs gets its own name, and a checker reports which are still running:
#!/bin/bash for j in plain nohup disowned marked setsid; do if pgrep -f "rec.sh $j\$" >/dev/null; then s=alive; else s=dead; fi printf '%-9s %s\n' "$j" "$s" done Three endings
#!/bin/bash cd /tmp/hup stop() { pkill -f '^/bin/bash ./rec.sh'; sleep 1; rm -f ./*.log nohup.out; } stop echo "== 1. terminal closed under the shell" python3 hangup.py close; sleep 3; ./check.sh a=$(wc -l < nohup.log); b=$(wc -l < plain.log); sleep 4 echo "lines written in the next 4 s: nohup $(( $(wc -l < nohup.log) - a )), plain $(( $(wc -l < plain.log) - b ))" ls -l nohup.out | cut -d' ' -f1,5,9 stop echo "== 2. typed exit, then terminal closed" python3 hangup.py exit; sleep 3; ./check.sh stop echo "== 3. login shell, shopt -s huponexit, typed exit" python3 hangup.py exit login; sleep 3; ./check.sh stop == 1. terminal closed under the shell plain dead nohup alive disowned alive marked alive setsid alive lines written in the next 4 s: nohup 4, plain 0 -rw------- 0 nohup.out == 2. typed exit, then terminal closed plain alive nohup alive disowned alive marked alive setsid alive == 3. login shell, shopt -s huponexit, typed exit plain dead nohup alive disowned alive marked alive setsid alive Ending 1: the terminal closes under the shell
This is the dropped connection, the closed laptop lid or the terminal window clicked shut. Only plain died, and its log stopped dead while nohup's kept growing.
The Bash manual describes the sequence. "The shell exits by default upon receipt of a SIGHUP. Before exiting, an interactive shell resends the SIGHUP to all jobs, running or stopped." A plain background job is in that list, has the default action for the signal, and goes down with the shell.
The other four each avoid the resend differently:
-
nohupdoes not stop the shell from sending the signal. It makes the command ignore it. The coreutils manual: "nohup runs the given command with hangup signals ignored, so that the command can continue running in the background after you log out." -
disowntakes the job off the shell's list, so there is nothing to resend to.disown -hkeeps it on the list with a mark. The Bash manual gives both: "remove it from the jobs table with the disown builtin (see Job Control Builtins) or mark it not to receive SIGHUP using disown -h." -
setsidstarts the loop in a new session with no controlling terminal, so the loop was not one of this shell's jobs. Bash still prints a job number for the&, but that job issetsiditself. In a separate check,jobsreported itDonea second later whilepgrepstill found the loop running.
Ending 2: typing exit
All five survived, plain included. This is the case that makes cmd & look safe. Typing exit ends the shell normally; it does not receive a SIGHUP, so it has nothing to pass on. The terminal that closed a moment later had no shell left to signal the jobs.
That makes the habit fragile in a specific way. The same command, started the same way, lives or dies depending on whether the session ended with a typed exit or with the network, the lid or the window. Testing it once with exit proves nothing about the night the connection drops.
Ending 3: a login shell with huponexit
The third run starts bash -l and sets one option first. plain died on a typed exit. The manual: "If the huponexit shell option has been set using shopt (see The Shopt Builtin), Bash sends a SIGHUP to all jobs when an interactive login shell exits."
shopt huponexit printed off in the shell I tested, so it has to be turned on, but a startup file on a shared or managed server can set it. A remote login over ssh normally starts a login shell, so on a machine with huponexit in its profile, ending 2 behaves like ending 1.
What nohup does that I did not ask for
After the first run, nohup.out sat in the working folder, empty and readable only by me. The loop writes its own log and prints nothing, so the file has no content, but nohup created it anyway because standard output was the terminal at launch. The manual: "If standard output is a terminal, the command's standard output is appended to the file nohup.out". On permissions it says: "Any nohup.out or $HOME/nohup.out file created by nohup is made readable and writable only to the user, regardless of the current umask settings." That matches the -rw------- above.
Two consequences for a long job:
- A
nohup.outappears wherever the job was started, and every laternohupfrom that folder appends to it. On a job that prints progress, that file grows for as long as the job runs. - Redirecting the output yourself avoids it. The manual's own example is
nohup make > make.log.
nohup also does not put the job in the background; the manual says the & has to be added by hand. Without the &, the prompt does not come back until the job ends, and the command line sits waiting.
The pattern I kept
- Start a long job with output redirected and hangups ignored:
nohup ./encode.sh > encode.log 2>&1 &. That covers all three endings above and leaves no straynohup.out. - Rescue a job already running as plain
&:disown -h %1, ordisown %1to drop it from the list. That is the fix for the case where the job was started before anyone thought about the connection. - Detach completely:
setsidwith output redirected. The job is not tied to the shell or its terminal at all, which also meansjobs,fgandkill %1cannot reach it. It needs apgrepor a pid file to find later. - Check instead of assuming: after reconnecting, look for the log still growing rather than for the process name. A log that stopped is the earliest sign the job did not make it.
Things I did not test
- A real
sshddisconnect. The terminal here was closed from Python. Closing the controlling terminal is the event that sends the shell its SIGHUP, but I did not pull a network cable on a live SSH session. -
tmuxandscreen. Both keep the shell itself alive after the connection goes, which is a different approach from protecting each job. I did not run them here. -
systemdsession cleanup. Some Linux systems are configured to end every process of a user when their last session closes, which would reach jobs started with any of the methods above. The test machine was not a fullsystemdlogin, so this is untested. - Shells other than Bash.
zshand others have their own rules for what they do with background jobs on exit. Everything here is GNU Bash 5.1 on Linux.
Sources. Bash quotes are from the GNU Bash manual page Bash Reference Manual: Signals. nohup quotes are from the GNU coreutils manual page nohup: Run a command immune to hangups. Outputs are pasted unedited from the scripts above.
If an old Android phone is the camera behind the clips that the overnight job works through, the app I make for keeping it recording in the background is Background Camera RemoteStream on Google Play. Nothing above depends on it.
Anything started with a bare & outlives a typed exit but not a dropped connection, so launch long jobs with nohup and a redirect, and rescue running ones with disown -h.
