We had a pipeline that could run for hours. GPU time is expensive. A crash at 95%
was the same as a crash at 0% — except we were angrier.

I got tired of it. Not in the "we need a framework" way. In the "this is stupid" way.

What we had before

A long script:

research → script → render (forever) → assemble → done

If the process died in render, we started over. Every time.
People say "use a job queue". We had workers. We still lost time, because
progress lived only in RAM.

The fix (boring on purpose)

After every stage, write down that you finished it.

research [done] render [done] ← crash here? next run skips research+render assemble [pending]

Then:

  • run() executes pending stages
  • resume() skips done, retries failed
  • state is written atomically (tmp file → rename), so a kill mid-write doesn't corrupt everything
  • workers send a heartbeat; watchdog notices when they go silent

That's it. No magic scheduler. Just refusing to forget finished work.

Why not "just checkpoint tensors"?

We don't need mid-layer optimizer state for every job. Most of our cost is
stage-level work: research, render, package. Stage checkpoints cover 95% of the pain
with 5% of the complexity.

Trade-off on purpose: a stage still re-runs from its start. That's acceptable.
Losing the whole pipeline is not.

What broke along the way

  1. Non-atomic state writes. Kill during save → corrupted JSON → resume lied to us. Fixed with write-to-temp + os.replace.
  2. Retries that duplicated side effects. Stage contract now says: re-entry must be safe (idempotent), or the stage owns its own local temp dir.
  3. "Busy" hung workers. Looked alive in the process list. Heartbeat file + staleness check.

Numbers

  • Long jobs: tens of minutes to hours
  • Before: crash = full re-run
  • After: crash = re-run from last finished stage
  • Tests: 12 behavioral contracts (resume, fail-fast, atomic state, stale heartbeat)

Practical checklist (copy/paste)

  • [ ] Checkpoint after each stage, not "sometime later"
  • [ ] Atomic write (tmp + rename). Kill mid-write must not corrupt state
  • [ ] resume() skips done, retries failed
  • [ ] Stages should be idempotent on re-entry
  • [ ] Heartbeat file + staleness check for hung workers
  • [ ] Fail-fast on poison stages
  • [ ] Test: crash after stage 1 → resume does stage 2 only

Rule of thumb

If re-running the job costs more than 5 minutes — checkpoint it.
If it costs GPU-hours — checkpoint it today.

Takeaways

  1. Checkpoint I/O is cheap. Re-running 3 hours of GPU is not.
  2. Atomic writes matter more than fancy progress bars.
  3. Resume is a feature, not an ops afterthought.
  4. Heartbeat ≠ "it works". It's only "it's alive".
  5. Fail-fast on poison stages beats "keep going and hope".

Open-sourced here: checkpointed-task-pipeline

Related reliability pieces: