To train past Modal's 24-hour function timeout, make the run resumable: checkpoint to a Modal Volume, stop each call just before the limit, and spawn a fresh call that resumes. Nodus takes the other route: a Nodus Job's time limit is optional, and Nodus checkpoints and restores your state directory itself, so a 72-hour run is one Job instead of a chain of calls.

The chain works well on Modal once it is built. The rest of this post is the working pattern, the three ways it loses money, the cost of those losses on a 72-hour run, and when we would stop building it.

What exactly does Modal's timeout limit?

Every Modal Function execution has a default timeout of 300 seconds, and you can set anything from 1 second to 24 hours (Modal timeouts guide). The limit is per execution attempt. With modal.Retries, each retry gets a fresh timeout, and once retries run out the caller gets a modal.exception.FunctionTimeoutError.

Two other limits shape the design. Modal's own long-training example notes that 24-hour timeouts with 10 retries give a job about 10 days before someone has to restart it by hand. The same example warns that calls made with .remote() expire after 24 hours, so a multi-day job should be started with .spawn().

Modal's preemption guide adds the third constraint. All Functions can be preempted, the chance grows with run length, and the non-preemptible option is not supported for GPU Functions. A preempted Function is restarted on the same input, from the beginning of your code.

What does a working Modal pattern look like?

The pattern that holds up is a self-stopping chain: each call trains until a time budget you set, saves, commits the Volume, spawns its successor and returns. Retries stay in place for crashes and preemptions, not as the main way past the 24-hour wall. We stop at 23 hours rather than 24 so the last save and commit finish well before the hard limit.

import os, signal, time from pathlib import Path import modal app = modal.App("long-train") image = modal.Image.debian_slim(python_version="3.12").uv_pip_install("torch") vol = modal.Volume.from_name("ckpts", create_if_missing=True, version=2) CKPT = Path("/ckpts") TIME_BUDGET_S = 23 * 3600 # stop well inside the 24 h ceiling SAVE_EVERY_S = 15 * 60 @app.function(gpu="H100:8", image=image, volumes={CKPT: vol}, timeout=24 * 3600, single_use_containers=True, retries=modal.Retries(max_retries=3, initial_delay=0.0)) def train_chunk(run_id: str) -> None: started = last_save = time.monotonic() run_dir = CKPT / run_id run_dir.mkdir(parents=True, exist_ok=True) model, opt, sched = build() # your model, optimizer, scheduler step = load_latest(run_dir, model, opt, sched) # 0 on a fresh run interrupted = False def on_sigint(*_): nonlocal interrupted interrupted = True # Modal sends an interrupt signal on preemption signal.signal(signal.SIGINT, on_sigint) while step < TOTAL_STEPS: train_step(model, opt, sched, step) step += 1 out_of_time = time.monotonic() - started > TIME_BUDGET_S if interrupted or out_of_time or time.monotonic() - last_save > SAVE_EVERY_S: save_atomic(run_dir, model, opt, sched, step) vol.commit() # durable before we move on last_save = time.monotonic() if interrupted: raise KeyboardInterrupt # end this attempt; Modal reruns the input if out_of_time: train_chunk.spawn(run_id) # a fresh call with a fresh 24 h timeout return 

The save and load helpers carry the real state, not just a step counter:

import torch def save_atomic(run_dir, model, opt, sched, step): tmp = run_dir / "last.pt.tmp" torch.save({"model": model.state_dict(), "opt": opt.state_dict(), "sched": sched.state_dict(), "step": step, "rng": torch.get_rng_state()}, tmp) os.replace(tmp, run_dir / "last.pt") # never leave a half-written file def load_latest(run_dir, model, opt, sched): path = run_dir / "last.pt" if not path.exists(): return 0 state = torch.load(path, map_location="cpu") model.load_state_dict(state["model"]); opt.load_state_dict(state["opt"]) sched.load_state_dict(state["sched"]); torch.set_rng_state(state["rng"]) return state["step"] 

Deploy the app with modal deploy, then start the chain with modal.Function.from_name("long-train", "train_chunk").spawn("run-001"). Nothing then depends on your laptop staying awake. If you use a dataloader, save its position too, or each resumed chunk replays data it has already seen.

A few details in that code come straight from Modal's docs.

single_use_containers=True is what the long-training example uses so every retry starts in a fresh container that mounts the latest Volume state. The explicit vol.commit() matters because, per the Volumes guide, changes become visible to other containers only after a commit.

Background commits run every few seconds and a final commit runs at shutdown, but we would rather know the checkpoint is durable than hope the last background commit finished. Volumes v2 is the version Modal recommends for new Volumes.

Where does a chained run lose time and money?

A chained run pays three costs a single long run does not: work lost since the last save, a startup on every new call, and duplicate work when two calls write the same run. Each one is billed at the full GPU rate.

Lost progress is the big one. If you rely on retries alone, the 24-hour timeout kills the call mid-step, and everything since the last checkpoint runs again. The self-stopping chain avoids that at planned boundaries, but a preemption still loses up to one save interval.

Startup is paid on every call. Each new call pulls the image, attaches 8 GPUs, loads the checkpoint from the Volume and warms up. Modal bills Functions by the second (Modal pricing), and the GPUs are billed through all of it.

Duplicate chains are the quiet failure. If a call crashes after spawn() but before it returns, its retry spawns a second successor, and two chains now write last.pt. The Volumes guide says last write wins on concurrent writes to the same file. Write a small marker with the next chunk's ID before spawning, and have each new call exit if the marker names someone else.

What does that cost on a 72-hour run?

We priced a 72-hour run on 8 NVIDIA H100 GPUs from Modal's pricing page, where an H100 is $0.001097 per second, or $3.9492 per GPU-hour. Eight of them cost $31.5936 per hour of wall time on the GPU meter alone. For the restart we assume 10 minutes from new call to first useful step. That is our illustrative figure, not a measurement, so substitute your own.

The GPU meter for 72 hours of useful training is 72 x 8 x $3.9492 = $2,274.74. Both chain designs need four calls, because each call fits only about 23 hours of useful work. Everything below is overhead on top of that $2,274.74.

Chain design Overhead per boundary Boundaries Overhead Cost at $31.5936/hr
Self-stopping, 23 h budget 10 min restart 3 30 min $15.80
Retries only, hourly saves (worst case) up to 60 min lost + 10 min restart 3 210 min $110.58

The retries-only design needs a fourth call too: each 24-hour attempt can lose up to an hour at the timeout kill, so three attempts cover at most about 68.5 hours of saved work.

Extra GPU cost of chaining a 72-hour 8 x H100 run on Modal: $15.80 for a self-stopping chain and up to $110.58 for retries only with hourly saves.
Stopping each call yourself turns the 24-hour wall into three short restarts instead of three lost hours.

Preemptions add to either design, and the save interval sets the price. On the same 8 x H100 run, one preemption costs up to the interval plus the 10-minute restart.

With hourly saves that is up to 70 minutes, or $36.86. Saving every 30 minutes caps it at 40 minutes, or $21.06. Saving every 15 minutes caps it at 25 minutes, or $13.16. Modal calls preemptions rare and publishes no rate, so we do not multiply by a count.

Worst-case cost of one preemption on 8 x H100 at Modal's list price: $36.86 with 60-minute saves, $21.06 with 30-minute saves, $13.16 with 15-minute saves.
Checkpoint every 15 minutes on multi-GPU runs; the save is cheap next to a lost hour on 8 GPUs.

Prices checked November 17, 2026.

How does Nodus run the same 72-hour job?

On Nodus the same run is one Job with no chain, and the checkpoint handling moves from your code into the platform. The Jobs guide treats the wall-clock timeout as an optional limit that you can raise, lower or remove on a running Job, so there is no 24-hour boundary to plan around.

Your program still writes its state, but only to /nodus/state, and loads it on start, exactly like load_latest above. Per the checkpoints guide, Nodus copies that directory on an automatic interval, at least four times over the run's expected length. It also checkpoints when the capacity gives notice that it is about to be reclaimed, and restores the directory on the replacement machine. The command comes from our docs:

nodus run --gpu H100:8 --max-cost 1600 -d -- torchrun train.py 

The checkpoint path defaults to /nodus/state, and --max-cost suspends the Job with its state saved if it would spend more than $1,600. The guide also covers a Hugging Face Trainer integration, so a Trainer script only needs output_dir pointed at the state directory.

The rate is the other difference. An H100 SXM is listed at $2.60/hr on the Nodus pricing page, so 72 hours on 8 GPUs is 72 x 8 x $2.60 = $1,497.60, against $2,274.74 on Modal's GPU meter. That is 34% less before the chain overhead.

Modal also bills CPU and memory on separate meters. Interruptions still cost a restart on Nodus; what changes is that you do not write or maintain the handoff.

GPU-meter cost of a 72-hour run on 8 x H100: $2,274.74 on Modal at $3.9492 per GPU-hour, $1,497.60 on Nodus at $2.60.
The same 576 GPU-hours cost 34% less at the Nodus list rate, before any restart overhead.

If you are moving an existing Modal app, our migration guide shows the smallest code change, and the Modal pricing explainer covers the rest of Modal's meters.

When is the Modal chain the right choice?

Stay on Modal and build the chain when your training already lives there and runs last a few days at most. Per-second billing, Volumes and the rest of your app in one place are real advantages. The long-training example is short, and the code above adds a few dozen lines to it.

Our decision rule, from the arithmetic above:

  1. Under 20 hours of training: set timeout to the expected time plus some headroom, checkpoint every 15 to 30 minutes anyway, and keep retries.
  2. From 20 hours to a few days on Modal: build the self-stopping chain with a 23-hour budget, explicit commits and a handoff marker. Do not use retries as the chain.
  3. Multi-day runs on 8 or more GPUs, or many long runs a month: run them as Nodus Jobs. The rate gap on 576 GPU-hours is $777.14, more than any overhead in the table above.

For a wider look at where to put long training, see our Modal alternatives for training and how to make training survive preemption.

FAQ

What do checkpoints cost to keep on a Modal Volume?

Modal's pricing page lists Volumes at $0.09 per GiB per month with the first 1 TiB free, so checkpoints cost nothing extra until your total Volume storage passes 1 TiB. The Volumes guide notes that deleted data can stay billed for up to four days, so pruning old checkpoints saves money a few days later, not at once.

Can I just catch FunctionTimeoutError and call the Function again?

You can, and the timeouts guide shows how to catch it. It is the weaker design: the work since the last save is already lost when the error arrives, and the caller has to stay alive for the whole multi-day run.

Can I make a Modal GPU Function non-preemptible?

Not today. The preemption guide says the nonpreemptible option is not supported for GPU Functions, so every GPU training run should expect a restart and checkpoint for it.

How often does Nodus checkpoint a long Job?

With the default interval: auto, Nodus checkpoints at least four times over the Job's expected run time and keeps the time spent checkpointing to about a tenth of it. You can set a fixed interval from 1 minute to 6 hours instead, and a suspend or reclaim notice always triggers a checkpoint.

Originally published on the Nodus blog.