GPU marketplaces like Vast.ai are a cheap way to fine-tune open models, as long as the data is not confidential: the host can see
the machine while your job runs. The harder problem is that training scripts assume the hardware under them stays alive.
When a host disappears, two things break:
- Checkpoints that only exist on the local disk are gone.
- If the machine hangs instead of dying, the rental keeps billing while nothing trains.
Here is what I ended up building, and what the measurements were.
1. Upload checkpoints outside the training loop. A small agent in the container watches the checkpoint directory and uploads
each finished checkpoint to Cloudflare R2. The training script writes a marker file when a checkpoint is complete, so half-written
ones are never uploaded. The agent uses a temporary key that only works for this job's folder.
2. Count traffic and start-up, not only the GPU price. One QLoRA 7B run was billed $0.81, most of it traffic for downloading
the model; after choosing hosts by the full hourly price including traffic, the next run was $0.19. In six later runs the first
training step came 4.5 to 8.5 minutes after renting, and those minutes are billed. My own rule of thumb, not a measurement:
checkpoint often enough that one lost interval costs less than one start-up.
3. Handle the dead host. A supervisor outside the GPU machine checks the instance and the agent's heartbeat. When the host
is gone it rents another machine, the agent restores the last checkpoint and the script resumes. In a test where I destroyed the
machine twice during one run, training continued after 3.5 and 13 minutes.
4. Handle the hung host. If the GPU stops responding for five minutes, or the job makes no progress for half an hour while the
GPU sits idle, the job is moved to another machine.
5. Handle the dead supervisor. If the supervisor itself goes away, nobody replaces failed machines or checks spending
limits. So the agent watches how long it has gone without a new storage key from the supervisor and, after a safety window, stops
the job and the machine from the inside. I tested this on Vast with the window shortened to a few minutes: the machine stopped,
and after that Vast billed only for disk. In normal settings the window is about 2.5–3.5 hours; I have not run a test of that
length.
Do it yourself or not. If your team has someone to run it, dstack and SkyPilot are good open-source tools for this, and you
probably don't need a service.
Disclosure: I run NomadGPU (nomadgpu.com), a small service that does the above on your own Vast.ai
account, with spending limits in dollars. Live numbers from our own test jobs, a small sample:
status.nomadgpu.com/s/status.
