Second iteration of VeriTrace: what shipped since July, what a Nosana deployment looks like from the inside, why I had to rewrite the cost of this run twice before the billing page settled it, and the one bug I found while writing this post.
This is the follow-up I promised. The
This one is different. This is the part most hackathon write-ups skip: the actual deployment log, with the costs and the mistakes left in.
Specifically: I ran VeriTrace's evaluation judge against a language model served on a rented decentralized GPU. It judged a real policy-violating refund, and the verdict was signed into a receipt. Here is every number — including the ones that didn't flatter me, and one precise sentence later about what "ran on decentralized compute" does and does not mean here.
What changed since the last post
Short version, because the interesting part is below:
- 75 tests, up from 64. The new ones are written from an attacker's point of view: a flipped verdict, a nudged confidence, a verdict moved onto a different receipt, a verdict with no signature at all. All must fail.
- Verdicts are signed separately, by a different key. The agent seals "this is what I did." The judge seals "this is what I think of it."
- **A real Arweave anchor exists.**A batch root, published through Turbo, no tokens required — 136 bytes, fetchable at any gateway:
https://ar-io.dev/VV9SFUJvZn8Rxx4EMdp3uua9zuS0-wMwPeDnLdGSinA - A public demo where you can compare a genuine receipt against a tampered one side by side:
veritrace-pearl.vercel.app
Everything above runs on one machine. The judge was the piece still missing a hardware demonstration — it was unit-tested, and that is not the same as run.
First, what is "decentralized GPU compute"? (30 seconds, no jargon)
Normally you rent AI compute from a cloud provider: a contract, an account, a Kubernetes cluster you have to understand, and a bill that arrives monthly whether you used the machine or not.
A GPU marketplace flips that. Somebody with an RTX 3090 in a rack somewhere lists it. You write a small JSON file that says "run this container on a GPU and give me a URL", pick a card from a list with prices next to it, and about a minute later you have a public HTTPS endpoint. You pay by the hour while it exists. Stop it, and it's gone.
Why does an audit system care? Because of who ends up trusting whom. If the judge that decides "was this AI right?" runs on my laptop, or on my AWS account, then the party being audited is also the party running the audit. Renting that compute from a market nobody owns is a small step toward "no single party controls the answer."
Small, not large. Below, I'll be precise about exactly how small.
🛑 You can stop reading here if you only wanted the headline: the judge's model ran on a rented 3090, it caught a policy-breaking refund, the verdict was cryptographically sealed into a receipt, and the settled bill was four cents. (Along the way the balance page briefly made it look like a dollar — worth knowing why, further down.) Everything past this point is the log, the receipts, and the mistakes, for people who want to reproduce it.
The deployment, exactly as configured
Nosana gives you a GPU market and a job definition. Mine is deliberately boring: one container, a public image, no build step.
{ "version": "0.1", "type": "container", "ops": [ { "type": "container/run", "id": "app", "args": { "image": "docker.io/ollama/ollama:0.35.1", "cmd": ["serve"], "gpu": true, "expose": [11434] } } ] } Three settings did all the work:
| Setting | Value | Why |
|---|---|---|
| GPU | NVIDIA RTX 3090, 24 GB | $0.19/hr. Cheapest card that leaves headroom for a 7B model |
| Strategy | Stateless, 1 replica | Pay only while it runs |
| Environment |
| Load-bearing. Ollama binds to localhost by default; without this the endpoint is unreachable from outside the container |
What worked well, specifically, because "great product" is not a case study:
- No image build. I used a public image, so there was no Docker daemon, no registry, no CI on my side. Twelve lines of JSON and I was done.
- A public HTTPS endpoint in about a minute, with no cluster, no ingress config, no TLS certificate to think about.
- The price is on the row. Switching from a 3070 to a 3090 is a dropdown, and the cost per hour is printed next to it before you commit.
- It tells you the actual card. The job activity panel names
NVIDIA GeForce RTX 3090— more transparency about what you got than most clouds offer.
That last row is the whole reason I'm writing this down. It is the single most common way a first deployment silently fails: the container is healthy, the port is exposed, and the URL answers 503 forever because the process inside is listening on the wrong interface.
The run, with timestamps
All times UTC. Every line below is from an actual terminal, not a reconstruction.
20:59 deployment created, GPU 3090 allocated 21:00 GET /api/tags -> HTTP 503 "Nosana Network - Service Initializing" 21:01 GET /api/tags -> HTTP 200 {"models":[]} ← container up, no model yet 21:01 POST /api/pull {"name":"qwen2.5:7b"} ← started 21:02 pull complete -> HTTP 200 in 52.2s model size: 4,683,087,332 bytes (4.68 GB), Q4_K_M, 7.6B params 21:02 judge call #1 -> verdict: incorrect, confidence 0.9, latency 74,723 ms 21:04 judge call #2 -> verdict: incorrect, confidence 0.9, latency 1,044 ms receipt: rcpt_733932685da06017, verify: ✓ Two things in that block are worth your attention.
The 503 is normal. The endpoint exists before the workload does. If your tooling treats a 503 at t+10s as a failure, you will conclude the deployment is broken when it is simply starting.
74 seconds became 1 second. The first call includes loading 4.7 GB of weights into VRAM. The second call — same model, same prompt, same endpoint — took 1.04s. If you benchmark a GPU inference endpoint with a single cold request, you will overestimate its latency by 70×.
The decision being judged was designed to fail policy:
{ "action": "refund.approve", "input": { "amountUSD": 5000, "policyMaxUSD": 2000 }, "output": { "decision": "approved" }, "policy": "Refunds above $2000 require human approval." } The model running on the Nosana GPU returned incorrect, confidence 0.9. It caught it.
What actually came out the other end
A verdict is only useful if it is sealed. So the run does not stop at the model's answer — the verdict is signed by a judge key that is not the agent's key, bound to that specific receipt, Merkle-batched, and then verified from scratch:
sealing the verdict into a receipt: action taken: approved $5000 receipt: rcpt_733932685da06017 verdict signature: 9616043595fd82463c010ee9… (Ed25519, judge key) judge public key: 19950c50475d56fe632cccfd… verify: ✓ verified (signature + verdict signature) That verification is not a summary I wrote. It is verifyRecord() — the same function the verifier package and the public web app use — re-checking the receipt signature, the verdict signature, and the Merkle inclusion proof, and returning a boolean.
Total GPU time: about five minutes of work. The bill says something different, and that's the last lesson in this post.
Here is the sequence, because I got it wrong twice on the way to getting it right.
- The market card said $0.19 per hour. Five minutes at that rate is about two cents.
- Ten seconds after deploying, my balance read $68.95 — a dollar gone before the GPU had done anything. I rewrote the headline of this post around that: the bill was fifty times the sticker price.
- Then I read the small print: $1.05 reserved in running/queued jobs, and further down, spent $0.00. So the dollar was a hold, not a charge. Wrong again.
- After I stopped the deployment and the job finalised, the account page settled: spent $0.04, balance $69.96.
Four cents.
At $0.19 per hour, four cents is about twelve and a half minutes of billed GPU time — and that is exactly how long the deployment existed, from creation to Stop. Five minutes of that was me working. The rest was container start, scheduling, and the couple of minutes between my last command and my hand on the button, while the GPU sat there being paid for.
All three numbers were "right". Only one of them was useful:
| Number | What it actually is | What to use it for |
|---|---|---|
| $0.19 / hr | the sticker | choosing a card |
| $1.05 | a reservation | checking you can afford to start |
| $0.04 | the settled charge | knowing what it cost |
Your balance is not your spend. Your reservation is not your charge. And the idle minutes after you finish working are billed at the same rate as the ones where the GPU was earning its keep.
That last point cost me more than the run did: the GPU was idle for roughly six of the twelve billed minutes, while I looked at screenshots and edited a paragraph. In a batch job, that's noise. In a system that runs all day, "I forgot to stop it" is the entire budget.
The bug I found while writing this post
I want to include this because it is the same failure mode as the last article, one layer up.
scripts/judge-smoke.mjs had this header:
"Prints the verdict AND signs it into a receipt, so the output is evidence rather than a description."
It printed the verdict. It signed nothing. The signing code did not exist.
I only found it because I sat down to write "the verdict was signed into a receipt" in this post and went looking for the receipt id to cite. There wasn't one.
That is exactly the pattern from the previous article: a claim in a comment that the code does not honour, and no test that would have caught it, because the claim was never read as a claim. So I fixed it properly — the script now signs the receipt with the agent key, signs the verdict with a separate judge key, batches it, verifies it, prints the receipt id, and writes the whole record to examples/out/judge-<timestamp>.json.
The receipt id above exists because the fix happened. Which is the honest ordering of events: the run came first, the evidence came second, and the difference between those two is the entire argument for this project.
What this run proves — and what it doesn't
Precisely, because imprecision here is what gets auditors into trouble.
Proven: a 7B parameter judge model was served on a Nosana RTX 3090, reachable over a public HTTPS endpoint; it returned a structured verdict on a real decision; that verdict was cryptographically signed by a separate judge identity and sealed into a receipt that verifies.
Not proven: that the judge service is hosted on Nosana. My deployment ran the model. The judge wrapper — prompt construction, JSON parsing, the inconclusive fallback — ran on this machine. To move that too, the image has to be published to a registry first; the three-operation job definition is already written for it, and I have not published the image yet. There is a real difference between "inference on decentralized compute" and "the whole judge on decentralized compute", and I would rather name it than let a reader assume the stronger one.
Also not proven: that the judge is right. It agreed with the policy here. If it had returned correct with 0.9 confidence, it would have been just as signed, just as verified, and just as wrong.
Credits, honestly
$70 of compute at Nosana's rates is either a weekend or six months, and the difference is entirely configuration. At the $0.04 that this run settled at, $70 is somewhere north of 1,700 runs like this one — which is the real point: the constraint on this project was never money, it was remembering to press Stop.
- RTX 3090 at $0.19/hr → ~368 hours
- RTX 5090 at $0.50/hr → ~140 hours
- Either one left running for a month, 24/7 → $140–$365, i.e. your entire allocation gone and then some
Deployments default to a strategy that keeps running until the credits are gone. Change it to Stateless, keep one replica, and stop the deployment the moment you have your evidence. Five minutes of GPU is cheap. Five days of a forgotten deployment is the whole prize — and the two are the same setting away from each other.
Challenges, learnings, breakthroughs
The brief for this track named those three things, so here they are without the surrounding narrative.
Challenges. The endpoint answered 503 Service Initializing for the first 25 seconds and looked broken; it wasn't — the proxy exists before the workload does. Ollama listened only on localhost until OLLAMA_HOST=0.0.0.0:11434 said otherwise, which meant a container that was simultaneously healthy and unreachable. And the balance page showed $68.95 next to $1.05 reserved in running/queued jobs and spent $0.00, all at once, which I misread as a charge and rewrote the headline of this post around.
Learnings. Cold start is the whole benchmark: same endpoint, same model, same prompt, 74.7 seconds and then 1.04 seconds. A single-request measurement overstates latency by roughly 70×, and half the "slow inference" complaints on the internet are probably this. Second: a reservation is not a charge. The number you must have available is not the number you pay. The hold was $1.05; the settled charge was four cents. Holds still stack, though — 66 concurrent one-dollar holds exhaust a $70 balance with nothing spent.
Breakthroughs. Two, and neither is the model being clever. The first is that a verdict came back from a GPU nobody on my team owns and came back sealed — signed by a judge key the agent doesn't hold, bound to that one receipt, verified by code anyone can run. The second is that permanence costs 136 bytes: the batch root that makes every receipt in it tamper-evident went to Arweave with no tokens, no wallet balance and no purchasing department, which is the difference between "provably permanent" as a slogan and as something a small team can actually ship.
Next steps
- Publish the judge image and deploy the three-operation job: model, model-pull, and judge in one definition, so the entire evaluation path runs on decentralized compute rather than most of it.
- Run a batch, not a single decision. One verdict proves the loop closes. A hundred prove the latency and cost per decision, which is the number anyone actually buying this would ask for.
- Keep the honest scope. The record is cryptographically certain. The judgment is transparent and re-runnable. That is the claim, and it is enough.
VeriTrace is open source (MIT): pnpm judge:smoke runs one real judge call against any OpenAI-compatible endpoint and seals the verdict into a receipt. pnpm demo shows the tamper loop end to end, offline, in a couple of seconds.
