A beginner-friendly walk through a security bug I found in my own system — and the one question that catches it, whether you write code or not.

Who this is for. You don't need to be a cryptographer, and you don't need to read code. If you've ever worried about whether an AI did something it shouldn't have, and whether you could prove it later — this is for you. The technical parts are clearly marked, and every one of them is explained in plain English first. Skip them and you'll still get the whole story.

The short version, if you only have 30 seconds:

  • I built a system that gives AI decisions "receipts" so you can prove later what the AI did.
  • I told the world the receipts were tamper-proof.
  • They were — but only half of them. The other half could be changed by anyone, and my software would still say "verified."
  • Nobody caught it. I caught it, by reading my own claim word by word.
  • This post is that story, because the mistake is far more common than the fix.

This is a follow-up to my earlier piece on why AI decisions need receipts. That one explained why this matters. This one is about something going wrong, which is usually a better story.

Part 1 — The $5,000 refund

A friend runs customer support automation for an online store. An AI agent is allowed to approve refunds, within limits.

One day a customer disputes a $5,000 refund. The bank wants to know: who approved this, when, and on what basis?

My friend opens his logs. And here's the part that stuck with me — he had logs. Lots of them. Detailed ones. But he couldn't use them as evidence.

Why not? Because he could have edited them. So could a disgruntled employee. So could the vendor whose dashboard the logs lived on. A record that the accused party can quietly change isn't proof of anything. It's a story you're asking someone to take on faith.

That's not a rare edge case. It's the default state of almost every AI system shipping today.

So I built something to fix it. I called it VeriTrace, and the promise was simple:

Every AI decision gets a receipt. That receipt is signed and published permanently, so nobody — not the company, not the vendor, not me — can change it afterwards.

Payments have receipts. AI decisions should too.

Part 2 — What an AI "receipt" actually is

Forget the technology for a second. Think about a paper receipt from a shop.

A receipt is useful because of three things. It's self-contained — everything is on the paper, you don't need the shop's computer. It's yours — you keep it, the shop can't take it back. And it's independent — if there's a dispute, a third party can look at the receipt and decide, without trusting either side.

An AI receipt works the same way. For every decision, it records:

  • Which agent made the decision
  • What action it took (approve refund, deploy code, execute trade)
  • Which model and exact version was used — models change silently, so the version matters
  • A fingerprint of the inputs the AI saw
  • A fingerprint of the output it produced
  • When it happened

Notice I said "fingerprint," not "copy."

That's deliberate, and it's the first clever bit. Instead of storing the customer's private data, the receipt stores a hash — a short code derived from the data. Change one character of the original data, and the hash changes completely.

Think of it like a wax seal on an envelope. The seal doesn't contain the letter. But if anyone opens the envelope and swaps the contents, the seal is broken and you can tell instantly.

So a receipt records evidence of what happened, without storing the sensitive details. You can hand receipts to an auditor, a regulator, or a court without handing over your customers' data.

Part 3 — The two questions

Here's where it gets interesting, and where most logging tools stop short.

When an AI does something, there are really two separate questions:

  1. What did it do? — "It approved a $5,000 refund."
  2. Was that the right thing to do? — "No, because policy says anything over $2,000 needs a human."

Ordinary logging tools answer the first question. You get a line in a dashboard saying "refund approved." Useful for debugging. Useless when a bank is asking whether the decision was defensible.

So VeriTrace answers both. Each receipt carries a verdict — a machine-readable assessment of whether the action was correct, produced by a second AI acting as a judge:

assessment: incorrect, confidence: 0.94, reasoning: "$5,000 exceeds the $2,000 policy cap"

That's a genuinely useful thing to have attached to a record. It turns "here's what happened" into "here's what happened, and here's whether it was defensible."

And because a judgement is only worth something if nobody can quietly change it, I wrote this in my README:

"A signed evaluation verdict, signed so it can't be quietly rewritten later."

I was proud of that sentence. I'd written it carefully. It sounds precise and serious and it is exactly the kind of sentence that makes a technical reader trust you.

It was also false.

Part 4 — What "signed" actually means

If you know how digital signatures work, skip to Part 5. If you don't, this is the only concept you need, and it's easier than it sounds.

A digital signature is a way of proving two things at once: who created something, and that it hasn't changed since.

Here's the intuition. Imagine I have a special stamp that's unique to me and that nobody else can copy. I press it onto a document. Now:

  • Anyone can look at the document and check the stamp is genuinely mine, using a public reference that proves it's mine without letting them copy it.
  • If anyone changes even a single character of the document after I stamped it, the stamp no longer matches. The document and the stamp are mathematically tied together.

The technical names: the thing that creates the stamp is the private key (stays secret), and the thing that checks it is the public key (shared freely). The process of applying the stamp is signing. The process of checking it is verification.

The crucial property is this: a signature doesn't protect "a document" in general. It protects exactly the contents you signed. Add one extra line after signing, and that line is not protected — it's just sitting there, outside the stamp.

That distinction is the entire bug. Hold onto it.

Part 5 — The hole

Now, the technical part. Both halves of this story are short.

A VeriTrace record contains two things:

1. the receipt → what the AI did 2. the verdict → whether a judge thought it was correct

When a record was created, the code did this:

signature = sign(receipt)

It signed the receipt. It never signed the verdict.

The verdict was sitting right there, in the same file, looking perfectly official — but it was outside the stamp. Unprotected. Anyone could reach in and change one word:

"assessment": "incorrect" → "assessment": "correct"

And then what happened? Nothing. The signature was still valid, because the receipt hadn't changed. Verification passed. My software displayed a green tick.

Let me make sure the severity lands. My system's whole purpose is to prevent an AI's mistakes from being quietly covered up. The single most safety-relevant piece of information — did the judge think this was wrong? — was the one piece anyone could edit freely.

I had built a tamper-evidence system with one unsealed drawer, and the drawer was labelled "was it correct?"

Part 6 — Why 64 tests missed it

I had 64 automated tests. Every single one passed. All the way up to the moment I published that README claim.

This is the part I'd most like other builders to read, because the reason is subtle.

The tests checked that altering the receipt broke verification. That was true and it was thoroughly tested.

There was no test for altering the verdict — because the idea that the verdict was unprotected never occurred to me. In my head, the verdict was part of the receipt. I'd been imagining one object. The code had two.

Here's the general lesson, and it applies well beyond cryptography:

A test suite can only describe the invariants you already believe you hold.

Tests protect the things you're worried about. They're blind to the thing you haven't thought of. So a fully green dashboard is not evidence that your system is safe — it's evidence that your system does what you currently think it does. Those are different claims, and the gap between them is where bugs live.

If I'd been more careful, I'd have noticed something simpler. My README said "signed so it can't be quietly rewritten." I never checked whether the code did that exact thing. Not "is there signing happening?" There was. But "does the signing cover the thing the sentence promises?"

I had never asked the second question.

Part 7 — The fix, piece by piece

Skip this section if you don't code — the plain-English version of each piece is in bold, and that's the important part.

Fix 1: Sign the verdict separately, with a different key

In plain terms: give the judge its own stamp, and have it stamp its verdict.

The agent stamps "this is what I did." The judge stamps "this is what I think of it." These are different claims from different parties, so they get different stamps. Sharing one key between them would be convenient and wrong.

There's also a subtle attack this prevents: replay. If the judge signed a verdict on its own, someone could take a genuine "correct" verdict from one receipt and attach it to a different receipt. So the signed content includes a fingerprint of the specific receipt the verdict is about:

signed content = ( fingerprint of receipt , the verdict )

Now a verdict is only valid for the one decision it was issued about.

Fix 2: Put the verdict inside the published batch

In plain terms: every receipt belongs to a batch, and the whole batch is summarised by one number that gets published publicly. Make sure the verdict is part of what goes into that number.

This is the part that ties the judgment to the permanent record. Once that summary number is written down publicly, you cannot change any verdict in the batch — the maths stops adding up.

There was one constraint I cared about: don't break anything that already exists. Older receipts that have no verdict needed to keep working exactly as before, so their previously published proofs still check out. A fix that invalidated all existing data would have been a worse bug than the one I was repairing.

Fix 3: Treat an unsigned verdict as a failure, not an absence

In plain terms: if a verdict shows up without a stamp, don't shrug and say "no stamp to check." Treat it as forged.

This is the tempting mistake. The natural code says: is there a signature to verify? No? Well then there's nothing to check, carry on. That's a hole with a polite name.

A verdict with no signature is not "a verdict we can't verify." It's a verdict that cannot be trusted, and the system must say so loudly.

Fix 4: Make the demonstration fail if the protection breaks

In plain terms: build the check into the pipeline so the failure is impossible to miss.

My demo now tampers twice: it alters the receipt, then it flips the verdict. It refuses to report success unless both are caught. That runs automatically on every change, so if the protection ever regresses, the build turns red instead of the README quietly becoming a lie again.

Tests went from 64 to 75. The new ones are written from an attacker's point of view: a flipped verdict must fail, a slightly adjusted confidence must fail, a verdict moved onto a different receipt must fail, a verdict signed by the wrong key must fail.

Part 8 — Why "permanent storage" completes the picture

One more piece, and again it's about who you have to trust.

If receipts live on my server, then proving they're unaltered requires trusting me. That's circular — I'm the party who might have an incentive to change them. The record has to outlive me and not be under my control.

That's what Arweave is for: a public storage network where data is written once and kept permanently. I publish the batch summary there. Even if my company vanished tomorrow, the record and its fingerprint would still be there for anyone to check.

The part nobody tells you: it can be free

Normally, writing to Arweave costs money, and you have to buy their token first. For a system that records thousands of AI decisions, that's a real barrier — it turns every logged action into a running bill.

But there's an alternative path. Turbo, the bundling service run by the ArDrive team in front of the Arweave network, subsidises small uploads — up to about 105 kilobytes each, with roughly 10 megabytes of allowance per wallet, no tokens required.

Here's the trick: my published payload is about 150 bytes. Not kilobytes. Bytes.

Because I don't upload receipts to Arweave. I upload one hash that summarises an entire batch of them. Thousands of AI decisions fold down into one small number, and that number goes on the permanent record.

So the marginal cost of making an AI decision permanently auditable is zero. You're not paying per decision. You're not paying per batch. The only thing that touches the network is a fingerprint.

(I did hit one practical snag worth passing on: these small uploads don't always appear on the main Arweave gateway immediately while they propagate. A different gateway served mine right away. My verifier now tries several in turn — a small detail that turns "it didn't work" into "it worked, try the other door.")

Part 9 — What this does and doesn't prove

I want to be careful here, because this is where projects like mine tend to oversell, and overselling is exactly how I ended up with an unsigned verdict in the first place.

What a signed verdict gives you: proof that the holder of the judge key issued that specific assessment, about that specific decision, and it hasn't been altered since.

What it does not give you:

  • It doesn't prove the judge was right. An AI judge can be wrong, and it will still sign a confident, well-formed, completely incorrect verdict.
  • It doesn't prove the model behind the judge was good. As I discovered while testing, a small model will sometimes return output so malformed that the only honest response is "inconclusive" — which my system now does, correctly, rather than guessing.
  • It doesn't prove the key wasn't stolen.

So what's the actual value? Something narrower, and more useful than "the AI was right."

You get proof of exactly what was claimed, by whom, about which decision, at what time — checkable by anyone without trusting the person who made the claim.

The judgment stays transparent, and it stays re-runnable. Because the inputs are frozen forever, anyone can run a different judge against the same evidence and see whether they agree. You're not buying certainty about correctness. You're buying an honest, permanent record of what was asserted — which happens to be the thing auditors and courts actually ask for.

Part 10 — A 10-minute check for your own system

You don't need to work on AI to use this. It applies to any system where you claim something can't be tampered with — audit logs, financial records, medical data, contract approvals.

  1. Write your promise down in one sentence. Mine was "signed so it can't be quietly rewritten later."
  2. List every field that sentence implies is protected. Mine implied two: the action, and the judgment about the action.
  3. For each field, ask: what line of code would fail if someone changed this? Not "is it stored?" Not "is it logged?" Specifically: what breaks?
  4. If you can't point to the line, you don't have the property.

I could point to the line for the first field. For the second, I had nothing — and I'd never noticed, because I'd never asked.

Step 4 is the whole exercise. It takes ten minutes and it would have saved me from publishing a false security claim to the world.

The takeaway

The uncomfortable question in this industry isn't "do you sign your AI's decisions?" Almost everyone will say yes, and many of them will be telling the truth.

The question is: what exactly is inside the signature?

A system that signs the action but not the assessment of the action has protected the easy half — the part that was never really in dispute. The part that matters is usually the part someone would have a motive to change.

That's what I got wrong, and finding it taught me more than building it did. The system now signs both halves, proves it with a public transaction, and fails loudly if either protection ever breaks.

The record is cryptographically certain. The judgment is transparent and re-runnable. And one more thing, which I now consider part of the product: the claim in the README is true.

VeriTrace is open source under the MIT licence. If you want to see the loop end to end, pnpm demo signs a receipt, batches it, verifies it, then tampers with it twice — and only reports success if both attempts are caught. No accounts, no network, no prior setup.