In short: Networks retry, so any API that moves data or money receives some requests
twice. I built an idempotent sync API and hit three instructive bugs: a JSON mismatch that
failed without any error, deduplication that worked in tests but not over HTTP, and the
design change that made retries free. The pattern applies to payments, webhooks and message
queues alike.

When my laptop sends its changes to the VPS, the request can fail in the most annoying way
possible: the VPS receives it, applies it, and the response gets lost. From the laptop's
side that looks identical to "never arrived", so it sends the batch again.

Any network protocol that retries delivers messages at least once. The receiver has to
make the second delivery harmless. That property is idempotency, and getting it right took
three attempts.

Attempt 1: batch IDs

Each sync batch carries an ID. The receiver remembers IDs it has processed. If one arrives
again, it answers "accepted, nothing applied" instead of applying it twice:

{"accepted": true, "applied": 0, "skipped": 1, "message": "Duplicate batch ignored"}

A duplicate is accepted, not rejected. The sender did nothing wrong; it just can't tell
whether its first attempt landed. Returning an error would make it retry forever.

Unit tests passed. Then I tried it over HTTP.

The bug that returned no error

The sender encoded batches like this:

{"from_node": "laptop", "batch_id": "b-1", "mutations": [...]}

The receiver's Go struct had no JSON tags, so Go's decoder looked for keys named after
the fields: FromNode, BatchID, Mutations. Nothing matched.

Here's the part that cost me time: json.Decode doesn't treat unknown keys as an error. It
returned a perfectly valid, completely empty struct. The handler then correctly rejected an
empty batch as invalid. The handler did its job on garbage input; the bug was that the
encoder and decoder had two different ideas of what a batch looked like.

The fix was to put the tags on the one real type and have both sides use it:

type SyncBatch struct { FromNode string `json:"from_node"` BatchID string `json:"batch_id"` Mutations []SyncMutation `json:"mutations"` }

One contract, used by both ends, can't drift apart. Two "equivalent" definitions eventually
will.

Attempt 2: deduplication that only worked in tests

With the wire format fixed, I sent the same batch twice to a running server and wrote down
my prediction first: second response, "Duplicate batch ignored". Actual result:

Accepted Applied Skipped Message -------- ------- ------- ------- True 1 0 Batch applied successfully True 1 0 Batch applied successfully

Applied twice. The cause was one line in the HTTP handler:

service := replication.NewSyncService() // inside the handler

A value created inside a request handler lives for one request. Every request got a
fresh service with an empty "seen batches" set, so no batch was ever a duplicate. The unit
tests reused one service across calls, so they never saw it.

The fix: create one service per node when the program starts, and share it with every
request (with a mutex, since requests run concurrently). The replay then returned
Applied: 0, Skipped: 1.

Attempt 3: IDs that make retries free

Where does a batch ID come from? A random ID works for network retries, but I wanted
something stronger: if the laptop sends the same contents again (say, after a restart),
the VPS should skip it without looking at a single change.

So the ID is a hash of the sender plus its contents:

func BatchIDFor(fromNode string, mutations []SyncMutation) string { hashes := make([]string, 0, len(mutations)) for _, mutation := range mutations { hashes = append(hashes, HashMutation(mutation)) } sort.Strings(hashes) // order-independent // ...sha256 of fromNode + hashes, first 8 bytes as hex }

Could two different batches share an ID? Only if a machine's state returned to an
earlier version, and vector clocks only move forward, so it can't.

Two layers, and why the second one matters

Deduplication now happens twice:

  1. Batch ID seen before? Skip the whole batch. Fast.
  2. Otherwise, per change: compare vector clocks. A change the receiver already has compares Equal or older and is skipped.

The second layer let me fix a slow memory leak safely. The set of seen batch IDs grew
forever, so I capped it at 10,000 and forget the oldest. Is forgetting safe? Only because of
layer 2: a forgotten batch that arrives again is processed, and every change in it is
skipped anyway. I wrote a test that forgets a batch, replays it, and checks that nothing
changes.

A test checks one replay. Later I wrote a TLA+ specification of the protocol and stated the
property directly: delivering any message a second time, in any state any node can reach,
changes nothing. The model checker verified it across every interleaving of three machines
(see part 9). That's the real justification for forgetting batch IDs.

Where this shows up in real systems

  • Payment APIs. Stripe's idempotency keys are the best-known example: resending a charge request with the same key can't charge the customer twice.
  • Message queues. Kafka, SQS and most brokers deliver at least once by default. Consumers that aren't idempotent eventually process something twice.
  • Webhooks. Providers retry when your endpoint times out, even if you'd already processed the event. Every webhook handler needs the "accepted, nothing applied" path.
  • Contract drift between services. The silent JSON mismatch is the small version of a common integration failure: two teams' "equivalent" schemas that quietly disagree. Shared types or contract tests catch it; decoders that ignore unknown fields hide it.

What I took away

  • Design for the second delivery from the start. A retry isn't an edge case.
  • A decoder that ignores unknown fields can turn a schema mismatch into silent empty data. Keep one definition of every message.
  • Test through the real transport. Unit tests with a long-lived object hid a bug that the first real HTTP request exposed.
  • A fast path is fine to make lossy when there's a correct slow path behind it.

Next: How Two Computers Find Their Differences Without Sending Everything.

About this series: I'm learning distributed systems by building a real application, with an
AI assistant as coach and pair programmer. It explains, reviews and sometimes writes code; I
verify every claim against tests and measurements, and every bug and number here is real.

I write about building reliable software in Go, from the code up. If you're working on similar
problems, I'd like to hear from you: get in touch or connect with me on LinkedIn.