A worker sends a paid search request. The connection drops before a reply arrives. On the next queue delivery, the worker creates a fresh UUID and sends the same body. If the first search completed, the second UUID can buy the same observation twice.
Now change one detail. The worker receives a confirmed 408 search_timeout from the API. Sending the same key can replay that failure. One retry policy has to handle both outcomes.
Keep the scheduled check and the API attempt in storage
Use a durable job ID for the scheduled observation. Persist the request body and a UUID for its current API attempt before sending it. If the worker restarts, the next worker can resume the same attempt. A later scheduled check gets its own job and attempt key, even if the query is identical.
In Serper.live, Idempotency-Key identifies an attempt. A replay can return the first attempt's result with Idempotent-Replayed: true. Sending a modified body with the same key can still return the original result. Save the key and body together. Serper.live retry contract
| Outcome | Next action |
|---|---|
| Reply missing or unreadable | Back off and use the saved key and body |
409 idempotency_in_progress | Back off and use the saved key and body |
Confirmed API 408, 424, or 503 | Back off, persist a new attempt key, and retry the same body |
409 idempotency_replay_unavailable | Stop automatic retries and reconcile the attempt |
| Successful search | Save the observation and finish the job |
| Other confirmed API error | Record the failure for handling by the caller |
The replay window is up to 15 minutes. Bound the retry budget accordingly. If the result is still unknown when the window expires, use the saved attempt details and any request ID to investigate before starting a fresh paid search. A new key starts a new search. Serper.live errors and replay behavior
Test the decision without making a search
Save this as retry.mjs and run node retry.mjs. The worker passes an HTTP status only when it can identify the response as coming from the API. A connection reset or a generic gateway reply leaves the outcome unknown. For a success, pass status 200 after decoding the result.
import assert from "node:assert/strict"; function retryAction({ status = null, code } = {}) { if (status === 200) return "store-result"; if (status === null || (status === 409 && code === "idempotency_in_progress")) { return "retry-same-key"; } if (status === 409 && code === "idempotency_replay_unavailable") { return "reconcile"; } if ([408, 424, 503].includes(status)) return "retry-new-key"; return "record-failure"; } assert.equal(retryAction(), "retry-same-key"); assert.equal( retryAction({ status: 409, code: "idempotency_in_progress" }), "retry-same-key", ); assert.equal( retryAction({ status: 409, code: "idempotency_replay_unavailable" }), "reconcile", ); for (const status of [408, 424, 503]) { assert.equal(retryAction({ status }), "retry-new-key"); } assert.equal(retryAction({ status: 200 }), "store-result"); assert.equal(retryAction({ status: 422 }), "record-failure"); console.log("Retry decisions passed"); A real worker executes the decision. Claim the job before reading its current attempt so two deliveries cannot both rotate the key. For a same-key retry, keep the body byte-for-byte and schedule the next attempt with backoff. For a new-key retry, write the new key to storage before the next POST. An attempt counter and next-run time belong in that same write.
When the worker sends the request, the persisted key goes in the header. The API allows a search to run up to 90 seconds and documents a client timeout of at least 100 seconds. A short deadline can leave the outcome ambiguous while the search is still running. Serper.live client timeout guidance
curl --include --max-time 100 https://api.serper.live/v1/search \ -H "Authorization: Bearer $SERPER_LIVE_TOKEN" \ -H "Idempotency-Key: $SERPER_LIVE_ATTEMPT_KEY" \ -H "Content-Type: application/json" \ --data-binary @attempt-body.json Use a stored JSON body such as {"query":"espresso scale","country":"US","language":"en","device":"mobile","limit":10}. On success, save the response and mark the job complete in one local transaction, or make the result write idempotent by job ID. That handles the other common crash window, when the worker gets the result and dies before acknowledging the queue message.
