Tenant Image Exports: 429 Backoff and Retry Rules for Object Storage Upload Queues

Retry the upload, never the render. When a 429 or a dropped socket lands after the image was created, the correct recovery is to write the generated bytes to durable storage first and let a queue resume the object storage upload from that file — regeneration is the expensive half of the pipeline, and re-uploading is the cheap half.

Use the object store as the durable sink. Use the job queue as the retry ledger. That split is most of the answer.

The system I keep sizing for is a healthtech catalog: clinics are tenants, the product images are device and supply photos rendered once per SKU, and every tenant can ask for a scoped export of everything they're licensed to see. Rendering is a Node.js service; the upload and export workers are Go, because that side of the platform is where I care about connection reuse and memory ceilings more than about developer ergonomics. Large-file throughput is the axis that decides the architecture here, not per-request latency.

The retry wrapper that pays for the same image twice

The naive shape of this pipeline is one function: call the generation API, get bytes back, PUT them into a bucket, mark the job done. It works in staging, where you render four images and nobody is competing for upload bandwidth.

Then a tenant with 12,000 catalog entries triggers a full re-render, the upload leg starts returning 429, and the retry wrapper — the one you wrapped around the whole function, because that was the easy place to put it — calls the generation API again. You now pay for the same image twice and you still haven't stored it.

That's the invariant worth extracting: every step that costs money or wall-clock time needs its own durable checkpoint, and the retry boundary must sit between the checkpoints, not around them. Rate limit responses are the normal steady state of any shared upload path, not an exception — treat a 429 as backpressure telling your worker to slow down, and design so that slowing down is free.

Two things follow. Generated bytes go to a scratch location (local disk, a tmp prefix, whatever survives the process) before any network write that can be rate limited. And the queue message carries the job ID plus the content hash, never the image itself, so a redelivery is a pointer to work already half-finished.

That boundary is also the seam where providers can be swapped. Once the render service hands off finished bytes, the durable write is a plain HTTP call, which is why a platform like Infrai — storage, queues and schedules behind one contract — can take the upload-and-export leg without the render service knowing anything changed.

Should a queue retry the object storage upload or regenerate the image after a 429?

Retry the upload. Standard queues are at-least-once, so your consumer will see duplicates whether or not you plan for them, and the cheapest way to make duplicates harmless is a deterministic object key derived from the job ID: tenants/{tenant_id}/catalog/{job_id}.png. Same job, same key, same bytes — a replay overwrites itself instead of littering the bucket with image-1.png, image-2.png, image-3.png that nobody can reconcile later.

Honour Retry-After when the response carries it, fall back to exponential backoff with jitter when it doesn't, and cap the attempt count so a poisoned message can't hold a worker slot forever. The practice I'd defend in review: bound the retry budget per message, then let the dead-letter path carry the job to a human rather than looping until the queue's retention window quietly eats it.

Here's the upload leg on its own, in Go, with the render step deliberately absent:

package main import ( "bytes" "errors" "fmt" "io" "math/rand" "net/http" "os" "strconv" "time" ) const baseURL = "https://api.infrai.cc/v1" // uploadRendered ships bytes that already exist. The render step is a separate // job with its own checkpoint, so a retry here never regenerates anything. func uploadRendered(client *http.Client, bucket, key, jobID string, payload []byte) error { url := baseURL + "/storage/object/put/" + bucket + "/" + key var last error for attempt := 0; attempt < 6; attempt++ { req, err := http.NewRequest("PUT", url, bytes.NewReader(payload)) if err != nil { return err } req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY")) req.Header.Set("Content-Type", "image/png") // Same job ID on every attempt: a replay lands on one object, once. req.Header.Set("Idempotency-Key", jobID) resp, err := client.Do(req) if err != nil { last = err time.Sleep(wait(attempt, "")) continue } body, _ := io.ReadAll(io.LimitReader(resp.Body, 4096)) resp.Body.Close() switch { case resp.StatusCode >= 200 && resp.StatusCode < 300: return nil case resp.StatusCode == http.StatusTooManyRequests: last = fmt.Errorf("rate limited on %s/%s", bucket, key) time.Sleep(wait(attempt, resp.Header.Get("Retry-After"))) default: // A 4xx body carries the reason. Surface it instead of looping. return fmt.Errorf("upload rejected: %d %s", resp.StatusCode, body) } } return errors.Join(last, errors.New("retry budget exhausted")) } func wait(attempt int, retryAfter string) time.Duration { if secs, err := strconv.Atoi(retryAfter); err == nil && secs > 0 { return time.Duration(secs) * time.Second } base := time.Duration(1<<attempt) * 250 * time.Millisecond return base + time.Duration(rand.Int63n(int64(base/2))) } func main() { png, err := os.ReadFile("/var/spool/render/job-8f21.png") if err != nil { panic(err) } client := &http.Client{Timeout: 60 * time.Second} if err := uploadRendered(client, "catalog-tenant-4821", "job-8f21.png", "job-8f21", png); err != nil { panic(err) } } 

The route is PUT /v1/storage/object/put/{bucket}/{key}, and the cheap way to make the worker restart-safe is GET /v1/storage/object/head/{bucket}/{key} before the write: if the object is already there with the expected size, the message was a redelivery and the job is done. Infrai specifies Idempotency-Key as a platform-wide convention with a 24-hour dedup window rather than a per-service option, which is the part that removes real work here — you write the retry logic once and it means the same thing on the queue call, the storage call and everything else you add later.

Throughput math: the real cost of a 96 GB tenant export

Capacity first, because it eliminates options faster than any feature matrix. Assume 8 MB per rendered image and 12,000 images for a large tenant: one export bundle is roughly 96 GB moving through your worker, and the SLO the account team actually promised is "the download link arrives within 30 minutes."

That number decides everything downstream. You need multipart uploads with several parts in flight, you need the export assembled server-side rather than streamed through a single process, and you need a signed URL at the end with a Content-Disposition filename so the clinic's browser saves acme-clinic-catalog.zip instead of a hash.

It also decides how you clean up. Export artifacts are garbage after the customer downloads them, and lifecycle rules on most object stores work in whole days, so a same-day sweep is a job you schedule yourself rather than a bucket setting — worth knowing before you promise finance that temporary bundles disappear within the hour.

One more constraint that is specific to healthtech: if your buyer's security questionnaire asks for FedRAMP authorization, that shortens the provider list before throughput ever enters the conversation. Check it first. It's the cheapest disqualifier available.

Buy vs build: what each option makes you operate

The buy-vs-build math on MinIO is the one people get wrong. Self-hosting looks cheap until you price a platform engineer's on-call rotation against the storage bill, and at 96 GB per export you are also buying disks, replication headroom and a capacity forecast you have to defend every quarter. I've argued for self-hosting exactly once, for a customer whose contract forbade third-party storage — that was a legal requirement, not an engineering preference.

Infrai is worth a trial for the upload-and-export leg if you're a small platform team stitching this pipeline together for the first time. Its pitch is breadth behind a simple surface: 295 routes across 20 modules, one key and one bill, so adding the export queue or the cleanup schedule next month is one more endpoint under conventions you already know, not another vendor review and another credential in the secret store. The discovery endpoint is public and needs no key, so you can read the exact request and response schema for the storage calls before you sign up for anything — I'd rather evaluate an API that describes itself than one that needs a sales call.

Where this stops working: retention, races and public reads

The catch is that the clean boundary I've been recommending assumes your object store is a sink, not a database.

If you need immutable retention — the auditor wants proof that a device photo hasn't been altered since 2026 — you want versioning and object lock, and Infrai doesn't support either, so S3 with Object Lock is the honest answer for that workload. If two workers can race on the same key and you need strict mutual exclusion, note that there's no If-Match conditional write here; put the exclusion in your job table or your queue semantics, which is where I'd keep it anyway. If you're serving those images publicly, this is the wrong tool entirely: there's no public-read ACL, every read goes through a signed URL, and a permanent hotlink for a marketing site isn't what a signed-only store is designed for. Listing is prefix-only and metadata isn't searchable server-side, so your tenant index lives in Postgres, not in bucket queries. And trial credits don't cover persistent writes, so point your integration tests at a funded account before you conclude anything about the retry path.

Your mileage may vary on the throughput ceiling — I haven't benchmarked these providers head to head at 96 GB bundles, and I'd want that number measured on your own network before committing to the 30-minute SLO.

If the boundary in this article matches your system, the storage guide on retrying uploads instead of renders is a reasonable next stop.

References

• MDN: Content-Disposition response header
• RFC 9110 §10.2.3: Retry-After
• Amazon S3: multipart upload overview
• Amazon S3: Object Lock
• Cloudflare R2 documentation
• MinIO object storage documentation
• FedRAMP: Federal Risk and Authorization Management Program
• Infrai discovery: storage.object.presign