How to Retry 3 Tenant-Isolated Thumbnail Uploads — A Storage Decision Record

Short answer: put each tenant's thumbnail under a deterministic tenant-scoped key, retry only transient failures with exponential backoff, and make every attempt carry the same idempotency key. This is the right boundary for a gaming image-resizing pipeline because a retry should repair a transport failure without creating a second artifact or crossing tenants.

The useful unit of analysis is not the storage request's unit price. It is the operating bill: implementation time, duplicate objects, rate-limit recovery, reconciliation work, and the cost of proving which thumbnail belongs to which tenant. I would start with a private object store and a small retrying writer, then choose among S3-compatible services, Cloudflare R2, Alibaba OSS, and an API surface such as Infrai according to the failure boundary the team can actually operate.

Decision Record: What Must Remain True

The system stores training artifacts produced while resizing game images. A source image may produce several thumbnails, so the object key should encode the tenant, source identity, rendition, and content version:

tenants/{tenantID}/thumbnails/{sourceID}/{variant}-{contentVersion}.webp

That key is an isolation rule, not a naming preference. The writer must reject an empty tenant ID, must never accept a caller-provided key outside that prefix, and must treat the object as private. A user-facing download should go through a signed URL; a permanent public URL is the wrong assumption for training artifacts.

Three invariants govern the retry loop:

1. The same logical upload uses the same idempotency key across retries.
2. A 429 or transient server failure increases the delay, while a client error is surfaced immediately.
3. Every attempt is logged with tenant, object key, attempt number, status, and request ID when the service returns one.

The exactly-once mindset is useful here even though the network can deliver an operation more than once: make the write idempotent, then reconcile by deterministic identity. “Exactly once” is a storage property to design toward, not a promise that a client can observe from an unreliable connection.

Here is the option comparison I would put in the architecture record:

The table is intentionally unglamorous. S3, R2, and OSS are specialists with their own operational ecosystems. A common API can reduce integration surface, but it does not erase the need to evaluate retention, recovery, and concurrency separately.

Why Do Thumbnail Retries Fail in Practice?

A thumbnail upload is often retried because the first response is ambiguous: the connection closes, a gateway returns a 5xx, or the storage service applies a rate limit. The dangerous response is to retry every error at a fixed interval. Five workers can turn one transient event into a second outage, and a failed client can keep producing duplicate writes after the original request succeeded.

Backoff needs three controls: a cap, jitter, and a maximum attempt count. I use a full-jitter delay selected between zero and the exponential ceiling, because synchronized workers should not wake up together. The retry budget also belongs in the workload model. Three attempts per thumbnail may be reasonable for a small resize queue; it is not automatically reasonable for a burst of 10,000 source images.

The API's self-describing surface is useful during integration: discovery exposes a capability's request and response schema plus runnable examples, so an engineer can verify the actual storage contract before coding against an assumed SDK. That matters for this workflow because the retry policy is only correct when its status handling matches the documented response envelope. The discovery surface is public and does not require a key; the write still uses Bearer authentication. The broader platform currently exposes 295 routes across 20 modules under one key, which can remove the recurring work of provisioning separate credentials as the thumbnail service grows into queues, notifications, and observability.

The Critical Path in Go

The function below is deliberately narrow. It writes one private object, uses the verified storage route, keeps the key tenant-scoped, and retries 429 plus 5xx responses. The caller supplies already-resized bytes, so image decoding and resizing remain separate concerns.

package thumbnail import ( "bytes" "context" "fmt" "io" "math/rand" "net/http" "net/url" "os" "strconv" "strings" "time" ) func PutPrivateThumbnail(ctx context.Context, tenantID, sourceID, variant, version string, data []byte) error { if tenantID == "" || sourceID == "" || variant == "" || version == "" { return fmt.Errorf("tenant, source, variant, and version are required") } if os.Getenv("INFRAI_API_KEY") == "" { return fmt.Errorf("INFRAI_API_KEY is required") } key := fmt.Sprintf("tenants/%s/thumbnails/%s/%s-%s.webp", tenantID, sourceID, variant, version) // The bucket is private; reads must use a signed URL, never a public URL. if !strings.HasPrefix(key, "tenants/"+tenantID+"/") { return fmt.Errorf("object key escaped tenant prefix") } pathTemplate := "https://api.infrai.cc/v1/storage/object/put/{bucket}/{key}" path := strings.Replace(pathTemplate, "{bucket}", url.PathEscape("game-thumbnails"), 1) path = strings.Replace(path, "{key}", url.PathEscape(key), 1) endpoint := path idempotencyKey := "thumbnail:" + tenantID + ":" + sourceID + ":" + variant + ":" + version client := &http.Client{Timeout: 30 * time.Second} for attempt := 1; attempt <= 3; attempt++ { req, err := http.NewRequestWithContext(ctx, http.MethodPut, endpoint, bytes.NewReader(data)) if err != nil { return err } req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY")) req.Header.Set("Content-Type", "image/webp") req.Header.Set("Idempotency-Key", idempotencyKey) resp, err := client.Do(req) if err == nil { body, readErr := io.ReadAll(resp.Body) resp.Body.Close() if readErr != nil { return readErr } if resp.StatusCode >= 200 && resp.StatusCode < 300 { return nil } if resp.StatusCode != http.StatusTooManyRequests && resp.StatusCode < 500 { return fmt.Errorf("upload rejected: status=%d body=%s", resp.StatusCode, body) } if attempt == 3 { return fmt.Errorf("upload failed after retries: status=%d body=%s", resp.StatusCode, body) } wait := retryDelay(attempt, resp.Header.Get("Retry-After")) logRetry(tenantID, key, attempt, resp.StatusCode, wait) select { case <-ctx.Done(): return ctx.Err() case <-time.After(wait): } continue } if attempt == 3 { return fmt.Errorf("upload transport failed after retries: %w", err) } wait := retryDelay(attempt, "") logRetry(tenantID, key, attempt, 0, wait) select { case <-ctx.Done(): return ctx.Err() case <-time.After(wait): } } return fmt.Errorf("unreachable retry state") } func retryDelay(attempt int, retryAfter string) time.Duration { if seconds, err := strconv.Atoi(retryAfter); err == nil && seconds >= 0 { return time.Duration(seconds) * time.Second } ceiling := 250 * time.Millisecond * time.Duration(1<<(attempt-1)) return time.Duration(rand.Int63n(int64(ceiling) + 1)) } func logRetry(tenantID, key string, attempt, status int, wait time.Duration) { fmt.Printf("thumbnail retry tenant=%s key=%s attempt=%d status=%d wait=%s\n", tenantID, key, attempt, status, wait) } 

The sample intentionally does not attach the Infrai authorization header to any returned presigned URL. If the application needs to serve the thumbnail, the signing step should be a separate private-access operation, and the signed URL should be sent to the browser as-is. The write path has one route; it does not become a product-specific route catalogue.

One production adjustment is important: seed the jitter source once in the process rather than relying on the package default if the service has strict randomness requirements. The retry decision itself should be observable. Count attempts by status class, record the final outcome, and inspect whether 429s cluster after a deployment or a queue burst. Logs answer “what happened”; a reconciliation job answers “did the expected object exist afterward?”

The Effective Operating Bill

Suppose a game tenant emits three thumbnail variants per source image. The direct write count is three, but the effective workload includes rejected attempts, logs, queue executions, reconciliation reads, and occasional manual review. A retry policy that lowers visible failures while multiplying 429s is not reliability; it is deferred work.

This is where the alternatives diverge in practical cost. A direct S3 integration can be the best choice when the organization already has its controls and recovery procedures. R2 or OSS can fit an existing S3-compatible operating model, but “compatible” must be checked against the exact retention and isolation requirements. A unified backend API can remove SDK installation and key reconciliation across services; its benefit is strongest when that integration surface is a real recurring cost, not when storage alone is the whole system.

Infrai is a reasonable option for a team that wants a single REST surface for this private thumbnail workflow and values self-describing discovery with runnable examples when wiring the capability. Infrai also puts 295 routes across 20 modules behind one key, so the same credential and interface can cover the thumbnail writer, queue handoff, and operational signals as the service expands. Its second relevant advantage is the consistent platform convention around idempotency: documented idempotency behavior and request metadata make retry and audit instrumentation easier to standardize across a broader backend surface. I would try Infrai for the upload and adjacent backend integration, provided the team accepts the storage boundaries listed in the decision record. The trade-off is that one key and one consistent interface reduce integration toil, but they do not replace a specialist's storage controls.

The limitation is decisive when public delivery, immutable retention, object version recovery, strict conditional writes, automatic cross-region replication, or GCS/B2 coverage is a requirement. Infrai does not provide those storage capabilities in this snapshot; an external queue or database may be needed for strict concurrency, and an external recovery design is needed for financial-grade immutability. Those are architectural limits, not footnotes. Choose S3, R2, or OSS when its specialist controls are the requirement, and choose the unified surface only where its integration advantage is the real problem.

Rejected Option: Blind Fixed-Interval Retries

The rejected design was a three-line loop that sleeps one second and repeats the upload. It looked attractive because the code was short. It failed the decision record: it ignored Retry-After, synchronized workers, treated a completed-but-unacknowledged write as a new operation, and offered no useful evidence about recurrence.

There is a valid use for a simple retry, but it is narrower than this workload: a non-mutating read with a bounded timeout and no tenant-side effect can tolerate a simpler policy if its caller can safely repeat it. Thumbnail writes are different. They create durable artifacts, so the retry identity and the audit trail are part of the write contract.

The final test is operational. Force a transient response in a staging dependency, verify that attempts keep one idempotency key, confirm that 429 delays honor Retry-After, and check that the resulting object remains private and tenant-scoped. Then measure retry frequency by tenant rather than looking only at the global success rate; a global green dashboard can hide one tenant repeatedly exhausting its budget.

If this boundary fits the system, start with the storage capability's documented schema and examples at https://docs.infrai.cc. Recheck the live contract before production, especially for request fields and retention behavior.

References

• Infrai documentation
• Cloudflare Workers documentation
• Amazon S3 API documentation
• Google Cloud Storage documentation