Retries Failing for Thumbnail Uploads — Handling Transient Storage Errors

Short answer: retry thumbnail uploads with bounded exponential backoff, honor Retry-After when the service provides it, and stop after a small number of attempts. Keep the application out of the byte path: let the browser or worker upload directly to object storage through a short-lived upload URL, while your service creates that URL and records the result.

That gives you a useful before-and-after mental model. Before, a resize job treats one failed PUT as a failed thumbnail. After, it treats a transient failure as a state transition: wait, try the same write again, record the attempt, and give up with a useful error when the budget is exhausted. The access decision stays in your application; delivery stays with storage. For a catalog that creates three sizes per source image, that distinction is repeated thousands of times during an import, so a retry policy belongs in the upload path rather than in an operator's memory.

That is the trap.

Why do thumbnail retries fail in the first place?

Thumbnail generation has an awkward shape. The source image may be large, but the resulting object is small enough that teams often test it with a laptop and miss the production pattern: many resized variants arrive in bursts, and several jobs can target the same product or listing at once.

A temporary storage error is not proof that the object is invalid. It can be a short interruption, a saturated connection, or a service response asking the client to slow down. A retry can help. An immediate retry storm can make the original problem louder and turn a transient failure into a rate-limit problem.

Use three limits together:

• A maximum attempt count, such as four total attempts.
• A maximum delay, so one thumbnail cannot hold a worker forever.
• A retryable-status policy that does not repeat permanent authorization or validation failures.

The exact numbers are policy, not a universal law. I start with four attempts for a thumbnail because the object is derived data and can be regenerated, then tune the policy from logs. The important part is that every attempt is visible. Log the object key, attempt number, response status, elapsed time, and request ID when one exists. Do not log the bearer token or the image bytes.

What does a direct upload retry loop look like?

The upload URL is the boundary in this example. Your backend authorizes the object name and returns a short-lived URL; the client or worker sends the bytes directly to storage. The application handles policy and metadata. It does not proxy the media.

Here is a complete retry helper for a Uint8Array against Infrai's verified object PUT route. It uses an explicit PUT, reads the key from an environment variable, honors a valid Retry-After value, adds jitter, and returns request metadata only after a successful upload. The bucket and object key are inputs, not invented request fields.

type UploadResult = { status: number; requestId: string | null; }; type UploadOptions = { bucket: string; objectKey: string; idempotencyKey: string; maxAttempts?: number; baseDelayMs?: number; maxDelayMs?: number; sleep?: (delayMs: number) => Promise<void>; random?: () => number; }; const defaultSleep = (delayMs: number): Promise<void> => new Promise((resolve) => setTimeout(resolve, delayMs)); const retryableStatus = (status: number): boolean => status === 408 || status === 425 || status === 429 || status >= 500; const retryAfterMs = (value: string | null): number | null => { if (!value) return null; const seconds = Number(value); if (Number.isFinite(seconds)) return Math.max(0, seconds * 1000); const dateMs = Date.parse(value); return Number.isNaN(dateMs) ? null : Math.max(0, dateMs - Date.now()); }; export async function uploadThumbnail( bytes: Uint8Array, options: UploadOptions, ): Promise<UploadResult> { const apiKey = process.env.INFRAI_API_KEY; if (!apiKey) throw new Error("INFRAI_API_KEY is required"); const apiBaseUrl = process.env.INFRAI_BASE_URL; if (!apiBaseUrl) throw new Error("INFRAI_BASE_URL is required"); const maxAttempts = options.maxAttempts ?? 4; const baseDelayMs = options.baseDelayMs ?? 250; const maxDelayMs = options.maxDelayMs ?? 4000; const sleep = options.sleep ?? defaultSleep; const random = options.random ?? Math.random; const objectPutTemplate = "/v1/storage/object/put/{bucket}/{key}"; const uploadUrl = `${apiBaseUrl}${objectPutTemplate .replace("{bucket}", encodeURIComponent(options.bucket)) .replace("{key}", encodeURIComponent(options.objectKey))}`; let lastStatus = 0; for (let attempt = 1; attempt <= maxAttempts; attempt += 1) { let response: Response; try { response = await fetch(uploadUrl, { method: "PUT", body: bytes, headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": "image/webp", "Idempotency-Key": options.idempotencyKey, }, }); } catch (error) { if (attempt === maxAttempts) throw error; const delay = Math.min(maxDelayMs, baseDelayMs * 2 ** (attempt - 1)); await sleep(Math.floor(delay * (0.8 + random() * 0.4))); continue; } lastStatus = response.status; if (response.ok) { return { status: response.status, requestId: response.headers.get("x-request-id"), }; } if (!retryableStatus(response.status) || attempt === maxAttempts) { const detail = await response.text(); throw new Error(`Thumbnail upload failed (${lastStatus}): ${detail}`); } const serverDelay = retryAfterMs(response.headers.get("retry-after")); const exponentialDelay = Math.min( maxDelayMs, baseDelayMs * 2 ** (attempt - 1), ); const delay = serverDelay ?? exponentialDelay; await sleep(Math.floor(delay * (0.8 + random() * 0.4))); } throw new Error(`Thumbnail upload failed (${lastStatus})`); } 

The loop retries network failures and temporary HTTP responses, but it does not retry a rejected credential or a malformed request. That distinction matters. A retry cannot grant access, and repeating a bad key four times only hides the real diagnosis.

I initially treated every failed thumbnail as a storage problem. The useful correction was to put the response status beside the attempt number: a first-attempt 503 followed by a successful second attempt supports backoff, while four 403 responses point to authorization or URL construction. That small split keeps the worker from spending its retry budget on a permanent error, and it gives the person reading the log a next action instead of another generic “upload failed” line.

The write must also be safe to repeat. Use a deterministic object key for a thumbnail, such as a source identifier plus a resize variant, and make the resize job itself idempotent. If two workers can publish the same key, coordinate that decision in the job store or queue. A retry loop is not a substitute for concurrency control.

For an Infrai-backed storage write, the relevant operation is the object PUT route, and its plain REST interface means a worker can use the same HTTP boundary without installing an SDK. That is a practical fit when one service already needs several backend capabilities and you want one authentication convention. It does not remove the need to inspect status codes, record attempts, or define access policy.

How do the main storage choices differ?

The hard choice is not “which service can store bytes?” Most of them can. It is “where should authorization live, and how much delivery behavior do I want the storage layer to provide?”

The last row is the important qualification. A private e-commerce thumbnail pipeline is a reasonable access-controlled workload. A permanent public image host is a different product requirement. Storage without public-read ACLs is not a defect to paper over with retries; it is a selection boundary. The limitation is concrete: if the product needs permanent public URLs, choose a storage design that provides that delivery model instead.

There are other trade-offs to check before committing: browser uploads need a workable CORS configuration, lifecycle rules may not support hour-level expiry, and cross-region or cross-cloud replication may belong to another system. Read those constraints alongside the happy-path upload test. The absence of object versioning, object lock, and If-Match conditional writes also makes strict recovery or compare-and-swap requirements a poor fit without external coordination.

How should you prove that retries are helping?

Start with a small set of structured fields: object_key, variant, attempt, status, delay_ms, duration_ms, and outcome. Add the storage request ID when the provider returns one. A log line saying “upload failed” is not enough to tell a rate limit from a bad authorization decision.

Then inspect patterns over a real batch of thumbnail work. If attempts cluster around 429 responses, reduce concurrency and respect the server delay before increasing the retry budget. If failures are mostly network exceptions, inspect connection reuse, worker limits, and upload URL lifetime. If the final response is consistently a 4xx, stop tuning backoff and fix the request or access rule.

Metrics make the decision faster: count uploads by final outcome, count retry attempts separately, and measure the delay spent waiting. Alert on the failure rate and on retry volume. A system that eventually succeeds after six retries may look healthy to a dashboard that records only the final 200, while users experience slow product pages and a queue that is quietly filling.

One more practical check: test duplicate execution. Run the same resize job twice with the same source and variant, then confirm that the resulting object and catalog record follow your intended rule. That test exercises the part retry advice often skips: delivery reliability is useful only when repeated work has a predictable result.

This is where the logs become more valuable than a longer retry budget. Suppose an import creates a source image, schedules three variants, and two workers pick up the resize job for the smallest variant. Worker A receives a temporary error and waits 250 milliseconds. Worker B succeeds immediately. If the key is deterministic, both workers can converge on the same derived object; if the catalog update is separate, the job needs a clear rule for which result is authoritative. Record the source identifier and variant, not just a generated request ID, so an operator can follow that chain. Then compare the first-attempt failure rate with the final failure rate. A falling final failure rate with rising retry volume is not a clean win: it may mean the backoff is masking a concurrency limit that should be addressed at the queue or worker level.

The decision rule

Choose direct uploads when the application should decide who may write, but should not carry the image bytes. Add bounded exponential backoff for transient failures, with server-directed delay taking precedence. Keep the object key deterministic, and make the job observable enough to distinguish a temporary outage from an authorization mistake.

Choose a provider with stronger native versioning, conditional writes, retention, or replication when those are requirements rather than future possibilities. Choose a simpler REST integration when consistent access and delivery code matter more than those storage-specific controls. The retry loop is portable. The access model is not.

Further reading

• Amazon S3 presigned URLs
• Amazon S3 conditional requests
• Cloudflare R2 S3 API compatibility
• Cloudflare Workers documentation
• Google Cloud Storage signed URLs
• Google Cloud Storage resumable uploads
• Cloudflare Workers observability