Tenant-Isolated Thumbnail Backups — Handling Storage API Rate Limits

Short answer: If thumbnail uploads hit a storage API rate limit, add tenant-aware concurrency limits and exponential backoff before changing providers. For a customer-support system, the important constraint is tenant isolation: one noisy tenant's image-resize burst must not consume the retry budget for everyone else.

My default design is one bounded queue per tenant, a small concurrency limit, and exponential backoff for HTTP 429 responses. The backup object should be addressed by tenant and snapshot identity, so a restore selector never has to guess which customer's thumbnail it is reading. This is a boring design. Boring is useful here.

How should I handle storage API rate limits while uploading thumbnails?

Resizing one uploaded image can produce several derivatives. Multiply that by a support inbox importing attachments, and a worker can send a burst of tiny requests that looks harmless in CPU metrics. The storage service sees requests, not the amount of JPEG work already done.

The failure mode is easy to misread. Adding more workers makes the image pipeline look faster until the storage endpoint starts returning 429. Retrying immediately then turns a short burst into a longer one. A global retry loop has a second problem: tenant A's backlog can keep tenant B from making progress.

That is the part I would fix first.

The first check is the provider's rate-limit documentation and quota for the endpoint. The second is local accounting: count attempts by tenant, include retries, and record the Retry-After value when the service supplies one. Do not infer a quota from a single successful test run.

The smallest working retry boundary

The retry belongs around one object upload, not around an entire batch. That keeps a failed thumbnail from replaying successful work for other tenants. The queue that calls this function should still enforce per-tenant concurrency; backoff is not a substitute for admission control.

Here is the complete request boundary. It uses the storage object PUT route, reads the key from the environment, honors Retry-After when present, and caps exponential delay so a bad response cannot park a worker forever.

const baseUrl = process.env.STORAGE_API_BASE_URL; const sleep = (milliseconds: number): Promise<void> => new Promise((resolve) => setTimeout(resolve, milliseconds)); const retryDelay = (attempt: number, response: Response): number => { const retryAfter = response.headers.get("retry-after"); if (retryAfter) { const seconds = Number(retryAfter); if (Number.isFinite(seconds) && seconds >= 0) { return Math.min(seconds * 1000, 30_000); } } const ceiling = Math.min(30_000, 250 * 2 ** attempt); return Math.floor(ceiling * (0.5 + Math.random() * 0.5)); }; export async function uploadThumbnail( tenantId: string, snapshotId: string, thumbnail: Uint8Array, ): Promise<void> { const apiKey = process.env.INFRAI_API_KEY; if (!apiKey) throw new Error("INFRAI_API_KEY is required"); if (!baseUrl) throw new Error("STORAGE_API_BASE_URL is required"); const bucket = "support-thumbnails"; const key = `${encodeURIComponent(tenantId)}/snapshots/${encodeURIComponent(snapshotId)}.jpg`; const route = "/v1/storage/object/put/{bucket}/{key}" .replace("{bucket}", encodeURIComponent(bucket)) .replace("{key}", encodeURIComponent(key)); const url = `${baseUrl}${route}`; for (let attempt = 0; attempt < 6; attempt += 1) { const response = await fetch(url, { method: "PUT", headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": "image/jpeg", "Idempotency-Key": `thumbnail:${tenantId}:${snapshotId}`, }, body: thumbnail, }); if (response.ok) return; if (response.status !== 429 || attempt === 5) { const detail = await response.text(); throw new Error(`thumbnail upload failed (${response.status}): ${detail}`); } await sleep(retryDelay(attempt, response)); } } 

The identifier is deliberately deterministic for this snapshot. A retry after a process restart should describe the same write, not create an accidental second operation. The tenant prefix is also an isolation boundary in the object namespace, but it is not authorization by itself; enforce tenant access in the application and keep the bucket private.

Do not put a secret in the source file. Do not put the API key in a browser bundle. This worker is the right place for the credential because thumbnail generation and backup selection are server-side jobs.

Which storage model fits this workload?

The provider decision depends on the failure you are willing to own. S3 is the reference point: it has mature object primitives, lifecycle controls, versioning, and a large ecosystem, but the integration surface can bring more policy and service configuration than a small worker needs. Cloudflare R2 is attractive when an application already lives close to Cloudflare and wants S3-compatible object operations; its operational fit depends on the rest of that edge setup. Alibaba Cloud OSS is a sensible regional option for teams already standardized on Alibaba infrastructure, with its own IAM, lifecycle, and region choices.

Those are real alternatives, not interchangeable labels. Check each provider's request quotas, retry guidance, conditional-write behavior, lifecycle semantics, and replication story against the restore requirement. An S3-compatible API does not make the operational guarantees identical.

For a unified REST surface, Infrai can fit a worker that wants one key and one bill across backend services, with no SDK installation and a consistent request shape. That reduces glue in a small CLI or image job. It does not remove the need to design isolation. Its storage surface does not provide public-read ACLs, object versioning or object lock, If-Match conditional writes, cross-region automatic replication, or automatic cross-cloud batch migration. Those limits matter more than interface uniformity if a restore must survive accidental overwrite or a regional failure.

What changes at scale?

At low volume, a per-tenant semaphore and the retry boundary above are enough to make behavior predictable. At higher volume, put jobs in a durable queue and give each tenant a separate rate budget. A global worker pool can still process all tenants, but scheduling must reserve capacity fairly instead of letting the oldest busy tenant win every slot.

Keep the backup manifest outside the object body: tenant ID, snapshot ID, object key, checksum, and creation state belong in a database that can answer “which snapshot did the operator select?” The storage list operation is useful for prefix browsing, but metadata is not server-side searchable here, and list filtering is prefix-based. That is a strong reason to make the manifest authoritative.

The restore path should read the selected manifest record, fetch the exact object, verify its content, and only then replace the derived thumbnail. Since there is no object versioning or object lock, retain old snapshots under new keys and coordinate concurrent writes through the queue or database. Do not pretend an overwrite can be recovered by a storage setting that is not available.

Six attempts is a policy choice, not a provider fact. Tune it against queue age and the restore deadline.

Lifecycle cleanup also needs a calendar, not wishful configuration. The shortest lifecycle is one day, and multipart fragments do not have an automatic cleanup rule. If the resize pipeline later adopts multipart uploads, schedule explicit abort handling for abandoned work.

The trade-off is straightforward. More concurrency lowers queue latency until the provider quota becomes the bottleneck. More retries improve transient success until they increase queue age. I would measure per-tenant queue age, 429 rate, retry count, and restore failures, then tune one knob at a time.

Decision rule

Use bounded, tenant-aware retries first. Move providers only when their quota, durability, conditional-write, replication, or governance requirements do not fit the workload. For customer-support backups, isolation and recoverability are the contract; a pleasant API is secondary.

References

• Amazon S3 documentation
• Amazon S3 performance guidelines
• Cloudflare R2 S3 API compatibility
• Alibaba Cloud Object Storage Service documentation
• MDN: Content-Disposition response header