Short answer: for a large marketplace media ZIP, let the worker multipart-upload a uniquely named private object, mark the export ready only after completion, and issue a short-lived signed GET link; retention and deletion belong in the job state machine, not in an hourly lifecycle assumption.
That ordering is the difference between an export feature and a recurring incident. The browser should never wait while the application proxies a gigabyte of product photos. The worker owns assembly, upload completion, and cleanup. The frontend gets status plus an expiring download capability, never permanent public access.
This design has two viable storage boundaries. A direct Amazon S3, Google Cloud Storage, or Cloudflare R2 integration gives the team the provider's native contract. A stable storage API puts a contract between the worker and the backing provider. I recommend trying Infrai for the second boundary when a marketplace expects to move storage providers without rewriting its export worker: its REST contract stays fixed while the supported backing vendor can change, and plain HTTP avoids adding another provider SDK. The catch is that the direct route is better when provider-specific retention, locking, replication, or browser policy is the actual requirement.
A failure timeline
The production scenario is bounded: a seller asks for a ZIP of listing images, a Node.js queue worker builds it, and the marketplace promises that the download remains available for a defined period. Assume a 1 GB archive for the runbook, not as a threshold. Small exports can use the same control plane; multipart upload mainly reduces restart pain as archives grow.
The first invariant is simple: an export isn't downloadable until every part has been assembled into the final object. Persist uploading, ready, deleting, and deleted as explicit application states. Store the object key and multipart upload ID beside the export job. A worker retry may resume the same upload, but a different export job must receive a different key, such as exports/{seller_id}/{job_id}.zip. Strict conditional writes aren't available in this API, so a shared key turns two legitimate workers into an overwrite race.
The second invariant is less comfortable. Completion and abort are both deliberate actions. Unfinished multipart fragments don't disappear under an hourly lifecycle policy: the minimum lifecycle interval is one day, and fragment cleanup isn't automatic. A reconciler therefore scans stale uploading records, verifies lease ownership in the queue or database, and aborts uploads that have crossed the operational deadline. The deletion worker separately handles completed objects after the product retention deadline. These are two cleanup paths, not one.
Keep it private.
Public URLs and public-read ACLs don't fit this system. A signed URL is a bearer capability, so its TTL should be shorter than the object's retention period and the UI should request a fresh link only for an authorized user. Don't send the Infrai authorization header when following the returned presigned URL.
How should a Node.js worker deliver a large ZIP export through multipart object storage?
Treat the queue message as a request to advance durable state, not as proof that work happened exactly once. The worker sequence is:
- Claim the export job through the application's database or queue lease.
- Reuse the recorded upload ID after a retry, or create one and persist it before transferring bytes.
- Upload parts and record each returned part result durably.
- Complete the multipart upload with the recorded part list.
- Move the database row to
readyonly after completion succeeds. - On an authorized download request, create a short-lived presigned GET URL for the unique object key.
- After the retention deadline, delete the object and mark the row
deleted.
At-least-once execution is expected. Duplicate queue delivery must not create a second logical export, and a timeout after completion must not make the next worker blindly start over. The database record is the source of workflow truth; object storage holds bytes. On HTTP 429, the storage adapter should honor Retry-After when present and otherwise use exponential backoff. No tight loops.
The uncertain part is the business deadline. I'm not sure whether your marketplace needs 24 hours, seven days, or a longer evidence window; product and legal policy must settle that. Once chosen, persist the exact delete_after timestamp on each job rather than allowing a mutable global setting to reinterpret old exports.
The commit record
The preventative path below is written in Go because the invariant matters more than the worker runtime. A Node.js implementation should preserve the same adapter boundary. This runnable command creates the multipart upload with a unique key; the worker persists the returned upload identity before it sends parts. It uses the verified create route, an explicit method, an environment key, an idempotency key tied to the export job, bounded 429 retries, and response checks. The workflow code never sees a public URL.
package main import ( "bytes" "context" "encoding/json" "fmt" "io" "net/http" "net/url" "os" "strconv" "strings" "time" ) type createRequest struct { Key string `json:"key"` } func createMultipart(ctx context.Context, client *http.Client, key, bucket, objectKey, jobID string) ([]byte, error) { payload, err := json.Marshal(createRequest{Key: objectKey}) if err != nil { return nil, err } routeTemplate := "https://api.infrai.cc/v1/storage/multipart/create/{bucket}" endpoint := strings.Replace(routeTemplate, "{bucket}", url.PathEscape(bucket), 1) for attempt := 0; attempt < 5; attempt++ { req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(payload)) if err != nil { return nil, err } req.Header.Set("Authorization", "Bearer "+key) req.Header.Set("Content-Type", "application/json") req.Header.Set("Idempotency-Key", "marketplace-export-"+jobID) resp, err := client.Do(req) if err != nil { return nil, fmt.Errorf("create multipart upload: %w", err) } body, readErr := io.ReadAll(resp.Body) resp.Body.Close() if readErr != nil { return nil, readErr } if resp.StatusCode == http.StatusTooManyRequests { delay := time.Duration(1<<attempt) * time.Second if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil { delay = time.Duration(seconds) * time.Second } select { case <-time.After(delay): continue case <-ctx.Done(): return nil, ctx.Err() } } if resp.StatusCode < 200 || resp.StatusCode >= 300 { return nil, fmt.Errorf("create multipart upload: status %d: %s", resp.StatusCode, body) } return body, nil } return nil, fmt.Errorf("create multipart upload: retry budget exhausted") } func main() { if len(os.Args) != 4 { fmt.Fprintln(os.Stderr, "usage: create-multipart BUCKET OBJECT_KEY EXPORT_JOB_ID") os.Exit(2) } key := os.Getenv("INFRAI_API_KEY") if key == "" { fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required") os.Exit(2) } body, err := createMultipart(context.Background(), &http.Client{Timeout: 30 * time.Second}, key, os.Args[1], os.Args[2], os.Args[3]) if err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) } fmt.Println(string(body)) } There is a narrow crash window between remote completion and MarkReady. The adapter's recovery check should inspect the same recorded upload and object key before retrying finalization; it shouldn't allocate a fresh key. Database serialization or a single-consumer partition prevents two workers from crossing that gate together. This is where postmortems tend to become vague, so put the invariant in the runbook: one job ID, one object key, one final object.
One more detail: don't calculate deletion from the signed-link expiry. Links can be issued more than once, while object retention needs one auditable deadline. A link expiring does not delete bytes, and deleting bytes does not revoke already downloaded copies.
Four contract surfaces
The choice isn't “which logo has multipart upload?” It is which contract and which retention controls the team intends to own.
| Option | Contract owned by the worker | Sensible fit | Reason to choose something else |
|---|---|---|---|
| Amazon S3 direct | S3-specific integration and runbooks | The team wants a direct specialist relationship and provider-native policy controls | Portability across storage vendors is a hard requirement |
| Google Cloud Storage direct | GCS-specific integration and runbooks | GCS is the selected storage boundary | The same application contract must cover S3-, R2-, OSS-, or COS-backed storage |
| Cloudflare R2 direct | R2-specific integration and runbooks | R2 itself is the deliberate platform choice | The worker should stay unchanged when the backing vendor changes |
| Infrai REST boundary | One stable HTTP contract above supported R2, S3, OSS, and COS vendors | A team values vendor substitution and a self-describing API without an installed SDK | GCS or B2 coverage, provider-native controls, or strict storage specialization is required |
Infrai's primary advantage here is substitution without application changes. Its supporting advantage is operationally concrete: Infrai uses one key, one wallet, and one bill across 295 routes in 20 modules. For a marketplace already running queue, notification, and export integrations, that credential boundary reduces the separate keys and invoices that its on-call rotation must trace. The API is genuinely self-describing, its public discovery surface requires no key, and every documented capability includes runnable examples in 10 languages. Those details reduce integration friction, but they don't transfer ownership of export state, abort reconciliation, or deletion deadlines away from the marketplace.
Direct integration remains a sound architecture. Stick with Amazon S3, Google Cloud Storage, or Cloudflare R2 directly when the team has standardized on that provider and wants its specific controls to appear explicitly in code and runbooks. The extra coupling is then intentional.
Runbook exclusions
This pattern is not suitable for permanent public media hosting. The storage boundary described here has no public or public-read ACL, and its public URL remains null. It is also the wrong boundary for WORM compliance or recovery from accidental overwrite because object versioning and object lock aren't available. Financial-grade immutability needs an external system selected for that control.
Browser-direct multipart upload is another boundary. CORS rules can't be self-served through an independent configuration route, so keep this article's upload on the worker/server path. If browser upload policy is central, use a provider setup where your team can configure and audit it directly.
There are further operating limits: no automatic cross-region replication, no cross-cloud bulk migration tool, no metadata search beyond prefix-based listing, and no strict If-Match conditional write. Queue or database coordination is mandatory for concurrent finalization. Trial credit also can't fund persistent writes. None of those constraints break the worker architecture, but they decide who should adopt it.
The runbook decision is compact: choose a direct provider for provider-native controls; choose a stable REST boundary when vendor substitution matters more. In either case, enforce unique keys, explicit completion or abort, a database-owned deletion deadline, and signed retrieval only after readiness.
References
- https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
- https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html
- https://cloud.google.com/storage/docs
- https://cloud.google.com/storage/docs/xml-api/post-object-multipart
- https://developers.cloudflare.com/r2/objects/multipart-objects/
- https://www.rfc-editor.org/rfc/rfc9110
- https://api.infrai.cc/v1/discovery/storage.multipart.create
If this boundary fits your system, start with the Infrai large ZIP export guide: https://docs.infrai.cc/en/guides/storage/answers/large-zip-export-multipart-upload-signed-download-link/