AI Receipt Images in a Node.js App: A Throughput-First Storage Decision
Short answer: for a customer-support system that processes receipts, keep the original file in private object storage, keep thumbnails as separate regenerable objects, and make the Node.js application publish database pointers only after the bytes are complete; signed URLs and lifecycle rules should enforce access and retention policy, not stand in for either one.
The operational constraint changes the answer. Large-file throughput matters more here than the smallest posted storage rate, because an audit original has a different failure boundary from a thumbnail. The original receipt is evidence. The thumbnail is a convenience.
The audit record starts with a deletion policy
Start with the erasure boundary, then define the bucket layout. For every receipt, the application should know the original key, derivative keys, content digest, processing state, and deletion state. Object storage is the byte store, not the business catalog.
Use distinct, immutable keys for the original and each thumbnail. A useful shape is receipts/{receipt_id}/original/{digest} and receipts/{receipt_id}/thumb/{width}/{transform_digest}. A hash-derived key prevents a late resize worker from silently replacing a newer output; the database pointer decides which completed variant the user sees.
Separate the upload path from the request that serves a support agent. Large receipt files should stream through bounded buffers, while resizing runs asynchronously. A web request that waits for upload, image processing, database commit, and URL signing has too many failure boundaries to retry as one unit.
Three words: bytes first, pointer second.
The signed URL is a temporary bearer credential, not a permissions model. Generate it only after an authenticated application request has checked the support case, keep its lifetime close to the download window, and do not put it in ordinary request logs. Content-Disposition can influence whether a browser displays a response inline or treats it as an attachment; it does not authorize the reader.
Throughput is a pipeline property, not a bucket feature
For large files, the bottleneck may be the customer upload, the application worker, the network path, image decoding, or the destination write. Measure each stage separately. Useful counters include bytes accepted, upload duration, queue age, resize duration, bytes written, retry count, and the time between object completion and database publication. Percentiles matter more than an average when a few very large receipts dominate the queue. Measure it.
A service can stream an incoming body to a storage adapter while keeping only bounded metadata in memory. The adapter below is deliberately generic: its contract is private storage, a unique key, and a complete write. It does not invent a provider-specific SDK or request path.
from dataclasses import dataclass from hashlib import sha256 from typing import BinaryIO, Protocol class ObjectSink(Protocol): def write_private(self, key: str, chunks: list[bytes], operation_id: str) -> None: """Persist the complete private object or raise a retryable error.""" @dataclass(frozen=True) class StoredObject: key: str digest: str size: int def store_receipt(source: BinaryIO, sink: ObjectSink, receipt_id: str) -> StoredObject: digest = sha256() chunks: list[bytes] = [] size = 0 while chunk := source.read(1024 * 1024): digest.update(chunk) chunks.append(chunk) size += len(chunk) value = digest.hexdigest() key = f"receipts/{receipt_id}/original/{value}" operation_id = sha256(key.encode("utf-8")).hexdigest() sink.write_private(key, chunks, operation_id) return StoredObject(key=key, digest=value, size=size) This sample keeps chunks in a list so the state transition is easy to see; a production adapter should stream or spool to bounded local storage rather than allowing a very large receipt to consume process memory. That distinction is important. A correct key does not make an unbounded worker safe. If the storage interface supports multipart transfer, use it for files large enough to justify parallel parts, and record unfinished operations for cleanup. If it does not, stream through a controlled worker pool and tune concurrency from measurements. More workers can increase throughput until the network, CPU, or remote rate limit becomes the actual queue. A 429 should cause bounded backoff, not an unbounded retry storm.
How should a Node.js app store generated images and thumbnails in object storage?
For generated images and thumbnails, the practical contract is simple: originals and derivatives are separate objects, only completed objects become catalog entries, and a signed URL is minted after authorization. The answer is not a public link with a cleanup job attached. It is a private byte path whose state is explicit enough to audit.
Retention rules are governance, not cleanup automation
A lifecycle rule is appropriate for a prefix containing failed jobs, temporary exports, or thumbnails whose source original still exists and can be regenerated. It is a poor substitute for a legal deletion workflow and a dangerous default for originals. Put originals and derivatives under different prefixes, and review the rule against the retention policy before enabling it.
The exact deletion path should be explicit. When a customer asks to erase a receipt, enqueue the object keys, mark the application record as pending deletion, delete the objects, and record completion. Make the consumer idempotent. If the database row is removed first, the application may lose the key it needs to delete; if the object is removed first and the process stops before the database update, the catalog can point at nothing. Neither partial order is a sufficient audit trail.
Consider one receipt with an original, three thumbnail widths, a failed resize attempt, and an export copied to an audit queue. A request to erase it cannot safely mean “delete the row and let a bucket rule catch up,” because that loses the relationship between the authoritative file and its derivatives while leaving the export unaccounted for. The record should instead enumerate every object key and its state, so the deletion worker can retry the original and each derivative independently, record which operation completed, and leave an auditable pending state when a policy exception prevents removal. The same inventory helps routine lifecycle design: a thumbnail prefix can expire after the original remains available for regeneration, while an original prefix can be excluded from automated expiration and handled through the approved retention workflow. This is slower to design than a single time-to-live, but it makes a large-file pipeline explainable when support, security, or compliance asks what happened to one receipt.
Keep it private.
GDPR Article 17 is a useful reminder that an erase button is a workflow with evidence, not merely a successful HTTP response. Retention exceptions, legal holds, backups, replicas, and exported audit material need explicit policy decisions. The regulation does not tell your team which prefix to expire; your data inventory has to do that work.
Options and their failure boundaries
| Design choice | Where it fits | What changes the decision |
|---|---|---|
| Private object storage plus an application catalog | High-volume originals and regenerable thumbnails | Requires a reliable state machine and deletion worker |
| Binary columns in the primary database | Small bounded files needing one transactional backup path | Large receipt reads can compete with transactional queries and backups |
| One mutable key per thumbnail size | Simple prototypes with no concurrent rendering | Late workers and caches can publish stale bytes |
| Public object URLs | Public, non-sensitive media | Customer receipts and audit files should not be public |
The trade-off is operational complexity in exchange for a storage path that can absorb large files without making every database operation carry the binary payload. The selected design is not suitable when the system requires a single atomic transaction that includes the object bytes, or when the team cannot operate an asynchronous retry and deletion queue. In those cases, keeping bounded files in the existing database may be the more honest choice.
Do not make the cheapest nominal storage rate the decision rule. Compare ingress and egress assumptions, request charges, retrieval behavior, replication, backup treatment, deletion requirements, and the engineering time needed to recover an orphaned object. Your mileage may vary because traffic geography and receipt-size distribution can change the result more than a posted storage number.
An ADR worth keeping after launch
Record the rejected option and the reason it was rejected. For this system, the rejected default is one mutable thumbnail path, because publication and cleanup become difficult to reason about when workers race. The valid use case is a prototype where a single worker owns each key and losing an older derivative has no audit consequence.
Also record the operational thresholds: maximum accepted receipt size, concurrency limits, retry budget, URL lifetime, retention class, and the alert for a record stuck between upload and publication. Review those values with real percentile data. I'm not sure which threshold will dominate before the first production distribution is visible, and pretending otherwise is how capacity plans acquire false precision.
The decision is narrow: keep originals private and durable enough for the audit policy, make derivative keys immutable, publish only completed objects, and let the application—not a guessed URL or an over-broad lifecycle rule—own authorization and deletion.
