Last March I lost a full weekend to two text files. Not to image generation, to CSVs and a spreadsheet I was keeping by hand.
I had a batch of 60 assets ready to go. I built the Adobe CSV by hand in a spreadsheet, uploaded, and got rejections back in minutes. The header was wrong. Adobe's upload CSV wants exactly 5 columns in a specific order, and if you hand-type the header or export from a spreadsheet that quietly renames a column, the whole file bounces. I fixed it, re-uploaded, then did the same dance for Vecteezy (4 columns) and Dreamstime (15 columns, the worst of the three). Three platforms, three different headers, one Saturday gone.
Then, tired and on my second coffee, I re-ran a batch I had already submitted. Double-posted a chunk of my catalogue to the same platform in one afternoon. That is the fastest way to get an account flagged, and I got to explain myself in a support ticket instead of making new art.
That weekend is why my pipeline now spends more code on metadata and tracking than on generation. Here is what I learned, with numbers.
Generation is the cheap part
People assume the image step dominates the cost. It does not. On my box, a 1 vCPU, 2 GB RAM server with no GPU, generation is a fixed, predictable cost per asset. The variable cost is everything around it: the metadata CSV per platform, the upload decision per file, the packaging. My parent pipeline is about 19,000 lines of code, and the scripts I reach for every single day are the boring ones: make_metadata.py and upload_tracker.py. Both use the Python standard library alone. No pip install, no API keys, no cloud account.
The math is simple. If a wrong header costs you one rejected upload cycle, and a rejected cycle costs you a day of waiting plus a manual fix, then a script that writes the correct 5-column Adobe header, the 4-column Vecteezy header, and the 15-column Dreamstime header pays for itself the first time it prevents one bounce. My 39 finished products did not get made faster because I generated images faster. They got made because nothing bounced.
One CSV builder, three headers, zero improvisation
The rule I follow now: the script writes the header the platform demands, and nothing else. Not a superset. Not "close enough". Adobe wants 5 columns. Vecteezy wants 4. Dreamstime wants 15. A spreadsheet export that adds a trailing column or a BOM will get rejected, and you will not see why from the error message.
So the builder takes a folder of files and a platform flag:
python make_metadata.py ./my-assets --platform adobe --out metadata.csv --ext .jpg Same folder, three runs, three correct CSVs. There is an optional --titles titles.json for real titles and keywords. Without it, the script title-cases the filename as a placeholder, which is fine for a first pass but you should replace it before submitting. Filenames as titles read like filenames, and reviewers notice.
The double-post problem is a memory problem
Here is the thing about double-posting: it is not a discipline problem, it is a state problem. If your pipeline does not remember what it already uploaded, then any re-run, any retry, any 2 a.m. "let me just push the rest" session can repeat work. Willpower does not fix that. A ledger does.
upload_tracker.py is about as plain as it gets:
python upload_tracker.py pending items.txt # what is NOT done yet python upload_tracker.py done "asset-001.jpg" python upload_tracker.py status Before any batch, I run pending on the working list. Anything already marked done does not get uploaded again. It is a few hundred lines of standard library code, and it has prevented exactly one career-limiting mistake per quarter since I built it.
Small server habits that kept this alive
Since everything ran on 1 vCPU and 2 GB RAM, a few habits mattered more than any framework:
- One process at a time. Never fan out image jobs in parallel on a single core.
-
gc.collect()after each asset, so memory does not creep across a 60-file batch. - Measure peak RSS with
resource.getrusage(RUSAGE_SELF).ru_maxrss, not your gut feeling. Average memory lies; the peak is what kills the process. - One file per subprocess for heavy render steps, so the OS actually gets the memory back.
- Inspect previews at 60 dpi instead of rendering at 300. Same information for a visual check, a fraction of the RAM.
None of this is clever. All of it is the difference between a batch that finishes and a batch that OOMs at file 47.
The folder layout is part of the cost control
One more thing that bit me: I deleted my research/ folder once to "clean up". My scrape logs and notes were the raw material for two later products and I had to redo the work. Now the structure is fixed: assets/ for raw output, output/ for submission-ready files plus the metadata CSVs, packs/ for finished zips and covers, research/ kept forever, scripts/ for the tools. A catalogue without a fixed structure turns into a junk drawer, and a junk drawer is where double-posting happens, because you cannot tell what you already shipped.
Where the tools live
Everything above is baked into the Pipeline Starter Kit: the same four scripts that packaged those 39 products (856 MB of output) on that 1 vCPU, 2 GB box. make_metadata.py writes the exact Adobe, Vecteezy and Dreamstime headers, upload_tracker.py makes double-posting structurally impossible, and md2pdf.py plus pack_product.py handle turning content into a sellable zip with a 1280x720 cover and a 600x600 thumbnail. Two third-party packages total, Pillow and reportlab, and the two scripts that matter most need no install at all.
If you want the metadata and tracking layer without rebuilding it from my mistakes, it is here:
Building digital products? Free guides & kits for digital product sellers.
