My upload pipeline turned every screenshot into JPEG. The text got noisy and the files were bigger than PNG-8

My small tool site gets more screenshots than photos. For a long time the upload handler treated them all the same way: decode, re-encode as JPEG, store. Then a user sent me a screenshot of a chat log and asked why the text looked fuzzy after upload. I went back and measured, and the short version is that JPEG was the wrong call for screenshots twice over. The text picked up a ring of noise, and the file was not even smaller than the lossless alternative.

Reproducing it without user data

I didn't want to test on a user's image, so I made two of my own. I wrote two throwaway pages and captured them at 1280×800. One is a page of Chinese meeting notes with made-up content (174 distinct colors). The other is a fake admin order table (1,402 colors). I encoded both with sharp 0.34.5 on an M4 Mac. JPEG uses sharp's defaults (4:2:0 chroma subsampling). The palette PNG comes from sharp's bundled libimagequant, capped at 256 colors.

So my old pipeline at q60 produced a 124 KB file for the text screenshot. The palette PNG was 116 KB, and because the source only had 174 colors it decoded back to exactly the same pixels. For the table screenshot the gap was wider: 58 KB as a palette PNG versus 112 KB as JPEG q80. Zoomed in, both JPEG versions show grey speckles in the white around each character, worse at q60. Colored status badges in the table (green text on a light green pill) get dirty edges with a faint color halo. My guess is the chroma subsampling hurts small colored text the most, but I didn't test a 4:4:4 JPEG to confirm that.

WebP q80 landed at almost the same size as the palette PNG, about 3% smaller for the text shot. I still went with the palette PNG. WebP is lossy here and the glyph edges are slightly soft, while people use my site to look things up in old screenshots.

Photos should stay lossy

My first instinct was to switch everything to PNG. Two 1280-wide landscape photos talked me out of it. As 256-color PNGs they came out 2.5 to 3.8 times larger than JPEG q80. Quality wasn't better either. A colorful fruit-stall photo scored SSIM 0.9309 as a palette PNG, below JPEG q60 at 0.9684. A sky gradient showed visible banding with dithering off and turned grainy with dithering on. Not every gradient breaks, though. A single-tone foggy area in the same photo looked fine at 256 colors.

Routing by color concentration

Counting unique colors sounded like the obvious classifier. It falls apart once a screenshot has already been through JPEG somewhere else, which can easily happen before a screenshot ever reaches your server. My table screenshot went from 1,402 colors to 21,500 after one JPEG q80 pass. A 413×579 portrait (a CC BY-SA headshot from Wikimedia Commons that I cropped for another test) has 26,827. Same ballpark.

What held up was how concentrated the colors are. I measured the share of pixels covered by the 16 most frequent colors. The two screenshots scored 0.920 and 0.954 as lossless originals, and still 0.838 and 0.745 after JPEG. None of the four photos I checked went above 0.135. I picked 0.5 as the cutoff because it sits far from both groups.

async function dominantShare(input) { const { data, info } = await sharp(input).removeAlpha().raw().toBuffer({ resolveWithObject: true }); const pixels = info.width * info.height; const hist = new Map(); for (let p = 0; p < pixels; p++) { const key = data.readUIntBE(p * 3, 3); hist.set(key, (hist.get(key) || 0) + 1); } const top = [...hist.values()].sort((a, b) => b - a).slice(0, 16); return top.reduce((sum, n) => sum + n, 0) / pixels; } 

Anything above 0.5 goes to png({ palette: true, colours: 256, compressionLevel: 9, effort: 10 }), the rest to jpeg({ quality: 80 }). On my four samples the screenshots came out at exactly 116,231 and 58,317 bytes. Leaving compressionLevel and effort at sharp's defaults made those two files about 1 KB and 2 KB bigger.

For a sanity check I compared against an existing tool. I used the image compressor in ImgIng (https://imging.ai/), which runs in the browser, and loaded the same files in Chromium 149. It labeled both screenshots as icon or line art and defaulted to an 8-bit palette PNG. The text one came out as a 174-color indexed PNG, identical to the source. The landscape photo was labeled a photo and defaulted to WebP at quality 75. It misjudged the fruit-stall photo as a screenshot or UI, but the only effect was quality going from 75 to 84 while the format stayed WebP.

What I haven't solved

This is four test images at one width, eight if you count the two JPEG re-saves and the two portraits I used for the color features. The 0.5 threshold is untuned, so I can't claim any accuracy number. Screenshots with a large embedded photo and dark-mode screenshots are the cases I expect to land near the cutoff. I don't have good samples of either yet. Edge density was the other feature I tried, and it only separated the fruit photo from the table screenshot by about 3x, so I dropped it.

If your upload path also converts everything to JPEG, pull a dozen real screenshots from your logs and run them through the function above. Then save each one as JPEG and as a palette PNG and compare sizes before you change anything.