I've been building a PDF-to-text tool that also has to cope with scanned pages. At some point "it works on my test PDFs" stopped meaning anything, so I set up a proper test: 36 public-domain book pages from Wikimedia Commons (1700s to 1900s) in German, English, Spanish, French, Hindi and Portuguese, six per language. Wikisource has proofread transcriptions of these pages, so I could score the OCR output with character error rate (CER) instead of squinting at it.

The engine is plain Tesseract 5 at 300 dpi. Everything below is about what happens around it.

One page is out of the averages: a handwritten Portuguese manuscript. Tesseract produced nothing usable (CER 100 %), and no preprocessing trick changes that. The numbers below are over the other 35 printed pages.

The baseline was fine. The phone photos weren't.

On the original scans the final pipeline lands at a mean CER of 5.5 % (median per language between 0.9 % for Portuguese and 6.8 % for Hindi). Old typefaces, long ſ, stains: a few wrong characters per hundred.

Then I simulated what people actually upload: a phone photo of a page. Uneven lighting, lower contrast, blur, heavy JPEG compression and a 2° skew. Same 35 pages, same pipeline: mean CER 20.8 %. With plain global (Otsu) thresholding and only deskew it was even worse, about 50 %. Whole regions of the page vanished because the shadowed half was darker than the "ink" threshold.

The obvious fix is adaptive thresholding (Sauvola, which Tesseract has built in via thresholding_method=2). That brought the phone photos from ~50 % to ~15 %. Better, but still bad, and it had a side effect I'll get to.

The fix that worked: divide by the background

What helped was boring and old: estimate the paper brightness and divide by it. Shrink the page, take a max filter so the text disappears and only the paper remains, blur it, scale it back up, divide.

import numpy as np from PIL import Image, ImageFilter, ImageOps def unevenness(im: Image.Image) -> float: """Spread of paper brightness over an 8x8 grid. 0 = evenly lit.""" small = np.asarray(im.resize((256, 256), Image.BILINEAR), dtype=np.float32) cells = small.reshape(8, 32, 8, 32).transpose(0, 2, 1, 3).reshape(64, -1) paper = np.percentile(cells, 90, axis=1) # brightest 10% of each cell ~ paper  return float(np.percentile(paper, 95) - np.percentile(paper, 5)) def flatten(im: Image.Image) -> Image.Image: """Divide by a blurred background estimate so uneven light becomes white paper again.""" small = im.resize((max(1, im.width // 8), max(1, im.height // 8)), Image.BILINEAR) bg = (small.filter(ImageFilter.MaxFilter(9)) # remove the text, keep the paper  .filter(ImageFilter.GaussianBlur(6)) .resize(im.size, Image.BILINEAR)) a = np.asarray(im, dtype=np.float32) b = np.maximum(np.asarray(bg, dtype=np.float32), 1.0) out = np.clip(a / b * 255.0, 0, 255).astype(np.uint8) return ImageOps.autocontrast(Image.fromarray(out), cutoff=0.5) im = Image.open("page.png").convert("L") if unevenness(im) > 30: # only touch pages that are actually unevenly lit  im = flatten(im)

The > 30 gate matters. Evenly lit scans don't get touched at all, so they can't get worse.

Result on the phone simulations: 20.8 % → 5.7 %, basically back to the 5.5 % of the originals. The same pages turned by 90° end at 5.1 %.

Why Sauvola is only a retry

Once Sauvola had rescued the phone photos, I was tempted to just use it everywhere. On the 35 original scans that didn't change the mean (5.7 % either way), but the median went from 3.8 % to 4.4 % and a clean Portuguese page roughly doubled its errors (1.0 % → 1.9 %). Clean pages don't need local thresholding, and it adds noise around faint letters.

So now it's a retry: Otsu first, and only if Tesseract's mean confidence is below 90 do I also run Sauvola and keep whichever result has more confidently recognized characters.

Throwing away lines Tesseract doesn't believe

Old books have stains, show-through from the other side, decorations. Tesseract happily "reads" those as lines of garbage. Its TSV output gives a confidence per word, so I average per line and drop lines below 40:

NOISE_CONF = 40 def is_noise(words: list[tuple[str, float]]) -> bool: """A line Tesseract itself barely trusts: stains, show-through, page edges, pictures.""" cs = [c for _, c in words if c >= 0] # conf = -1 means "no word here"  return bool(cs) and sum(cs) / len(cs) < NOISE_CONF # words come from: tesseract page.png stdout tsv (level 5 rows = words) lines = [join(words) for words in lines_of_paragraph if not is_noise(words)]

That was the single biggest win on the originals: mean CER 6.8 % → 5.5 %. One French page went from 9.4 % to 3.3 %, another from 47.9 % to 8.8 % (it had a large illustration that came out as a wall of junk characters).

Small things that cost me time

  • Fraktur needs its own model. For the German blackletter pages, the regular deu model gave 11–25 % CER, the frk model 4–14 %. Tesseract's script detection recognizes Fraktur reliably, so switching automatically is cheap.
  • Rotate: 90 from Tesseract's OSD means "turn it clockwise by 90". PIL's rotate() is counter-clockwise, so it's im.rotate(-90, expand=True). I had it backwards at first and turned sideways pages upside down.
  • Don't trust OSD's script guess on bad pages. On a poor scan it called Latin text "Cyrillic" at confidence ~1. I now only trust the script if its confidence is ≥ 3; otherwise I do a quick English pass and guess the language from the text.
  • Born-digital PDFs have their own trap. With pdfplumber's default tolerance, LaTeX papers came out as "Thedominantsequence…". extract_text_lines(..., x_tolerance_ratio=0.15) fixed the word spacing.
  • License check before picking a PDF library. PyMuPDF is great but AGPL, which I didn't want in a paid tool, so it's pdfplumber + pypdfium2 + Tesseract, all under permissive licenses.

What it costs to run

It runs on Apify with 1 GB of memory, which is a quarter of a CPU, and Tesseract on one thread. A dense OCR page takes roughly 18–25 seconds there, about $0.0011–0.0014 per page in compute (measured over three 10-page runs). A page that already has a text layer costs around $0.00004, which is why the tool only OCRs pages that need it.

Caveats

36 pages is a small sample, and the phone photos are simulated, not real photos. Old book typography is also harder than a modern invoice: on a clean German EU regulation rendered at 300 dpi I get 0.3–1.7 % CER. If you have a set of real phone photos with ground truth, I'd genuinely like to know how this holds up.

What do you use against uneven lighting: background division, morphology, or something learned?

The pipeline is available as a pay-per-use tool on Apify if you want to try it on your own PDFs: PDF to Text, Markdown & OCR.