I run a visa agency in Bali. The other half of my week goes into Apify Actors, and one of them does OCR: photos, screenshots or scanned PDFs in, text page by page out, with a confidence score and optionally the box and confidence of every single word. I built the confidence part because the agency work demanded it — a passport page read wrong is worse than one not read at all, and I wanted a number that said which scans to put in front of a human.

Then I spent a morning measuring what that number actually measures. It is not what I assumed, and the way everybody uses it makes the output worse while making the score look better.

One page, 266 words, 91.6%

Here is a real run against a real scan — page one of the Einstein–Szilárd letter to Roosevelt, as hosted on Wikimedia Commons. A 765 × 991 JPEG, 114 KB, typewritten in 1939, scanned at something close to screen resolution: the letterforms come back 11 pixels tall.

{ "urls": ["https://upload.wikimedia.org/wikipedia/commons/f/f7/Einstein_Szilard_p1.jpg"], "language": "eng", "includePages": true, "includeWordBoxes": true, "minWordConfidence": 0 }

5.1 seconds, $0.011, one row:

{ "format": "image", "pageCount": 1, "ocrPageCount": 1, "language": "eng", "wordCount": 266, "charCount": 1497, "confidence": 91.6, "status": "ok" }

91.6 out of 100. On any dashboard that is a green page. Here is how it opens:

The author's surname is wrong. The building is the "Waite House". "Which" is "waich". Further down, "a nuclear chain reaction in a large mass of uranium, by which…" arrives as uranium,oy. And the last token on the page is abdaOl — whatever the engine made of the handwritten signature in the bottom-right corner, at x=553, y=939, pasted into the text like a word.

For comparison, my own clean 200-DPI test invoice — a page where every one of its 26 words comes back correct — scores 93.7. Two points and one tenth separate a perfect page from a page that misspells Einstein. If you route scans by average confidence, there is no threshold that puts these two on opposite sides.

The average cannot move

The reason is arithmetic. Across those 266 words:

mean confidence 91.6 median confidence 96.0 words below 95 83 words below 90 33 words below 80 17 words at exactly 0 4

Eighty-eight percent of the page sits at 90 or above, because most of a typewritten page is ordinary lowercase text and Tesseract reads it comfortably. The errors live in the thin tail: proper nouns, abbreviations, anything with a digit, anything touching the edge of the paper. Twelve bad words out of 266 cannot drag a mean more than a point or two — and a page-level score is a mean.

The same word makes the point on its own. bombs appears twice on this page, same typewriter, same ribbon, six lines apart:

{ "text": "bombs", "confidence": 95, "left": 106, "top": 787 } { "text": "bombs", "confidence": 44, "left": 115, "top": 880 }

Fifty-one points for identical ink. Confidence is a statement about one patch of pixels, not about the document — and averaging it over a page throws away the only part you needed.

The floor that deletes the evidence

So the obvious next move, the one in every OCR tutorial including my own Actor's field help: raise the minimum word confidence to cut the noise. Same file, one changed field, minWordConfidence: 80.

 floor 0 floor 80 wordCount 266 249 charCount 1497 1375 confidence 91.6 94.4

The score went up 2.8 points. The text got worse. Here it is:

Albert Old Grove 2d. ... Waite House Washington, Some recent work by and L. waich has been com- municated to me in leads me to expect that the element uran- ium may be turned into a new and important source of energy in the mediate future. ... through the work of Joliot in as well as and in America

Seventeen words were removed. Here is every one of them, with its confidence:

Binstein 61 D.C. 28 Sir: 0 ! 0 E.Fermi 78 Szilard, 70 manuscript, 70 im- 78 recomendations: 62 - 73 France 56 Fermi 67 Szilard 73 uranium,oy 0 bombs 44 air, 75 abdaOl 0

Three of the seventeen were genuine OCR errors (Binstein, uranium,oy, abdaOl). One is ambiguous — recomendations: is one m short of the word, and I cannot tell from this scan whether the engine or the typist dropped it. The other thirteen were correct, and they were the thirteen words a reader would most want: Fermi, Szilard twice, France, Washington's D.C., manuscript, bombs, the salutation.

Meanwhile the floor kept Waite — at confidence exactly 80, surviving because the filter is a strict less-than — plus waich at 83 and 2d. at 83, which should be Rd.

So the floor deleted seven proper-noun tokens, kept three of the six visible mistakes, and reported a higher score for the privilege. It is not denoising. It is selecting against rare words, and rare words are the content.

Read the tail, not the mean

What I do now: ignore the page average as a routing signal and count the tail instead. Word boxes are off by default; switch them on and the number is cheap to compute.

import requests rows = requests.post( "https://api.apify.com/v2/acts/lergassy~ocr-text-extractor/run-sync-get-dataset-items", params={"token": TOKEN}, json={"urls": [url], "language": "eng", "includeWordBoxes": True, "minWordConfidence": 0}, ).json() for row in rows: if row["status"] != "ok": print(row["source"], row["errorCode"]); continue for page in row["pages"]: weak = [w for w in page["words"] if w["confidence"] < 80] print(row["fileName"], "page", page["number"], "avg", page["confidence"], "weak", len(weak), f'{len(weak)/max(1,len(page["words"])):.1%}') for w in weak: print(" ", w["text"], w["confidence"], w["left"], w["top"])

On this page that prints avg 91.6 weak 17 6.4%, and the seventeen coordinates tell a human exactly where to look on the original. That is a review queue; an average of 91.6 is not.

Two notes on the same mechanism. Keep minWordConfidence at 0 in what you store and apply thresholds at read time, where they are reversible — once a word is filtered out during the run, nothing downstream can tell a deleted word from a word that was never on the page. And watch for confidence 0: four tokens here are at exactly 0, and because the filter is less-than, the default floor of 0 keeps them. abdaOl is in the stored text of a page whose status is ok.

What confidence is genuinely good for

One thing, and it is a big one: catching the wrong script. I ran my clean test invoice through four language packs.

language confidence wordCount text eng 93.7 26 correct deu 93.0 26 byte-identical to eng ind 92.5 26 byte-identical to eng rus 52.2 26 "М\/ОТСЕ 2026-114 / Мата Баа / Чаап Вауа..."

Two things fall out of that table. The Cyrillic pack on a Latin page collapses to 52.2 — so the average does catch a catastrophe, and 52 against 93 is a gap you can threshold on. But wordCount stays at 26 in all four runs, the garbage one included: the engine finds the same 26 ink blobs either way and only disagrees about what they say. Word count is not a sanity check.

The second thing is that I was wrong in my own documentation, where I wrote that picking the right language is "the single biggest factor in accuracy". On Latin-script print it is not: German, Indonesian and English produced byte-identical text here and sat within 1.2 points of each other. The dropdown matters for script — Cyrillic, Arabic, CJK, Devanagari — not for language. If your scans are English, German, Dutch, Indonesian or Portuguese in any mixture, stop tuning it and go find a higher-resolution scan. Resolution is what moved my numbers. If you truly do not know the script in advance, autoDetectLanguage runs Tesseract's script-detection pass first and switches packs when the script is clearly non-Latin.

Small print worth knowing

  • The same page as PNG and as PDF do not score the same. My invoice reads 93.7 as a PNG and 92.0 as a one-page PDF — identical text, identical 26 words. The PDF page is rasterised at 220 DPI before reading, so it is a different set of pixels. Do not compare confidence across formats.
  • Failed files are rows, not a failed run, and they are free. Wikimedia's thumbnail service answered 400 to three of my test URLs. I got three rows with "errorCode": "http-error" and usageTotalUsd of exactly 0 on each, while the run itself stayed SUCCEEDED. Check status per row; never infer per-file success from run status.
  • PDF pages that already contain text are copied, not re-recognised. pages[].source says which happened, ocr or text-layer, and copied pages are not billed as OCR pages. A copied page's confidence is not an OCR score at all.

When this is the wrong tool

  • Your files already contain text. A born-digital PDF, a .docx, a spreadsheet — rasterising those to read them back with OCR is slower, more expensive and strictly less accurate. Document Text Extractor reads the text layer instead.
  • You need handwriting. The signature on this page came back as abdaOl. The engine is built for printed type, and no confidence floor fixes that.
  • You need tables as tables. You get text in reading order plus word coordinates. Reconstructing columns from left values is doable and it is your code, not the Actor's.
  • You need an accuracy guarantee. For anything legal or medical the product is OCR plus a review queue. The confidence tail is how you build that queue, not how you skip it.

The Actor is OCR Text Extractor — images and scanned PDFs in 19 languages, per-page and per-word confidence, word boxes, $0.002 per file plus $0.009 per page that actually needed OCR.

If you keep one thing from this: a page-level OCR confidence is a mean, and a mean over 266 words cannot see the twelve that are wrong. Count the words below your threshold; never delete them.