How to straighten and binarize a photo of a document with OpenCV

A document scanner app does three things to a phone photo: it finds the paper, warps its four corners into a rectangle, and turns the result into ink and paper. None of the three is exotic, but each has a place where it quietly goes wrong. The PDF Editor in the osFoundry desktop app does this with four corner handles and a Black and white button. This post rebuilds the same pipeline in Python with OpenCV and measures it, because there is a way to score it exactly: the photo is one I generated myself, so the page that was "printed" and the corners it was placed at are both known.

The test photo

make_scan.py renders page one of a made-up lease with Pillow (Times New Roman, 1275 × 1650, Letter at 150 dpi), then shoots it virtually: a perspective transform onto a 1500 × 2000 canvas, a wood-grain table, a soft shadow under the paper, brightness falling off toward the bottom right, a little noise and a JPEG at quality 82. The photo has 253 distinct grey levels, and the true corners are (212,236) (1318,184) (1392,1652) (150,1748).

Everything below runs with:

uvx --with opencv-python-headless --with numpy --with pillow --with pymupdf python -I measure.py 

OpenCV 5.0.0, NumPy 2.5.4, Python 3.12.15, on an Apple M1. Timings are single runs, so read them to the nearest order of magnitude.

Find the four corners

The paper is the big bright blob. Blur the photo so the text stops mattering, let Otsu pick the threshold, close the gaps, and keep the largest contour:

blur = cv2.GaussianBlur(gray, (0, 0), 6) t, mask = cv2.threshold(blur, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU) # t = 144 mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, np.ones((25, 25), np.uint8)) contours, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) c = max(contours, key=cv2.contourArea) hull = cv2.convexHull(c) 

Reducing the hull to four points with approxPolyDP is the usual next step, and it is where the first pitfall is. The epsilon that gives exactly four points depends on the photo, so loop over a range of epsilons and take the first that returns four. And even then the corners are only approximately right. Against the true corners, the four points from approxPolyDP were off by 3.0, 5.7, 9.9 and 6.4 px (top-left, top-right, bottom-right, bottom-left). At 1500 px wide that doesn't look like much, but after the warp the error is magnified by the scale factor, and text a few pixels thick is sensitive to it.

A better corner is the intersection of two well-fitted edges. Take the contour points that lie near each side of the rough quad, fit a line to them with a robust loss so that a few stray points don't tilt it, and intersect neighbouring lines:

def refine(q, pts): lines = [] for i in range(4): a, b = q[i], q[(i + 1) % 4] e = (b - a) / np.linalg.norm(b - a); n = np.array([-e[1], e[0]]) along = (pts - a) @ e; dist = np.abs((pts - a) @ n); L = np.linalg.norm(b - a) sel = pts[(dist < 12) & (along > 0.08 * L) & (along < 0.92 * L)] # skip the corners themselves  vx, vy, x0, y0 = cv2.fitLine(sel, cv2.DIST_HUBER, 0, 0.01, 0.01).ravel() lines.append((np.array([x0, y0]), np.array([vx, vy]))) out = [] for i in range(4): (p1, d1), (p2, d2) = lines[i - 1], lines[i] t = np.linalg.solve(np.array([d1, -d2]).T, p2 - p1)[0] out.append(p1 + t * d1) return np.array(out, np.float32) 

The error against the true corners drops to 2.0, 2.2, 6.5 and 2.0 px. The whole search takes 12 to 20 ms. The one corner that stays 6.5 px off is the bottom right, which is also where the shadow under the paper falls in this photo.

Warp to a rectangle

getPerspectiveTransform takes the four corners and the four corners of the output, and warpPerspective does the rest:

w = round((np.linalg.norm(q[1] - q[0]) + np.linalg.norm(q[2] - q[3])) / 2) h = round((np.linalg.norm(q[3] - q[0]) + np.linalg.norm(q[2] - q[1])) / 2) M = cv2.getPerspectiveTransform(q, np.float32([[0, 0], [w - 1, 0], [w - 1, h - 1], [0, h - 1]])) flat = cv2.warpPerspective(gray, M, (w, h), flags=cv2.INTER_CUBIC) 

The warp takes 2 to 6 ms. The second pitfall is the output size. Averaging the lengths of opposite edges is the obvious way to pick w and h, but perspective makes the near edge look longer than the far one and the average doesn't recover the true aspect ratio. Here it gave 1174 × 1488, a ratio of 0.789, for a page that is really 0.7727 (Letter). That is a 2% stretch. If you know the paper size, use it; otherwise, accept a slightly stretched page, as scanner apps do.

Warping first is not optional. Otsu on the unwarped photo, table included, marks 42.9% of the pixels as ink. The wood is darker than the paper and gets counted with the text.

One side effect shows up in the histogram. The photo has 253 grey levels; after the bicubic warp there are 255, because interpolation invents values between the old ones. It is a good reminder that "number of grey levels" only means something once binarization has run.

Binarize

Three ways to turn the flat grey page into two values:

def binarize(g, how): if how == 'global': # one threshold for the whole page  return cv2.threshold(g, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1] if how == 'adaptive': # threshold relative to each pixel's neighbourhood  return cv2.adaptiveThreshold(g, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 41, 12) if how == 'flatfield': # divide out the paper's own lighting, then one global threshold  bg = cv2.medianBlur(cv2.dilate(g, np.ones((7, 7), np.uint8)), 51) n = np.clip(g.astype(np.float32) / np.maximum(bg, 1) * 255, 0, 255).astype(np.uint8) return cv2.threshold(n, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1] 

To score them, I scaled each result to the ground-truth page and compared ink pixels. Exact pixel overlap (IoU) is harsh for thin text, since a stroke shifted by one pixel counts as a miss, so the table also gives precision and recall with a 3 px tolerance: the share of predicted ink that has true ink within 3 px, and the share of true ink that has predicted ink within 3 px. The true page is 1.85% ink.

On this photo they are indistinguishable. After the warp, the paper is much brighter than the ink everywhere, and a single threshold is enough. All three keep every stroke (recall 1.000) and make them a little too thick (2.6 to 3.2% ink against 1.85%).

The last row is the page the osFoundry PDF Editor saved for the same photo, scored the same way. I read the page image back out of the saved PDF with PyMuPDF, and it has exactly 2 grey levels and 2.36% ink. I haven't looked at how the editor does it, so this row is a comparison of outputs only. It shows that a result from a tool with four draggable handles lands in the same range as the scripted pipeline.

When the global threshold breaks

The photo's lighting was mild, which hides the reason the other two methods exist. To see it, I multiplied the flat page by a ramp that dims the lower right to 40% brightness, like a hand shadow across a form, and ran the three methods again:

Global Otsu turns the shaded half black, so recall stays perfect while precision falls to 0.07. Adaptive and flat-field results barely move. Adaptive is the one to reach for first: it costs about 8 ms more than global and needs no extra step. Flat-field gives nearly the same result and is slower (46 ms here), so there is no reason to prefer it on this page.

The code above is the whole pipeline; the scoring only compares ink pixels with the page the photo was made from. The editor is part of the free osFoundry desktop app, and the local-first page describes how osFoundry keeps work on your own machine.