Most face-shape tools on the web have the same problem: whatever photo you give them, they hand back a confident label. Ours did too. For a while our classifier called 9 out of 9 real test photos "Diamond". After I fixed that, it looked fine on every sample photo I tried, and it was still wrong for most real users.
I work on The Face Analysis, a free in-browser face analysis site. This post is about how we found out: running the site's own measurement code on a public research face set and changing the engine based on the numbers, not on how the output looked.
Everything below runs client-side: MediaPipe Face Landmarker (@mediapipe/tasks-vision 1.0.1, WASM) places 478 points on one photo, and plain TypeScript turns those points into ratios.
The pipeline in one paragraph
The landmarker returns 468 face points plus 10 iris points, each with normalized x, y and a relative z. Every measurement is 2D geometry over a handful of those indices; z is ignored on purpose because its scale differs from x/y. The WASM bundle and model download only when the user starts an analysis (about 3 MB, which would wreck LCP if it loaded with the page), and the photo stays in the browser the whole time.
The indices that matter for face shape (from lib/analysis/geometry.ts):
| Measurement | Landmarks | Note |
|---|---|---|
| Face length | 10 → 152 | 10 is the top of the forehead surface, not the hairline |
| Cheekbone width | 234 ↔ 454 | the widest point on every face we measured |
| Forehead width | 21 ↔ 251 | temples; 70/300 (brow tips) were the original choice and were wrong |
| Jaw width | 172 ↔ 397 | gonial area |
| Pupils (iris centres) | 468, 473 | only present in the 478-point mesh |
Bug #1: the forehead was measured at the eyebrows
The first version measured forehead width between 70 and 300, the outer ends of the eyebrows. On 27 faces that distance came out at 0.84–0.89 of cheekbone width. Diamond's first condition is "cheekbones clearly wider than forehead", so it was true for practically every face. Switching to the temple points 21/251 gave 0.92–1.00 on the same faces.
There was a second problem. The length thresholds (1.2 / 1.5 / 1.6) came from the usual face-shape taxonomy, but that taxonomy measures length from the hairline, and landmark 10 sits well below it. Our length-to-width values on 32 faces ran from 1.119 to 1.347, so the "> 1.5 is oval", "< 1.2 is round/square" and "> 1.6 is oblong" rules mostly couldn't fire. Diamond was the only label left. The thresholds came from someone else's ruler.
I recalibrated on those 32 faces and all seven labels became reachable. But most of the 32 were synthetic, and none were labelled. That turned out to matter.
Measuring the engine on real faces
For real data we used the Face Research Lab London Set (DeBruine & Jones, figshare doi:10.6084/m9.figshare.5047666, CC BY 4.0). It has 102 adults (49 women, 53 men, ages 18–54), neutral and smiling front photos, attractiveness ratings from 2,513 raters, and researcher-placed 189-point templates. The photos are not in our repo and we don't republish them.
The harness has two rules: use the same model build the site ships, and run the same analyzeLandmarks function, compiled from the site's TypeScript. Extraction drives a real Chrome through Playwright, because the engine needs WASM and WebGL and doesn't run in plain Node:
// extract-landmarks.mjs (study folder, trimmed) const b = await chromium.launch({ channel: "chrome", headless: true }); const p = await b.newPage(); await p.evaluate(async () => { const v = await import("https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@1.0.1/vision_bundle.mjs"); const fs = await v.FilesetResolver.forVisionTasks( "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@1.0.1/wasm"); window.__lm = await v.FaceLandmarker.createFromOptions(fs, { baseOptions: { modelAssetPath: MODEL_URL, delegate: "GPU" }, runningMode: "IMAGE", numFaces: 1, }); }); // ...for each photo: decode, detect, keep { x, y, z } for all points measure.mjs then feeds the stored landmarks into the compiled engine under several conditions: neutral, smiling, rolled 5° and 10°, downscaled to 338 px, darkened to 45%, a three-quarter view, and a 3:4 portrait crop. The model found a face in all 102 photos under every condition.
Bug #2: the one that never threw
The 3:4 crop condition is where the worst bug showed up. MediaPipe normalizes x by image width and y by image height. Our distance function treated them as the same unit:
// lib/analysis/geometry.ts export function dist(a: Landmark, b: Landmark): number { return Math.hypot(a.x - b.x, a.y - b.y); } On a square photo that's correct. On a 3:4 phone portrait, a horizontal distance in normalized units is 4/3 of the same distance measured vertically. Every face looks wider than it is. Length-to-width ratios drop, jaw angles and eye-tilt angles shift, and symmetry moves too.
When we cropped the same 102 faces to 3:4, the old engine changed the face-shape label on 27 of them and called 59 of them "round".
It never crashed or returned NaN; every label was a valid label. Our sample photos were all square, so the dev loop looked correct, while phone selfies are almost never square.
The fix is one function that rescales x into height units before any measuring happens. Ratios are scale-invariant, so nothing else in the engine had to change:
// lib/analysis/geometry.ts export function toUniformScale(l: Landmarks, width: number, height: number): Landmarks { if (!width || !height || width === height) return l; const k = width / height; return l.map((p) => ({ ...p, x: p.x * k })); } It has to be applied at the boundary, and only to the measurement path. Drawing code (the mesh overlay on the photo) still needs the original normalized points:
// lib/analysis/index.ts — analyzeImageDetailed() const landmarks = await detectLandmarks(img); if (!hasAllPoints(landmarks)) throw new NoFaceError(); // Measure with uniform-scale points; draw and sample colour with the normalized ones. const result = analyzeLandmarks(toUniformScale(landmarks, img.naturalWidth, img.naturalHeight)); After the fix, the 3:4 crop changes the label on 11 of 102 faces. That's in line with the landmark noise you get from simply rotating the photo, which I come back to below. The camera alignment path and the showcase got the same correction. The lesson I took away: test geometry code with non-square images. A square sample image means this whole class of bug can't show up.
Bug #3: thresholds that didn't generalize
With the aspect ratio fixed, the face-shape split on the 102 neutral faces was still badly off. We had calibrated on 32 mostly synthetic faces, and that didn't carry over to real ones:
| Label | Old thresholds | Recalibrated |
|---|---|---|
| Oval | 16 | 38 |
| Round | 32 | 13 |
| Triangle | 34 (30 of 53 men) | 10 |
| Square | 15 | 10 |
| Diamond | 5 | 9 |
| Heart | 0 | 12 |
| Oblong | 0 | 10 |
Two labels never occurred at all. The distributions explain why:
| Ratio (London set, n = 102) | p10 | Median | p90 |
|---|---|---|---|
| Length ÷ cheekbone width | 1.12 | 1.18 | 1.23 |
| Jaw ÷ cheekbone | 0.77 | 0.79 | 0.82 |
| Forehead (temples) ÷ cheekbone | 0.91 | 0.94 | 0.97 |
| Forehead ÷ jaw | 1.12 | 1.19 | 1.25 |
The old oblong cut-off was 1.32 and nobody's length ratio got there. The heart cut-off (forehead ÷ jaw ≥ 1.30) sat above the 90th percentile too.
The new thresholds come from this set's percentiles, chosen by a search that balanced two goals: every label should be reachable at a plausible rate, and small photo changes (5° and 10° rotation, downscaling, darkening) should flip labels as rarely as possible. They live in one exported object:
// lib/analysis/faceShape.ts (trimmed) export const FACE_SHAPE_RULES = { oblongRatio: 1.229, triangleJaw: 0.794, triangleForeheadToJaw: 1.125, roundRatio: 1.16, roundJaw: 0.791, squareJaw: 0.796, squareForeheadToJaw: 1.158, heartForeheadToJaw: 1.239, diamondForehead: 0.934, diamondJaw: 0.796, } as const; if (ratio >= R.oblongRatio) return "oblong"; // Must come BEFORE square, or square swallows it. if (jaw >= R.triangleJaw && foreheadToJaw <= R.triangleForeheadToJaw) return "triangle"; if (ratio <= R.roundRatio && jaw >= R.roundJaw) return "round"; if (jaw >= R.squareJaw && foreheadToJaw <= R.squareForeheadToJaw) return "square"; if (foreheadToJaw >= R.heartForeheadToJaw) return "heart"; if (forehead <= R.diamondForehead && jaw <= R.diamondJaw) return "diamond"; return "oval"; Two things here took me too long to learn. First, rule order is part of the model. A rule placed after a broader rule never fires, and an earlier version had a dead branch for exactly that reason. A small verification script in our make check now builds synthetic landmarks and asserts that every label in our eye classifier can be reached, because that kind of bug also fails silently. Second, the page copy that explains the rules on the face shape detector reads its numbers from FACE_SHAPE_RULES, so the text and the engine can't drift apart again.
The same pattern elsewhere: lips and iris scale
The same dataset caught two more bugs.
Downturned lips. The threshold was 0°, which looks like the obvious "flat mouth" line. But on neutral faces the corner angle is negative because of how the landmarks sit: p10 −7.2°, median −3.0°, p90 +1.4°. With a 0° cut-off, 82 of 102 neutral faces were labelled downturned. The constant is now DOWNTURNED_MAX_DEG = -6.5, roughly the bottom 15% of the neutral distribution, and 19 faces get the label.
Pupillary distance. This is the only absolute (mm) measurement on the site. It uses the iris as a ruler, assuming a horizontal visible iris diameter of about 11.7 mm. Comparing against the researchers' hand-marked iris borders showed that MediaPipe's iris diameter reads a median 5.9% too large, while the pupil-to-pupil distance matches the hand marks within 0.3%. So the error was in the scale, not in PD itself. After a ×1.059 correction, the median PD moved from 58.4 mm to 61.9 mm, and the error band we state went from ±2 mm to ±3 mm, which is the honest figure.
fWHR: publishing the reference instead of a verdict
On its own, an fWHR of 1.85 tells you nothing. Published means depend on the definition (brow or eyelid), the camera and the sample. So the fWHR calculator places your number in a distribution measured with the same engine: bizygomatic width (234/454) divided by the height from the upper lip to the brow midpoint, both measured along axes aligned to the eye line so that head roll doesn't mix width into height.
On the London set: median 1.78 (women 1.72, men 1.82), p10 1.64, p90 1.95. All 102 values ship as a sorted array in lib/analysis/ratioReference.ts, and the percentile function deliberately refuses to claim precision that 102 faces can't support:
// lib/analysis/ratioReference.ts export function referencePercentile(ref: readonly number[], v: number): number { let below = 0, equal = 0; for (const x of ref) { if (x < v) below++; else if (x === v) equal++; } return Math.min(99, Math.max(1, Math.round(((below + equal / 2) / ref.length) * 100))); } The same check produced a result we didn't like. The site's overall composite score did not predict human attractiveness ratings on this set (Spearman ρ ≈ −0.07, confidence interval spanning zero). We didn't tune it until it did. The score is now presented as a percentile against the 102 faces, and the methodology page says plainly that it isn't an attractiveness rating.
What never leaves the browser
- The photo. Detection and measurement run locally. The free tools make no upload request.
- The mesh.
analyzeImageDetailedreturns landmarks beside the result object, not inside it. The overlay draws them and then drops them. A 478-point mesh is closer to a face template than a set of ratios, so it isn't persisted. - History. The last 50 results go to
localStorageundertfa.measurements.v1, and they're only numbers: score, symmetry, shape labels, tilt, PD. No image and no landmarks.
The embeddable widgets run the same engine inside an iframe, with no analytics and no cookies in the frame.
Honest limits
- No labelled ground truth. Nobody in the London set has a "true" face shape. The thresholds spread real faces across the seven labels in a plausible way, and that's all we claim. We don't claim an accuracy rate.
- Borderline faces flip. Across the four small perturbations (two rotations, downscale, darken), about 17% of faces change face-shape label after recalibration. The pages say so.
- Pose and expression matter a lot. In the first robustness run, a smile changed the face-shape label on 36 of 102 faces and a three-quarter view changed 98 of them. Use a straight-on, neutral, evenly lit photo.
- One dataset. 102 adults photographed in London in 2012 isn't the world. The cohort is mostly white, and the engine doesn't know sex even where medians differ by sex.
- Not medical. These are ratios from a photo. They aren't a diagnosis, and they aren't a recommendation to change anything about your face.
Takeaways for anyone shipping landmark geometry
- Normalize coordinates to one unit at the boundary; test with non-square images.
- Don't borrow thresholds from a taxonomy that measures different points.
- Calibrate on real faces with the exact model build you ship.
- Assert every label is reachable. Rule order is logic.
- When the data contradicts a feature, publish the contradiction.
If you want to see the result, the free tools are on thefaceanalysis.com. I'm happy to answer questions about the harness or the geometry in the comments.
This article was written with AI assistance and reviewed and edited by the author.
