Munchable scans barcodes, but a restaurant menu has no barcode. So the app has a second camera mode: point it at the menu, drag a box around a dish, let go. There is no shutter button. You can watch it in the third panel under How it works on munchable.app, where the demo drags a box around "Creamy garlic mushroom tagliatelle" the way a thumb would.
The interesting part is not the gesture. It is that a box drawn in view points has to become a rectangle in photo pixels, and every way of getting that slightly wrong fails quietly: the crop lands somewhere plausible, misses the last two lines of the dish description, and the user is told we could not read that. There is no broken screen to look at. So the geometry is 194 lines in a file that imports one type, and it is tested in numbers.
A drag has four corners and no direction
The first function exists because people do not drag top-left to bottom-right:
export function rectFromDrag( startX: number, startY: number, endX: number, endY: number, view: Size, ): Rect Both points are clamped to the view, then the rect is min/abs of the two. Dragging up and to the left produces the same rectangle as dragging down and to the right, and a finger that leaves the viewfinder mid-drag stops at the edge instead of producing a negative width.
Then there is a floor on what counts as a deliberate selection:
export const MIN_SELECTION: Size = { width: 44, height: 28 }; 44 by 28 points. Below that it was a tap, not a box, and nothing is read. That is the same 44 that every touch target guideline uses, for the same reason: it is roughly the smallest thing a finger means to do on purpose.
The mapping is a cover fit, and the offsets are negative
The preview is a centre-cropped cover fit of the sensor frame on both platforms (resizeAspectFill on iOS, FILL_CENTER in CameraX on Android), with preview and still capture sharing one aspect ratio. That makes the mapping one uniform scale plus a centring offset:
const scale = Math.max(view.width / photo.width, view.height / photo.height); const offsetX = (view.width - photo.width * scale) / 2; const offsetY = (view.height - photo.height * scale) / 2; Math.max rather than min is the entire difference between cover and contain, and it is the one character in this file I would most expect somebody to "fix". On a 390 by 844 phone screen showing a 3024 by 4032 photo, the scale is 0.20933, the photo is drawn 633 points wide, and 121.5 points hang off each side of the screen. The offsets are negative. A box at x = 24 on screen is at pixel 695 in the photo, not pixel 116.
The result is floored and ceilinged outward, clamped to the photo, and then one last guard:
if (width < 8 || height < 8) return null; Null, not a tiny rectangle. A caller that gets null says so, rather than sending an 8 by 3 pixel sliver off to be read and reporting back that the menu was unreadable.
The pad is a percentage, with a floor, because boxes differ by an order of magnitude
A drag ends where the finger lifts, which on a thumb is a couple of millimetres inside where the person thought they were pointing. So the box grows before it is mapped. Menu mode adds a flat 12 points. The label capture flow, where the same function crops a frozen still, adds a fraction instead:
export const CROP_PAD_FRACTION = 0.06; export const MIN_CROP_PAD = 8; Six per cent, never less than 8 points. The boxes people draw in that flow are not comparable in size: a two line "Ingredients: oats, water" on a porridge pot against the full wrap of a smoothie bottle. A fixed 12 points is a third of the height of the small box and invisible on the big one. And a percentage alone is worse at the bottom end: a box only just over the minimum would get 44 * 0.06 = 2.6 points of margin, which is less than the width of the box's own border and shaves the ascenders off the top line of text. Hence the floor.
The pad is applied in view points, before the mapping, which is the part worth stealing. The user gets the margin they would have drawn with their finger, rather than a margin that grows with the megapixels of whatever phone they bought.
Clamping comes after: a box drawn hard against the edge of the screen cannot ask for pixels that are not there. The pad is a courtesy, not a promise.
Eight tests, all of them arithmetic
The test file is 123 lines of numbers on the exact shapes phones produce (a 12MP 3:4 sensor, a 19.5:9 screen), and it asserts things no amount of looking at a device would tell you:
// 6% of 100 is 6, under the 8pt floor, so the small box grows by the floor. assert.deepEqual(padRect(small, view), { x: 92, y: 92, width: 116, height: 116 }); One test asserts that the padded crop strictly contains the unpadded one on all four sides and still lies inside the photo. One asserts that a 0 by 800 view and a 0 by 0 photo both produce null rather than a NaN rectangle. One asserts that the whole-photo escape hatch returns whole pixels, because a photo size can arrive fractional and originX: 0, width: 4032.7 is not a crop anything will accept.
These run with node --test in milliseconds, which matters more here than anywhere else in the app: this is the one piece of camera code whose correctness is a number, and the device is the slowest possible way to check a number.
The drawing and the deciding are on different threads
The box itself never goes through React. It is drawn on the UI thread from Reanimated shared values, so it sticks to the finger frame by frame. Only when the drag ends does runOnJS hand four coordinates and the canvas size to the JS thread, which builds the rect, asks whether it is big enough, and takes the picture. Everything outside the box dims to rgba(27, 18, 12, 0.55) so the chosen dish pops.
One more gesture detail, easy to get wrong in a camera on a scrolling page: the page's scroll view is told to wait for the drag to fail before it may scroll. A swipe over the viewfinder draws a box. A swipe anywhere else on the page scrolls it normally.
The user who cannot drag
A drag is not a gesture a screen reader user can perform, and "draw a box around the dish" is not an instruction that can be read out usefully. So the whole canvas is one accessible element, and activating it reads everything in view:
export function wholePhotoRect(photo: Size): CropRect | null That is the no-drag path and the accessible path at once, and it is four lines because the crop that keeps the whole photo is not a special case of anything. It is less precise than a box, which is a real cost. It is also the difference between a feature that has an alternative and a feature that simply excludes people.
Two things I would not have predicted when this started: that the hardest part of a Lens style selection would be a Math.max and a pad fraction rather than the gesture, and that the honest version of the feature needed a function that ignores the gesture entirely.
Menu mode comes with a hard cap that is worth reading on the live page rather than taking from me, since menus do not print full ingredients: the homepage panel says a clean read there is never a green light, and the conditions the app checks against are listed at munchable.app/conditions.
