[P] Built a 100% Client-Side Vision Pipeline for Real-Time Chessboard & Multi-Board Detection (Chrome/Firefox Extension) [P]

Hi everyone,

Inspired by tools like Chessvision.ai, I wanted to take a different architectural approach and build a browser extension (ChessInsights AI) that performs chessboard detection and piece recognition 100% client-side using local inference—with zero image data ever leaving the user's machine, support for detecting multiple boards in a single frame, and entirely free features.

The main goal was to bridge passive chess content (YouTube, Twitch, PDFs, articles) with active engine analysis without context switching: capture what's on screen and get a FEN string + engine eval in a couple of clicks.

System Architecture & Technical Approach

• On-Demand Capture (Multi-Board Support):
  • Instead of continuously sampling video frames, the extension captures a screenshot of the visible tab via the browser's tab-capture API, triggered by the user (either a one-click "Analyze" on the current view, or a "Photo mode" where you draw a selection rectangle to crop a specific region).
  • That screenshot is fed to a locally-run object-detection neural network (YOLO-style architecture via TensorFlow.js, WebGL/CPU backend) that outputs bounding boxes and confidence scores for chessboard-shaped regions, filtered with non-max suppression.
  • Because detection runs over the whole frame rather than assuming a single board, it can find and return several distinct chessboards in one screenshot (e.g., multi-diagram PDFs, news articles, or broadcast splits). Boards are currently expected to be roughly axis-aligned rectangles (perspective/homography correction for heavily skewed boards is planned).
• Piece Classification & Artifact Robustness:
  • Each detected board is cropped, split into its 8x8 grid, and each of the 64 cells is passed to a separate local CNN classifier (also TensorFlow.js) that predicts the piece type or empty square.
  • To handle video compression noise, stream overlays, arrows, and different 2D/3D board themes, the classifier was trained with augmentations focused on UI artifacts and low-resolution captures.
• Everything Runs In-Browser:
  • Both the detection and classification models run entirely inside the extension (in an offscreen document on Chrome MV3) via TensorFlow.js—no image or frame data is ever sent to a server.
  • Position analysis uses Stockfish compiled to WebAssembly, running locally in a Web Worker, so engine evaluation also happens fully offline.
  • Results are converted into a FEN string and shown in the extension's dashboard/board editor, where you can play out lines against the local engine.

Key Differences vs. Existing Tools

• Private by Design: No board images or video frames are ever uploaded—detection, classification, and engine analysis all run locally on your device.
• Multi-Board Processing: Native support for capturing multiple diagrams at once.
• Zero Cost / No Paywalls: Full feature set available out of the box.

I’d love to gather technical feedback from the community on client-side vision optimizations! For those building in-browser CV tools: what edge-case augmentation strategies or lightweight architectures have worked best for you when dealing with compression artifacts and overlay occlusions in real-time frame parsing?