Whenever I tried using cloud LLMs to get through 500-page technical textbooks or multi-thousand-chapter web serials, I ran into the same two failure modes:
- Lossy summarization flattens nuance: Standard AI summaries strip out equations, drop edge cases, and hallucinate connective tissue.
- Zero source traceability: When a summary makes a surprising claim, there is no deterministic link back to the exact paragraph in the original PDF or EPUB.
Over the past 9 months, I built Omnitome—an open-source (AGPL-3.0), local-first Windows desktop application that performs loss-bounded semantic compression (with a documented target of ~7:1) where every synthesized sentence carries a clickable citation back to the immutable source paragraph.
🎥 3-Minute Architecture & UI Walkthrough: https://youtu.be/Nmb-qiZpnrk
Here is how the multi-process architecture works under the hood, and the four hardest systems bugs I had to solve to make the entire pipeline run offline on a 4 GB NVIDIA RTX 3050 laptop GPU.
1. Multi-Process Isolation & The Sole SQLite Writer Law
Packaging a modern React UI, an asynchronous orchestration layer, and heavy CUDA PyTorch models into a single desktop app without freezing the UI required strict process boundaries:
- Tauri v2 Desktop Shell (Rust): Native window supervision, OS lifecycle management, and child process teardown.
- Node.js 22 LTS Orchestrator (
better-sqlite3): Pipeline coordinator and sole writer todata.dbin SQLite WAL mode. - Python FastAPI Sidecar (PyTorch FP16 CUDA): Runs BGE-M3 (1024-dim embeddings), BGE Reranker v2, and Kokoro-82M neural TTS.
Why Node.js Is the Sole SQLite Writer
While SQLite in WAL (Write-Ahead Logging) mode supports concurrent readers, allowing Rust, Node.js, and Python processes to write directly to data.db triggers SQLITE_BUSY lock contention during heavy ingestion. In Omnitome, Python sidecars and the React frontend have zero direct database write access—every mutation flows strictly through Node.js REST boundaries.
2. Crushing the 14.6 GB Node.js RAM Cliff (F65 Sequential Await Barrier)
In early builds, ingesting a 10-book, ~508,000-word multi-format test corpus caused Node.js heap memory to spike to 14.6 GB, crashing consumer laptops.
- Root Cause: Unbounded
Promise.allconcurrency queued hundreds of document chunks, embedding payloads, and DOM parsing buffers in memory simultaneously while waiting on the GPU sidecar. - The Fix (F65 Sequential Await Barrier): I refactored the 5-stage ingestion pipeline (
Parse -> Vectorize -> Classify -> Plan -> Compress & Attribute) behind a strict chunk-by-chunk asynchronous await gate with explicit buffer release. - Verified Result: Peak Node.js RSS dropped by 98%—from 14.6 GB down to 296 MB—processing all 508,000 words in 5.9 minutes.
3. Running 3 Neural Models on a 4 GB GPU (LifecycleManager VRAM Eviction)
Omnitome's local neural stack runs three models:
- BGE-M3 (FP16): ~2.1 GB VRAM
- BGE Reranker v2: ~2.2 GB VRAM
- Kokoro-82M Neural TTS: ~0.6 GB VRAM
Keeping all three resident requires ~4.9 GB of VRAM, immediately crashing a 4 GB RTX 3050 laptop GPU.
To solve this without tearing down the Python sidecar process between tasks, I implemented an asynchronous LifecycleManager state machine (UNLOADED -> READY -> EVICTED) with a sliding-window TTL governor:
- Models load in FP16 precision on demand.
- When a stage completes or sits idle past its TTL window, the governor evicts the model weights and flushes the CUDA cache, reclaiming 3.2 GB of VRAM.
4. Calibrating BGE-M3 Retrieval & Fixing the CJK Token Explosion
- CLS Pooling Calibration: Switching BGE-M3 vector extraction to calibrated CLS pooling improved our retrieval-separation metric 1.68x (from
0.2520to0.4225) across our 19-suite regression harness. - CJK Multilingual Token Safety: When testing unspaced Japanese texts, naive whitespace word counters treated entire paragraphs as a single "word," bypassing chunk guards and causing context-window overflows. Adding script-aware regex weighting (
~0.7words per CJK glyph) and sentence-boundary splitting (。!?\n) eliminated the overflow completely.
Try Omnitome v1.1.1 (Open Source)
Omnitome v1.1.1 is live today as a single 1.95 GB Windows x64 installer (bundling the Python + CUDA runtime so no local Python setup is needed):
- 📂 GitHub Repository & Installer: https://github.com/ayumu-ml/Omnitome
- 🚀 Product Hunt Launch Page: https://www.producthunt.com/products/omnitome
- 🎬 3:44 Video Walkthrough: https://youtu.be/Nmb-qiZpnrk
If you're building local-first AI tools with Tauri, Rust, or PyTorch on constrained hardware, I'd love to hear how you handle sidecar supervision and VRAM residency!
