Perforvisor: a 5★ review said "Needs to be more clean." Open models caught it.

This is a submission for the Hacktoberfest Open-Source AI Challenge: Week 2 - Side Hustle.

A small restaurant owner has hundreds of reviews and no time to read them. The star average says 4.1. It doesn't say what to fix this week.

I built Perforvisor to answer that one question: what should I fix first? It reads every review with a small open-weight model, turns what customers say into a knowledge graph and gives the owner a verdict, up to three concrete actions and the reviews behind each one.

The review that made it click

While testing on a real café in Indore (Indian Coffee House, DAVV, synced live from Google Maps), Perforvisor flagged this one:

★★★★★ "Needs to be more clean."

Five stars. One complaint. A star-average dashboard counts that as a happy customer.

Perforvisor reads the words, not the stars. The model read the text as negative and the Trust page flagged it as "Star rating contradicts the text". Because it's flagged, it can't inflate the praise list or the advice either.

I had tested this on purpose earlier: I added a 5★ review that was really a complaint and asked "What do users complain about most?" The answer counted it as a complaint, with the review cited. Seeing the same thing happen on a real review, unprompted, is when I knew the idea worked.

What I Built

Connect a business (search Google Maps, load a sample or paste reviews from any platform, one per line with an optional rating) and Perforvisor:

• Reads every review. For each one, Qwen3-4B pulls out the aspects people talk about (food, service, value, wait time, cleanliness…), the sentiment for each aspect and short concrete claims ("staff were rude at the counter").
• Builds a knowledge graph of claims. Claims that say the same thing in different words merge into one node. On the café, "The service staff members were exceptionally polite and attentive" and "staff was very polite" became one claim backed by two reviews.
• Writes the advice. A verdict under 40 words and up to three actions for the owner, most-mentioned complaint first.
• Shows trends. Rating and sentiment over time and review bursts: days with at least 3 reviews and 3× the usual daily volume, labelled as a 5★ burst, a 1★ burst, or a volume spike.
• Flags reviews you shouldn't trust. Near-copy-paste text and ratings that contradict the words. Flagged reviews are kept out of praise, complaints and advice.
• Compares you with competitors on the same aspects, side by side.
• Answers questions like "Is the wait long on weekends?" from the reviews it retrieves, with those reviews cited.

Who pays and why

The buyer is the owner of one location: a restaurant, café, salon, clinic or shop with a few hundred reviews. They know reviews matter. They don't read all of them every week and reputation-management suites are priced and built for chains.

What they get for the money is a weekly to-do list grounded in their own customers' words, plus an early warning when something changes.

Pricing (a proposal, not live yet):

Demo

• Live app: https://perforvisor.vercel.app
• Code: https://github.com/KrAG2000/perforvisor

How It Works

Vercel runs the app, the API and background syncs. Two vLLM servers share one L4 GPU on GKE. MongoDB Atlas holds the records and the knowledge graph; Tiger Data holds vectors and time series.

One sync of a business goes: fetch new reviews → store in MongoDB and embed into Tiger Data → extract each review → merge claims into the graph → rebuild the report. The app streams progress the whole way.

Five decisions I'd make again

1. A claim takes its aspect's sentiment, not the review's. "Taste was fine" inside an angry review about the wait isn't a complaint about taste. Each claim is tagged with an aspect and it inherits the sentiment the model gave that aspect. The review's overall sentiment is only the fallback.

2. Claim merging is two-stage, because one threshold can't work. I measured EmbeddingGemma 2 on pairs of claims: same-meaning pairs scored 0.867 to 0.957, different-meaning pairs 0.772 to 0.912. Those ranges overlap, so any single cut-off either merges different things or misses real matches. So:

• Candidates must share the business, sentiment and aspect.
• Cosine similarity ≥ 0.93 merges automatically.
• 0.86 to 0.93 goes to Qwen3-4B as a yes/no question ("If they name different dishes, products or features, answer no").
• Below 0.86 stays separate.

The cheap path handles the clear cases and the model only judges the grey zone. In testing, the judge leaned towards keeping claims apart, which is the safer mistake for counts.

3. I removed the reranker after measuring it. My first design had a small LLM reranker. On my labelled set it lowered rerank score from 0.86 to 0.63. It scored "overheated on a flight" 2/10 for the question "Does it overheat?". Now retrieval is plain cosine top-8 over EmbeddingGemma 2 and Qwen3-4B, while writing the answer, cites only the reviews that actually support it. Citations are checked against what the model was shown, so it can't cite a review it never saw.

Answers: citation precision 0.79, recall 0.79, median 1.0 s per answer (measured on my laptop's RTX 4060). It's a small test set, so I treat it as a direction, not a benchmark. You can rerun it with pnpm --filter @perforvisor/backend eval:retrieval.

4. Only extreme mismatches count as suspicious. On real reviews, 4★ with "the gravy could be better" is normal. So a review is flagged only at the extremes: 5★ with negative text, or 1★ with positive text. Copy-paste detection compares word 3-grams (Jaccard ≥ 0.7) and skips reviews under 5 words, because lots of genuine people just write "Nice place".

5. Code decides the facts, the model writes the words. Small models mixed up strengths and weaknesses when I asked them to list both. Now strengths and weaknesses come straight from the classified claims in code and the model only writes the verdict and actions, with the rule "never suggest fixing something customers praise". Thinking mode is off for every call: each one is schema-constrained JSON and thinking would multiply latency for little gain.

Two problems that took longest

Serverless time limits. A first sync reads dozens of reviews with a model and a Vercel function stops at 300 s. Each run stops starting new reviews at 240 s and queues a continuation. Every step skips finished work (analysed reviews, reviews already in the graph), so the next run picks up exactly where the last one stopped. Failed runs retry with backoff and an idempotency key per business per minute collapses double-clicks and overlapping crons into one sync.

EmbeddingGemma 2 crashed vLLM on the L4. vLLM sizes its prefill attention tiles for A100/H100 shared memory. The L4 has about 99 KB per block, so the embedding server died at startup. I patched the kernel file and mount it into the container with a ConfigMap until it's fixed upstream.

Why Open Innovation Matters Here

The cost has to be flat, because the product re-reads everything. Every review gets extracted, every new review gets embedded, every grey-zone claim pair gets a judge call and every question gets an answer. With per-token pricing, each of those is a line on the bill and the bill grows with every owner and every sync. With my own models, the GPU is a fixed cost that every business shares. The only thing left to do for me is to make sure that models are available and can handle requests from the growing clientele.

Here is the real arithmetic, from public prices (estimates, October 2026, us-central1):

So at the proposed $9/month, even an always-on GPU and load balancer are covered by about 40 paying owners and each owner after that mostly adds SerpApi cost. The GPU doesn't charge me more when the model reads more. I'll be honest about the other side: with only a few businesses, a cheap hosted API would cost less than an idle GPU. The open setup wins once the volume is steady and it keeps the cost predictable either way.

Small models are enough when the job is narrow. Pulling aspects, sentiment and claims out of one review is a well-defined task. A 4B model with schema-constrained output does it and a 0.74B embedding model handles retrieval and claim matching. Both fit on one 24 GB GPU with room to spare.

I control the knobs. I could switch Qwen3's thinking mode off, pin exact model versions, patch vLLM's kernel for my GPU and use EmbeddingGemma 2's task prefixes and Matryoshka truncation. Chat, extraction and embeddings are separate roles with their own URL and model name, so swapping the extraction model for a fine-tuned one later is a config change.

No model provider sees the data. Reviews and owners' questions go to models running on infrastructure I control. No third-party model provider gets them and no one can deprecate my model out from under me.

Prize Categories

Best Use of Gemma

EmbeddingGemma 2 is the retrieval and matching layer, served with vLLM on GKE (Google Cloud):

• It embeds every review and every owner question. Cosine top-8 over these vectors beat keyword ranking on reranker picks (0.86 vs 0.56) and was good enough that I deleted the reranker.
• It decides which claims merge in the knowledge graph. I measured its similarity ranges for same and different claims and set the 0.93 / 0.86 thresholds from them.
• I use it the way it was trained: task: question answering | query: for questions, title: … | text: for reviews and task: sentence similarity for claim-to-claim comparison. Vectors are 768-d, with Matryoshka truncation built in for smaller sizes.
• Getting it onto an L4 meant patching vLLM's prefill attention kernel (above).

Best Use of MongoDB Atlas

Atlas is the source of truth and the knowledge graph behind an app built on open-weight models:

• Collections for businesses, reviews, per-review analyses (with the model name) and reports.
• The knowledge graph lives in nodes (business, aspect, claim) and edges (HAS_ASPECT, HAS_CLAIM, ABOUT, COMPETES_WITH). Merged claims keep every wording as variants and every supporting review id, with first/last-seen dates. The café's graph has 62 nodes and 114 edges after one sync.
• The SerpApi cache is a collection with a TTL index, so expired responses delete themselves, plus a per-month usage counter for the budget guard.
• Sync progress is written to the business document and streamed to the app.

Best Use of Tiger Data

Tiger Data holds everything that's vectors or time:

• pgvector: review_search (one embedding per review) for retrieval and claim_vectors for finding merge candidates. Retrieval is always scoped to one business, so a btree on the business key plus an exact vector scan is fast and it avoids HNSW's recall problems under filters at this scale.
• I tested hybrid search and dropped it. Full-text plus vectors scored MRR 0.81 on my set; cosine alone scored 0.89.
• TimescaleDB: review_events is a hypertable (30-day chunks). Trends use time_bucket (day, week or month depending on the range) and burst detection is one query over daily buckets.

Best Use of SerpApi

SerpApi keeps the advice grounded in fresh reviews:

• Google Maps search to find a business, then Google Maps reviews with pagination: up to 3 pages on the first sync, 1 page on each daily re-sync.
• Every call goes through a guard: identical requests within 24 h are served from the MongoDB cache and a monthly budget (default 200 of the free plan's 250) makes calls fail cleanly instead of overspending.
• The Ask flow answers from those freshly synced reviews, with citations.

What's Next

• A consumer side: "is this place actually good at X?" lookups. I've designed it to be location-aware, because there are a lot of restaurants called "Paradise" and the answer has to be about the one near you.
• More sources: Amazon products and CSV import with column mapping.
• Alerts when a burst of complaints starts.
• A fine-tuned extractor: extraction is already its own model role, so a small fine-tuned model can replace it without touching the rest.

First User

Wrapping up

Perforvisor is a small, specific tool: one owner, one location, one question. Open models make it work as a business. They're cheap enough to read every review, I could tune them to the job and the data stays on infrastructure I run. A 5★ review that says "Needs to be more clean." is exactly what an owner needs to hear about and now they will.