This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle.

I Built an AI That Checks Whether Digital Products Deliver What They Promise

A digital-product listing can promise “48 pages”, “A4 and US Letter editions”, and “debt repayment worksheets.” Then the customer opens the ZIP and finds something else.

For a small seller, that turns into refunds, support messages, rough reviews, and the awful feeling that you missed something obvious. PromiseDiff is a local-first pre-publish check for Etsy, Gumroad, template, and educational-PDF sellers.

Paste in the listing. Upload the actual PDF or ZIP. PromiseDiff shows what the files can back up, what they contradict, and what they simply cannot prove.

The rule I cared about most: missing text is not a failure. A measured mismatch is a contradiction; a claim the files cannot prove is just unverified.

What I built

Every claim gets one of four plain outcomes:

Result	Meaning
`SUPPORTED`	The supplied evidence supports this exact promise.
`CONTRADICTED`	A reproducible check established a conflicting fact.
`UNVERIFIED`	The files do not establish the claim. Missing evidence is not a contradiction.
`OUT_OF_SCOPE`	The claim needs a capability the MVP cannot test.

Every result keeps the method and the evidence that led to it.

For example, a page-count claim is deterministic:

Listing: “The main PDF contains exactly 48 pages.” File: budget_planner_A4.pdf → 48 pages Result: SUPPORTED — DETERMINISTIC

“Includes debt repayment worksheets” is different. A PDF might call that page a “Loan Balance Reduction Tracker.” Here Gemma looks at the extracted text, but PromiseDiff only accepts the result after checking the suggested filename, page, and quote against the actual PDF.

Demo

The demo uses a real 48-page A4 Budget Planner PDF. Nothing is mocked.

I used ProgressCut to cut the screen recording down to a short walkthrough.

The listing promises:

The main PDF contains exactly 48 pages. Includes both A4 and US Letter PDF editions. Contains monthly budget review worksheets. Includes savings goal trackers.

The upload only has the A4 edition. PromiseDiff confirms the page count, flags the missing US Letter version, and lets the seller open the evidence or export the report as Markdown.

How I built it

Why a hybrid system instead of “ask the model”?

Some questions have a simple answer; others need interpretation.

Question	Correct tool
How many PDF pages exist?	PyMuPDF metadata
Is this page actually A4?	PDF geometry with a 3-point tolerance
Does a listed file exist?	Complete file inventory
Does “Loan Balance Reduction Tracker” plausibly satisfy a debt-repayment claim?	Gemma plus validated quotation

Asking a language model to count pages is slower and worse than reading the PDF metadata. But keyword search alone will not reliably connect “debt repayment” and “loan balance reduction.” So each tool only gets the jobs it can support with evidence.

The rule is:

missing evidence != contradiction

If the document is scanned, the claim scope is ambiguous, a citation points to a nonexistent page, or a quote does not occur in extracted text, the answer is UNVERIFIED.

Architecture and safety

It is a small React + FastAPI app. The verification-flow diagram above shows how the pieces connect.

Uploads are untrusted, so the backend puts tight boundaries around them: size, PDF count, total pages, extracted text, ZIP ratio, file types, and expanded ZIP size. It rejects traversal paths, duplicate entries, encrypted PDFs, nested archives, and corrupt files. The original upload is never saved.

The model work is deliberately kept small:

Gemma extracts structured promises from the listing.
Python validates source quotations and normalizes measurable claims.
Local retrieval selects compact, potentially relevant PDF excerpts.
Gemma evaluates semantic correspondence in batch.
Evidence validation checks filename, page, and exact quote before serialization.

Why does open innovation matter?

Many sellers are working on unpublished material: course notes, templates, client downloads, and draft listings. With the local setup, Ollama keeps the listing text and extracted PDF text on the seller’s machine.

Open-weight Gemma also makes this affordable for a small operation. A local run has no per-audit third-party API bill, although someone still pays for the hardware or hosting. Sellers can repeat the same check before every release without handing their draft material to another service.

Reproducible benchmark, including a failure

I made ten independently labelled synthetic PDF cases and ran them through the same local API the UI uses. The runner saves the raw responses and timings in benchmark/results.json.

exact-label accuracy = correct labels / total cases = 9 / 10 = 90%

xychart-beta title "Exact classification by category (10 synthetic cases)" x-axis [Supported, Contradicted, Unverified, OutOfScope] y-axis "Correct cases" 0 --> 4 bar [3, 2, 3, 1]

Metric	Result	Caveat
Exact-label accuracy	90% (9/10)	Tiny synthetic evaluation, not a generalization claim.
Contradiction precision	100% (2/2)	Both measured failures were page/dimension facts.
Contradiction recall	100% (2/2)	Same small set.
Supported recall	75% (3/4)	One semantic paraphrase was conservatively left unverified.
Median API latency	2.11 s	Local Gemma on this machine.
Mean API latency	3.50 s	Prompt-injection case took 13.81 s.

The miss is the useful bit. The listing said debt repayment worksheet and the PDF said Loan Balance Reduction Tracker. The model did not have strong enough evidence to connect them, so PromiseDiff returned UNVERIFIED instead of pretending it knew. That is a limitation to fix, not a result to hide.

Known limitations

The benchmark is small and synthetic. It is not a claim about every kind of seller PDF.
Scanned documents without extractable text can still receive metadata checks, but semantic claims remain unverified.
The current retrieval can miss paraphrases. In the benchmark, “debt repayment worksheet” and “Loan Balance Reduction Tracker” stayed unverified rather than becoming a false positive.
Public deployments should isolate PDF processing with strict resource limits.