When I started checking whether an AI search product could find a small tool, I ran into a measurement problem before a search problem: three different tasks were producing superficially similar answers.
A system might summarize a page after receiving its URL. It might find the same product from its name. Or it might recommend that product when someone only describes a need. These outcomes need different labels.
I maintain 拾玖搜证台, a browser-local recorder for small manual AI search experiments. This article explains the recording method, including a standalone JavaScript example. You can use the same method in a spreadsheet; the tool is optional.
Freeze the question before collecting the answer
Write down the exact question and its route:
| Route | What the question supplies | What a successful observation would establish |
|---|---|---|
| Name | A product name, without its URL | Correct identity, additional supported facts, and a source |
| Need | A task, without the product or author | An applicable recommendation with correct facts and a source |
| URL | A specific address | Content recognition and supported attribution for that address |
A passing URL check does not enter the independent-discovery denominator. Giving a platform, article title, or domain adds another retrieval hint, so I keep those diagnostics separate too.
Before making changes, save a small fixed set of questions. A question added later has no earlier baseline. Comparing its first answer against an older question is a different experiment.
Save evidence before assigning a score
For every attempt, preserve the question, full answer, and the links actually supplied by the assistant. Open the links and check whether they support the claimed identity and functions.
Record the date, product, entry point, mode, login condition, and known network condition. Mark unknown conditions as unknown. A fresh conversation under one account does not exclude account-level personalization.
Keep two fields separate:
- An observed answer: what appeared in the interface.
- A reviewed outcome: what the answer and its sources support.
If an assistant says it found an article in a feed, that is a claim to check. A correct title plus a general feed URL is not automatically a verified article source. Feeds also change, so preserve the observation time and avoid claiming that a current page reconstructs an older search index.
An interrupted attempt is not a negative answer
Here is a small executable teaching example. Its records are synthetic fixtures, not results from our project. This illustrative schema is not an import file for the recorder.
const rows = [ { id: 'name-not-found', route: 'name', status: 'complete', identity: false, facts: false, sourceVerified: false, recommended: false }, { id: 'url-read', route: 'url', status: 'complete', identity: true, facts: true, sourceVerified: true, recommended: false }, { id: 'waiting-for-verification', route: 'need', status: 'verification_required' } ]; function summarize(rows, routes) { const eligible = rows.filter(r => routes.includes(r.route)); const answered = eligible.filter(r => r.status === 'complete'); const passed = answered.filter(r => r.identity && r.facts && r.sourceVerified && (r.route !== 'need' || r.recommended) ); return { answered: answered.length, passed: passed.length, excluded: eligible.length - answered.length }; } console.log('independent', summarize(rows, ['name', 'need'])); // { answered: 1, passed: 0, excluded: 1 } console.log('URL diagnostic', summarize(rows, ['url'])); // { answered: 1, passed: 1, excluded: 0 } The synthetic URL success does not improve the independent result. The unanswered verification attempt does not lower the success rate either.
In a real browser test, a human must handle any site verification. Observe the original conversation afterwards instead of sending the question again. Preserve the interruption in an event log and count the eventual answer once. A historical interruption and an answered sample can describe the same attempt without creating two answers.
These booleans are reviewed judgments. The code does not inspect a source or decide whether a recommendation is suitable. That work remains explicit.
A practical browser-local workflow
In 拾玖搜证台, I paste the answer and actual citations, choose the route and batch, and record the review. Records stay in the current browser's localStorage; the application has no endpoint that uploads these records.
JSON preserves records for backup and transfer. CSV supports table analysis and GEO Doctor's sampling format. Markdown produces a readable report. JSON import merges by id without overwriting existing records, with a 3 MB file limit and a 2,000-record limit.
Export JSON before clearing site data or changing browsers. This is file-based transfer, not cloud sync. Opening the hosted application and following external links still makes normal network requests.
The recorder itself does not log into or send questions to an AI service, call model APIs, highlight semantic differences, or provide real-time collaboration. Browser execution used by the maintainer is a separate part of the experiment. If a spreadsheet is sufficient for your sample, keep using it.
What our current experiment actually shows
In our November 5, 2026, 17:49 (Asia/Shanghai) snapshot, the formal dataset contains 23 answered consumer-web tests: eight name-route answers, twelve need-route answers, and three URL diagnostics. None of the name or need answers passed our identity, facts, and source criteria. One of the three formal URL checks passed content and source review. The public experiment page continues to track later observations separately.
For comparison over time, the fixed questions matter more than the mixed totals: the repeated name question has five answers, while two repeated need questions have ten answers combined. The other five questions were sampled for the first time later. We have not treated those first observations as improvements over a baseline they did not have.
These tests use one logged-in account, with a new conversation for each question. Account-level effects are not excluded. Network region is unverified, and the results are not an API benchmark or a mobile-app result. An AI assistant reviewed the records; there has not been a second human review of every judgment.
There are additional, separately labeled retrieval diagnostics. They do not change these formal route counts. Public pages and platform publication establish available sources, not independent discovery in a particular search product.
Try the method and inspect its boundaries
The local recorder is an addition made for this demo. We use Fangx-AI's GEO Doctor for fact cards, question management, publishing records, and technical checks; that upstream work is credited to its maintainers. Technical scores, notifications, and a readable page do not promise indexing, recommendation, or ranking.
Disclosure: I maintain 拾玖搜证台. This article was produced and edited with AI assistance. Product claims were checked against the implementation, and experiment counts against saved records. The synthetic code fixture above is excluded from real results.
