Everyone has a screenshot of what an AI said about their product. Almost nobody has a method.
Before I run mine, here's the protocol, so you can tell me where it's broken.
This post has no results. That's deliberate. If I publish the method first, I can't quietly adjust it after seeing numbers I don't like. Think of it as a pre-registration, the way good studies do it, just much smaller.
Why screenshots aren't data
A single screenshot of an assistant answer tells you almost nothing, because:
- Answers are sampled. Ask the same question twice and you can get different sources.
- Answers are personalized. History, location, account and settings can all change what you see.
- Answers drift. Models, indexes and retrieval change over weeks.
- People screenshot the wins. Nobody posts the ten runs where they weren't mentioned.
So the goal isn't "am I mentioned?". It's "across controlled, repeated runs, which kinds of pages get cited, and how often?"
The five question types
I'll use one question of each type, written in the words a real user would type. To avoid making this about any specific product, the subject is a generic category: daily puzzle and quiz apps.
- Category: "What are some good daily puzzle games for learning?"
- Comparison: "What's the difference between a daily puzzle and a quiz app?"
- How-to: "How do daily puzzle games decide the puzzle of the day?"
- Definition: "What is a daily puzzle game?"
- Trust: "Are daily learning games actually effective?"
Each type stresses a different kind of page: listicles, comparison posts, docs, definitions, research summaries.
The controls
For every question on every assistant:
- Fresh session. New chat, no prior messages.
- Same wording. Copy-pasted, no follow-ups.
- Web search on where it's a toggle, and the setting recorded.
- Five runs per question. Spread over at least two days.
- Logged-out or clean account where the product allows it, and recorded either way.
- Region and language recorded. I won't pretend my results generalize to other regions.
Five questions × three assistants × five runs = 75 answers. Small, but enough to see patterns instead of anecdotes.
What I'll record for each answer
A flat spreadsheet, one row per cited URL:
| Field | Example |
|---|---|
| assistant | assistant A |
| question_type | definition |
| run | 3 |
| date_time | ISO timestamp |
| cited_url | full URL |
| domain_type | encyclopedia / news / blog / vendor / forum / video |
| position | 1st, 2nd, 3rd citation |
| quoted_sentence | the passage it lifted, if visible |
The last column is the one I care about most. If I can see which sentence was lifted, I can check whether it matches the "one liftable sentence" pattern I keep writing about, or whether that idea is wrong.
How I'll score it
Nothing fancy:
- Citation frequency per domain type, per assistant.
- Stability: how often the same URL appears across the five runs of the same question.
- Overlap: how many cited URLs are shared between assistants for the same question.
- Sentence shape: for lifted sentences, does it start with a name, contain a category, stand alone?
No p-values. With 75 answers, I'm looking for big obvious differences, not subtle ones.
The biases I already know about
- I chose the questions. Someone else would pick different ones.
- One region, one language. Results will differ elsewhere.
- Snapshot in time. This will be stale in a few months.
- Interface differences. Some assistants show citations inline, others in a list. Position isn't strictly comparable.
- I have a stake. I build in this category, which is exactly why I'm publishing the method first.
What would change my mind
I currently believe assistants favour pages with short, self-contained definitional sentences over long, "comprehensive" pages. If the lifted sentences in this test are mostly long, context-dependent, or from pages with no clear definition, that belief is wrong and I'll say so in part two.
Why three assistants, and why only five questions
Three assistants because they represent different approaches to sourcing: some lean on their own search index, some on a partner index, some put citations front and centre. If the same kind of page wins across all three, that's a stronger signal than any single one.
Five questions because the bottleneck isn't asking, it's recording. Every answer means copying several URLs, classifying each domain and, where possible, finding the exact sentence that was lifted. Seventy-five answers is already a few evenings of careful work. A bigger set done sloppily would be worse than a small set done carefully.
What part two will contain
To keep myself honest, here's what I'm committing to publish, whatever the outcome:
- the raw spreadsheet, every row
- the exact wording of the five questions
- dates, times and settings for each run
- a short list of the most-lifted sentences and their source pages
- a plain statement of whether the hypothesis held
If anything in the method changes after today, part two will list the change and the reason, before any results.
Takeaways
- Screenshots are anecdotes. Repeated, controlled runs are data.
- Fix the wording, the session, the settings and the date before you start.
- Record which sentence was lifted, not just which URL.
- Publish the method before the results, so you can't move the goalposts.
- State what result would prove you wrong.
Your turn
Where is this protocol weakest? If you could add one control or swap one question before I run it, what would it be?
I write about practice, feedback and the AI-readable web while building NEXO, a daily learning game.
