Your brand disappeared from a report. Did the AI answer omit it, or did the collection fail? If both situations become the same zero, the report can send an editor to fix the wrong thing.
Summary
Store completed answers, explicit no-answer outcomes, and collection errors as different states. This local Python example uses synthetic records and shows which denominator each rate uses. Prepared with AI assistance for TheLogic.
Contents
- Define the three states
- Calculate one clearly named rate
- Inspect the report before comparing runs
- FAQ
Define the three states
Here, answer means a completed answer string is present, no_answer means the collection completed and recorded an explicit absence of an answer, and error means the observation failed. A valid answer can still contain no brand mention. This schema is a local reporting convention, not an OpenAI response format.
A missing brand name and a failed observation need different labels.
Calculate one clearly named rate
Use the code below to keep the counts visible. The mention rate is defined only over answer rows. Report no-answer results and errors alongside it so readers can see coverage.
def summarize(rows): counts = dict(answer=0, no_answer=0, error=0, mentioned=0) for row in rows: state = row['state'] if state not in {'answer', 'no_answer', 'error'}: raise ValueError('Unknown state') if state == 'answer': if not isinstance(row.get('answer'), str) or not row['answer'].strip(): raise ValueError('An answer row needs nonempty text') if type(row.get('mentioned')) is not bool: raise ValueError('mentioned must be a boolean') counts['mentioned'] += int(row['mentioned']) counts[state] += 1 rate = counts['mentioned'] / counts['answer'] if counts['answer'] else None return counts, rate rows = [ dict(state='answer', answer='Example with brand', mentioned=True), dict(state='answer', answer='Another answer', mentioned=False), dict(state='answer', answer='Third answer', mentioned=False), dict(state='no_answer'), dict(state='error'), ] counts, rate = summarize(rows) assert counts == dict(answer=3, no_answer=1, error=1, mentioned=1) assert rate == 1 / 3 assert summarize([])[1] is None try: summarize([dict(state='unknown')]) except ValueError: pass else: raise AssertionError('Unknown states must fail') print(counts, rate) Five attempts produce three answer rows, one explicit no-answer outcome, and one error. One mention divided by three answer rows is 33.3%. These are synthetic examples, not a customer result.
| State | Included in mention-rate denominator | Also report |
|---|---|---|
| answer | Yes | Mentioned or not mentioned |
| no_answer | No, by this definition | Count of completed no-answer outcomes |
| error | No | Count and reason if available |
Inspect the report before comparing runs
Keep the same question set, platform, search setting, and period when comparing runs. A smaller error count can change how many answer rows you have. Show both counts and rates before interpreting a difference. TheLogic's public GEO methodology distinguishes collected observations and their source records.
FAQ
Is this the only correct denominator
No. You can define a rate over all completed observations, but name the denominator and keep failure counts visible.
Can no_answer be inferred from an empty answer field
Only if your collector guarantees that meaning. Otherwise use an error or unknown state until it is classified.
Does the script query an AI service
It summarizes local records and makes no network requests.
