Count AI answer mentions and official citations separately with Python

A brand can appear in an AI answer while the citations point entirely to other sites. A URL can also appear among the sources even when the answer does not name the brand. When auditing AI search, store both signals and keep collection errors out of the denominator.

This article was prepared by TheLogic with AI assistance. The data below is synthetic, and the script works locally with Python 3.10+ and no third-party packages.

Define the input

Use one JSON object per line. Here, answer is the visible answer text, with citation URLs extracted separately into citations. An error row represents a collection failure. A completed answer with no links uses an empty array. Use a separate file per question, platform, market, and observation period if those settings differ.

{"status":"ok","answer":"Harbor Ledger supports branch exports.","citations":["https://directory.example/harbor-ledger"]} {"status":"ok","answer":"The platform supports branch exports.","citations":["https://docs.harborledger.example/export"]} {"status":"ok","answer":"Harbor Ledger supports CSV exports.","citations":["https://harborledger.example/docs/csv"]} {"status":"ok","answer":"Harbor Ledger appears in this directory.","citations":["https://harborledger.example.attacker.example/"]} {"status":"error"} 

Run a small audit

The hostname check compares DNS labels, so a URL that merely starts with the official hostname does not pass. The name check uses one exact name with case-insensitive word boundaries. A production workflow can maintain reviewed aliases and flag ambiguous names for human review.

import json import re import sys from urllib.parse import urlsplit def official_url(url, domain): try: parsed = urlsplit(url) host = (parsed.hostname or '').rstrip('.').lower() except ValueError: return False return parsed.scheme in {'http', 'https'} and ( host == domain or host.endswith('.' + domain) ) def summarize(records, brand, domain): pattern = re.compile(r'(?<!\w)' + re.escape(brand) + r'(?!\w)', re.IGNORECASE) totals = dict(completed=0, errors=0, mentioned=0, cited=0, both=0) for row in records: if row['status'] != 'ok': totals['errors'] += 1 continue mentioned = bool(pattern.search(row['answer'])) cited = any(official_url(url, domain) for url in row['citations']) totals['completed'] += 1 totals['mentioned'] += mentioned totals['cited'] += cited totals['both'] += mentioned and cited return totals if __name__ == '__main__': with open(sys.argv[1], encoding='utf-8') as source: rows = (json.loads(line) for line in source if line.strip()) print(summarize(rows, 'Harbor Ledger', 'harborledger.example')) 

Save the fixture as observations.jsonl and the script as audit.py, then run python audit.py observations.jsonl.

Keep the answer, its citations, and the observation conditions together. A total without those inputs is difficult to audit.

Interpret the output

The fourth answer mentions the brand, but its source is not on the official domain. The second answer cites an official subdomain without naming the brand. A matching domain also does not show that the source supports a specific claim: open the page and compare the relevant passage manually.

FAQ

Will this collect ChatGPT answers? It summarizes records you already have. ChatGPT’s web search can show citations and a Sources panel. Save the source destinations from the interface if available. OpenAI’s guide to ChatGPT Search

Should redirects count as official citations? Record the final destination separately after checking the link. This local example does not resolve redirects, so it only classifies the URL in the record.

Can I use a brand’s total mentions across unrelated questions? Keep the question set, platform, language, market, and time window in the report. Compare runs with equivalent settings before drawing a trend.

TheLogic uses the distinction between mentions and citations in its Korean-language GEO workflow. See its public GEO methodology.