I built SeenLens, a one-time AI readiness audit for websites. To see what the tool would flag across different kinds of sites, I scanned one URL on each of 30 sites. Most were home pages. The sample included retail, documentation, news and public-sector sites. The sites are anonymous in this article.
I ran the scans on 7 November 2026, between 09:06 and 09:20 UTC, under SeenLens rules 0.4.1, which define 43 checks. One check, a comparison of responses to different user agents, was turned off in these scans. The median score was 78 out of 100 among 30 scored pages. Five pages refused the scanner with HTTP 403 or 202. Among the 25 pages that returned HTTP 200, the median was 79. I counted a gap only when the scanner could evaluate the relevant check. The full results, including how often each AI crawler is blocked, are on the SeenLens AI readiness statistics page. Where some pages could not be evaluated, I give the denominator alongside the count.
Content-Signal preferences were absent
I found no Content-Signal line on 26 of 28 sites whose robots.txt could be read. A site can use this line to express preferences about search, use as model input, and training. Decide which uses you want to permit before you add it. Start with the plain-language choices and check the generated line against your policy.
Keep this check separate from crawler access. When you add the line, inspect the rest of robots.txt to confirm that any crawler groups still express your intended access rules. Test the served file after you deploy it. If your CDN manages robots.txt, verify what the public URL actually returns. The scanner records the presence of a line. This test does not establish that any crawler applied it. Content-Signal fix · Content Signal generator
Few pages exposed WebMCP tools or a Markdown version
The scanner detected no WebMCP tools on 13 of 14 pages where it ran the check. It skips the check on pages with no forms. It found no Markdown version on 19 of 25 pages. I report these separately because they are different capabilities. WebMCP lets a compatible browser agent call tools a site exposes. A Markdown version gives clients a text representation of a page.
Both are optional. A documentation site might offer Markdown if it already has a reliable way to generate and update it. A product that wants agents to use specific workflows can consider WebMCP where a supported client can use the tools. For an ordinary marketing page, start by making sure its main content is available in readable HTML. A missing optional format says little about whether an AI answer will cite the page.
Most robots.txt files named no AI crawler tokens
On 18 of 28 readable robots.txt files, I found no rules naming the AI crawler tokens tested (13 tokens). That observation alone says nothing about access. A generic User-agent: * group can still apply. An explicit group can allow a path, block it, or make no difference for the URL tested.
Read the group that applies to the crawler and check it against the exact path. Decide separately how you want to handle search crawlers and training crawlers. If the file's policy is correct and a crawler still receives a challenge, investigate the response in your firewall or CDN logs. The tests tell you whether the rules and the fetch agree for a client like the scanner. SeenLens robots.txt guide
About half the pages had no JSON-LD
The scanner found no JSON-LD on 13 of 25 pages it could inspect. It does not detect microdata or RDFa, so some of these pages may carry structured data in another format. Structured data, often published as JSON-LD, gives machines explicit fields such as a product name or publisher. Start with fields the page can support. For a product page, use facts that match the visible text. For a home page, identify the site and its publisher when that is useful. Check the output after deployment because the page that a crawler receives may differ from your template.
Choose a schema type that describes the page. Check that the data parses and that its URLs identify the intended entities. The value of the work is clearer data about your page. This scan does not measure whether a search engine used the markup or whether it changed citations. SeenLens JSON-LD fix
llms.txt was missing or invalid
On 13 of 27 sites, the file was missing or failed the scanner's validation. 12 returned HTTP 404. One returned a file with no Markdown links. On three sites, the probe was blocked or got a challenge, so I left them out. If your site offers llms.txt, publish it at the root of the domain. Check the HTTP response and the text. A custom page that returns HTTP 200 for any path can look like a file exists until you read the body. Link to pages that are public and still work.
Keep the file short enough to maintain. A concise index can help clients that choose to read it. Whether those clients fetch the file is a separate question. You can look for requests to /llms.txt in your server logs. A file or a fetch does not establish a citation benefit. SeenLens guide to the evidence on llms.txt
What the five highest-scoring pages shared
The five highest scores ranged from 92 to 96. All five pages returned HTTP 200 without a bot wall, served their main content in the HTML, had a title and Open Graph tags, and had a readable robots.txt. All five published valid JSON-LD with an Organization or WebSite node. Of the other 20 pages that returned HTTP 200, 5 did. All five also passed the llms.txt check, against 8 of the other 20. The optional agent formats were less common: 3 of the five offered a Markdown version, 3 named AI crawlers in robots.txt, 2 had a Content-Signal line, and 1 exposed WebMCP tools. The WebMCP check did not run on 2 of the five, which had no forms. 3 of the five passed the Lighthouse agentic audits. None passed the Core Web Vitals check: four got a warning, and one had no PageSpeed result, so its Lighthouse and Core Web Vitals checks were not evaluated.
The score summarizes the rules applied to the page at scan time. A high score shows that the tested URL satisfied many of those rules. It does not describe every URL on the domain. Start by comparing individual findings with what you expect the page to serve, especially if a test was skipped or a fetch hit a challenge. That will tell you which change to investigate first.
What this test can tell you
This was a small set of sites, with one page on each site and one run per page. The selection was not random. The site, its CDN, and the network can all change the response on the next scan. The rules produce the same result when their inputs and version are the same. Keep the URL and rules version in your notes when you compare results later.
Use the list to decide what to verify on your own pages. Start with the URLs you want people to find. Confirm that a crawler can fetch and read them before you spend time on optional formats. The scan did not measure AI answer citations, so it cannot establish whether any of these changes would affect citations.
