Most on-page SEO problems are boring and invisible: a template that ships noindex to production, a canonical pointing at the staging domain, 40 product pages sharing one title, or a relaunch that dropped all the JSON-LD. Nobody notices until traffic drops.
This post covers:
- What an on-page audit should actually check.
- A minimal single-page checker in plain Python (no dependencies).
- A hosted Actor that crawls a site or sitemap, scores every page 0–100 and summarises duplicates, with Python and Node.js examples.
Disclosure: I built the SEO On-Page Audit Actor used in part 3. Parts 1 and 2 don't need it.
1. What to check
| Area | What can go wrong |
|---|---|
| Indexability | noindex in meta robots or the X-Robots-Tag header, page blocked by robots.txt, 4xx/5xx |
| Canonical | missing, or not self-referencing when it should be |
| Title / description | missing, too long (truncated), too short, duplicated across pages |
| Headings & content | no H1, several H1s, thin content (very low word count), missing lang / viewport |
| International | hreflang alternates missing or wrong |
| Social | missing Open Graph / Twitter card tags (ugly link previews) |
| Structured data | no JSON-LD, or JSON-LD that doesn't parse |
| Images & links | images without alt, internal/external/nofollow link counts |
2. A minimal one-page checker (Python standard library)
import re import sys import urllib.request from html.parser import HTMLParser class SEO(HTMLParser): def __init__(self): super().__init__() self.title, self.meta, self.h1, self.canonical = "", {}, 0, None self._in_title = False def handle_starttag(self, tag, attrs): a = dict(attrs) if tag == "title": self._in_title = True elif tag == "meta" and a.get("name"): self.meta[a["name"].lower()] = a.get("content") or "" elif tag == "link" and (a.get("rel") or "").lower() == "canonical": self.canonical = a.get("href") elif tag == "h1": self.h1 += 1 def handle_endtag(self, tag): if tag == "title": self._in_title = False def handle_data(self, data): if self._in_title: self.title += data url = sys.argv[1] if len(sys.argv) > 1 else "https://www.scrapethissite.com/" req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0 (seo-check script)"}) with urllib.request.urlopen(req, timeout=20) as r: html = r.read().decode("utf-8", "replace") x_robots = r.headers.get("X-Robots-Tag", "") p = SEO() p.feed(html) title, desc = p.title.strip(), p.meta.get("description", "") issues = [] if not title: issues.append("Missing title") elif not 30 <= len(title) <= 60: issues.append(f"Title length {len(title)} (aim for 30-60)") if not desc: issues.append("Missing meta description") elif not 70 <= len(desc) <= 160: issues.append(f"Description length {len(desc)} (aim for 70-160)") if p.h1 != 1: issues.append(f"{p.h1} H1 headings (expected 1)") if not p.canonical: issues.append("No canonical") if "noindex" in (p.meta.get("robots", "") + x_robots).lower(): issues.append("noindex!") if 'application/ld+json' not in html: issues.append("No JSON-LD") print(url) print("\n".join(f" - {i}" for i in issues) or " no issues") python seo_check_stdlib.py https://your-site.example/ That's fine for one URL. It gets painful when you need it for 500 URLs from a sitemap, duplicate detection across pages, robots.txt checks and a sortable report.
3. Hosted option: SEO On-Page Audit on Apify
The SEO On-Page Audit Actor accepts single pages, a homepage to crawl, or a sitemap (.xml / .xml.gz, sitemap indexes are followed). You get one dataset row per page with all the tags above, a 0–100 score and plain-English issues. A SUMMARY record in the key-value store lists the top issues across the site, duplicate titles and descriptions with their URLs, and the average score. It uses plain HTTP with no browser, API key or proxy, so runs are cheap.
Set APIFY_TOKEN from a free Apify account (Console → Settings → API & Integrations).
Input
| Field | Description |
|---|---|
startUrls | Pages, a homepage to crawl, or sitemap URLs |
crawlSite | Follow internal links from the start URLs (default false) |
maxPages | Max pages audited in total (default 100) |
maxDepth | Crawl depth (default 3) |
respectRobotsTxt | Skip pages disallowed for Googlebot while crawling (default true) |
onlyPagesWithIssues | Save only pages with at least one issue |
maxConcurrency, timeoutSecs | Politeness and limits (defaults 5 and 20 s) |
Crawl a site:
{"startUrls": ["https://www.example.com/"], "crawlSite": true, "maxPages": 200} Audit every URL in a sitemap and keep only problem pages:
{"startUrls": ["https://www.example.com/sitemap.xml"], "maxPages": 1000, "onlyPagesWithIssues": true} Python (pip install apify-client)
"""Run an on-page SEO audit of a site and print the worst pages and top issues. Actor: https://apify.com/paperlinenstudio/seo-onpage-audit Usage: APIFY_TOKEN=... python seo_audit.py https://your-site.example """ import os import sys from collections import Counter from apify_client import ApifyClient site = sys.argv[1] if len(sys.argv) > 1 else "https://www.scrapethissite.com/" client = ApifyClient(os.environ["APIFY_TOKEN"]) run = client.actor("paperlinenstudio/seo-onpage-audit").call( run_input={ "startUrls": [site], # pages, a homepage or a sitemap.xml "crawlSite": True, # follow internal links from the start URL "maxPages": 15, # small crawl for the demo } ) pages = list(client.dataset(run.default_dataset_id).iterate_items()) summary = client.key_value_store(run.default_key_value_store_id).get_record("SUMMARY") print(f"{len(pages)} pages audited, average score {summary['value']['averageScore']}/100") print("Lowest-scoring pages:") for p in sorted(pages, key=lambda p: p["score"])[:5]: print(f" {p['score']:>3} {p['url']}") print("Most common issues:") counts = Counter(i.split(" (")[0] for p in pages for i in p["issues"]) for issue, n in counts.most_common(5): print(f" {n:>3} x {issue}") Node.js (npm i apify-client)
// Run an on-page SEO audit of a site and print the worst pages and top issues. // Usage: APIFY_TOKEN=... node seo_audit.mjs https://your-site.example import { ApifyClient } from 'apify-client'; const site = process.argv[2] ?? 'https://www.scrapethissite.com/'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('paperlinenstudio/seo-onpage-audit').call({ startUrls: [site], // pages, a homepage or a sitemap.xml crawlSite: true, maxPages: 15, }); const { items: pages } = await client.dataset(run.defaultDatasetId).listItems(); const summary = await client.keyValueStore(run.defaultKeyValueStoreId).getRecord('SUMMARY'); console.log(`${pages.length} pages audited, average score ${summary.value.averageScore}/100`); console.log('Lowest-scoring pages:'); for (const p of [...pages].sort((a, b) => a.score - b.score).slice(0, 5)) { console.log(` ${String(p.score).padStart(3)} ${p.url}`); } const counts = {}; for (const p of pages) for (const i of p.issues) { const k = i.split(' (')[0]; counts[k] = (counts[k] ?? 0) + 1; } console.log('Most common issues:'); for (const [issue, n] of Object.entries(counts).sort((a, b) => b[1] - a[1]).slice(0, 5)) { console.log(` ${String(n).padStart(3)} x ${issue}`); } Sample output
One page record (real, from a run on https://www.heise.de/ on 03.11.2026; shortened):
{ "url": "https://www.heise.de/", "statusCode": 200, "responseTimeMs": 498, "blockedByRobotsTxt": false, "title": "heise online - IT-News, Nachrichten und Hintergründe | heise online", "titleLength": 67, "metaDescriptionLength": 140, "noindex": false, "canonical": "https://www.heise.de/", "canonicalIsSelf": true, "h1Count": 0, "h2Count": 23, "lang": "de", "viewport": true, "hreflang": [ { "lang": "en", "url": "https://www.heise.de/en" }, { "lang": "x-default", "url": "https://www.heise.de" } ], "jsonLdTypes": ["ItemList", "ListItem", "Thing"], "jsonLdErrors": 0, "wordCount": 3930, "imagesCount": 509, "imagesWithoutAlt": 32, "internalLinks": 279, "externalLinks": 102, "issues": [ "Title too long (67 characters, may be truncated in Google; aim for 30-60)", "Missing H1 heading", "Images without alt attribute (32)" ], "score": 85 } The SUMMARY record contains pagesAudited, pagesWithIssues, averageScore, topIssues (issue → count), duplicateTitles and duplicateDescriptions (each with the affected URLs).
How the score works
Each page starts at 100, and points are deducted per issue, weighted by importance. For example, noindex or a robots.txt block costs −30, a missing title −20, a missing meta description −10, HTTP instead of HTTPS −10, a missing viewport or H1 −8, and smaller issues like title length or missing alt text −3 to −5. 4xx/5xx pages score 0. Use the score to sort pages, and fix what the issues list says.
Weekly monitoring
Point it at your sitemap.xml with onlyPagesWithIssues: true and add a schedule in the Apify Console. You'll catch pages that quietly become noindex, start returning 404 or lose their structured data. It pairs well with the Sitemap URL Extractor and the Broken Link Checker.
Limitations
- Pages are fetched without a browser, so tags injected only by JavaScript aren't visible. That's roughly what search engines see on their first, non-rendered pass.
- Sites with aggressive bot protection may return 403 or challenge pages; these are reported with their status code.
- Thresholds (title 30–60 chars, description 70–160, thin content under 200 words) are common guidelines, not official Google rules.
- On-page only: no backlinks, rankings or Core Web Vitals.
Wrap-up
- One page, quick look: the ~50-line script in part 2.
- Whole site or sitemap, duplicates, scores, CSV export, scheduled checks: SEO On-Page Audit. It's free; you only pay Apify platform usage, which is very low because no browser is used.
Feedback and feature requests are welcome, either in the comments or on the Actor's Issues tab.
