Every scraping project starts the same way: one site, a few CSS selectors, done. Then a second site arrives with different markup. Then a third redesigns its product page. A few weeks later you are maintaining a pile of per-site parsers whose only job is to produce the same five fields.

This post shows a schema-first approach: you describe the shape of the data you want once, then extract it from many different pages. We will:

  1. look at the DIY version (fetch + LLM + validation) and where it gets tricky,
  2. run the same workflow with a hosted Actor and read the real output,
  3. cover cost and the caveats you should know before trusting AI extraction.

The DIY version

The core loop is simple:

import json, requests from jsonschema import validate SCHEMA = { "type": "object", "properties": { "title": {"type": "string"}, "price": {"type": "string"}, "availability": {"type": "string"}, }, "required": ["title", "price"], } html = requests.get(url, timeout=30).text text = html_to_text(html) # strip tags, scripts, nav raw = call_your_llm(prompt(text, SCHEMA)) data = json.loads(raw) validate(data, SCHEMA) # raises if the shape is wrong

The hard parts aren't in that snippet:

  • Bad JSON. Models sometimes return prose, trailing commas or the wrong types, so you need a repair-and-retry loop with a cap.
  • Pages that can't work. Login walls, captchas, 429s and JavaScript-only apps return junk text. If you don't detect them, you will "extract" garbage and pay for it.
  • Long pages. You have to truncate input, and that silently drops fields near the bottom.
  • Missing vs. empty. A field the page doesn't state should be null and flagged, not hallucinated.

The hosted version

AI Page to JSON: URL + Schema Extractor handles that loop for you. Disclosure: I build this Actor at SteadyScrape. It fetches each public URL over plain HTTP (no browser), sends the page text to an LLM, validates the result against your schema, and returns one row per URL with a clear status.

Input matching the current schema (urls is required; give fields and/or jsonSchema):

{ "urls": ["https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"], "fields": ["title", "price", "availability"] }

Fields can carry a hint, e.g. "price: Product price in USD". For stricter pipelines, pass a full jsonSchema object instead; rows that miss required fields come back as partial with missingFields listed.

Calling it from Python with the official client (token read from an environment variable, never hard-coded):

import os from apify_client import ApifyClient client = ApifyClient(os.environ["APIFY_TOKEN"]) run = client.actor("steady-scrape/ai-page-to-json").call(run_input={ "urls": [ "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html", ], "fields": ["title", "price", "availability"], }) for row in client.dataset(run["defaultDatasetId"]).iterate_items(): print(row["status"], row["url"], row["data"])

Real output from a verified run (trimmed):

{ "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html", "status": "ok", "data": { "title": "Tipping the Velvet", "price": "£53.74", "availability": "In stock (20 available)" }, "missingFields": null, "nullFields": null, "truncated": false, "pageTitle": "Tipping the Velvet | Books to Scrape - Sandbox", "chargeEvent": "page-extracted" }

A real workflow: a weekly price sheet across different sites

  1. Keep a CSV of product URLs from several shops.
  2. Run the Actor weekly with the same fields (Apify's scheduler works fine).
  3. Keep only rows where status is ok, and append url, title, price, availability, run date to a sheet.
  4. Diff against last week and alert on price or stock changes.

Because you ask for the same schema every time, step 3 can be the same code for every site, and a shop redesign usually doesn't mean writing a new parser. Still spot-check results after a site changes, since extraction quality can shift with the page.

The statuses matter here: refused (login/paywall), blocked (captcha/429), js_required (needs a browser) and failed rows incur no page-extraction charge. Filter them out and review them separately; don't treat them as "price missing".

What it costs

Pricing is pay-per-event and shown on the Store page:

  • $0.010 per extracted page (ok or partial)
  • $0.001 once per run, only if at least one page was extracted
  • Apify's standard $0.00005 Actor start event

So a weekly run where all 100 product pages are extracted costs $1.00105 (100 × $0.010 + $0.001 + $0.00005). Failed, refused, blocked, JavaScript-only and empty pages incur no page-extraction charge, and a run where every page fails doesn't pay the $0.001 run fee; Apify's $0.00005 start event still applies to every run. You can also set a maximum cost per run in Apify's run options.

Caveats (read these)

  • AI extraction is not guaranteed to be correct. Validation checks the shape, not the truth. Spot-check values, especially prices and dates, before you act on them automatically.
  • HTTP only in this version. Single-page apps that render content with JavaScript return js_required.
  • Public pages only. It doesn't bypass logins, paywalls or captchas, and it has no personal-data harvesting features. Check each site's terms before collecting at scale.
  • Long pages may be cut at maxInputChars; those rows carry truncated: true.

Integrations

You can run it through the Apify API, schedules and webhooks, or connect it to Make, Zapier or n8n through Apify's own integrations. AI agents can call it through the Apify MCP server.

If you try it on a page that should have worked and didn't, open an issue on the Actor page. I read them.

SteadyScrape (steady-scrape) builds small, pay-per-result data tools on Apify.