You want to know when a competitor ships a new landing page, product URL, or blog post. Crawling the whole site every day is slow and expensive. Most sites already publish an XML sitemap for search engines, and that list updates when pages do. Diffing the sitemap day over day is enough.

This post shows how to turn that into a daily "what's new" feed: baseline the sitemap once, then every later run returns only URLs that weren't there before. No browser, no proxy, and no full crawl.

Why sitemaps beat a crawl for this

A sitemap is the site's own list of indexable URLs. Compared with a crawler:

  • It is usually complete for the pages they care about ranking.
  • Nested sitemap indexes and .xml.gz files are normal; you have to follow them.
  • Runs finish in seconds for tens of thousands of URLs.
  • You only pay for URLs you keep after filters.

The missing piece is memory between runs. You need a stored set of URLs you've already seen, so today's run can drop everything that was there yesterday.

The one input that makes it a monitor

Disclosure: I packaged this as XML Sitemap URL Extractor on Apify (source on GitHub). The input that matters for monitoring is onlyNewUrls: true.

On the first run with a given stateKey, every matching URL is treated as new and written to the dataset with isNew: true, and the tool saves those URLs in a named key-value store. On later runs with the same stateKey, URLs already in that store are filtered out. The dataset then holds only the newcomers.

For a clean demo, narrow the sitemap with includeUrlPatterns so the first run can baseline the whole path you care about (if you stop early with maxUrls, later runs will still see the rest of the sitemap as "new").

import os, json from apify_client import ApifyClient # pip install apify-client  client = ApifyClient(os.environ["APIFY_TOKEN"]) run = client.actor("swillz/sitemap-url-extractor").call( run_input={ "startUrls": [{"url": "https://docs.apify.com"}], "onlyNewUrls": True, "stateKey": "competitor-academy-nodejs-daily", # one key per monitor  "includeUrlPatterns": ["*/academy/node-js*"], "maxUrls": 0, # 0 = no cap; baseline the whole matched set  }, max_total_charge_usd=0.05, ) items = [i for i in client.dataset(run["defaultDatasetId"]).iterate_items() if i.get("url")] print(run["statusMessage"]) print(len(items), "new URLs") print(json.dumps(items[0], indent=2) if items else "(none)")

A first capped run against the Apify docs academy Node.js section returned 27 rows. One of them looked like this:

{ "url": "https://docs.apify.com/academy/node-js", "lastmod": null, "changefreq": "weekly", "priority": 0.5, "imageCount": 0, "isNew": true, "sitemapUrl": "https://docs.apify.com/sitemap_base.xml", "site": "https://docs.apify.com", "extractedAt": "2026-11-06T18:18:37.424Z" }

Running the same call again immediately (same stateKey) wrote 0 rows and reported No new URLs since the last run. Everything matched was already in the saved state, so the dataset stayed empty. That empty result is the normal "nothing launched today" outcome.

Use a different stateKey for each competitor or each path you monitor (stripe-blog, rival-pricing-pages, and so on). Sharing a key mixes their histories.

Schedule it daily

In the Apify Console:

  1. Open the Actor, set onlyNewUrls to true, pick a stable stateKey, and optionally narrow with includeUrlPatterns (for example ["*/blog/*", "*/products/*"]).
  2. Save the input as a task, or create a Schedule that runs the Actor once a day.
  3. On the Schedule or Task, add an integration: email, Slack, a webhook, Zapier, Make, or n8n.

The first scheduled run builds the baseline. From the second day onward, a quiet day means an empty dataset; a launch day means a short list of new URLs you can paste into Slack or a sheet.

Rough cost for a site that adds about 20 URLs a day is well under a dollar a month on paid plans (from about $0.10 per 1,000 URLs extracted on higher tiers, charged only for URLs written after filters).

Optional: filter, then check status

Useful extras on the same Actor:

  • includeUrlPatterns / excludeUrlPatterns to keep product or blog paths only.
  • lastmodFrom: "7 days" to ignore stale entries when you are not using onlyNewUrls.
  • checkUrlStatus: true to add httpStatus and finalUrl (separate per-URL charge; skip it on the daily monitor if you only care that the URL appeared).

Tip: the first baseline must finish without hitting your max cost per run, or the Actor skips saving state. Prefer includeUrlPatterns (or a direct sitemap URL) over a tiny maxUrls when you want an empty second run.

Hand it to an AI agent (MCP) 

{ "mcpServers": { "apify": { "url": "https://mcp.apify.com?tools=swillz/sitemap-url-extractor,swillz/google-sheets-import-export" } } }

Then:

"Check docs.apify.com academy/node-js sitemap URLs that are new since yesterday using stateKey competitor-academy-nodejs-daily, then append any new rows to my sheet tab 'launches' without duplicates."

The agent runs the monitor, and if the dataset has rows it can pipe them into Google Sheets Import & Export with deduplicateBy: ["url"].

Wrap-up

  • Diff the sitemap, don't recrawl the site, when you only need new URLs.
  • onlyNewUrls plus a stable stateKey is the whole monitor.
  • Baseline the full path you care about on run one; empty datasets after that mean a quiet day.
  • Narrow with path patterns, and notify via Slack or a sheet when the dataset is not empty.