If you are building a chatbot or a search tool over someone's documentation, the first job is boring: get the text out of the website. Not the menus, not the cookie banner, not the footer. Just the content, with its headings and code blocks intact, because those are what make chunks meaningful.
Most guides reach for a headless browser here. For documentation sites, blogs and help centres you rarely need one. These sites send their content in the HTML, so a plain HTTP request is enough, and it is far faster and cheaper than driving Chrome.
Disclosure: I built the tool used in this article. It is a paid tool, priced per page; the numbers below are from a real run.
What "clean" has to mean
For retrieval, the Markdown needs four things:
- Only the main content. Navigation text repeated on every page poisons your embeddings: every chunk starts to look similar.
- Headings kept as headings. You will split on them.
- Code blocks kept whole, with their indentation.
- Links as full addresses, so an answer can cite its source.
Crawling a site
I use the Website to Markdown Crawler on Apify. You give it a start page; it follows links to other pages of the same site and returns one record per page.
This input reads the Python tutorial and nothing else on python.org:
{ "urls": ["https://docs.python.org/3/tutorial/"], "maxPagesPerWebsite": 12, "stayUnderStartPath": true } stayUnderStartPath keeps the crawl inside /3/tutorial/. Without it, the crawler would wander into the rest of the documentation.
That run returned 12 pages, about 33,700 words, in 11 seconds. Each record looks like this:
{ "url": "https://docs.python.org/3/tutorial/controlflow.html", "title": "4. More Control Flow Tools — Python 3.15.2 documentation", "depth": 1, "foundOn": "https://docs.python.org/3/tutorial/", "wordCount": 5662, "markdown": "# 4. More Control Flow Tools\n\nAs well as the [`while`](https://docs.python.org/3/reference/…) statement just introduced, Python uses a few more…", "text": "4. More Control Flow Tools\n\nAs well as the while statement just introduced, Python uses a few more…" } markdown is what you want for chunking. text is the same content with no markup, for when you only need words.
Running it from Python
from apify_client import ApifyClient client = ApifyClient("<YOUR_APIFY_TOKEN>") run = client.actor("properscraper/website-to-markdown-crawler").call(run_input={ "urls": ["https://docs.python.org/3/tutorial/"], "maxPagesPerWebsite": 12, "stayUnderStartPath": True, }) pages = list(client.dataset(run["defaultDatasetId"]).iterate_items()) print(len(pages), "pages,", sum(p["wordCount"] for p in pages), "words") Chunking on headings
Because the headings survive, you can split each page into sections and keep the heading with its text. That gives every chunk a title and a source link, which is most of what good retrieval needs.
import re def chunks(page, max_words=400): sections = re.split(r"\n(?=#{1,3} )", page["markdown"]) for section in sections: heading = section.splitlines()[0].lstrip("# ").strip() words = section.split() for i in range(0, len(words), max_words): yield { "source": page["url"], "heading": heading, "text": " ".join(words[i:i + max_words]), } all_chunks = [c for page in pages for c in chunks(page)] Embed text, store source and heading beside it, and show them with every answer.
What it will not do
Being honest about limits saves you an afternoon:
- It does not run JavaScript. A site that builds its pages entirely in the browser returns little or no text. Documentation sites, blogs, shops and company sites almost always work; some single-page apps do not.
- It obeys robots.txt by default and identifies itself honestly. If a site blocks automated visitors, you get a "not read" in the summary, not a workaround.
- It stays on one website. A subdomain counts as a different site; add it as a second start page.
Cost
The tool charges $0.50 per 1,000 pages saved, and nothing for pages it could not read. The 12-page run above cost less than a cent. Apify's free plan includes $5 of credit a month, which is enough to try this on a real site.
A copyright note to finish: most website text is protected. Indexing your own documentation, or content you have permission to use, is straightforward. For anyone else's, check their terms first.

