For agents and RAG, converting webpages to Markdown is the easy part. Getting the page is harder. A plain fetch can return a bot-check page or an empty shell that needs JavaScript to fill in, and the request still looks successful. The model then gets no page content.
This guide shows how to get the real page as Markdown with a short Python script and the Decodo Web Data API.
TL;DR
- A POST to the Decodo Web Data API's /scrape endpoint with "markdown": true returns a page as Markdown. You can start on the free plan.
- The script starts with the cheapest request, checks each result for page content, and adds JavaScript rendering or the premium proxy pool only when needed.
- On 22 public test pages, the script converted 20, headless Chromium 18, and plain requests 15.
What you'll need
- Python 3.10+ and the Requests library
- A Decodo account on the Web Data API Free plan
The /scrape endpoint in 60 seconds
The /scrape endpoint handles proxy rotation, JavaScript rendering, and anti-bot protection.
Send the URL with "markdown": true. The page is returned in results, with the site's own status_code, so you can detect error pages.
Step 1: Get your API key
Activate the Web Data API free plan and copy your API key from the dashboard.
Install Requests and export the key:
pip install requests export DECODO_API_KEY="your-api-key" In PowerShell, use $env:DECODO_API_KEY="your-api-key" instead.
Step 2: Convert a page to Markdown
Save this as url_to_markdown.py. The script tries each request type in LADDER, cheapest first, and stops at the first usable result:
import os import re import sys from pathlib import Path from urllib.parse import urljoin, urlsplit import requests from requests.adapters import HTTPAdapter from urllib3.util import Retry API_URL = "https://data.decodo.com/v1/scrape" KEY = os.environ.get("DECODO_API_KEY") or sys.exit("Set DECODO_API_KEY") HEADERS = {"Authorization": f"Bearer {KEY}"} # A page with fewer words probably needs rendering or the premium pool MIN_WORDS = 100 # Cheapest first, by the per-request prices on the billing page LADDER = [ ("standard", False), ("standard", True), ("premium", False), ("premium", True), ] KNOWN_STEP = {} # Domain -> index of the LADDER entry that worked # Chinese and Japanese (about 2 characters a word) and Thai (about 6) # don't put spaces between words NO_SPACES = {r"[\u3040-\u30ff\u4e00-\u9fff]": 2, r"[\u0e00-\u0e7f]": 6} SESSION = requests.Session() # urllib3 skips POST retries by default # (see Retry.DEFAULT_ALLOWED_METHODS). # A /scrape call is safe to repeat. SESSION.mount( "https://", HTTPAdapter( max_retries=Retry( total=3, backoff_factor=5, status_forcelist=[429, 500, 524], allowed_methods=None, ) ), ) def word_count(text): return len(text.split()) + sum( len(re.findall(p, text)) // n for p, n in NO_SPACES.items() ) class ScrapeError(Exception): def __init__(self, code, message): super().__init__(f"{code}: {message}") self.code = code def scrape(url, pool, render_js): body = {"url": url, "markdown": True, "proxy_pool": pool} if render_js: body["headless"] = "html" resp = SESSION.post(API_URL, json=body, headers=HEADERS, timeout=180) data = resp.json() if resp.ok and data.get("status") != "failed": return next((r for r in data["results"] if r["type"] == "markdown"), None) raise ScrapeError(data.get("status_code", resp.status_code), data.get("message")) def url_to_markdown(url, render_js=False, pool="standard"): domain = urlsplit(url).netloc step = max(LADDER.index((pool, render_js)), KNOWN_STEP.get(domain, 0)) while step < len(LADDER): pool, render_js = LADDER[step] step += 1 try: result = scrape(url, pool, render_js) except ScrapeError as err: # Some sites need the premium pool for Markdown. After a # 400 from standard, try premium. A 400 from premium raises. if err.code == 400 and pool == "standard": step = LADDER.index(("premium", render_js)) # Codes for "try the next attempt", explained at # help.decodo.com/docs/web-data-api-status-codes elif err.code not in (612, 613, 999): raise continue if result is None: # Try the next attempt continue if result["status_code"] in (404, 410): raise ScrapeError(result["status_code"], "page not found") ok = result["status_code"] < 400 and "help" not in result if ok and word_count(result["content"]) >= MIN_WORDS: KNOWN_STEP[domain] = step - 1 break else: raise ScrapeError("all attempts", "no page content found") content = result["content"] # Remove the navigation above the title title = re.search(r"^# ", content, re.M) # A "#" after a code fence may be a code comment, not a title fence = re.search(r"^`{3}", content, re.M) if title and fence and fence.start() < title.start(): title = None if title and word_count(content[title.start() :]) >= MIN_WORDS: content = content[title.start() :] # Absolute links keep citations working outside this site content = re.sub(r"\]\(([^)\s]+)", lambda m: "](" + urljoin(url, m[1]), content) slug = re.sub(r"\W+", "-", url.split("://")[-1]).strip("-")[:150] path = Path("pages") / (slug + ".md") path.parent.mkdir(exist_ok=True) path.write_text(content, encoding="utf-8") words = word_count(content) print(f"saved {path} ({words:,} words, {pool}, js={render_js})") return content if __name__ == "__main__": for url in sys.argv[1:]: try: url_to_markdown(url) except (ScrapeError, requests.RequestException) as err: print(f"FAILED {url}: {err}", file=sys.stderr) Run it on a static docs page:
python url_to_markdown.py https://docs.python.org/3/library/json.html The script converts the page on the cheapest attempt:
saved pages/docs-python-org-3-library-json-html.md (4,359 words, standard, js=False) The saved file starts at the page title, and its links are absolute:
# `json` — JSON encoder and decoder[¶](https://docs.python.org/3/library/json.html#module-json "Link to this heading") **Source code:** [Lib/json/\_\_init\_\_.py](https://github.com/python/cpython/tree/3.15/Lib/json/__init__.py) ... Steps 3 to 5 explain parts of this script. You don't need to add code.
Step 3: Handle JavaScript-rendered pages
Swagger UI renders API reference pages in the browser. Here's the Petstore demo:
The server's HTML body has only an empty div and scripts:
The /scrape response marks a page like this with a help field in its result. When a page has that field, or fewer than MIN_WORDS words, the script moves to the next attempt, which renders the page with "headless": "html":
python url_to_markdown.py https://petstore.swagger.io/ Rendering returns every endpoint:
saved pages/petstore-swagger-io.md (172 words, standard, js=True) Step 4: Pick the proxy pool
Standard handles simple pages, and premium is built for complex anti-bot protection. Each LADDER entry sets the proxy_pool field to standard or premium. Standard costs less per request, so the script starts there. See the billing page for current prices. For a site that needs premium, call url_to_markdown(url, pool="premium").
The free plan includes both pools and JavaScript rendering, so the whole script runs on it.
Step 5: Run it on long URL lists
These 4 parts of the script keep the output clean and the cost low:
- Status checks. The script checks the site's own status_code and doesn't save error pages. For a site you scrape often, also check for a phrase that every real page has. Then you can safely lower MIN_WORDS for its short pages.
- Per-domain cache. KNOWN_STEP remembers the attempt that worked on each domain, so later pages there usually need 1 request. In a long-running process, clear KNOWN_STEP regularly, because sites change.
- Navigation removal. The script removes the navigation above the page's # title, so your chunks don't contain it.
- Absolute links. Relative links become absolute, so citations work outside the site.
Pass several URLs:
python url_to_markdown.py \ https://www.law.cornell.edu/wex/contract \ https://docs.python.org/3/library/no-such-page.html The missing page is logged, not saved:
saved pages/www-law-cornell-edu-wex-contract.md (809 words, standard, js=False) FAILED https://docs.python.org/3/library/no-such-page.html: 404: page not found How it compares with Requests and Playwright
Each method fetched the same 22 public pages whose robots.txt allows AI crawlers such as GPTBot and ClaudeBot. Only this script used proxies. A page was converted if its Markdown contained a phrase from the main content.
| Method | Pages converted |
|---|---|
| This script with Decodo's Web Data API | 20 |
| Headless Chromium with Playwright | 18 |
| Plain Requests with markdownify | 15 |
The script also converted every page that the other methods did. Most of its 20 conversions needed only the cheapest attempt.
Edge cases and common problems
- Prompt injection. Attackers can use alt text, display:none blocks, or white-on-white text to hide instructions for models. Treat page Markdown as data, and put it between clear delimiters. Don't trigger a tool call only because page text asks for it.
- Choosing pages to convert. The Web Data API is built for publicly accessible pages. Check each site's terms of service.
Final thoughts
The Decodo Web Data API returns Markdown with 1 parameter, and the script checks each result before saving the page.
Next steps:
- Convert a whole docs site with /crawl, which also returns Markdown. To list its URLs first, use /map.
- Send large URL lists as asynchronous requests through /task/batch.
- Give an agent /scrape as a tool from our Model Context Protocol (MCP) server.
- Load pages with the LangChain integration or llama-index-readers-decodo, and set auth_mode="token" to authenticate with your API key.
Start with the Web Data API free plan, and compare the script's Markdown with your current pipeline's output.
