Raw HTML is a poor thing to feed a model. A typical docs page is mostly navigation, scripts, styles and wrapper divs, and the actual text is a fraction of the bytes. Markdown keeps what matters (headings, lists, links, tables, code) and drops the rest, which makes it a good format for prompts, RAG chunks and chatbot knowledge files.
This post builds a small HTML-to-Markdown converter on Python's built-in html.parser, with no pip install. It reads a local file or a URL, skips page chrome, keeps tables and code blocks, and turns relative links into absolute ones.
Disclosure: Appie.run publishes this post. Appie has free HTML and webpage to Markdown converters, mentioned near the end. The script below does not depend on them.
Earlier posts in this series cover converting PDFs to Markdown for RAG and converting Notion to Markdown.
What a useful converter has to do
- Drop the chrome.
<head>,<script>,<style>,<nav>,<header>,<footer>,<aside>, forms and inline SVGs add noise and no meaning. - Catch navigation that is not a
<nav>. Many sites build sidebars from plain<div>s markedrole="navigation". - Keep structure: headings, nested lists, bold and italic, inline code and
<pre>blocks. - Make links absolute.
/docs/configmeans nothing once the text is pasted into a prompt. - Keep tables as tables, with pipes inside cells escaped.
- Collapse whitespace without breaking code blocks.
The script
Save this as html2md.py. It was tested with Python 3.13.5 on the cases below.
#!/usr/bin/env python3 """Convert an HTML file or URL to Markdown with the standard library only.""" import sys import urllib.request from html.parser import HTMLParser from urllib.parse import urljoin SKIP = {"head", "script", "style", "nav", "footer", "header", "aside", "form", "noscript", "svg"} SKIP_ROLES = {"navigation", "banner", "contentinfo", "search"} BLOCK = {"p", "div", "section", "article", "main", "br", "table", "ul", "ol"} class MarkdownParser(HTMLParser): def __init__(self, base_url=""): super().__init__(convert_charrefs=True) self.base = base_url self.out = [] self.skip_tag = None # tag that opened the skipped block self.skip_count = 0 # nesting of that tag inside the block self.lists = [] # stack of "ul" / "ol" with counters self.href = None self.link_text = [] self.in_pre = False self.row = None self.rows = [] def emit(self, text): if self.href is not None: self.link_text.append(text) elif self.row is not None: self.row[-1] += text else: self.out.append(text) def handle_starttag(self, tag, attrs): a = dict(attrs) if self.skip_tag: if tag == self.skip_tag: self.skip_count += 1 return if tag in SKIP or a.get("role") in SKIP_ROLES: if tag not in {"img", "br", "input", "meta", "link"}: self.skip_tag, self.skip_count = tag, 1 return if tag in {"h1", "h2", "h3", "h4", "h5", "h6"}: self.out.append("\n\n" + "#" * int(tag[1]) + " ") elif tag in BLOCK: if tag == "br": self.out.append(" \n") elif tag in {"ul", "ol"} and self.lists: self.out.append("") # nested list: no blank line else: self.out.append("\n\n") if tag in {"ul", "ol"}: self.lists.append([tag, 0]) elif tag == "li": kind = self.lists[-1] if self.lists else ["ul", 0] kind[1] += 1 indent = " " * (len(self.lists) - 1) marker = f"{kind[1]}." if kind[0] == "ol" else "-" self.out.append(f"\n{indent}{marker} ") elif tag in {"strong", "b"}: self.emit("**") elif tag in {"em", "i"}: self.emit("*") elif tag == "code" and not self.in_pre: self.emit("`") elif tag == "pre": self.in_pre = True self.out.append("\n\n" + "`" * 3 + "\n") elif tag == "a" and a.get("href"): self.href = urljoin(self.base, a["href"]) self.link_text = [] elif tag == "img" and a.get("src"): self.emit(f"})") elif tag == "tr": self.row = [] elif tag in {"td", "th"} and self.row is not None: self.row.append("") def handle_endtag(self, tag): if self.skip_tag: if tag == self.skip_tag: self.skip_count -= 1 if self.skip_count == 0: self.skip_tag = None return if tag in {"ul", "ol"} and self.lists: self.lists.pop() self.out.append("\n") elif tag in {"strong", "b"}: self.emit("**") elif tag in {"em", "i"}: self.emit("*") elif tag == "code" and not self.in_pre: self.emit("`") elif tag == "pre": self.in_pre = False self.out.append("\n" + "`" * 3 + "\n\n") elif tag == "a" and self.href is not None: text = "".join(self.link_text).strip() or self.href href, self.href = self.href, None self.emit(f"[{text}]({href})") elif tag == "tr" and self.row is not None: self.rows.append([c.strip().replace("|", "\\|") for c in self.row]) self.row = None elif tag == "table" and self.rows: width = max(len(r) for r in self.rows) rows = [r + [""] * (width - len(r)) for r in self.rows] lines = ["| " + " | ".join(rows[0]) + " |", "|" + " --- |" * width] lines += ["| " + " | ".join(r) + " |" for r in rows[1:]] self.out.append("\n\n" + "\n".join(lines) + "\n\n") self.rows = [] elif tag in {"h1", "h2", "h3", "h4", "h5", "h6", "p"}: self.out.append("\n\n") def handle_data(self, data): if self.skip_tag: return if self.in_pre: self.out.append(data) return text = " ".join(data.split()) if text: # keep a space between inline pieces if data[:1].isspace(): text = " " + text if data[-1:].isspace(): text += " " self.emit(text) def markdown(self): md = "".join(self.out) lines = [line.rstrip() if not line.endswith(" ") else line for line in md.splitlines()] cleaned, blank = [], 0 for line in lines: if line.strip() == "": blank += 1 if blank > 1: continue else: blank = 0 line = line if line.startswith((" ", "|")) else line.lstrip() cleaned.append(line) return "\n".join(cleaned).strip() + "\n" def load(source): if source.startswith(("http://", "https://")): req = urllib.request.Request(source, headers={"User-Agent": "html2md/1.0"}) with urllib.request.urlopen(req, timeout=20) as resp: charset = resp.headers.get_content_charset() or "utf-8" return resp.read().decode(charset, errors="replace"), source with open(source, encoding="utf-8", errors="replace") as f: return f.read(), "" def main(): if len(sys.argv) < 2: sys.exit("usage: html2md.py <file.html | URL> [base_url]") html, base = load(sys.argv[1]) if len(sys.argv) > 2: base = sys.argv[2] parser = MarkdownParser(base) parser.feed(html) parser.close() sys.stdout.write(parser.markdown()) if __name__ == "__main__": main() Usage:
python3 html2md.py page.html https://example.com/docs/start # local file, with a base URL for links python3 html2md.py https://docs.python.org/3/library/html.parser.html > html-parser.md How it works
-
HTMLParseris event based. It callshandle_starttag,handle_dataandhandle_endtagas it walks the document, and tolerates broken markup.convert_charrefs=Trueturns&and friends into plain characters. - Skipping by element. When a tag in
SKIPopens, or any element has aroleinSKIP_ROLES, the parser records that tag name and counts nested tags of the same name. Everything is ignored until the matching close tag brings the count back to zero. Counting matters: a sidebar<div>usually contains more<div>s, and closing on the first</div>would leak the rest of the sidebar into your output. - Lists use a stack, so nested lists get two spaces of indentation per level and ordered lists keep their own counters.
- Links are buffered: the text inside
<a>is collected and written as[text](absolute-url)on</a>.urljoinresolves relative paths against the base URL, which is the page URL when you pass a URL. - Tables are collected row by row and written at
</table>. Short rows are padded so every row has the same number of cells, and the first row becomes the header. -
<pre>switches off whitespace collapsing and wraps the block in a fence. -
markdown()trims trailing spaces (keeping the two-space hard line break), removes repeated blank lines and returns the result.
Test it
A small page with every awkward part:
<html><head><title>Docs</title><style>body{}</style><script>var x=1;</script></head> <body><nav><a href="/">Home</a> <a href="/pricing">Pricing</a></nav> <div class="sidebar" role="navigation"><div><a href="/docs/a">Page A</a></div><a href="/docs/b">Page B</a></div> <main> <h1>Getting started</h1> <p>Install the <strong>CLI</strong>, then read the <a href="/docs/config">config guide</a>.</p> <h2>Plans</h2> <table><tr><th>Plan</th><th>Seats</th></tr><tr><td>Free</td><td>1</td></tr><tr><td>Team | Pro</td><td>10</td></tr></table> <ul><li>Fast setup</li><li>Works with <em>any</em> site<ol><li>Step one</li><li>Step two</li></ol></li></ul> <pre><code>pip install example example init</code></pre> <img src="/img/flow.png" alt="Flow diagram"> </main> <footer>Copyright 2026</footer></body></html> python...
