We build and maintain web scrapers that run for hours at a time. In October one of them ran out of memory, and the cause was a detail of lxml that we had skimmed past in its docs.

The crawler walked the listing pages of a large online store, about 4 MB of HTML each. On every page it ran a few xpath() queries, pushed the item links into a queue, and dropped the parsed tree. On a store with about 46,000 listings, memory went from about 100 MB to 3,010 MB, and runs with a 1 GB limit were killed.

The links were the problem. xpath() returns "smart strings" that hold a reference to the element they came from, so every queued link kept its whole page tree alive, at about 12.6 MB per page. Adding smart_strings=False to those calls kept the same crawl at about 100 MB from start to finish.

What does xpath() actually return? 

from lxml import html tree = html.fromstring('<html><body><a href="/item/1">Item</a></body></html>') href = tree.xpath("//a/@href")[0] print(type(href).__name__) # _ElementUnicodeResult print(href.getparent()) # <Element a at 0x...>

_ElementUnicodeResult is a subclass of str. It prints, compares and hashes like a normal string, so nothing in a crawler looks wrong. It also has getparent(), which means it holds a reference to its element, and the element keeps its document alive. The lxml docs are direct about the cost: "the tree will be kept alive by the string" (XPath return values).

While you are still working on the page, that costs nothing extra. It turns into a leak once the string goes somewhere that outlives the page: a crawl queue, a set of seen URLs, a dict of results.

How much memory does one kept string cost?

To check this away from our crawler, we wrote a short benchmark script. Each page is a synthetic HTML document of 0.93 MB with 2,000 product cards. The script parses it with lxml.html, keeps the first 20 links, deletes the tree and moves on to the next page. Each mode runs in its own process, and at the end the script reads the process's memory with macOS footprint.

kept = [] for i in range(pages): tree = html.fromstring(make_page(i)) # about 0.93 MB of HTML  if mode == "smart": links = tree.xpath("//a/@href") # lxml default  elif mode == "str": links = [str(x) for x in tree.xpath("//a/@href")] else: links = tree.xpath("//a/@href", smart_strings=False) kept.extend(links[:20]) # keep 20 links per page  del tree, links gc.collect()

Results from 2026-11-08 (macOS, Python 3.13.0, lxml 6.1.1, libxml2 2.14.6):

How links were stored Pages Links kept Pages still in memory phys_footprint
default xpath() 50 1,000 50 221 MB
default xpath() 100 2,000 100 422 MB
str(x) before storing 100 2,000 0 33 MB
smart_strings=False 100 2,000 0 27 MB

We counted "pages still in memory" by following every kept link back with getparent() and reading its page title. With the default, every page the script had parsed was still there, at about 4 MB per page. Storing 2,000 short links took 422 MB instead of 27 MB, 15.6 times as much.

Why didn't our local test catch it?

We did run the crawler locally before shipping it, and memory looked fine. The number we watched was RSS from ps, and on macOS that number can sit far below what a process really holds. macOS compresses memory pages that have not been touched for a while. Compressed pages still belong to the process, but they no longer count toward RSS. During the leak, ps showed 176 MB while the process held about 3 GB.

We now read phys_footprint from footprint. Its man page describes it as the kernel ledger that tracks the dirty memory a process owns:

footprint -p <pid> | grep phys_footprint

In the benchmark above, RSS and phys_footprint stayed close (429 MB against 422 MB at 100 pages). Each run took under a second, so most likely nothing had time to be compressed. The gap opens up in long runs, and long runs are where a leak like this one shows up.

What did we change?

Every xpath() call whose results are stored past the current page now passes smart_strings=False:

links = tree.xpath("//a/@href", smart_strings=False) # compiled expressions take the same argument find_links = etree.XPath("//a/@href", smart_strings=False)

Code that cleans a value up before storing it was already fine. We checked what the common string operations return:

Expression Returns Points back to the page
tree.xpath("//a/@href")[0] _ElementUnicodeResult yes
tree.xpath("//title/text()")[0] _ElementUnicodeResult yes
tree.xpath("string(//title)") _ElementUnicodeResult no, getparent() returns None
value.strip(), "".join(...), re.search(...).group() str no
str(value), f"{value}", value + "", value[:] str no

So the leak only bites code that stores the raw result, such as a link or an attribute pushed straight into a queue with no cleanup in between. For a crawler, that is the most common thing to do with a link.

If you use parsel, the selector library behind Scrapy, this is already handled: its Selector passes smart_strings=False to lxml by default (_lxml_smart_strings = False in parsel 1.11.0). We only ran into the problem because this crawler called lxml directly.

What we took away

  • Treat an xpath() result as a reference to its page until you convert it.
  • Convert where values move into longer-lived structures. The line that adds to a queue, a set or a results dict is where smart_strings=False or str() belongs.
  • When memory grows with the number of pages, count how many documents are still reachable. Following stored strings back with getparent() answered that for us in a few lines.
  • On macOS, watch phys_footprint during long runs. RSS from ps showed 176 MB while we were holding about 3 GB.

Drafted with AI assistance; the numbers and traces come from our own runs and were checked by us.