A browser scraper needs to know when the records exist, how to identify the next page, and what to do with missing fields. Changing the browser library does not remove those decisions.
Nodriver is an asynchronous Python library that controls a Chromium-based browser through the Chrome DevTools Protocol. It does not require Selenium's ChromeDriver process. We use it here for a bounded collection job on a public practice site: render quote cards, follow pagination, and export records with their source pages.
What Nodriver Does
Nodriver can open pages, select elements, interact with them, and inspect the rendered document. Its asynchronous interface lets a coroutine wait for browser operations without turning every step into a fixed delay.
Direct CDP control describes the connection to the browser. It does not prove that a target will accept an automated session. A site's response can also depend on the network, account state, browser environment, and the action being performed. Use a public practice site to establish that your extraction logic works before investigating a target-specific difference.
The official Nodriver documentation covers its API, while the project repository contains implementation and license details. Check dependency and license requirements before including it in a distributed application.
Install and Check Your Runtime
This example uses Nodriver with BeautifulSoup for the parsing stage. Install the packages in an isolated environment:
python -m pip install nodriver==0.50.6 beautifulsoup4==4.15.0 You also need a compatible local Chrome or Chromium installation. Nodriver normally discovers the executable; an application can supply browser_executable_path when it needs an explicit path. Keep that path in deployment configuration rather than assuming every machine uses the same operating system layout.
Use a separate automation profile. Reusing your personal browser profile can mix private sessions, extensions, and state with the collection job. It also makes a supposedly reproducible example depend on whatever happened in your everyday browser.
Launch, Wait, and Extract Record Cards
Save the following as collect_quotes.py. The page bound is intentional: the exercise reads two pages, then closes the browser.
import asyncio import json from pathlib import Path from urllib.parse import urlsplit import nodriver as uc from bs4 import BeautifulSoup START = "https://quotes.toscrape.com/js/" async def main(): browser = await uc.start(headless=True) records, visited = [], set() try: tab = await browser.get(START) for _ in range(2): await tab.select("div.quote", timeout=15) current_url = await tab.evaluate("window.location.href") if current_url in visited: raise ValueError("Pagination returned to an earlier page") if urlsplit(current_url).netloc != urlsplit(START).netloc: raise ValueError("Unexpected collection host") visited.add(current_url) soup = BeautifulSoup(await tab.get_content(), "html.parser") cards = soup.select("div.quote") if not cards: raise ValueError("Expected rendered quote cards") for card in cards: text = card.select_one("span.text") author = card.select_one("small.author") if text is None or author is None: raise ValueError(f"Incomplete card at {current_url}") records.append({ "text": text.get_text(" ", strip=True), "author": author.get_text(" ", strip=True), "source_url": current_url, }) if soup.select_one("li.next a") is None: break if len(visited) < 2: next_link = await tab.select("li.next a", timeout=5) await next_link.click() await tab.wait_for("div.quote", timeout=15) Path("nodriver-quotes.json").write_text( json.dumps(records, ensure_ascii=False, indent=2), encoding="utf-8" ) print(f"Saved {len(records)} records from {len(visited)} pages") finally: browser.stop() await asyncio.sleep(0.5) if __name__ == "__main__": asyncio.run(main()) tab.select() waits for a selector rather than immediately assuming that the page's first document contains records. The scraper then takes one HTML snapshot and parses fields relative to each card. That arrangement makes it easy to save the acquired HTML for diagnosis without changing the field extraction rules.
The short pause after stop() lets scheduled browser cleanup progress before the event loop exits. It is not a content-loading strategy. Page extraction waits on a record selector, which is the meaningful readiness condition for this exercise.
Make Pagination a State Transition
The script locates the next link in the rendered document and clicks the actual element. After navigation, it waits for quote cards and reads the current URL again. That gives the loop a page identity rather than relying on an incremented counter alone.
For a site that updates records without changing the URL, the readiness check must compare another stable property: a page index, a result identifier, or a content change you observed in the application. Do not copy the URL-based duplicate check into that kind of interface and assume it will work.
Avoid a broad text search for “Next” when the page has several controls with that label. A selector scoped to the pagination region gives the action a clearer target. If the site has a documented structured source for its records, inspect that source before building a longer click sequence.
Treat the Output as a Contract
Every exported record has text, author, and source_url. Required fields are checked before a record is appended, and Unicode is preserved when the file is written. Downstream code can therefore distinguish a valid empty optional field from a broken extraction rule.
A production job may need additional fields such as observation time and a source record ID. Add them deliberately. Browser titles and page URLs are useful provenance, but they do not necessarily identify a unique record across category pages or repeat visits.
Keep partial results out of the final export if the collection fails midway and the consumer expects a complete snapshot. An alternative is a separate partial-results file with an explicit status. The consumer should never have to infer completeness from the number of rows.
Decide Who Should Operate the Browser
Local Nodriver gives your application control over the browser process. That also means your deployment owns the executable, profile storage, dependencies, and resource limits. Start with a bounded session and observe its actual memory and process lifetime before increasing concurrency.
Scrapeless Agent Browser is an option when the application should connect to managed browser infrastructure. Its documented Playwright path is a separate client workflow. Do not assume a Nodriver script can connect to it by replacing a URL: a supported CDP client and a tested library connection are different claims.
If the application only needs the final page content, Web Unlocker may fit without an interactive session. Preserve the parser and change the acquisition layer when that meets the task.
What to Check Before Expanding the Job
Run the bounded example first. Inspect the exported authors and source pages, confirm that the second page differs from the first, and verify that the browser process exits. Then test a deliberately missing selector so the failure message is understandable.
Expand collection scope only after those checks work. A browser that opens successfully has passed the launch stage; record validation and complete output remain separate parts of the job.
FAQ
Does Nodriver Require ChromeDriver?
Nodriver controls the browser through CDP and does not use Selenium's ChromeDriver process for this workflow.
Can a Headless Success Guarantee Access to Another Site?
No. It confirms the example's acquisition and extraction path on that environment. A different target requires its own authorized test.
Can Nodriver Replace the Parser?
It can inspect elements directly. Here we use a rendered HTML snapshot and BeautifulSoup to keep browser interaction separate from field parsing.
Legal note: automate only sites and actions you are authorized to access. Respect applicable terms, data rights, and the site's collection boundaries.
