The simple definition of web scraping is the automated process of extracting data from websites. It also usually involves saving that data into a structured format, like a database, CSV or JSON file, or text document. That way, you can parse large collections of pages (or entire websites) in a much more efficient way than a person browsing the pages and manually copying the desired information.

A web scraper first needs to make requests to a web server, and then fetch the contents of the requested page, just like a person using a web browser normally would. Then, it needs to parse the page into whichever format is required. Both of those steps can be tricky at scale—many servers will throttle or block IP addresses making bulk requests, and parsers have to understand various page layouts and rendering methods.

In this guide, we'll go over how to build your own web scraper in Python, JavaScript, and Ruby, and how to use the easier option—the Extract API from SerpApi.

Building your own web scraper

Web scraping can be done using a variety of programming languages and tools. Whether you're comfortable with JavaScript, Ruby, Go, or another language, libraries and frameworks are available to help you scrape websites.

The first step in web scraping is to fetch the HTML content of the web page you want. This is similar to how your web browser works when you visit a website—it sends a request to the server, and the server responds with the HTML, CSS, JavaScript, and other resources needed to display the page.

In web scraping, you'll use a programming language or tool to send an HTTP request to the website's server. This can be done using various methods:

  • Programming Libraries : Languages like Python, JavaScript, Ruby, and others have libraries that make it easy to send HTTP requests.
  • Built-in Functions : Some languages have built-in functions for making web requests.

By requesting the HTML content, you're retrieving the raw data that makes up the web page, which you can then process and analyze.

Once you've obtained the HTML content, the next step is to parse it to extract the information you're interested in. Web pages are structured documents that use HTML tags to define elements like headings, paragraphs, links, images, and more.

Some common methods of navigating through HTML pages are CSS selectors or XPath. For example, if you wanted to find all links in a page with "product" in the destination URL, you could use a CSS selector like this:

a[href*="product"]

Libraries like Beautiful Soup for Python, jsdom or Cheerio for Node.js, Nokogiri or Nokolexbor for Ruby, and others can parse HTML and XML documents. That way, you can navigate the document elements using standard selectors. Another option is using an AI model to parse the HTML.

Once you've identified the correct elements, you can extract the text, attributes, or other relevant information. That gives you the data for analyzing trends, feeding into another application, building a dataset, or other tasks.

Practice scraping a real website

It's time to build a web scraper! We will use the Hacker News home page as the example site, with the goal of extracting the title, external link, and visible link label for each post.

The Hacker News home page in a web browser.

First, we need to figure out where the links are in the page's HTML. Try right-clicking one of the links, then clicking the "Inspect Element" or "Inspect" menu option.

🧭 Using Safari on Mac? You may need to enable development features first. From the top menu bar, select Safari > Settings, click the Advanced tab, then check the box for "Show features for web developers."

This will open the HTML page's structure, and jump to where that link element is located.

The Elements view in Safari, showing the page structure.

As of the time of writing, each submission is a <tr> table row element with a athing CSS class. In each of those table rows, there's a .titleline element containing everything we need: a link element (a) with the external link and post title, and a .sitestr tag with the visible link label.

Now it's time to scrape that information!

Web scraping using Python

If you want to build a web scraper in Python, you can start with the Requests and Beautiful Soup libraries. For more complex web pages beyond this example, you can try Selenium or Scrapy.

First, install Python on your computer if you haven't already, then create the folder you want to use for this project. Next, open the Terminal (Mac/Linux) or PowerShell (Windows) application, and change the current directory to that project folder:

cd ~/Desktop/scraper

📁 Not sure how to find a folder's path? Type cd and a space, then drag the folder into your Terminal or PowerShell window.

Next, create a Python virtual environment for the project, so all dependencies and other data stays within the folder:

python -m venv venv

💻 Python command not found? Try "python3" instead. The name of the executable depends on how it was installed.

Now, activate the Python virtual environment. On a Mac with the default bash/zsh shell, run this:

source venv/bin/activate

On a Windows PC in PowerShell, run this:

venv\Scripts\Activate.ps1

Next, install the Beautiful Soup and Requests libraries in the project:

pip install requests beautifulsoup4

Now it's time to start writing code! Create a new Python script called main.py in your project with the following contents, then try running it with the python main.py command:

import requests URL = 'https://news.ycombinator.com/' response = requests.get(URL) html_content = response.text print(html_content)

This imports the Requests library, sends a GET request to the URL, and stores the HTML content in a variable.

Now, let's parse the HTML to extract the data. Add the following import statement to the top of the script:

from bs4 import BeautifulSoup

Replace the print(html_content) statement with this line, which creates a Beautiful Soup object for the parsed HTML content:

soup = BeautifulSoup(html_content, 'html.parser')

Next, the script needs to get a list of all table rows with the athing class, and extract each row's link URL, inner text, and visible link. You can do this with a soup.find_all function and a for loop at the end of the script:

rows = soup.find_all('tr', class_='athing') for row in rows: titleline = row.find('span', class_='titleline') link_tag = titleline.find('a', href=True) title = link_tag.text link = link_tag['href'] visible_link_tag = row.find('span', class_='sitestr') # This avoids an error when the visible_link_tag is None  visible_link = visible_link_tag.text if visible_link_tag else "No visible link" print("Title:", title) print("Link:", link) print("Visible Link:", visible_link) print('---')

Running the entire script should give you an output like this:

Title: StreetComplete on iOS is now in public beta Link: https://github.com/streetcomplete/StreetComplete/issues/5421 Visible Link: github.com/streetcomplete --- Title: RIP, vector database Link: https://turbopuffer.com/blog/rip-vector-database Visible Link: turbopuffer.com --- Title: How to speed up the Rust compiler in September 2026 Link: https://nnethercote.github.io/2026/09/30/how-to-speed-up-the-rust-compiler-in-september-2026.html Visible Link: nnethercote.github.io ---

The website has multiple pages. If we want to grab all the data, we must first find the pattern. In this case, this is what the patterns look like:

You can check each page by wrapping the entire script in another for loop that counts from 1 to 3, and then runs the GET request with that number as the page count in the URL. The full script should now look like this:

import requests from bs4 import BeautifulSoup URL = 'https://news.ycombinator.com/' max_pages = 3 # I gave the limit to only 3 pages  for i in range(1, max_pages+1): response = requests.get(f'{URL}?p={i}') html_content = response.text soup = BeautifulSoup(html_content, 'html.parser') rows = soup.find_all('tr', class_='athing') for row in rows: titleline = row.find('span', class_='titleline') link_tag = titleline.find('a', href=True) title = link_tag.text link = link_tag['href'] visible_link_tag = row.find('span', class_='sitestr') visible_link = visible_link_tag.text if visible_link_tag else "No visible link" print("Title:", title) print("Link:", link) print("Visible Link:", visible_link) print('---')

The output will now include results from the first three pages.

You probably want to save the data somewhere other than the terminal output. Python's built-in CSV library can be used to write a CSV spreadsheet file, which can then be opened by Google Sheets, Microsoft Excel, LibreOffice Calc, and many other applications.

Here's how you could write a CSV file with the title, link, and visible link as the columns, and each post as a row:

import requests from bs4 import BeautifulSoup import csv URL = 'https://news.ycombinator.com/' max_pages = 3 # Open a CSV file in write mode with open('data.csv', mode='w', newline='') as file: writer = csv.writer(file) # Write the header row  writer.writerow(['Title', 'Link', 'Visible Link']) for i in range(1, max_pages+1): response = requests.get(f'{URL}?p={i}') html_content = response.text soup = BeautifulSoup(html_content, 'html.parser') rows = soup.find_all('tr', class_='athing') for row in rows: titleline = row.find('span', class_='titleline') link_tag = titleline.find('a', href=True) title = link_tag.text link = link_tag['href'] visible_link_tag = row.find('span', class_='sitestr') visible_link = visible_link_tag.text if visible_link_tag else "No visible link" # Write the data row to the CSV file  writer.writerow([title, link, visible_link])

Now, running the script will create a data.csv file in your project folder with all the data. The file looks like this when opened in a spreadsheet application:

A spreadsheet open in Apple Numbers, containing rows of titles, links, and visible links.

You now have a basic web scraper running in Python! The below resources can help you take on more complex scraping tasks:

You can also skip to the Extract API for a simpler method of web scraping.

Web scraping using JavaScript

JavaScript is also an option for web scraping projects, though not in the same way you might use the language in front-end development. Web browsers will block access to most external domains with CORS restrictions, which is required for scraping, so you need to use Node.js or another non-browser runtime. Those runtimes typically don't have a built-in Document Object Model (DOM), so common functions for parsing HTML like querySelector or getElementById() are unavailable unless you add additional libraries.

For this web scraping project, we'll use the Node.js runtime with the jsdom library. Node.js has a built-in Fetch API for retrieving web content, so additional libraries for that purpose (like Axios) are no longer required.

First, you'll need Node.js and the NPM package manager installed on your computer. Then create a folder somewhere for your project, open a Terminal window (Mac/Linux) or PowerShell prompt (Windows), and navigate to that folder. For example, this will change the current directory to a "scraper" folder on your Desktop:

cd ~/Desktop/scraper

📁 Not sure how to find a folder's path? Type cd and a space, then drag the folder into your Terminal or PowerShell window.

Next, we need to install the jsdom library in the project, with this command:

npm install jsdom

Next, create a new JavaScript file called main.js in the project folder, and start with this code that fetches the HTML for the home page and displays it in the console:

const response = await fetch("https://news.ycombinator.com"); const text = await response.text(); console.log(text);

You can run it with the node main.js command. Next, we can parse the HTML by importing the jsdom library, passing the text to a JSDOM constructor, and making a for loop that goes through every .athing table row:

import jsdom from "jsdom"; const response = await fetch("https://news.ycombinator.com"); const text = await response.text(); const dom = new jsdom.JSDOM(text); // Get every table row representing a post for (let el of dom.window.document.querySelectorAll("tr.athing")) { // Get the title, link, and visible link const title = el.querySelector(".titleline a").textContent; const link = el.querySelector(".titleline a").href; const visibleLink = (el?.querySelector(".sitestr")?.textContent || "No visible link"); // Display the data console.log(`Title: ${title}\nLink: ${link}\nVisible Link: ${visibleLink}\n---`); }

The website has multiple pages. If we want to grab all the data, we must first find the pattern. In this case, this is what the patterns look like:

This is a modified version of the script that fetches the first three pages, and then displays all links from all pages:

import jsdom from "jsdom"; const baseUrl = "https://news.ycombinator.com" for (let i = 1; i <= 3; i++) { const response = await fetch(`${baseUrl}/?p=${i}`); const text = await response.text(); const dom = new jsdom.JSDOM(text); // Get every table row representing a post for (let el of dom.window.document.querySelectorAll("tr.athing")) { // Get the title, link, and visible link const title = el.querySelector(".titleline a").textContent; const link = el.querySelector(".titleline a").href; const visibleLink = (el?.querySelector(".sitestr")?.textContent || "No visible link"); // Display the data console.log(`Title: ${title}\nLink: ${link}\nVisible Link: ${visibleLink}\n---`); } }

You might want to save the scraped data outside of the terminal output. We can write the results to a CSV file by importing Node's fs library and constructing the CSV format:

import jsdom from "jsdom"; import fs from "node:fs/promises"; const baseUrl = "https://news.ycombinator.com" let csvText = `"Title","Link","Visible Link"` for (let i = 1; i <= 3; i++) { const response = await fetch(`${baseUrl}/?p=${i}`); const text = await response.text(); const dom = new jsdom.JSDOM(text); // Get every table row representing a post for (let el of dom.window.document.querySelectorAll("tr.athing")) { // Get the title, link, and visible link const title = el.querySelector(".titleline a").textContent; const link = el.querySelector(".titleline a").href; const visibleLink = (el?.querySelector(".sitestr")?.textContent || "No visible link"); // Add the data as a new row csvText += `\n"${title}","${link}","${visibleLink}"` } } await fs.writeFile("data.csv", csvText);

Now, running the script will create a data.csv file in your project folder with all the data. The file looks like this when opened in a spreadsheet application:

A spreadsheet open in Apple Numbers, containing rows of titles, links, and visible links.

You now have a basic web scraper running in Node.js with JavaScript! The below resources can help you take on more complex scraping tasks: