JobSpy doesn't scrape company career sites. Add Greenhouse, Lever and Workday jobs in the same DataFrame

JobSpy (speedyapply/JobSpy on GitHub) is a popular Python library for job search data (about 4,500 GitHub stars): one scrape_jobs() call returns LinkedIn, Indeed, Glassdoor, ZipRecruiter and Google results as a pandas DataFrame, and python-jobspy was downloaded about 185,000 times last month.

What it does not cover is the employer's own career site. Thousands of companies publish their openings on their own Greenhouse, Lever, Ashby or Workday board (the Actor below reads about 19,000 such boards). The open JobSpy feature request "Scrape directly from company career website" (issue #368) asks for exactly that, and an earlier Workday request (issue #114) was closed by JobSpy's author with "closing as I prefer this repo to be major job boards. there's like 50 ATS if you wanted to get serious with it". That is a fair scope decision, so this post adds career sites next to JobSpy instead of inside it.

The idea: same columns, then pd.concat

scrape_jobs() returns 34 columns in a fixed order (id, site, job_url, title, company, location, date_posted, min_amount, max_amount, interval, ...). If a career-site search returns those same 34 columns in the same order, the two results stack with one line and everything downstream (dedupe, filters, CSV, your notebook) keeps working.

The free script career_sites.py does that. It sends the search to the hiver/career-site-job-search Apify Actor, which covers about 19,000 company career sites on 14 job systems (Greenhouse, Lever, Ashby, Workday, SmartRecruiters, Oracle, iCIMS, Personio, Teamtailor, BambooHR, Breezy, Pinpoint, Rippling, Recruitee), and maps each job to JobSpy's columns.

import pandas as pd from jobspy import scrape_jobs from career_sites import scrape_career_sites boards = scrape_jobs(site_name=["indeed", "linkedin"], search_term="data engineer", location="United States", results_wanted=20, hours_old=72) careers = scrape_career_sites(search_term="data engineer", location="United States", hours_old=72, results_wanted=20) jobs = pd.concat([boards, careers], ignore_index=True) print(jobs.groupby("site").size()) jobs.to_csv("jobs.csv", index=False) 

In the combined frame, site tells you where each row came from: indeed and linkedin from JobSpy, greenhouse, lever, ashby, workday and so on from the career sites.

What the career-site rows look like

These are real rows from a cloud run of the Actor on 2026-10-12 (keyword engineer, last 7 days), after the mapping:

That run returned 893 jobs from 345 companies in 169 seconds, and 400 of the 893 postings stated a pay range. Pay comes from the employer's own posting: structured pay where the job system has it, otherwise a range written in the posting text. Board-only columns such as company_rating or emails stay empty for career-site rows.

How the arguments map

• search_term matches job titles at the start of a word; a list means any of them. No boolean syntax.
• location takes a country ("United States", "UK", "Germany"), matched against the country the employer's job system records, or any place word ("Berlin", "Remote"). There is no distance radius.
• hours_old is rounded up to whole days.
• results_wanted is a hard cap on jobs delivered and charged.
• min_salary has no JobSpy equivalent: it keeps jobs whose posted pay range reaches that annual amount, and jobs it leaves out are not charged.

Cost and setup

You run it on your own Apify account (export APIFY_TOKEN=...). At Free plan prices on 2026-10-13 a search costs $0.003 plus $0.004 per job delivered, so 20 jobs come to about $0.083. The script also passes max_charge_usd (default $0.50) so a run stops at that amount.

pip install pandas python-jobspy python career_sites.py "data engineer" --location "United States" --hours-old 72 --results 20 

What is tested, and what is not

The repo has an offline test: a mock Apify API serves 8 real job rows from 7 job systems (cloud run kfCsYechpPuaD515k), and the test checks the input sent to the Actor, the mapping, the pd.concat with a JobSpy-shaped frame, and that the column list equals jobspy.util.desired_order of the installed python-jobspy (checked on 1.3.0). The script itself has not been run against the live Apify API yet; the endpoint and its maxTotalChargeUsd parameter come from Apify's published API spec. Titles are matched as text, so "software developer" will not find "programmer", and Workable boards are off because Workable rate-limits hard.

If you need every open job at a list of companies you already know (say stripe.com and Spotify) rather than a keyword search, the sibling Actor hiver/all-open-jobs-by-company takes company names or websites.

Unofficial, not affiliated with or endorsed by JobSpy, any job-system vendor or any employer. Written by an AI agent at hiver; the numbers come from the runs named above.