Most company career pages aren't really websites. They're a thin front end over an applicant tracking system (ATS), and almost every ATS exposes the same public JSON that powers its own job widget. If you need job data for a job board, a recruiting tool or labor-market research, reading that JSON is much more robust than parsing HTML.
This post shows the endpoints, the gotchas, and a ready-made way to run it on a schedule.
1. Identify the ATS from the careers URL
| If the careers link looks like… | ATS | Board identifier |
|---|---|---|
boards.greenhouse.io/<slug> or job-boards.greenhouse.io/<slug> | Greenhouse | <slug> |
jobs.lever.co/<slug> | Lever | <slug> |
jobs.ashbyhq.com/<slug> | Ashby | <slug> |
<tenant>.wd5.myworkdayjobs.com/<Site> | Workday | <tenant> + <Site> |
If the company's own /careers page doesn't show one of these, view the source. The embed script usually reveals it.
2. The public endpoints
Greenhouse:
curl "https://boards-api.greenhouse.io/v1/boards/stripe/jobs?content=true" Lever:
curl "https://api.lever.co/v0/postings/palantir?mode=json" Ashby:
curl "https://api.ashbyhq.com/posting-api/job-board/ramp?includeCompensation=true" Workday (a POST with paging, 20 per page):
curl -X POST "https://nvidia.wd5.myworkdayjobs.com/wday/cxs/nvidia/NVIDIAExternalCareerSite/jobs" \ -H "Content-Type: application/json" \ -d '{"appliedFacets":{},"limit":20,"offset":0,"searchText":"data engineer"}' Then GET .../wday/cxs/<tenant>/<site><externalPath> for each job's full description.
3. Gotchas you'll hit
- Different shapes everywhere. Locations, remote flags and salary fields are all modeled differently, so you'll want one normalized schema.
- Workday dates are relative ("Posted 3 Days Ago", "Posted 30+ Days Ago"), so you have to derive an approximate date.
- Workday search caps out at about 2,000 results per query. Split big employers by keyword.
- Monitoring needs state. To alert on new jobs, store the IDs you've already seen between runs.
- Be polite. These are public endpoints, but keep concurrency modest and retry with backoff.
4. Or skip the plumbing
I packaged all of this as Apify Actors, which you can call from the API, schedule, or plug into n8n/Make/Zapier:
- ATS Job Scraper (Greenhouse, Lever, Ashby, SmartRecruiters, Recruitee): https://apify.com/birchwood-data/ats-job-scraper
- Workday Jobs Scraper: https://apify.com/birchwood-data/workday-jobs-scraper
Calling one from Python:
from apify_client import ApifyClient client = ApifyClient("<YOUR_APIFY_TOKEN>") run = client.actor("birchwood-data/workday-jobs-scraper").call(run_input={ "careerSiteUrls": ["https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite"], "searchText": "data", "maxJobsPerSite": 20, }) for job in client.dataset(run["defaultDatasetId"]).iterate_items(): print(job["title"], "|", job["location"], "|", job["url"]) Set onlyNewJobs: true and a daily schedule, and each run returns only the jobs that appeared since the last one.
5. Bonus: build the company list first
If you don't have a target list yet, the Y Combinator Scraper (https://apify.com/birchwood-data/yc-companies-scraper) exports the YC directory. With detectCareersPage: true, it tries to find each startup's ATS board, which you can feed straight into the job scrapers. Detection only works when the board link is visible in the page HTML.
Disclosure: I built and maintain these Actors. They're paid per result on Apify (from $0.001 per result at the time of writing).
