What we learned scraping SeatGeek past DataDome

You point your scraper at a SeatGeek event and get a 403, a "Please enable JS" page, or the DataDome slider. You switch to a residential proxy and get the same thing. You add a stealth plugin and spoof a fingerprint, and the slider shows up more often.

Most of that effort goes to the wrong place. DataDome checks the connection, the browser, and the session, and a better IP only fixes the first one. In our tests, spoofing made things worse, because every value you override has to agree with every other value, and DataDome's tag is built to find the ones that don't. Our biggest improvement came from removing overrides.

Once the page loads, you're only halfway there. The event name, venue, and date come with the page. The listings come from a separate JSON request:

https://seatgeek.com/api/event_listings_v2?... 

That response has the section, row, quantity, purchase options, and prices. DataDome can refuse it while the page itself loads fine, and we've seen that happen on workers that passed every HTML check we had.

We run Tickets.dev and collect SeatGeek listings in production. Our collector lets the browser load the page and captures the listing responses it makes.

Why we run a real browser instead of reverse engineering DataDome

We think reverse engineering DataDome is the wrong bet for SeatGeek, so we run real browsers.

The page makes the listings request after DataDome's JavaScript has run, and DataDome scores the device, the browser environment, and interaction along with the request. Replaying that request from Python means keeping the context consistent: the TLS fingerprint, HTTP/2 settings, header order, client hints, the datadome cookie, the IP that cookie was issued to, and whatever the tag measured on the page. Mismatches can send you back to another challenge.

When DataDome changes the flow, that implementation needs updating. A real browser runs the updated JavaScript. We put our maintenance into the environment instead: a real display, native device settings, real input, and a fresh profile on every launch. We don't solve CAPTCHAs in production. A persistent challenge fails the attempt.

A browser costs more per capture, a few seconds and a few hundred megabytes. For us that's still cheaper than an engineer chasing tag versions. When a marketplace has no bot defense and its listings JSON answers a plain GET, we skip the browser and fetch it.

What DataDome checks on SeatGeek

A Chrome user agent doesn't make a Python HTTP client look like Chrome, and a browser with contradictory device values still has them behind a residential IP. DataDome's TLS write-up describes catching clients whose handshake doesn't match the browser they claim to be. Their JavaScript Tag docs list browser, OS, GPU, and behavioral signals, plus consistency checks between them.

DataDome doesn't say how it weighs these. Here's what made a difference for us.

Test readiness with a listings response

A warm-up that only checks whether the HTML loaded can put a broken worker into service. We've had normal homepage and event page loads while /api/event_listings_v2 was refused. Make the readiness check a successful listings response for a known event.

A first 403 can still turn into listings

In our captures, a refused listings request is sometimes followed by a device check and a successful retry. DataDome's Device Check runs without user interaction. It checks the device and environment, then allows the request, blocks it, or escalates to a CAPTCHA.

So give the page a fixed window to finish, and record the whole response sequence before you decide what happened. An interstitial that clears after a few seconds and one that stays on screen are different outcomes. A timeout or a missing request on its own doesn't tell you the IP is blocked.

Native fingerprints performed better in our tests

We ran tests on SeatGeek with the same box and proxy. With native device settings, the browser passed the device check. With a seeded fingerprint, it kept getting the slider. That changed where we spent our time: removing overrides helped more than adding another spoofed attribute.

Browser language, request headers, rendering environment, and platform values all need to agree. Adding more spoofed values just gives DataDome more chances to find one that doesn't match.

On TickPick, another DataDome site, the same browser and proxy also behaved differently on the host and in a container. Fonts and display configuration differed between the two, so we can't say which one decided it, only that the environment mattered.

We run headed Chrome on a real display and inspect the renderer and device settings on the actual machine. Modern Chrome has unified headless and headed implementations, so the configuration matters more than the label.

Navigation and input are part of the session

We compared navigation methods with the browser, profile configuration, and proxy held fixed. Scripted navigation landed on an interstitial. A trusted input click reached the listings.

A JavaScript element.click() and a click from real input are different browser events, and setting location.href isn't a click at all. That's why the SeleniumBase example uses a GUI click. DataDome's JS Tag keeps collecting interaction signals through the session.

Residential IPs get flagged too

DataDome has published a residential-proxy detection model that looks at aggregate traffic patterns, including how much clients and session characteristics vary. So a shared ISP exit can have a bad reputation even though it's a residential address.

So don't blame the proxy for every failure. Check whether it was the IP, the browser, or the navigation.

Capture the response the page already makes

Our collector reads from two places:

Attach the response listener before you navigate. Otherwise you can miss the first inventory response and sit waiting for a request that already happened.

The SeleniumBase example below uses the same UC/CDP approach as our collector. It warms up the browser, attaches response interception, navigates with a GUI click, and collects the listing payloads.

These are the interception calls from the full script below:

page.add_handler(mycdp.fetch.RequestPaused, on_paused) sb.cdp.get_event_loop().run_until_complete( page.send(mycdp.fetch.enable( patterns=[mycdp.fetch.RequestPattern( url_pattern="*/api/event_listings_v2*", request_stage=mycdp.fetch.RequestStage.RESPONSE, )] )) ) 

The handler resumes the paused request in a finally block, including when reading or parsing the body fails.

What the abbreviated fields mean

These are the fields our collector reads:

Keep the raw response. If a field changes, you can fix the parser and reprocess old captures.

Prices: keep the displayed total and fee separately

Use dp to compare the displayed total per ticket, and don't overwrite the returned fee with dp - p. Across both live captures we checked, every displayed total matched ceil(p + fee). The total is rounded up to the dollar, while the price and fee keep their cents. A ticket price of 344 and a fee of 57.85 add up to 401.85, and SeatGeek shows 402. Work the fee back from 402 and you get 58.

Keep all three values. If dp is missing and you have the price and fee, the example saves their sum as a calculated fallback. That fallback keeps cents; it is separate from SeatGeek's rounded-up display price.

Leave missing prices missing. If you default them to zero, broken records end up at the top of your cheapest-ticket results. Read the currency from the response when it's there, and don't label an unknown currency USD.

Quantity: a listing of three may not sell you two

q is how many tickets the listing has. sp is which quantities you can buy. If you want a pair, check that 2 is in sp.

Count listings and tickets separately too. Ten listings can hold far more than ten tickets.

Seats: watch for placeholder numbers

Seat numbers can be missing. Our implementation notes also flag placeholder sequences that start at 9001, and sometimes at 111. Real venues have high seat numbers too, so treat these as a reason to look at the data, and don't mark every seat over 111 as fake.

The code keeps the raw labels and sets suspectedPlaceholderSeats when every label is numeric and at least 9000. It's a heuristic, and false doesn't mean the seats are real. It doesn't try to catch sequences starting at 111. Don't use a suspected placeholder to match seats across marketplaces or to place tickets on a seat map.

Section and row describe a group of seats. Two listings with the same section and row can still be different tickets.

Section and row labels

Named sections can come through lowercased in s. Use sf for display instead of rebuilding the label yourself.

For rows, our parser takes rf, then r, then rr, and strips a leading "Row ". Keep the original fields as well, since the alternate spellings sometimes carry information.

General admission can have no row or seat at all.

Primary inventory and parking

The example treats m == "open" as primary inventory and leaves other markers unclassified. inventoryMarker keeps the source value, and inventoryType is the example's own mapping. Don't assume every other marker means resale.

Parking can show up alongside admission tickets. If you filter out sections containing "parking," record that you did, because your count will no longer match the raw response. Text matching is only a rough filter.

One response, not pages

In our live captures, the listings came back in one response. The requests had no page, limit, or offset parameters, and scrolling didn't trigger another listings call.

A second listings response is usually DataDome's retry after a 403 (its URL adds a retry parameter). The script merges responses by listing ID, so a retry doesn't double-count anything.

Inventory changes between captures, so record when each capture started and ended. The example records both, plus a receive time for each response in statuses.

Separate an empty event from a failed capture

An HTTP status alone won't tell you which one you have.

In our collector, a first refusal is sometimes followed by a successful retry once the page finishes its checks, so wait for the actual outcome before you call the attempt failed. It works the other way too: a 200 can carry a challenge payload instead of inventory, so a body without listings isn't automatically an empty event.

A redirect, an empty page, or a sudden drop in inventory doesn't tell you what happened by itself, so keep the raw response.

Full SeleniumBase example

Install the dependencies on a desktop machine with Chrome and a display:

pip install seleniumbase mycdp pyautogui 

The script starts CDP mode at about:blank, loads the homepage once, and checks the performer page for event links or page data before moving on. SeleniumBase's CDP docs cover the browser methods used here.

Save the script below as scrape_seatgeek.py, then run it with the event's full URL:

python scrape_seatgeek.py "https://seatgeek.com/your-event-url" --output snapshot.json 

--output writes the JSON directly to a file. Add --include-raw to keep successful raw payloads. Failed attempts write their details to seatgeek_debug.json.

The collector this example comes from returned listings in live runs. This copy adds file output and removes the scroll loop. We use fresh profiles and do not solve CAPTCHAs; the example follows that approach. It leaves out our production worker pool.

#!/usr/bin/env python3 """ scrape_seatgeek.py: capture SeatGeek listings from the page's own API call. python scrape_seatgeek.py "https://seatgeek.com/...-tickets/.../17965048" Requires headed Chrome and a working desktop display. Our deployment uses residential/static-ISP exits; these do not guarantee access. """ import argparse import base64 import json import random import math import re import time from datetime import datetime, timezone from pathlib import Path from urllib.parse import parse_qs, urlparse from seleniumbase import SB import mycdp HOMEPAGE = "https://seatgeek.com" LISTINGS = "/api/event_listings_v2" EVENT_ID = re.compile(r"/(\d{6,})/?(?:[?#]|$)") PERFORMER_HREF = re.compile(r'^/[a-z0-9][a-z0-9-]*-tickets(?:\?|$)') NEXT_DATA = re.compile( r'<script id="__NEXT_DATA__" type="application/json">(.*?)</script>', re.S ) BLOCK_MARKERS = ("captcha-delivery.com", "geo.captcha-delivery", "interstitial.captcha") def wait_until(sb, predicate, timeout, interval=0.25): end = time.monotonic() + timeout while time.monotonic() < end: try: if predicate(): return True except Exception: pass sb.cdp.sleep(interval) try: return bool(predicate()) except Exception: return False def html(sb): try: return sb.cdp.get_page_source() or "" except Exception: return "" def find_links(sb, pattern): try: source = sb.cdp.get_page_source() or "" except Exception: return [] found = [] for match in re.finditer(r'href="([^"]+)"', source): href = match.group(1) if pattern.search(href) and href not in found: found.append(href) return found def looks_blocked(sb): # Throttle full-source reads, including repeated wait_until calls. now = time.monotonic() previous = getattr(sb, "_sg_block_probe", None) if previous is not None and now - previous[0] < 2: return previous[1] source = html(sb).lower() result = any(marker in source for marker in BLOCK_MARKERS) sb._sg_block_probe = (now, result) return result def persistent_block(sb): if not looks_blocked(sb): return False sb.cdp.sleep(5) return looks_blocked(sb) def usable_page(sb, kind): # Return a boolean from the page instead of transferring its full markup. return bool(sb.cdp.evaluate(r""" (() => { const links = Array.from(document.querySelectorAll('a[href]')); const eventLink = links.some(a => /\/\d{6,}(?:[/?#]|$)/.test(a.href)); const performerLink = links.some(a => /-tickets(?:[?#]|$)/.test(a.href)); let pageProps = null; try { const node = document.querySelector('script#__NEXT_DATA__'); pageProps = node && JSON.parse(node.textContent)?.props?.pageProps; } catch (_) {} return KIND === 'homepage' ? performerLink : eventLink || !!(pageProps && Object.keys(pageProps).length); })() """.replace("KIND", json.dumps(kind)))) def click_href(sb, href): """Click a real same-site link through the OS pointer, falling back to navigation.""" before = sb.cdp.get_current_url() sb.execute_script(f""" (() => {{ let a = document.getElementById('__sg_nav'); if (!a) {{ a = document.createElement('a'); a.id = '__sg_nav'; document.body.appendChild(a); }} a.href = {json.dumps(href)}; a.style.cssText = 'position:fixed;left:0;top:0;width:100vw;height:100vh;' + 'z-index:2147483647;opacity:0.01;display:block'; a.textContent = ''; }})() """) try: # pyautogui: moves the real cursor in steps, then clicks. sb.cdp.gui_click_element("#__sg_nav") except Exception: sb.execute_script(f"window.location.href = {json.dumps(href)}") return "scripted" if not wait_until(sb, lambda: sb.cdp.get_current_url() != before, 6): sb.execute_script(f"window.location.href = {json.dumps(href)}") return "scripted" return "gui" def warm_up(sb): # CDP starts at about:blank; this is the only homepage navigation. sb.cdp.get(HOMEPAGE) if not wait_until(sb, lambda: usable_page(sb, "homepage"), 30, interval=2): raise RuntimeError("homepage did not expose performer links") if looks_blocked(sb) and not wait_until( sb, lambda: not looks_blocked(sb), 12, interval=2 ): raise RuntimeError("persistent challenge on homepage") performers = find_links(sb, PERFORMER_HREF) if not performers: raise RuntimeError("homepage listed no performers") mode = click_href(sb, random.choice(performers)) if not wait_until(sb, lambda: usable_page(sb, "performer"), 30, interval=2): raise RuntimeError("performer page has no event links or page data") if looks_blocked(sb) and not wait_until( sb, lambda: not looks_blocked(sb), 12, interval=2 ): raise RuntimeError("persistent challenge on performer page") return mode def request_event_ids(request): # Recognize common URL/body locations; unknown schemas use a logged fallback. ids = set() parsed = urlparse(request.url) match = EVENT_ID.search(parsed.path) if match: ids.add(match.group(1)) def collect(value, keys): if isinstance(value, dict): for key, item in value.items(): if key in keys: for v in item if isinstance(item, list) else [item]: ids.update(re.findall(r"\d+", str(v))) elif isinstance(item, (dict, list)): collect(item, keys) elif isinstance(value, list): for item in value: collect(item, keys) event_keys = ("event_id", "eventId", "event_ids", "eventIds") # SeatGeek sends the event as ?id=<event id> on the listings URL. collect(parse_qs(parsed.query), event_keys + ("id",)) post = getattr(request, "post_data", None) if post: try: collect(json.loads(post), event_keys) except (TypeError, ValueError): collect(parse_qs(post), event_keys) return ids def challenge_details(body, payload=None): # Best-effort parsing only. Do not execute a challenge's embedded JavaScript. details = {} url = payload.get("url") if isinstance(payload, dict) else None if isinstance(url, str): host = (urlparse(url).hostname or "").lower() if host == "captcha-delivery.com" or host.endswith(".captcha-delivery.com"): parsed = urlparse(url) details = {"url": url, "path": parsed.path} query = parse_qs(parsed.query) if query.get("t"): details["t"] = query["t"][0] dd = re.search(r"\bdd\s*=\s*(\{.*?\})", body or "", re.S) if dd: for key in ("t", "rt"): pattern = r"(?:^|[,{])\s*[\"']?" + key + r"[\"']?\s*:\s*[\"']([^\"']+)[\"']" match = re.search(pattern, dd.group(1)) if match: details[key] = match.group(1) return details def record_response(state, record, body): # HTTP statuses remain separate from parsing and association problems. state["statuses"].append(record) try: payload = json.loads(body) except (TypeError, ValueError): payload = None challenge = challenge_details(body, payload) if challenge: state["challenges"].append({**record, **challenge}) valid = ( record["status"] == 200 and isinstance(payload, dict) and isinstance(payload.get("listings")Source：https://dev.to/kgbop_54/what-we-learned-scraping-seatgeek-past-datadome-44ln
My favorites

Related posts

Sherman Anti-Trust Act (1890

Article URL: https://www.archives.gov/milestone-documents/sherman-anti-trust-act Comments URL: https://news.ycombinator.

0Hacker News

Your Phone Can Now Listen and Talk Completely Offline

Your phone can now listen to you and talk back without ever touching the internet. Not a demo. Not a research paper. Goo

0dev.to AI

Cloud Computing

Introduction Cloud computing has changed how individuals, developers and businesses access computing resources. In the p

0dev.to AWS

I Mostly Consume APIs. Here Is What I Wish Their Publishers Had Decided Before v1

Most of the APIs I deal with, I deal with from the wrong side of the door. I build VoiceDash, a white-label portal that

0dev.to WebDev

Building WUUUH: The First Week Needs Finishable Days, Not Bigger Goals — What Worked, What Broke

The first week after a layoff breaks a specific piece of machinery: the sense that a day can be finished. In a job, the

0dev.to WebDev

Teaching a robot to feel what is inside a sealed cup

At the Viam Physical AI hackathon in San Francisco on October 29 and 30, Sree Dhyuti Nimmagadda and I taught a robot arm

0dev.to WebDev