How Do AI Companies Get Their Training Data?

Originally published at aidata.broker.

AI companies get their training data through six channels: they crawl the open web, license content from publishers and platforms, pay experts and contractors to make data to order, buy or license records that companies already hold, build simulated environments and synthetic data, and use what their own users write and upload. The first channel built today's models and is still the largest by volume: one monthly Common Crawl archive holds 2.17 billion web pages. It is also the one closing fastest, with 11.4% of the sites read so far in our census of the 300,000 most visited shutting at least one AI crawler out. The money has moved to the other five: the large data vendors' newest annual revenues and run rates add up to $8.1B, Reddit signed $203.0 million of data licences in a single month, and Google bid $10 million for one bankrupt airline's emails and chats.

This guide takes each channel in turn: how it works, how big it is, what it costs, what legal exposure it carries and which way it is moving. It is written for the founder, CFO or counsel of a company that holds data and wants to understand where its data fits, and for anyone watching the market who wants the mechanics rather than the headlines. Our supply map shows where data sits, kind by kind; this guide shows how it moves, channel by channel. It closes with how the mix is shifting and what that means for a holder.

AIData.Broker acts as the holder's agent: the holder itself licenses or sells its data to the buyer, and we do not take ownership of it. This guide is research from public sources, not legal, tax or investment advice.

Contents

1. The six channels at a glance
2. Crawling the open web
3. Licensing content from publishers and platforms
4. Commissioning human and expert data
5. Buying company records and archives
6. Building environments and synthetic data
7. Their own users' data
8. How the mix is shifting
9. Questions people ask
10. What this means for you

The six channels at a glance

A large language model learns in stages. Pretraining feeds it trillions of words so it learns language and facts; Alibaba's Qwen3 models, for example, were pretrained on approximately 36 trillion tokens (a token is a word or a piece of one). Post-training then teaches it to follow instructions, reason and do tasks, using far less data, but data that is much harder to get: worked answers written by experts, judgments about which answer is better, and practice in simulated software. Each stage draws on different channels, and the channels differ in what they cost and who gets paid.

The table summarises the six channels before the sections take each apart. "Who gets paid" is the column that matters most to a holder: only three channels pay the people or companies that hold the data, and they are the three growing fastest.

1. Crawling the open web

How crawling works

A crawler is a program that downloads web pages by following links, the same way a search engine indexes the web. Two kinds feed AI. The first is Common Crawl, a nonprofit archive that has crawled the web for years and publishes each crawl for anyone to use; its September 2026 archive holds 2.17 billion web pages, or 361.4 TiB of uncompressed content, from 40.4 million hosts. Most large models start from cleaned versions of archives like it. The second kind is each lab's own crawler: OpenAI's GPTBot, Anthropic's ClaudeBot, Meta's meta-externalagent, ByteDance's Bytespider, plus the switches Google and Apple offer (Google-Extended, Applebot-Extended) for refusing AI training specifically.

Crawling is cheap to collect and expensive to clean, and it is running out. Epoch AI estimated the stock of quality public human text at around 300 trillion tokens and projected that models would fully use it between 2026 and 2032. Of the 2.17 billion pages in the September archive, only 587.2 million URLs had never been crawled before: most of each new crawl is the web the models have already read.

How many sites now block AI crawlers

A site tells crawlers what they may fetch in a public file called robots.txt. Our AI crawler census reads that file for the 300,000 most visited sites, 4,000 a morning, and records which AI crawlers each one shuts out of the whole site. In the current pass it has read 6,094 sites, and 11.4% block at least one AI crawler; 8.4% block two or more, one of them a training crawler. Most of the web still lets the crawlers in. The refusals matter more than their share suggests, because they are concentrated among the sources with the most valuable text.

The blocking is concentrated where it hurts. A 2024 academic audit of the C4 dataset (one of the standard cleaned web corpora) found that robots.txt restrictions added in a single year cut off about 5% or more of all its tokens, and 28% or more of its most actively maintained sources, while 45% of C4 was restricted by terms of service. Publishers' reason is arithmetic: Cloudflare measured Anthropic's crawlers fetching as much as 500,000 pages for every visitor they referred back in 2025, against a peak of 3,700 to one for OpenAI. A crawler that takes and sends nothing back is a cost.

What it costs, and the legal exposure

Crawling carries no licence fee, so its cost is legal risk, and that risk is now priced. The clearest price so far came from books rather than web pages: in Bartz v. Anthropic, the court held that training on lawfully acquired books was fair use but sent the pirated library copies to trial, and the case settled for $1.5 billion, about $3,000 per work. In Kadrey v. Meta the court granted partial summary judgment for Meta on fair use. In Thomson Reuters v. Ross, a non-generative legal search tool, the Third Circuit affirmed for Thomson Reuters against fair use. The New York Times' case against OpenAI and Microsoft sits at cross-motions for summary judgment, and the US Justice Department filed a statement urging that training on written works is fair use, which our record notes is non-binding. We track 83 lawsuits, rulings and settlements on the cases page; the fair use guide explains how the courts are splitting.

Robots.txt itself is a request, not a lock. In Ziff Davis v. OpenAI, the court dismissed a claim that ignoring robots.txt breached the anti-circumvention rules, on the reasoning that "robots.txt directives are merely requests". Platforms have turned to contract and scraping law instead: Reddit sued Perplexity and three scraping firms, and a judge largely rejected their attempt to dismiss; News Corp sued Brave alleging it disguises its crawlers to evade publisher blocks. The guide to whether scraping data for AI is legal covers the law in detail.

Direction

Crawling will remain the base layer, because nothing else supplies trillions of tokens. But the share of valuable text a crawler can take without asking is shrinking, the sites that matter most are the ones refusing, and every major lab now faces at least one suit over what it took. The labs' answer has been to pay for the content they most need, which is the second channel.

2. Licensing content from publishers and platforms

When a publisher blocks a crawler or sues, the lab's alternative is a licence: a contract that pays for access to an archive, a live feed, or both. Our deals tracker records 111 disclosed licensing deals, 56 of them for training models; another 40 license content to be shown in AI answers rather than to train on, and only 19 disclose a dollar amount. OpenAI alone has signed 29.

These deals pay the large holders of text, images and video: news publishers, academic publishers, stock-image libraries and user-generated platforms. They rarely pay the people who wrote the content directly, and when they do, the share depends on how the licence was set up: where authors had to opt in, HarperCollins paid them $2,500 a title, while opt-out arrangements paid authors far less.

The largest disclosed deals

Sources for the deal structures below: recurring access, minimum plus variable, paid up front, one-time deals.

How the deals are structured

Three patterns recur. Most large licences grant recurring access for two to three years rather than a one-off copy: Reddit's January 2024 licences ran two to three years with at least $66.4 million recognised in the first year. Many pair a guaranteed minimum with a variable part: OpenAI pays Dotdash Meredith a fixed amount with a variable component on top, and Adweek reported that most of OpenAI's publisher deals use fixed and variable components. And a large share of the money arrives up front: Getty recognised about 62 percent of two multiyear deals in the quarter they were signed, then guided the next year's revenue lower, attributing the decline to one-time revenue from those licensing deals.

The licensor side has become a real business line for some. Shutterstock's data, distribution and services revenue, which includes licences for AI training, reached $203.3 million in 2025. Wiley reports lifetime AI revenue over $110 million, and its research business sells AI licences that include content Wiley licensed from other publishers, paying them royalties: a smaller publisher can reach the market through a larger one.

Legal exposure and direction

For the lab, a licence replaces copyright risk with contract terms, which is the point of signing. For the licensor, the exposure is whether it holds the rights it licenses: an academic press licensing authors' books, or a platform licensing its users' posts, has to rely on its own terms with those authors and users.

The direction is toward more deals, smaller and more specific, and toward live feeds for display as much as archives for training. Google's News AI pilot pays participating publications for the delivery and use of enhanced content; Meta signed a run of news deals with terms not publicly disclosed. For the history and numbers of the two largest groups, read the guides to Reddit's AI data licensing deals and news publishers' AI licensing deals.

3. Commissioning human and expert data

Post-training needs data that does not exist anywhere until someone makes it: a doctor's worked answer to a hard diagnostic question, a lawyer's ranking of two contract summaries, an engineer's solution to a coding task with tests that check it. Labs buy this through human data vendors, companies that recruit experts and contractors, design the tasks, check the work and deliver it. The technique most associated with it is RLHF (reinforcement learning from human feedback): people judge which of a model's answers is better, and the model is trained toward the preferred ones. Today the work runs well beyond ranking, into writing full tasks, red-teaming (trying to make a model misbehave) and grading long agent runs.

This is the paid channel with the most revenue on the record, and the one where the money is best documented. The large vendors' newest annual revenues and run rates add up to $8.1B, and the experts themselves are paid well by the hour. Our expert work page and rates page carry the detail; the summary below shows the shape.

Scale and price

The workforce is large. Handshake was paying ~$60M a month to over 30,000 experts by late 2025; Mercor shows $4M+ in daily payouts on its homepage and refers to nearly five million experts on its platform; Scale's Outlier says it has 900K+ AI trainers. Across advertised roles, experts earn $55 to $101 per hour of work, and Mercor states an average contracted rate of $112/hr.

The vendors keep a large share of what labs pay. For one expert hour, a lab pays Mercor $160 to $187 an hour, of which Mercor keeps $48 to $75 an hour, or 30% to 40% of each dollar. Appen, which reports its margin after paying its crowd, kept 40.3 percent in 2025. Prices per task are rising as tasks get harder: Mercor's chief executive described tasks priced from $50 to $10,000, and coding tasks across several vendors pay contributors the equivalent of about $45 to $100 an hour.

Legal exposure and direction

The legal exposure in this channel mostly sits with the vendor, which recruits the experts, contracts with them and holds a great deal of personal data about them. Mercor's own update on a security incident said that of its nearly five million experts, only a limited subset had sensitive information affected. For the lab, the risk is concentration: one customer accounted for about 58 percent of Innodata's 2025 revenue, which shows how few buyers set the price.

The direction is toward longer, more expert tasks and toward environments (channel 5), which the same vendors now build. It is also toward the records companies already hold (channel 4): several of these vendors, having paid experts to recreate how work is done, have started buying the real thing. The guide to AI training data companies maps who does what.

4. Buying company records and archives

The newest channel is the one this site was built around. A company's internal records (its email, chat, tickets, code with its review history, CRM and documents) show how real work gets done over years: who asked what, what went wrong, how it was fixed. No crawler can reach them and no expert can recreate them at scale. Since 2025, labs and the vendors that supply them have started paying companies for these records, licensing them under contract and de-identifying them before use.

The channel is small next to the vendors' revenue, but it is where much of the newest money is going. Our buyer board lists 24 buyers that publish a way in for company records. micro1 committed to spend $1 billion over the next 12 months acquiring and licensing them, and told a bankruptcy court it had closed more than 50 data transactions in 45 days. SimpleClosure, which helps startups shut down, counts 9 times more AI buyers than in 2025.

What the records fetch

The best-documented sale is Spirit Airlines' archive, sold in its bankruptcy. The archive held about 100 million emails and 500,000,000 Teams items. Google opened at $5 million, three qualified bidders took it to $10.0M in one session, Mercor was named the $7.5 million alternate, and micro1 later filed $12.5 million. At the small...