Table of Contents

Chapter 1 of 4. This post covers:

  1. Module 1.1 — The Big Picture: What a Model Actually Learns
    1. 1.1.1 The One Sentence Everything Builds On
    2. 1.1.2 The Pipeline Loop
    3. 1.1.3 The Quantity, Quality, Diversity Trade-off
  2. Module 1.2 — The Mathematics of the Training Objective
    1. 1.2.1 Cross-Entropy, Split in Two
    2. 1.2.2 Proof in Code: Clean vs Spam-Polluted Training
    3. 1.2.3 The ln(V) Sanity Check
  3. Module 1.3 — What the Model Is Truly Optimizing For
    1. 1.3.1 From Tokens to Loss to Perplexity
    2. 1.3.2 Scaling Laws, in One Line
    3. 1.3.3 Where the Numbers From the Intro Come Back

Chapters 2, 3 and 4 (cleaning the web, teaching filters to think, and the final mile into tokens and batches) publish as their own posts.

Introduction

I have been quiet for a while, not because I had nothing to say, but because I kept turning one question over in my head: how do people like us, without a fraction of the money the big labs spend, actually contribute to building AI that reflects who we are. The honest answer took time to arrive at, and the number that finally pushed me to write this down is too large to sit with quietly.

Did you know? Meta's own published model card for Llama 3.1 states that training the 405-billion-parameter version alone used 30.84 million GPU-hours on H100 chips drawing 700 watts each. Run that math and it comes out to roughly 21.6 million kilowatt-hours of electricity, enough to power about 2,000 average households for a full year. Meta's own figures put the location-based emissions from that one training run at 8,930 tons of CO2, close to what 1,900 cars emit in a year. Spread across the 16,384 GPUs used, that workload still took about 78 days of continuous training, and at a modest $2 per GPU-hour cloud rate, the compute bill alone comes to roughly $62 million, before a single researcher, dataset, or building is paid for.

Sit with that $62 million for a second, then sit with this. In 2021, OpenAI contracted a company called Sama to have Kenyan workers label disturbing content so ChatGPT would not reproduce it, paying those workers roughly $1.50 to $2 an hour while internal documents showed OpenAI itself was billed $12.50 an hour for that same work. Separately, Scale AI, the company Alexandr Wang led as CEO until Meta bought 49 percent of it for $14.3 billion in 2025, has faced its own wave of US lawsuits after reports that its Remotasks platform paid some workers as little as a few cents per task. Two different companies, two separate scandals, the same pattern underneath: the humans doing the most psychologically demanding part of building this technology are paid the least, while the compute bill for the model they helped clean runs into the tens of millions.

Who this is for. This series is written for builders, students, self-taught engineers, researchers and curriculum designers anywhere who want to understand how large language models are actually built, not just how to call an API, especially those of us working with limited compute, limited funding and limited institutional backing. You do not need a GPU cluster or a research lab to read this. You need curiosity, a laptop, and the willingness to look closely at data most people scroll past.

exploting Kenyan people to label data

And while that gap sits unresolved, the people steering this industry are moving at a different altitude entirely. In September 2025, Donald Trump told the United Nations General Assembly that the United States would start calling artificial intelligence "superintelligence," or SI, later signing an executive order requiring federal agencies to use the new term. That is the altitude at which this race is being run, nations rewriting policy and vocabulary around who leads next, while much of the data that trains these systems, our languages, our conversations, our culture, keeps flowing outward for free.

This is where the series begins, not with despair at the size of the gap, but with the one part of it we can actually close ourselves. Over the next four chapters we move from the big picture of why training data matters so completely, through where real data comes from and how it gets cleaned, into the filtering and quality techniques that separate a frontier lab's dataset from a pile of scraped pages, and finally into how that data becomes the tokens and batches a model actually trains on. None of it requires a data center. All of it requires showing up.

Module 1.1: The Big Picture: What a Model Actually Learns

1.1.1 The One Sentence Everything Builds On

Here is the one sentence everything else in this series builds on: a language model becomes whatever you feed it. Not a feeling, not a simplification, a provable fact. Whatever pattern sits in the training text, grammar, facts, boilerplate, spam, a stereotype repeated often enough, gets absorbed with the same hunger. The model has no way to tell your intentions apart from the data's contents. It only ever sees the data.

That single fact is why data preparation is not a chore that happens before the real work of building AI. It is the real work. The architecture matters, but far less than people assume. Most open models today use nearly the same transformer design. What actually separates a strong model from a weak one is almost always what it was fed, how clean, how diverse, how deliberately chosen.

1.1.2 The Pipeline Loop 

flowchart TD A["Sources: web crawl, books, code, research, community data"] --> B["Extraction: raw pages to clean text"] B --> C["Filtering: remove junk, duplicates, unsafe content"] C --> D["Mixing: decide how much of each source to use"] D --> E["Tokenizing and batching"] E --> F["Training: the model learns to predict the next token"] F -->|"evaluate, then adjust a filter or a weight"| B

Notice the loop at the end. Nobody gets the mixture right on the first try. Every strong dataset in use today is the record of a team running that loop dozens of times, training small models, checking what improved, and adjusting. That loop is something anyone with a laptop can run at a small scale, long before anyone can afford a GPU cluster.

1.1.3 The Quantity, Quality, Diversity Trade-off

There is also a trade-off worth holding in your head for the rest of this series: quantity, quality, and diversity pull against each other. Aggressive filtering raises quality but shrinks quantity and diversity. Keeping everything preserves quantity but drowns the model in noise. Every technique in this series is really a negotiation between these three, and knowing that negotiation exists is most of what separates a deliberate data pipeline from a pile of scraped pages.

Module 1.2: The Mathematics of the Training Objective

Image prompt: Close-up photograph of a person's hands writing simple mathematical notation in a physical notebook next to an open laptop showing a blurred terminal window. Warm desk lighting, a coffee cup nearby, a genuine workspace. No digital overlays or glowing text. Shallow depth of field. 4:3.

1.2.1 Cross-Entropy, Split in Two

Training a language model means minimizing one quantity: the average surprise the model feels at the real next word, measured across the whole dataset. In the language of information theory, this quantity is called cross-entropy, and it splits cleanly into two pieces.

cross-entropy = entropy of your data + KL divergence (model's distance from your data)

The first piece, the entropy of your data, is a floor nothing can move. It is the randomness genuinely present in real language. The second piece, the KL divergence, is the only part training can ever reduce. Training pushes the model toward your data's distribution and nothing else. Read that sentence again with data quality in mind: the best possible outcome of training is a model that perfectly reproduces whatever is in your dataset. If 10 percent of your dataset is spam, the best the model can ever do is spam 10 percent as convincingly as the rest.

1.2.2 Proof in Code: Clean vs Spam-Polluted Training

This is not an abstract claim. I built the smallest possible version of a language model, one that predicts the next character using nothing but counts, and trained two copies of it on the same public-domain text, one clean and one with spam lines inserted after roughly a quarter of the lines.

held-out loss, trained on clean text: 2.4838 held-out loss, trained on spam-polluted text: 2.6550

Both models were tested on the same clean held-out text. The one trained on polluted data scored worse, a perplexity of about 14.2 against 12.0 for the clean model, because it wastes some of its probability on patterns that never show up in honest writing. Sample text generated from the polluted model shows fragments of the inserted spam, words like CLICK, BUY, FREE, www, sitting inside what should have been ordinary prose. A transformer with billions of parameters is a far better copier than this toy model. It will learn the spam more precisely, not less.

1.2.3 The ln(V) Sanity Check

One more number worth keeping in your pocket: at the very start of training, before the model has learned anything, its loss should sit almost exactly at the natural logarithm of the vocabulary size, written ln(V). For a tokenizer with 50,257 possible tokens, that is 10.82. If a real training run's starting loss is far from that number, something upstream is broken, often in the data pipeline, long before it is worth blaming the model architecture. This single check has caught more pipeline bugs than almost any other.

What this means practically: the moment you accept a document into your training set, you are voting for the model to reproduce it. Every filtering decision in the rest of this series exists because of this one mathematical fact.

Module 1.3: What the Model Is Truly Optimizing For

Image prompt: A wide photograph contrasting human scale against infrastructure: a single person sits silhouetted on a rooftop at golden hour, laptop on their knees, looking out over a sprawling city below. Cinematic natural golden-hour lighting, no data-center or sci-fi imagery. 21:9.

1.3.1 From Tokens to Loss to Perplexity

A document is a sequence of tokens, and the probability of the whole sequence is the product of the probability of each token given everything before it. Multiplying many probabilities together collapses toward zero extremely fast, so in practice everything is done in log space instead, where multiplication becomes addition. The average of those log-probabilities, flipped positive, is the loss. Raise e to the power of the loss and you get perplexity, a number with an intuitive reading: a perplexity of 15 means the model is, on average, about as unsure as if it were choosing between 15 equally likely options at every step.

1.3.2 Scaling Laws, in One Line

Scaling laws describe how this loss falls as a model and its training data both grow. The widely cited Chinchilla work (Hoffmann et al., 2022) fits a simple form:

loss = irreducible floor + (a term that shrinks as parameters grow) + (a term that shrinks as training tokens grow)

The practical lesson from that paper was that model size and data size should grow together, and the field has since pushed even further: modern 8-billion-parameter open models are now trained on more data per parameter than Chinchilla's own rule suggested, because a smaller model trained longer is cheaper to run for everyone who uses it afterward. That single design choice is why the field's appetite for clean, usable text keeps climbing even as models stay a similar size.

1.3.3 Where the Numbers From the Intro Come Back

This is also precisely where the opening numbers of this post connect back to the mathematics. Meta's own published figures state that training the 405-billion-parameter version of Llama 3 alone consumed 30.84 million GPU-hours. All of that compute exists to shrink one single number, the loss, by pushing the model closer to the data it was shown. None of it can make the model smarter than its data. A perfectly optimized model trained on a flawed dataset is still a flawed model, just a very confident one.

That is the thread this whole series pulls on. Chapter 2 picks up exactly here, at the question of where real training data actually comes from, and the first layer of work required before any of this mathematics gets to do something useful.

References

Next up, Chapter 2: From the Open Web to a Clean Corpus.