The LLM engineering stack, from the inside out: TEXT ──> TOKENIZER ──> EMBEDDINGS + POSITIONS ──> [ TRANSFORMER BLOCK x N ] ──> LOGITS ──> SAMPLER ──> next token ──┐ │ self-attention (with KV cache) │ │ feed-forward │ └───────────────────────────── repeat, token by token <───────┘ And around it: quantization (shrink it) · distillation (compress its skill) · LoRA (adapt it cheaply) · RAG (give it knowledge) · routing (pick the right model) · fallback (survive failures) · multimodal inputs (see more than text). 

ENCODER-ONLY (BERT-style) Each token attends to ALL tokens (bidirectional). Produces rich representations; great for classification, embeddings, NER, reranking. Doesn't generate text. DECODER-ONLY (GPT-style) Each token attends only to PREVIOUS tokens (causal mask). Trained to predict the next token; generates text autoregressively. The architecture of today's chat LLMs. ENCODER-DECODER (T5-style) An encoder reads the input; a decoder generates output while attending to the encoder. Common in translation and some summarization systems. 

 input text │ tokenizer v token IDs [1042, 318, 257, ...] │ embedding table (vocab_size x d_model) v token vectors + positional information (Section 3) │ v ┌───────────────────── TRANSFORMER BLOCK (repeated N times, e.g. 32 – 120) ─────────────────────┐ │ x ──> Normalize ──> MULTI-HEAD SELF-ATTENTION (causal) ──> + residual ──┐ │ │ ┌─────────────────────────────────────────────────────────────────────┘ │ │ └──> Normalize ──> FEED-FORWARD NETWORK (MLP) ──────────────────────> + residual ──> out │ └─────────────────────────────────────────────────────────────────────────────────────────────────┘ │ v final normalization ──> LM head (linear layer, d_model → vocab_size) ──> LOGITS (one score per vocabulary token) │ softmax (with temperature, top-k, top-p — Section 7) v next-token probabilities ──> sample ──> append to the sequence ──> repeat 

EMBEDDINGS A learned table mapping each token ID to a vector of size d_model (for example 4,096). The LM head often reuses (ties) the same matrix. RESIDUAL CONNECTIONS Each sub-layer's output is ADDED to its input (x + f(x)), letting gradients flow and letting layers refine rather than replace the representation. NORMALIZATION LayerNorm or RMSNorm stabilizes activations. Modern models usually normalize BEFORE each sub-layer ("pre-norm"), which trains more stably at depth. ATTENTION Mixes information ACROSS tokens (Section 2). FEED-FORWARD (MLP) Transforms each token's vector INDEPENDENTLY: expand to a larger hidden size, apply a nonlinearity (modern models often use gated variants such as SwiGLU), project back. The MLP holds a large share of the parameters (roughly two-thirds in classic designs) and much of the model's "knowledge." LM HEAD Projects the final vector to one score (logit) per vocabulary entry. 

Per block (classic design, MLP expansion of 4x): attention projections (Q, K, V, output): 4 x d² feed-forward (two matrices, d -> 4d -> d): 8 x d² ≈ 12 x d² parameters per block Total ≈ 12 x layers x d² (+ embeddings: vocab x d) Example (GPT-3-scale): 96 layers, d = 12,288 -> 12 x 96 x 12,288² ≈ 174 billion (published figure: ~175B). 

In an MoE model the single feed-forward network in a block is replaced by many "expert" networks plus a small ROUTER that sends each token to just a few of them (for example 2 of 64). + Total parameters (knowledge capacity) can be huge while ACTIVE parameters per token — and therefore compute per token — stay modest. - ALL experts must still be held in memory; routing adds engineering complexity; serving is harder (expert parallelism, load balancing). When you see "47B total / 13B active," that's MoE: memory cost follows total parameters, compute cost follows active parameters. 

1. PRETRAINING Predict the next token over trillions of tokens of text/code. Produces a "base" model with broad knowledge but no instruction-following manners. Enormously expensive. 2. SUPERVISED Fine-tune on curated (instruction, ideal response) examples so the model follows instructions and uses chat formats. FINE-TUNING (SFT) 3. PREFERENCE Align behavior with human or AI preferences (RLHF — reinforcement learning from human feedback; DPO and related methods optimize preferences more directly): helpfulness, honesty, TUNING safety, tone. 4. REASONING / RL Newer "reasoning" models add reinforcement learning on verifiable tasks to produce extended step-by-step reasoning before answering — trading latency and cost for accuracy. Application developers almost never pretrain; they use base-to-chat models as-is, adapt them cheaply (Sections 10-11), or call hosted models. 

- The model is a NEXT-TOKEN PREDICTOR; everything else (reasoning, tool use, JSON) is behavior learned or elicited on top. That's why validation, grounding, and constraints matter. - Cost scales with PARAMETERS (memory) and with TOKENS processed (compute), which is why prompt length, output length, and model size drive your bill and latency. - The context window is bounded by how the architecture handles position and by the memory cost of attention (Sections 3, 6). 

QUERY (Q) "What am I looking for?" KEY (K) "What do I contain / what can I be matched on?" VALUE (V) "What information do I hand over if I'm attended to?" For a sequence X: Q = X·W_Q K = X·W_K V = X·W_V Q · Kᵀ Attention(Q, K, V) = softmax( ────── + mask ) · V √d_k 

STEP BY STEP 1. SCORES Each token's query is dot-producted with every key: how well does token i match token j? (an n x n matrix) 2. SCALE Divide by √d_k so scores don't grow with dimension and saturate the softmax. 3. MASK In decoder-only models, token i may not attend to tokens AFTER it: set those scores to -infinity (the "causal mask"). 4. SOFTMAX Turn each row of scores into weights that sum to 1. 5. MIX Output for token i = weighted sum of the VALUE vectors, using those weights. 

// Educational single-head causal self-attention. Q, K, V are [sequenceLength][headDim]. // Real implementations use batched, fused GPU kernels (and avoid materializing the n x n score matrix). static float[][] CausalSelfAttention(float[][] Q, float[][] K, float[][] V) { int n = Q.Length; int d = Q[0].Length; var output = new float[n][]; for (int i = 0; i < n; i++) { // Token i may attend to tokens 0..i only (the causal mask) var scores = new float[i + 1]; for (int j = 0; j <= i; j++) { float dot = 0; for (int k = 0; k < d; k++) dot += Q[i][k] * K[j][k]; scores[j] = dot / MathF.Sqrt(d); // scaled dot product } // Softmax (subtract the max for numerical stability) float max = scores.Max(); float sum = 0; for (int j = 0; j < scores.Length; j++) { scores[j] = MathF.Exp(scores[j] - max); sum += scores[j]; } for (int j = 0; j < scores.Length; j++) scores[j] /= sum; // Weighted sum of value vectors var o = new float[V[0].Length]; for (int j = 0; j <= i; j++) for (int k = 0; k < o.Length; k++) o[k] += scores[j] * V[j][k]; output[i] = o; } return output; } 

d_model = 4,096, heads h = 32 -> each head works in head_dim = d_model / h = 128 dimensions. head_1 .. head_32 each compute attention independently -> concatenate (back to 4,096) -> output projection W_O. Different heads learn different roles (interpretability research finds heads that track previous tokens, match brackets, copy earlier text, and more). 

COMPUTE AND MEMORY: the score matrix is n x n -> cost grows QUADRATICALLY with sequence length n. Double the context and attention work roughly quadruples (the MLP part grows linearly). This is the fundamental reason long contexts are expensive and slow, and why so much engineering goes into attention efficiency. 

MULTI-QUERY ATTENTION (MQA) All query heads SHARE one key/value head -> the KV cache (Section 6) shrinks ~h-fold; slight quality cost. GROUPED-QUERY ATTENTION (GQA) Query heads are split into groups, each group sharing one K/V head. A middle path (for example 32 query heads, 8 KV heads) that most modern open models use: near-MHA quality with a much smaller KV cache and faster decoding. MULTI-HEAD LATENT ATTENTION Compresses keys/values into a low-dimensional latent to shrink the cache even further (used in some recent models). FLASHATTENTION NOT an approximation — an exact attention algorithm reorganized to minimize slow GPU memory traffic (tiling in fast on-chip memory), giving large speedups and avoiding the n x n matrix in memory. Standard in modern training and serving. SLIDING-WINDOW / LOCAL / SPARSE Each token attends only to a recent window (or a pattern), cutting cost from quadratic toward linear, at the price of direct long-range access (often combined with some global layers or "attention sinks"). CROSS-ATTENTION Queries from one sequence attend to keys/values from ANOTHER (decoder over encoder output; text over image features in some multimodal designs). 

SINUSOIDAL (ABSOLUTE) The original transformer added fixed sine/cosine waves of different frequencies to each token embedding: PE(pos, 2i) = sin(pos / 10000^(2i/d)) PE(pos, 2i+1) = cos(pos / 10000^(2i/d)) Each position gets a unique pattern; no learned parameters. LEARNED ABSOLUTE A trainable embedding per position (used by earlier GPT models). Simple, but can't extrapolate beyond the lengths seen in training. RELATIVE POSITION BIAS Encode the DISTANCE between tokens (added to attention scores), as in T5-style models. ALiBi Add a fixed penalty to attention scores that grows linearly with distance, with no position embeddings at all — good length extrapolation. ROTARY (RoPE) Rotate the query and key vectors by an angle proportional to their position. Because the dot product of two rotated vectors depends on the DIFFERENCE of their rotation angles, attention scores naturally depend on RELATIVE position. The dominant choice in modern open models. 

static float[][] SinusoidalPositionalEncoding(int maxPositions, int dModel) { var pe = new float[maxPositions][]; for (int pos = 0; pos < maxPositions; pos++) { pe[pos] = new float[dModel]; for (int i = 0; i < dModel; i += 2) { double angle = pos / Math.Pow(10000, (double)i / dModel); pe[pos][i] = (float)Math.Sin(angle); if (i + 1 < dModel) pe[pos][i + 1] = (float)Math.Cos(angle); } } return pe; } 

// Rotate consecutive pairs of dimensions by a position-dependent angle. Applied to Q and K (not V) before the dot product. static void ApplyRope(float[] vector, int position, float theta = 10000f) { int d = vector.Length; for (int i = 0; i < d; i += 2) { float frequency = MathF.Pow(theta, -(float)i / d); // low dimensions rotate fast, high dimensions slowly float angle = position * frequency; float cos = MathF.Cos(angle), sin = MathF.Sin(angle); float x0 = vector[i], x1 = vector[i + 1]; vector[i] = x0 * cos - x1 * sin; vector[i + 1] = x0 * sin + x1 * cos; } } 

A model only reliably handles positions it saw during training. Pushing beyond that range degrades quality. Techniques extend context after training: POSITION INTERPOLATION Compress positions so a longer sequence maps into the trained range, then briefly fine-tune. NTK-AWARE / YaRN SCALING Rescale RoPE frequencies so high-frequency detail is preserved while low frequencies stretch. LONG-CONTEXT TRAINING Continue training on longer sequences. Advertised context length is therefore a property of the TRAINING and POSITION SCHEME, not just a config number — and quality at the far end of a long window is often weaker than the headline suggests (see "lost in the middle," Section 5). 

CHARACTERS Tiny vocabulary, no unknown words — but sequences become very long (expensive attention) and each token carries little meaning. WORDS Short sequences — but a huge vocabulary, no way to represent unseen words, typos, or new names ("out-of-vocabulary"). SUBWORDS The compromise: common words are single tokens; rare words split into reusable pieces ("un" + "believ" + "ably"). Open vocabulary, moderate length. All modern LLMs use subword tokenization. 

BYTE-PAIR ENCODING (BPE) Start from individual symbols (characters or BYTES). Repeatedly find the most frequent adjacent pair in the training corpus and MERGE it into a new symbol. Stop at the target vocabulary size. Encoding applies the learned merges in order. BYTE-LEVEL BPE (GPT-2 onward) starts from the 256 bytes, so ANY text — any language, emoji, binary — can be encoded with no "unknown" token. Used by GPT-family models (via the tiktoken implementation), Llama 3 and others. WORDPIECE Like BPE, but chooses merges that maximize the likelihood of the training data rather than raw frequency. Continuation pieces are marked ("##ing"). Used by BERT-style models. UNIGRAM LANGUAGE MODEL Start with a LARGE candidate vocabulary and repeatedly PRUNE the pieces that hurt corpus likelihood the least. Encoding finds the most probable segmentation; it can also SAMPLE alternative segmentations (useful as regularization). Implemented in SentencePiece. SENTENCEPIECE A library/framework (supporting BPE and Unigram) that treats text as a raw character stream — no language-specific pre-splitting; spaces become a visible marker (▁). Used by Llama 2, T5, Gemma and many others. PRE-TOKENIZATION Most tokenizers first split text with a regular expression (separating words, numbers, punctuation, whitespace) so merges don't cross certain boundaries. The regex is part of the tokenizer's identity. 

// Learns merge rules from a word list — the heart of BPE. (Educational: real trainers work on bytes, at corpus scale, with careful data structures.) public static List<(string A, string B)> LearnBpeMerges(IEnumerable<string> words, int numMerges) { // Represent each word as space-separated symbols plus an end-of-word marker, with its frequency var vocab = words.GroupBy(w => w).ToDictionary( g => string.Join(" ", g.Key.Select(c => c.ToString())) + " </w>", g => g.Count()); var merges = new List<(string A, string B)>(); for (int step = 0; step < numMerges; step++) { // Count every adjacent symbol pair, weighted by word frequency var pairCounts = new Dictionary<(string, string), int>(); foreach (var (word, freq) in vocab) { string[] symbols = word.Split(' '); for (int i = 0; i < symbols.Length - 1; i++) { var pair = (symbols[i], symbols[i + 1]); pairCounts[pair] = pairCounts.GetValueOrDefault(pair) + freq; } } if (pairCounts.Count == 0) break; // Merge the most frequent pair everywhere it occurs var best = pairCounts.MaxBy(kv => kv.Value).Key; merges.Add(best); string pattern = $@"(?<!\S){Regex.Escape(best.Item1 + " " + best.Item2)}(?!\S)"; // whole-symbol matches only string replacement = best.Item1 + best.Item2; vocab = vocab.ToDictionary(kv => Regex.Replace(kv.Key, pattern, replacement), kv => kv.Value); } return merges; } // LearnBpeMerges(["low", "low", "lower", "newest", "newest", "widest"], 10) learns merges like ("e","s") -> "es" -> ("es","t") -> "est" -> ... // i.e. frequent character sequences become single vocabulary entries. 

// dotnet add package Microsoft.ML.Tokenizers using Microsoft.ML.Tokenizers; Tokenizer tokenizer = TiktokenTokenizer.CreateForModel("gpt-4o"); // BPE tokenizer matching an OpenAI model family IReadOnlyList<int> ids = tokenizer.EncodeToIds("Tokenization matters for cost."); int count = tokenizer.CountTokens("Tokenization matters for cost."); string roundTrip = tokenizer.Decode(ids); foreach (EncodedToken token in tokenizer.EncodeToTokens("unbelievably", out string? _)) Console.WriteLine($"{token.Id,8} '{token.Value}'"); // see exactly how a word is split into pieces 

COST AND LIMITS ARE IN TOKENS "1,000 words" is not a fixed price: ~750 tokens in English, noticeably more in many other languages and in code. THE MULTILINGUAL "TOKEN TAX" Tokenizers trained mostly on English split other languages (and scripts such as Devanagari, Thai, Arabic) into more, smaller pieces — so the same content costs more, fills the context faster, and can run slower. Measure token counts on YOUR languages before pricing or sizing. ARITHMETIC AND SPELLING The model sees tokens, not letters or digits as such: "strawberry" may be 2-3 tokens, so counting letters is genuinely hard; numbers split inconsistently ("12345" vs "1 2345"), hurting arithmetic. Do these tasks in code (tools). WHITESPACE AND CASE MATTER " hello" and "hello" and "Hello" are different tokens; leading spaces and trailing whitespace in prompts change behavior and token counts. CODE AND STRUCTURED TEXT Indentation, long identifiers, and JSON punctuation consume many tokens; minified or compact formats can save cost. SPECIAL TOKENS AND CHAT TEMPLATES Roles (system/user/assistant), end-of-turn, and tool markers are special tokens inserted by a chat template. Getting the template wrong for an open-weight model (a mismatched template) is a classic cause of poor output. Never let user text inject special-token strings. GLITCH TOKENS Rare or undertrained tokens can cause erratic behavior. TOKEN HEALING / BOUNDARIES Cutting a prompt mid-word (for completion) can place a token boundary the model rarely saw in training and degrade results. 

- Count tokens with the right tokenizer BEFORE sending (budgets, rate limits, cost estimates). - Test token efficiency for your languages and data formats; consider compact formats for large structured payloads. - Keep tokenizer, vocabulary, and chat template pinned with the model version (see AI Production / MLOps, Section 2). - Don't ask the model to do character-level or digit-level work it can't see — give it a tool. 

POSITION SCHEME The model is reliable only over position ranges seen in (long-context) training (Section 3). COMPUTE Attention cost grows quadratically with length; prefill for a 100k-token prompt is slow and expensive. MEMORY The KV cache grows linearly with length x concurrent sequences (Section 6) — often the binding constraint when serving many users. ATTENTION QUALITY More tokens means attention is spread thinner, and irrelevant text competes with relevant text. 

NEEDLE-IN-A-HAYSTACK TESTS Check whether a model can retrieve one planted fact from a long document. Many models do well — but a single easy lookup doesn't predict performance on tasks that need to combine many pieces, reason over the whole text, or ignore distractors. "LOST IN THE MIDDLE" Research has shown that information in the middle of a long prompt is often used less reliably than information near the beginning or end. Improved in newer models, but don't assume it's gone. CONTEXT ROT / DISTRACTION Quality on a fixed task can DEGRADE as irrelevant or conflicting text is added — even well within the window. INSTRUCTION DILUTION Instructions at the top can lose force after tens of thousands of tokens of content; restate critical rules near the end. SO: measure YOUR task at YOUR typical and maximum context lengths. The usable window is whatever length your evaluation shows holds quality. 

LONG CONTEXT ("just put it all in") + Simple; the model sees everything; good for tasks needing the whole document (summarize, review, cross-reference). - Cost and latency scale with length on EVERY request (mitigated by prompt caching); quality can degrade; no access control per passage; bounded by the window. RAG (retrieve what's relevant) + Cheap per request, scales to huge corpora, supports citations and per-user permissions and deletion. - Retrieval can miss; needs infrastructure and evaluation (see the RAG guide). HYBRID Retrieve broadly, then feed a gener...