How RAG Actually Works: Chunking, Embeddings, Retrieval, and Reranking Under the Hood

RAG feels simple on paper. Split documents into chunks, embed them, fetch the closest ones at query time. Done.

That description hides half a dozen hard sub-problems. Split too coarsely and you retrieve paragraphs that bury the relevant sentence in noise. Split too finely and no single chunk carries enough context to be useful. Use only dense retrieval and you miss documents where an exact product code or rare proper noun is the critical signal. Use only keyword search and you miss paraphrases. Retrieve top-k blindly and the final context window fills with redundant or marginally relevant chunks.

Most tutorials stop at "embed, store, retrieve." This article goes deeper — what actually happens at each stage, where the trade-offs live, and what production systems do differently.

The Pipeline at a Glance

INGESTION (offline, whenever documents change) sources → parse/clean → deduplicate → chunk → embed (dense) + index terms (BM25) → store vectors, text, metadata QUERY (online, per request) question → (optional) rewrite/expand → hybrid retrieval (BM25 + dense, top 50–200) → rerank with cross-encoder (keep top 5–10) → select for relevance/diversity within token budget → build prompt with numbered sources → generate with citation instructions → check citations → return answer with links

Two pipelines meet at the index. The offline path runs when documents change. The online path runs per request. Keeping them aligned is where most RAG systems break.

1. Chunking: The Hardest Part Nobody Talks About

Every embedding model has a maximum token length — 512 tokens for older bi-encoders, 4,096–8,192 for modern ones (text-embedding-3-large, nomic-embed-text-v1.5). But even if your model supports 8k tokens, shoving an entire chapter into a single vector is a mistake: the embedding averages the semantics of every sentence, so a query about one narrow fact scores poorly against a dense multi-topic chunk that happens to contain that fact.

Chunking controls a fundamental quality–recall trade-off:

Too large chunks — high recall (the fact is somewhere in there), but low precision (the LLM gets noise and may hallucinate or ignore the signal)
Too small chunks — high precision, but context loss: "It increased by 12%" is meaningless without its referent

Getting chunking wrong is probably the single most common cause of poor RAG performance in production.

Fixed-Length Chunking with Overlap

The simplest approach: split every chunk_size tokens, with an overlap of tokens shared between adjacent chunks so sentence boundaries don't cut off context.

def fixed_chunk(text: str, chunk_size: int = 512, overlap: int = 64, tokenizer=None) -> Iterator[str]: if tokenizer is None: tokens = text.split() decode = lambda ids: " ".join(ids) else: tokens = tokenizer.encode(text) decode = tokenizer.decode step = chunk_size - overlap if step <= 0: raise ValueError("chunk_size must be strictly greater than overlap") start = 0 while start < len(tokens): end = min(start + chunk_size, len(tokens)) yield decode(tokens[start:end]) if end == len(tokens): break start += step

Typical production values: chunk_size=256–512 tokens with overlap=32–64. The overlap parameter is crucial — without it, a sentence split across two chunks loses its grammatical antecedent in both halves.

Semantic Chunking

Fixed-length chunking is agnostic to content structure. Semantic chunking exploits embedding similarity to find natural topic breaks:

Split text into sentences
Embed each sentence
Compute cosine similarity between consecutive sentence embeddings
Mark a chunk boundary wherever similarity drops below a threshold (or is a local minimum)

def semantic_chunk(sentences: list[str], embed_fn, threshold: float = 0.75, min_sentences: int = 3) -> list[str]: embeddings = embed_fn(sentences) # shape (N, D)  sims = [cosine_similarity(embeddings[i], embeddings[i+1]) for i in range(len(sentences)-1)] chunks = [] current = [] for i, sent in enumerate(sentences): current.append(sent) if i < len(sims): is_boundary = sims[i] < threshold and len(current) >= min_sentences if is_boundary: chunks.append(" ".join(current)) current = [] if current: chunks.append(" ".join(current)) return chunks

Semantic chunking produces more coherent chunks but is slower (requires an embedding call) and more sensitive to the threshold. A common tuning strategy: run the chunker on a held-out set, visualize the similarity curve, and set the threshold at the 15th percentile of observed similarities.

Structure-Aware Chunking

Real documents have structure: headings, paragraphs, bullet lists, code blocks. Respecting that structure almost always outperforms purely statistical methods.

# The production equivalent: LangChain's MarkdownHeaderTextSplitter from langchain_text_splitters import MarkdownHeaderTextSplitter md_splitter = MarkdownHeaderTextSplitter( headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")] ) chunks = md_splitter.split_text(markdown_document) # Each chunk carries heading metadata for downstream filtering

The libraries you actually ship. Nobody writes chunkers from scratch twice — you write them once to understand the trade-off, then use maintained splitters. The two dominant ones are LangChain (langchain_text_splitters) and LlamaIndex (llama_index.core.node_parser), and both express exactly the strategies above.

Late Chunking (The Context-Preserving Trick)

Late chunking, introduced by Günther et al. (2024) and popularized with jina-embeddings-v2, inverts the usual order:

Pass the entire document through the transformer encoder to produce contextualized token embeddings
Mean-pool over the token spans corresponding to each logical chunk

The attention mechanism lets every token see its full document context before the pooling boundary is applied. "It increased by 12%" gets an embedding that encodes what "it" refers to, because self-attention already saw the preceding paragraph.

# late_chunking.py — illustrative from transformers import AutoTokenizer, AutoModel import torch def late_chunk_document(document: str, chunk_boundaries: list[tuple[int, int]], model_name: str = "jinaai/jina-embeddings-v2-base-en") -> list[LateChunk]: tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) model = AutoModel.from_pretrained(model_name, trust_remote_code=True).eval() encoding = tokenizer(document, return_tensors="pt", return_offsets_mapping=True, truncation=True, max_length=8192) offset_mapping = encoding.pop("offset_mapping")[0] input_ids = encoding["input_ids"] attention_mask = encoding["attention_mask"] with torch.no_grad(): token_embeddings = model(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state[0] chunks = [] for start_char, end_char in chunk_boundaries: token_mask = (offset_mapping[:, 0] >= start_char) & (offset_mapping[:, 1] <= end_char) & (offset_mapping[:, 1] > offset_mapping[:, 0]) span_embeddings = token_embeddings[token_mask] if span_embeddings.shape[0] == 0: continue chunk_embedding = span_embeddings.mean(dim=0) chunk_embedding = chunk_embedding / (chunk_embedding.norm() + 1e-9) chunks.append(LateChunk(text=document[start_char:end_char], embedding=chunk_embedding, start_char=start_char, end_char=end_char)) return chunks

Constraint: the full document must fit within the model's context window (~8k tokens). For longer documents, apply late chunking within sliding windows.

Late chunking has a generative cousin: contextual retrieval (Anthropic, 2024) asks an LLM to write a one-sentence situating preamble for each chunk and prepends it before embedding. Late chunking is cheap (one encoder pass, no LLM) but needs a long-context encoder; contextual retrieval works with any encoder but costs one LLM call per chunk at index time. They compose.

2. Hybrid Search: BM25 + Dense Retrieval

Dense retrieval excels at semantic similarity but struggles with exact-match recall. If a user asks about "CVE-2023-44487" or a specific product SKU, the query embedding may not land near the document containing that exact string — because the model generalizes. BM25 handles these cases natively.

BM25 score for query q against document d:

BM25(q,d) = Σ IDF(t) * (f(t,d) * (k1+1)) / (f(t,d) + k1 * (1 - b + b * |d|/avgdl))

where f(t,d) is term frequency, |d| is document length, avgdl is average document length, k1≈1.2–2.0, b≈0.75.

Use a real BM25 engine. A naive Python implementation rescores the entire corpus on every query — O(N⋅|q|) — fine for teaching, hopeless beyond a few thousand chunks. Real lexical retrieval uses an inverted index: for each query term, walk only the posting list of documents that actually contain it.

Open-source ladder:

bm25s — sparse-matrix BM25 in Python; right default for single-process index up to millions of short chunks
Pyserini — Python bindings over Lucene; research-grade path
Elasticsearch / OpenSearch — distributed inverted index when corpus outgrows one machine; both ship native hybrid retrieval and built-in RRF
Most vector databases now do the sparse leg too: Qdrant, Weaviate, Milvus all accept sparse vectors alongside dense and expose server-side RRF fusion

Reciprocal Rank Fusion (RRF)

Given a BM25 ranking and a dense ranking, how to combine them? Cormack et al. (2009) introduced RRF:

RRF(d) = Σ 1 / (k + rank_r(d))

where k = 60 (smoothing constant). Documents not in a ranking are omitted from that term. Highest combined RRF score wins.

RRF does not require score normalization — it only uses rank ordinals, making it immune to the scale mismatch between BM25 scores (0–20) and cosine similarities (0.5–1.0).

def reciprocal_rank_fusion(rankings: list[list[tuple[int, float]]], k: int = 60) -> list[tuple[int, float]]: rrf_scores = defaultdict(float) for ranked_list in rankings: for rank, (doc_idx, _score) in enumerate(ranked_list, start=1): rrf_scores[doc_idx] += 1.0 / (k + rank) return sorted(rrf_scores.items(), key=lambda x: -x[1])

Worked example: BM25 ranks [D2, D1, D3, D4, D5], dense ranks [D1, D3, D2, D5, D4]. With k=60:

D1: 1/62 + 1/61 = 0.0325 → rank 1 (wins despite BM25 rank 2)
D2: 1/61 + 1/63 = 0.0323 → rank 2
D3: 1/63 + 1/62 = 0.0320 → rank 3

D1 wins in fused ranking because its dense rank of 1 contributes a large term. The k=60 constant prevents any single top-1 ranking from dominating completely.

Learned Sparse Retrieval (SPLADE)

BM25 and dense retrieval fail in opposite directions. Learned sparse retrieval keeps the representation sparse (a vector over the tokenizer's vocabulary, lives in an inverted index) but learns the weights with a transformer instead of counting term frequencies.

SPLADE (Formal et al., SIGIR 2021) runs text through a masked-language-model head, producing a logit for every vocabulary term at every position. Then saturate and max-pool over positions:

w_j = max_i log(1 + ReLU(w_ij))

The log(1+·) is a saturation function borrowed from BM25's term-frequency damping. Sparsity is trained in with a FLOPS regularizer — typical passage keeps ~100 non-zero terms.

Payoff: term expansion. A passage about "myocardial infarction" gets non-zero weight on heart and attack even though those strings never appear, so a lexical index can now match a paraphrase.

# sentence-transformers v5+ added SparseEncoder from sentence_transformers import SparseEncoder model = SparseEncoder("naver/splade-v3") doc_emb = model.encode_document(corpus) # sparse tensors, |V|-dimensional query_emb = model.encode_query("http2 rapid reset attack") scores = model.similarity(query_emb, doc_emb) # sparse dot product  # Inspect *why* a document scored print(model.decode(doc_emb[0], top_k=8)) # e.g. [('http', 2.1), ('reset', 1.9), ...]

SPLADE is a first-stage retriever — drop-in replacement for (or third leg alongside) BM25. Fuse its ranking with dense ranking using RRF, then rerank with cross-encoder. Costs are real: encoding requires a transformer forward pass at both index and query time, and expansion terms lengthen posting lists. The honest 2026 summary: BM25 + strong dense encoder remains the pragmatic default; learned sparse is the upgrade when exact-match recall matters and queries are paraphrase-heavy, or when you need interpretability without giving up semantic matching.

3. Cross-Encoder Rerankers: The Precision Stage

The retrieval stage (BM25 or dense) must be fast — processing millions of documents in tens of milliseconds. Speed requires a bi-encoder architecture: query and document embedded independently, similarity = cheap dot product. Downside: query and document tokens never "see" each other during encoding, so nuanced relevance signals (multi-hop, contrastive relevance) can be missed.

A cross-encoder receives the concatenation [query; document] as a single sequence and produces a scalar relevance score. Every query token attends to every document token, enabling fine-grained relevance reasoning at the cost of O(N) forward passes (one per candidate). This makes cross-encoders unsuitable for first-stage retrieval but ideal for reranking a small shortlist of 20–100 candidates.

There's a well-defined middle: late interaction (ColBERT) keeps one vector per token rather than one per document, precomputes all document token vectors offline like a bi-encoder, and scores with a MaxSim operator recovering much of the cross-encoder's token-level matching. Costs ~10–100x more index storage in exchange for cross-encoder-like quality at first-stage speed.

Two-Stage Pipeline

Query → Hybrid Retrieval (BM25 + dense) → top-100 candidates fast, cheap recall → Cross-Encoder Reranker → scores all 100 [query; passage] pairs slower, expensive, precise scoring → top-5 chunks → LLM Context Window

Only the 5 winning chunks reach the LLM context window, keeping prompt costs low while the cross-encoder's joint query-passage attention ensures relevance quality.

from sentence_transformers import CrossEncoder import numpy as np def load_reranker(model_name: str = "cross-encoder/ms-marco-MiniLM-L-6-v2") -> CrossEncoder: return CrossEncoder(model_name, max_length=512) def rerank(query: str, candidates: list[str], model: CrossEncoder, top_k: int = 5, batch_size: int = 32) -> list[tuple[str, float]]: pairs = [(query, passage) for passage in candidates] scores = model.predict(pairs, batch_size=batch_size) ranked = sorted(zip(candidates, scores), key=lambda x: -x[1]) return ranked[:top_k] # Latency numbers (A100): # cross-encoder/ms-marco-MiniLM-L-6-v2, 100 candidates, avg 200 tokens: ~60ms # On CPU (Intel Xeon): ~400ms for same 100 candidates

Practitioner tip: Two APIs, one job. Classical BERT-style rerankers (ms-marco-*, bge-reranker-*, jina-reranker-*) use CrossEncoder. The 2025–2026 generation of LLM-initialized rerankers — Qwen3-Reranker, mxbai-rerank-v2 — read relevance from a causal LM as the logit of a "yes"/"no" token given a prompt with instruction, query, and document. They buy prompt conditioning ("rank by whether the passage contains a numeric answer") and long context, and cost a full decoder forward pass per pair. Load with AutoModelForCausalLM per model card, or use a wrapper: rerankers (AnswerDotAI) gives one Reranker(...).rank(query, docs) call over cross-encoders, ColBERT, LLM rerankers, and hosted APIs alike. In production, serve the reranker off the request path — HuggingFace Text Embeddings Inference hosts reranker models behind a /rerank endpoint with continuous batching.

Training Your Own Reranker

If you have domain-specific relevance labels (click logs, human annotations, distillation from a more powerful model), fine-tune a cross-encoder with binary cross-entropy on (query, positive, negative) triples, or listwise ranking loss (LambdaLoss, ListNet).

from datasets import Dataset from sentence_transformers.cross_encoder import CrossEncoder, CrossEncoderTrainer, CrossEncoderTrainingArguments from sentence_transformers.cross_encoder.losses import BinaryCrossEntropyLoss train_dataset = Dataset.from_dict({ "query": ["What is RRF?", "What is RRF?"], "passage": [ "Reciprocal rank fusion combines ranked lists using 1/(k+rank).", "The capital of France is Paris.", ], "label": [1.0, 0.0], # 1 = relevant, 0 = hard negative }) model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2", num_labels=1) loss = BinaryCrossEntropyLoss(model) args = CrossEncoderTrainingArguments( output_dir="./my-domain-reranker", num_train_epochs=1, # 1 epoch usually enough for domain adaptation  per_device_train_batch_size=16, learning_rate=2e-5, warmup_ratio=0.1, fp16=True, ) trainer = CrossEncoderTrainer(model=model, args=args, train_dataset=train_dataset, loss=loss) trainer.train() model.save_pretrained("./my-domain-reranker/final")

The data matters more than the loss. Pointwise BCE on random negatives teaches almost nothing — random negatives are trivially separable. Mine hard negatives: run your existing first-stage retriever, take documents ranked 10–100 that are not labelled relevant, and use those as zeros — they are exactly the confusions the reranker exists to fix. Usual ratio: 4–8 hard negatives per positive. With graded relevance or teacher scores, prefer listwise loss (LambdaLoss, ListNet) over BCE: it optimizes the ordering of a candidate list, which is what nDCG actually measures. Distilling a large reranker's scores into a MiniLM-sized student with MarginMSELoss is the standard way to get most quality at a fraction of latency.

4. Query Rewriting and HyDE

The query the user types is rarely optimal for retrieval — short, colloquial, or implicit.

Multi-Query Expansion

Generate several paraphrases of the original query, retrieve for each, and union the result sets (deduplicating by document ID). Dramatically improves recall for queries that can be stated multiple ways.

REWRITE_PROMPT = """You are a retrieval expert. Given a user question, produce {n} alternative phrasings that together cover the semantic space of the question. Return a JSON list of strings. User question: {question} Alternative phrasings (JSON list):""" def expand_query(question: str, llm_call: callable, n_variants: int = 3) -> list[str]: prompt = REWRITE_PROMPT.format(question=question, n=n_variants) response = llm_call(prompt) # extract JSON list from LLM output...  return [question] + variants[:n_variants] # include original

HyDE — Hypothetical Document Embeddings

Instead of embedding the (short, vague) query, ask the LLM to generate a hypothetical document that would answer the query, then embed that document and retrieve using its embedding.

The intuition: a hypothetical answer document and a real answer document inhabit closer regions of embedding space than the query and the answer document, because they share vocabulary, entity mentions, and syntactic structure.

HYDE_PROMPT = """Write a short, factual paragraph (2-4 sentences) that directly answers the following question. Do not say "I don't know" — write the best answer you can, even if uncertain. Question: {question} Answer:""" def hyde_embed(question: str, llm_call: callable, embed_fn: callable, n_hypotheses: int = 1) -> np.ndarray: hypotheses = [] prompt = HYDE_PROMPT.format(question=question) for _ in range(n_hypotheses): hypotheses.append(llm_call(prompt).strip()) embeddings = embed_fn(hypotheses) mean_embedding = embeddings.mean(axis=0) return mean_embedding / (np.linalg.norm(mean_embedding) + 1e-9)

HyDE hallucination risk: The LLM's hypothetical document will contain plausible-sounding but possibly incorrect facts. This is fine for retrieval (you only use the embedding, not the text), but be careful not to accidentally include the hypothesis in the LLM context. Always retrieve from the real corpus using the HyDE embedding, then pass the retrieved real documents to the generator.

Use HyDE when: corpus uses formal/technical language differing from query style; queries are short and ambiguous; you can afford 1–2 extra LLM calls per query (~200ms latency budget). Avoid when: queries are already long and specific; LLM hallucinations could mislead retrieval; low-latency SLA (<50ms) prevents extra LLM calls.

5. Metadata Filtering and Parent-Child Retrieval

Purely semantic retrieval treats all documents as equally eligible. In practice, queries have hard constraints: "only Q4 2024", "only 'legal' category", "only product model XYZ-500". Metadata filtering applies these constraints before or during ANN search, drastically reducing the candidate pool and improving precision.