PDF ↓ Text Extraction ↓ Chunking ↓ Embeddings ↓ Vector Database <- the INGESTION half: runs when content changes ───────────────────────────────── User Question ↓ Similarity Search ↓ Relevant Context <- the QUERY half: runs on every question ↓ LLM ↓ Answer 

┌─────────────────────────── INGESTION (offline, on content change) ───────────────────────────┐ │ │ │ Source documents -> Extract text -> Clean -> Chunk -> Embed -> Store in index │ │ (PDF, DOCX, HTML, (+ structure, (strip (focused (vectors) (vector + text │ │ wiki, tickets...) tables, OCR) boilerplate) passages) + metadata) │ │ │ └──────────────────────────────────────────────────┬────────────────────────────────────────────┘ │ the INDEX (vectors + chunks + metadata) ┌──────────────────────────────────────────────────┴────────────────────────────────────────────┐ │ │ │ Question -> (rewrite) -> Embed -> Retrieve (+ filters) -> Rerank -> Build prompt -> LLM -> │ │ (context injection) │ │ Validate (citations) -> Answer │ │ │ └─────────────────────────────── QUERY (online, per request) ───────────────────────────────────┘ 

vs. relying on the model's memory The model doesn't know your data, may be out of date, and hallucinates. vs. fine-tuning Fine-tuning shapes STYLE and BEHAVIOR; it is a poor, expensive way to teach FACTS, can't easily "unlearn" a deleted document, and gives no citations. RAG updates by re-indexing a document — no training. vs. pasting everything in the prompt Costly, slow, limited by the context window, and quality can DEGRADE as irrelevant text is added (see LLM Application Development, Section 6). 

FRESHNESS Update a document and the next answer reflects it. GROUNDING Answers are tied to source text, which sharply reduces hallucination. CITATIONS You can show WHERE an answer came from, and verify it. ACCESS CONTROL Retrieval can be filtered per user, so people only get answers from documents they're allowed to see (Section 8). COST CONTROL Send a few relevant passages instead of an entire corpus. 

Extraction UglyToad.PdfPig (text PDFs) · Azure AI Document Intelligence or OCR (scanned/complex PDFs) · DocumentFormat.OpenXml (DOCX) · HtmlAgilityPack (HTML) · Markdown parsers Tokenizing/chunking Microsoft.ML.Tokenizers + your chunker (or a library) Embeddings OpenAI / Azure OpenAI embedding models · open-source models via ONNX Vector store Azure AI Search · PostgreSQL + pgvector · Redis · MongoDB Atlas · dedicated vector DBs Orchestration Plain C# services · Microsoft.Extensions.AI · Semantic Kernel / Microsoft Agent Framework LLM OpenAI / Azure OpenAI chat models Hosting ASP.NET Core API · BackgroundService / queue workers / Azure Functions for ingestion Observability OpenTelemetry · structured logging of every stage 

A robust ingestion pipeline is: IDEMPOTENT Re-running it on the same document yields the same index state. INCREMENTAL Only reprocess documents that CHANGED (compare a content hash). OBSERVABLE Log what was ingested, how many chunks, which model, which failures. RESILIENT Retry transient failures; one bad PDF must not stop a batch. VERSIONED Records which embedding model / chunking version produced each chunk. REVERSIBLE Deleting or updating a document removes its OLD chunks. 

doc_id, chunk_id identity — so updates and deletes can target a document's chunks source, title, url for citations and for prefixing chunk text page / section "see page 14" citations; position for context expansion tenant_id, ACL groups who may see this (security trimming — Section 8) category, doc_type business filters language, created/updated date, version freshness and filtering embedding_model which model produced the vector (needed for migrations) content_hash change detection 

File shares / SharePoint / Blob Storage scheduled or event-driven ingestion (e.g. a blob-created event) Wikis, ticketing, knowledge bases connectors or APIs, with change feeds where available Databases turn rows into text documents with clear structure Web pages crawl, extract main content, strip navigation and boilerplate User uploads ingest on upload; keep the file private to that user/tenant 

// A hosted background worker pulling ingestion jobs from a queue public class IngestionWorker(IIngestionQueue queue, IngestionService ingestion, ILogger<IngestionWorker> logger) : BackgroundService { protected override async Task ExecuteAsync(CancellationToken stoppingToken) { await foreach (var job in queue.ReadAllAsync(stoppingToken)) { try { await ingestion.IngestPdfAsync(job.Path, job.DocId, job.TenantId, stoppingToken); } catch (Exception ex) { logger.LogError(ex, "Ingestion failed for {DocId}", job.DocId); /* dead-letter, don't crash the worker */ } } } } 

// dotnet add package PdfPig using UglyToad.PdfPig; using UglyToad.PdfPig.DocumentLayoutAnalysis.TextExtractor; static IEnumerable<(int Page, string Text)> ExtractPages(byte[] pdfBytes) { using var pdf = PdfDocument.Open(pdfBytes); foreach (var page in pdf.GetPages()) { // ContentOrderTextExtractor tries to follow reading order better than the raw page.Text string text = ContentOrderTextExtractor.GetText(page); if (!string.IsNullOrWhiteSpace(text)) yield return (page.Number, text); // keep the page number for citations } } 

TEXT-BASED PDF Contains real text. Libraries like PdfPig extract it directly. Fast and free. SCANNED PDF Pages are images of text. Text extraction returns NOTHING. Needs OCR (Azure AI Document Intelligence, Tesseract, or a cloud OCR service). MIXED / COMPLEX Tables, multi-column layouts, forms, figures. Plain extraction mangles them; a layout-aware service (e.g. Azure AI Document Intelligence's layout model) can return structure — headings, paragraphs, tables — often as Markdown. 

// Cheap sanity check: if a "document" extracts to almost no text, it is probably scanned bool probablyScanned = pages.Sum(p => p.Text.Length) < 100 * pdf.NumberOfPages; 

- REMOVE REPEATED HEADERS/FOOTERS and page numbers that appear on every page (they pollute chunks and embeddings with identical noise). - FIX HYPHENATION and broken line wraps ("informa-\ntion" -> "information"; single newlines inside a paragraph -> spaces). - NORMALIZE whitespace and strange Unicode. - PRESERVE STRUCTURE that carries meaning: headings, list items, and tables. Converting a table to a text row-per-line (or Markdown) beats a jumble of cell text; flattening a table into unlabeled numbers destroys its meaning. - HANDLE tables, figures, and images deliberately: extract tables as structured text, caption or describe important images (a vision model can generate descriptions), or flag them as not searchable. - REDACT or exclude content that must never reach the index (secrets, regulated personal data). 

DOCX DocumentFormat.OpenXml — preserves headings, lists, and tables directly HTML HtmlAgilityPack — strip navigation/scripts; keep the main content and heading hierarchy MD/TXT Read directly; Markdown headings give free structure for chunking (Section 4) Email Parse headers into metadata; strip quoted reply chains and signatures 

- Embedding models have INPUT LIMITS, and one vector for a whole document blurs every topic together. - Retrieval should return the PASSAGE that answers the question, not a 50-page document. - Prompts have a TOKEN BUDGET; you can only afford a few focused passages. 

TOO SMALL (a sentence or two) Precise matches, but each chunk lacks context ("it must be renewed annually" — what is "it"?), and answers get split across chunks. TOO LARGE (several pages) Contains the answer but also lots of noise; the embedding is a blurry average of many topics, so similarity is less precise; wastes prompt tokens. Typical starting point: a few hundred tokens per chunk (roughly 200-500) with 10-20% overlap — then TUNE on your own data using retrieval metrics (Section 12). There is no universally best size. 

FIXED-SIZE Every N tokens, with overlap. Simple and predictable; can cut mid-sentence or mid-idea. RECURSIVE / Split on the biggest natural boundary that fits: sections -> paragraphs -> sentences -> BOUNDARY-AWARE words. Respects structure. The usual sensible default. STRUCTURE-AWARE Use the document's own structure: split on headings, keep a table or code block whole, keep list items together. Best when the source has reliable structure (Markdown, HTML, layout-aware PDF output). SEMANTIC Split where the TOPIC changes (e.g. detect large drops in similarity between adjacent sentences). Can produce coherent chunks; costs extra embedding work and is less predictable. PARENT-CHILD Embed small chunks for precise retrieval but return their larger parent section to the (small-to-big) model for context (Section 13). 

using Microsoft.ML.Tokenizers; public static class Chunker { public static IEnumerable<string> Chunk(string text, Tokenizer tokenizer, int maxTokens = 400, int overlapTokens = 50) { var paragraphs = text.Split(["\r\n\r\n", "\n\n"], StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries); var current = new List<string>(); int currentTokens = 0; foreach (var paragraph in paragraphs) { int tokens = tokenizer.CountTokens(paragraph); if (currentTokens + tokens > maxTokens && current.Count > 0) { yield return string.Join("\n\n", current); // Start the next chunk with trailing paragraphs from this one (the overlap) var overlap = new List<string>(); int overlapCount = 0; for (int i = current.Count - 1; i >= 0; i--) { int t = tokenizer.CountTokens(current[i]); if (overlapCount + t > overlapTokens) break; overlap.Insert(0, current[i]); overlapCount += t; } current = overlap; currentTokens = overlapCount; } current.Add(paragraph); currentTokens += tokens; } if (current.Count > 0) yield return string.Join("\n\n", current); } } 

- PREFIX each chunk with its context: document title and section heading ("Employee Handbook > Leave Policy > Parental Leave"). This helps both the embedding and the model understand what the chunk is about. - Store provenance: doc id, page, position, and URL for citations. - Don't split tables, code blocks, or numbered procedures across chunks. - CONTEXTUAL ENRICHMENT (advanced): use an LLM to generate a one- or two-sentence description situating each chunk in its document, and prepend it before embedding (Section 13). 

using OpenAI.Embeddings; EmbeddingClient embeddings = new("text-embedding-3-small", apiKey); // Batch many chunks per request — far fewer round trips than one call per chunk var texts = chunks.Select(c => c.Text).ToList(); OpenAIEmbeddingCollection result = (await embeddings.GenerateEmbeddingsAsync(texts, cancellationToken: ct)).Value; for (int i = 0; i < texts.Count; i++) chunks[i].Vector = result[i].ToFloats(); 

SAME MODEL for chunks and queries. Always. Different models live in different vector spaces. RECORD THE MODEL on every chunk. Changing models later requires re-embedding the whole corpus; the stored model name tells you what's stale. BATCH, BUT BOUND THE BATCH. Providers cap inputs and total tokens per request. Batch in groups (for example a few dozen chunks) and respect rate limits, with retries and backoff (see AI APIs with .NET, Sections 10-11). EMBED THE ENRICHED TEXT. What you embed should include the title/section prefix, not just the bare paragraph — but store the display text separately if you don't want the prefix shown to users. COST IS MODEST, BUT NOT ZERO. Embedding is far cheaper than generation, yet a big corpus re-embedded repeatedly adds up — which is why incremental ingestion (content hashing) matters. QUERIES ARE EMBEDDED AT RUNTIME. Every question costs one small embedding call on the hot path; cache embeddings for repeated queries if volume warrants it. 

HYBRID CAPABILITY RAG queries mix concepts and exact terms ("error 40412 when resetting my password"), so a store that supports vector + keyword retrieval (Azure AI Search natively; PostgreSQL with full-text search) is a major advantage. FILTERING You will filter by tenant/permissions and business metadata on nearly every query — pick a store whose filter-with-vector behavior you've tested (see the pgvector and pre-filtering notes in the vector search guide). SCALE AND OPS Start with the database you already operate; add a dedicated vector DB only when scale or features demand it. REBUILDABILITY The index is DERIVED data. Keep source documents authoritative so you can rebuild it (new model, new chunking) at any time. 

public record ChunkRecord( string Id, string DocId, string Text, ReadOnlyMemory<float> Vector, int TenantId, string Source, int Page, string EmbeddingModel); public record SearchFilter(int TenantId, string? Category = null); public record ScoredChunk(ChunkRecord Chunk, double Score); public interface IChunkStore { Task UpsertAsync(IEnumerable<ChunkRecord> records, CancellationToken ct); Task DeleteByDocAsync(string docId, CancellationToken ct); Task<IReadOnlyList<ScoredChunk>> SearchAsync( string queryText, ReadOnlyMemory<float> queryVector, SearchFilter filter, int top, CancellationToken ct); } 

UPDATE a document: ingest the new version, then remove the old version's chunks. Doing "delete then insert" leaves a window where the document is missing from search; giving chunks version-stamped IDs and deleting superseded versions AFTER the new ones land avoids the gap. DELETE a document: remove ALL its chunks. A deleted document that still answers questions is a compliance problem, not just a bug — and it's a major argument for RAG over fine-tuning. REINDEX for a new embedding model or chunking strategy: build a NEW index alongside the old one, validate it with your evaluation set, then switch traffic (an index alias or config flag makes the cutover instant and reversible). 

1. EMBED the user's question (same model as the chunks). 2. SEARCH the index for the nearest chunks — vector search, ideally combined with keyword search (hybrid). 3. FILTER by tenant, permissions, and business metadata (Section 8). 4. RETRIEVE a generous candidate set (e.g. top 20-50) — then narrow it by reranking (Section 9). 

float[] queryVector = (await embeddings.GenerateEmbeddingAsync(question, cancellationToken: ct)).Value.ToFloats().ToArray(); IReadOnlyList<ScoredChunk> candidates = await store.SearchAsync( question, queryVector, new SearchFilter(TenantId: tenantId), top: 30, ct); 

If the right passage is NOT retrieved, the model cannot use it — and will either say "I don't know" (best case) or improvise from its own memory (worst case). Most "RAG gives wrong answers" bugs are retrieval bugs, not generation bugs. That is why Section 12 measures retrieval separately. 

HYBRID SEARCH Fuse vector and keyword (BM25) results with Reciprocal Rank Fusion. Rescues exact- term queries that embeddings handle poorly (IDs, codes, names). BETTER CHUNKS Right size, structure-aware, with title/section context prefixes. QUERY REWRITING Turn conversational or vague questions into good search queries (Section 13). METADATA FILTERS Narrow the search space to the relevant tenant, document type, or time range. RERANKING Re-score a wide candidate set with a stronger model (Section 9). BETTER EMBEDDINGS A stronger or domain-suited embedding model, evaluated on your own questions. 

Apply a SCORE THRESHOLD (tuned on real queries) so that when nothing relevant is retrieved the system declines instead of feeding the model irrelevant passages and receiving a confident-sounding, ungrounded answer. Score scales differ between models, metrics, and stores — calibrate per system. 

Examples: tenant_id = 42 (multi-tenant isolation) department IN ('HR', 'Legal') (scope by area) doc_type = 'policy' AND status = 'current' (exclude drafts and superseded versions) updated_at >= '2025-01-01' (freshness) language = 'en' product = 'X200' AND version >= 3 

In an enterprise RAG system, permissions are NOT optional. If the index contains documents with different audiences, retrieval MUST return only chunks the CURRENT USER may read. Otherwise the model becomes a very helpful way to leak confidential documents to people who shouldn't see them. Rules: 1. Store the ACL (user IDs / group IDs / tenant) on every chunk at ingestion time. 2. At query time, DERIVE the filter from the AUTHENTICATED identity on the server — NEVER from a request parameter the client can edit, and NEVER from the LLM. 3. Apply the filter INSIDE the search query (pre-filter), not by post-filtering results afterward — post-filtering can leave you with fewer than k results, or none, and a bug there can expose data. 4. Keep ACLs in sync: when a document's permissions change, update its chunks' ACLs promptly. 5. TEST it: automated tests where user A must never retrieve user B's documents. 

// The filter comes from the authenticated principal — not from the request body, and not from the model var filter = new SearchFilter( TenantId: User.GetTenantId(), Category: request.Category); // a business filter the user MAY choose, validated against an allow-list var candidates = await store.SearchAsync(question, queryVector, filter, top: 30, ct); 

EXPLICIT FILTERS The UI offers product/date/category dropdowns; the app passes them as filters. Predictable and safe. SELF-QUERY An LLM reads "what changed in the 2024 travel policy?" and extracts structured filters (year = 2024, topic = travel policy) plus a clean search string. Powerful for natural-language interfaces — but VALIDATE its output against an allow-list of fields and values, since it can be wrong or manipulated. Use it for BUSINESS filters only; security filters always come from identity. 

- Filter fields must be declared/indexed in the store (e.g. "filterable" in Azure AI Search, a "filter"-typed field in MongoDB Atlas, indexed columns in PostgreSQL). - Highly selective filters can interact badly with approximate (ANN) indexes in some stores — test that filtered queries still return a full result set (see pgvector notes in the vector search guide). - Metadata you didn't capture at ingestion can't be filtered on later without reprocessing. 

Stage 1 — RETRIEVE Hybrid vector + keyword search -> top 20-50 candidates (fast, high recall) Stage 2 — RERANK Cross-encoder / semantic ranker / LLM -> reorder candidates (slower, high precision) Stage 3 — SELECT Keep the best 3-8 passages for the prompt -> small, focused context 

CROSS-ENCODER MODELS Models that score (query, passage) pairs jointly. Hosted rerank APIs exist, and open-source rerankers can be run via ONNX. SEMANTIC RANKER Azure AI Search offers a built-in semantic reranking stage that rescores the top results — often the simplest option on Azure. LLM-AS-RERANKER Ask a chat model to rate each passage's relevance (or to pick the best ones). Flexible, no extra infrastructure — but slower and costlier; use a small model and a short candidate list. DIVERSITY (MMR) "Maximal marginal relevance" reorders to balance relevance against redundancy, so you don't spend your context budget on five near-identical chunks. 

public interface IReranker { Task<IReadOnlyList<ScoredChunk>> RerankAsync( string question, IReadOnlyList<ScoredChunk> candidates, int keep, CancellationToken ct); } 

FIXES Cases where the right chunk was in the candidate set but ranked 14th instead of 2nd. DOESN'T Cases where the right chunk was never retrieved at all. Reranking can only reorder what retrieval brought back, so keep the candidate set generous (top 30-50), and fix recall first. COSTS Added latency (an extra model call over N passages) and money. Measure the quality gain against both before committing. 

const string SystemPrompt = """  You are a documentation assistant for ACME. Answer the question using ONLY the sources in <context>. Cite every claim with its source number in square brackets, like [1] or [2][3]. If the sources do not contain the Source：https://dev.to/rhuturaj_takle/rag-retrieval-augmented-generation-2816
My favorites