Every chatbot product says the same sentence: "trained on your own content." It sounds like fine-tuning. It almost never is.

What nearly all of them do is retrieval-augmented generation — the model is never trained on anything. Your content is cut into pieces, each piece is turned into a vector, and at question time the pieces that look closest to the question get pasted into the prompt. The model then answers from text it was handed half a second ago.

That distinction matters, because it tells you where the bugs live. When the answer is wrong, the model is usually innocent. The retrieval step handed it the wrong paragraph.

Here is what each step actually does, and the knobs that change the outcome.

Step 1: getting the text out

Before anything clever happens, something has to read your pages. This is the least glamorous step and the one that quietly ruins the most deployments.

A few things that bite:

  • Navigation and footers. Keep them and every chunk on the site contains your menu. Every chunk then looks slightly similar to every other chunk, and similarity search gets measurably worse. Strip to the main content region.
  • JavaScript-rendered pages. A plain fetch of a React site returns an empty <div id="root"></div>. You need a headless browser, or the site's API, or its sitemap of pre-rendered pages.
  • PDFs with two columns. Naive extraction reads straight across, so you get the first line of column one followed by the first line of column two. The text is technically present and completely meaningless.
  • Tables. A price table flattened to one line of numbers loses which number belonged to which row. Converting tables to Markdown before chunking is worth the effort, because a model reads a Markdown table perfectly well.

A sanity check that costs nothing: print the extracted text for five pages and read it yourself. If you cannot answer a question from it, no model can either.

Step 2: chunking

You cannot put the whole site in the prompt. Not mainly because of the context limit — modern models have room — but because accuracy drops when the relevant sentence is buried in two hundred pages of noise, and because you would pay for those tokens on every single question.

So the text gets split, and the split size is a real trade-off:

  • Too small (a sentence or two) and the chunk loses its subject. "It costs 1,499 per month" is useless without the sentence before it naming the plan.
  • Too large (a whole page) and one chunk covers five topics. It matches many questions weakly and few questions strongly, so it crowds out better matches.

In practice something in the 400–800 token range works for prose, with a 10–15% overlap between consecutive chunks so a fact that straddles a boundary survives in at least one piece.

Better than tuning the number is splitting on structure. Headings exist because the author already decided where topics begin:

// Split on headings first; fall back to size-based splitting // only when a section is genuinely too long. function chunkByHeading(markdown, maxTokens = 700) { const sections = markdown.split(/\n(?=#{1,3}\s)/); const out = []; for (const section of sections) { if (estimateTokens(section) <= maxTokens) { out.push(section); continue; } // Long section: accumulate paragraphs up to the limit. let buffer = ''; for (const para of section.split(/\n{2,}/)) { if (buffer && estimateTokens(buffer + para) > maxTokens) { out.push(buffer.trim()); buffer = ''; } buffer += para + '\n\n'; } if (buffer.trim()) out.push(buffer.trim()); } return out; }

One cheap trick that pays for itself: prefix each chunk with its page title and heading path before embedding it. A chunk that begins Pricing > Starter plan — matches a question about the starter plan far better than a bare paragraph does, because the words the user typed are literally in the text.

Step 3: embeddings

An embedding model turns a piece of text into a list of numbers — 1536 of them for OpenAI's text-embedding-3-small, 3072 for the large variant, 768 or 1024 for most open models. Text with similar meaning lands near other similar text in that space.

Three things worth knowing.

The same model must be used on both sides. Chunks embedded with model A and questions embedded with model B produce coordinates in two unrelated spaces. The similarity scores will look plausible and mean nothing. If you upgrade the embedding model you re-embed everything, so store the model name next to every vector and you will know what needs rebuilding.

Embeddings are cheap; generation is not. Embedding a few hundred pages costs cents. The recurring cost of a chatbot is answer generation, not indexing. This is why re-indexing a whole site nightly is perfectly affordable, and why "the bot still quotes last month's price" is a choice rather than a constraint.

Similarity is not relevance. Two chunks about two different plans sit very close together in embedding space, because they share almost all their vocabulary. Embeddings are good at topic and bad at distinguishing near-identical things. Which leads straight to the next step.

Step 4: retrieval, and why pure vector search disappoints

Ask a vector index "what is the SKU for the 40mm variant" and it will happily return the chunk about the 50mm variant. The two sentences are nearly identical in meaning-space, and the one number that matters contributes almost nothing to the vector.

The standard fix is hybrid search: run a keyword search (BM25, Postgres full-text, SQLite FTS5 — whatever you already have) alongside the vector search and combine the rankings. Keyword search is precisely the tool that cares about the literal token 40mm. Reciprocal rank fusion is the usual combiner and is about six lines:

// Combine two ranked lists of ids. k = 60 comes from the original RRF paper // and is a sane default: it flattens the influence of deep ranks. function fuse(vectorHits, keywordHits, k = 60) { const scores = new Map(); for (const list of [vectorHits, keywordHits]) { list.forEach((id, i) => { scores.set(id, (scores.get(id) ?? 0) + 1 / (k + i + 1)); }); } return [...scores.entries()] .sort((a, b) => b[1] - a[1]) .map(([id]) => id); }

Two more retrieval details that matter more than people expect.

Retrieve more than you keep. Pull twenty candidates, re-rank them, keep the best four to six. A cross-encoder re-ranker reads the question and the chunk together instead of comparing two independent vectors, so it is much better at exactly the near-duplicate problem above.

Have a floor. If the best match scores below a threshold, the honest answer is "I don't have that in my content." Without a floor your bot is always handed something, and a model handed irrelevant text will dutifully build an answer out of it. Most hallucinations I have chased down were really retrieval failures that nobody blocked.

Step 5: the prompt

By now this part is almost boring, which is the goal. The retrieved chunks go into the prompt with instructions that do three jobs:

  1. Scope it. Answer from the provided content only, and say so when the content does not cover the question.
  2. Make it citable. Number the chunks and ask for the number alongside each claim. You can then show a source link in the UI, and — more usefully — grade answers later, because an answer whose cited chunk does not contain the claim is a detectable failure.
  3. Fix the register. Length, language, and what to do at the end of an answer. This is where "reply in the language the customer wrote in" lives.

What does not belong in the prompt is your knowledge. Every fact hardcoded into a system prompt is a fact that will be wrong after the next price change and that nobody will remember to update.

How to tell whether any of it works

Write down thirty real questions — pull them from your inbox, not your imagination — and the page that answers each one. Then measure one number: how often the correct page appears among the retrieved chunks. That is retrieval recall, and it is the ceiling on everything else. If the right page is not retrieved 95% of the time or better, a smarter model will not save you.

Only once that number is healthy does it make sense to argue about answer quality.

And keep the log of questions where retrieval found nothing. That log is the most useful document the whole system produces: a list, written by your actual customers, of the pages your website is missing.

I build ChatNory, an AI chatbot that answers from a business's own content on its website, WhatsApp and Telegram — so I have had plenty of chances to get all of the above wrong first. Curious where other people landed on chunk size and re-ranking; it is the part where everyone's numbers seem to differ.