What Actually Happens When You Build a Production RAG Chatbot on Amazon Bedrock

How we built a document Q&A system with NestJS, PostgreSQL, pgvector, Hybrid Search (RRF), Prompt Caching, and Claude on Amazon Bedrock.
Building a demo chatbot is easy.

Building one that consistently answers correctly after weeks of document updates, changing requirements, and real user traffic is where things become interesting.

Most Retrieval-Augmented Generation (RAG) failures don't look like failures.

The API returns 200.

The model responds.

Infrastructure dashboards stay green.

Yet answer quality gradually gets worse.

You ask about error code E-4021 and receive documentation for E-4012.

A recently updated document never appears in search results.

Prompt caching is enabled, but costs don't go down.

Hybrid search is implemented, but retrieval quality barely improves.

Nothing crashes.

Nothing raises an alert.

The chatbot simply becomes less useful over time.

For teams involved in application software development, the real work begins after the first successful response. Retrieval quality, chunking strategy, embedding consistency, prompt caching, document lifecycle management, and hallucination control quickly become more important than model selection itself.

This article walks through a complete production-oriented RAG implementation built with:

• Amazon Bedrock
• Claude
• Cohere Embed Multilingual v3
• NestJS
• PostgreSQL + pgvector
• Hybrid Search using Reciprocal Rank Fusion (RRF)

More importantly, it explains why each component exists, what problem it solves, and which production issues it prevents.

1. What is RAG

RAG (Retrieval-Augmented Generation) does not ask the LLM directly. Before each question, the system:

1. Finds a few internal document passages most relevant to the question.
2. Pastes them into the prompt together with the question.
3. Asks the LLM to answer based only on those passages, with citations.

The consequence of this design:

The problem "how does the LLM know my data" becomes a search problem. If retrieval is weak, the chatbot is weak, because no prompt can produce information that isn't in the context.

This distinction is easy to underestimate.

Many teams spend weeks experimenting with prompts while retrieval quality remains unchanged.

In reality, most enterprise application software development projects that incorporate AI succeed or fail based on retrieval architecture rather than prompt engineering.

That is why most of this article is about chunking documents and search, not about prompts.

Compared with the alternatives:

• Stuffing all documents into the prompt: only works for a very small corpus (a few dozen pages). Beyond that you hit the context limit and pay for the tokens on every turn.
• Fine-tuning: teaches style, not facts. A fine-tuned model still makes things up, just in your company's voice, which makes it harder to spot. Every data change means retraining.
• RAG: updating knowledge only requires re-ingesting, and answers can cite their sources. This is the default choice for business chatbots.

2. Overall architecture

A RAG system has three stages:

• Indexing (offline, re-run whenever documents change): read documents → chunk → generate embeddings → store in an index.
• Retrieval (online, per question): embed the question, find the most relevant chunks.
• Generation (online, per question): put the chunks in the prompt, the LLM writes the answer.

The three stages of RAG: documents are indexed ahead of time, then each question goes through retrieval and then generation

 ══ OFFLINE ══════════════════════════════════════════════════════════ data/in/*.md|txt|json │ ▼ LoaderService ──► ChunkerService ──► Bedrock (Cohere Embed) ──► Postgres read files 2-level split + input_type = chunks table header injection search_document (vector + tsvector) ══ ONLINE ═══════════════════════════════════════════════════════════ POST /api/chat { question } │ ▼ SearchService.hybridSearch ├─ embedQuery (Cohere, input_type = search_query) ├─ DenseSearch (HNSW index) ─┐ run in parallel ├─ KeywordSearch (GIN index) ─┘ └─ reciprocalRankFusion ──► top 5 chunks │ ▼ PromptBuilder ── [system + few-shot: CACHED] + [context + question: not cached] │ ▼ Bedrock (Claude, InvokeModel) │ ▼ { answer, citations, usage } 

Folder layout, one folder per stage:

src/ ├── config/ all config, read from env ├── database/ schema.sql + pg.Pool wrapper (no ORM) ├── llm/ LlmService interface + Bedrock implementation ├── ingest/ loader → chunker → repository → ingest.service ├── search/ dense · keyword · rrf · search.service ├── prompt/ system prompt · few-shot · prompt builder └── chat/ wires the three steps into one RAG loop + HTTP API scripts/ db-init · ingest · search (CLI) test/ unit tests for chunker, rrf, prompt builder 

This architecture may look surprisingly simple.

That is intentional.

Every stage has a single responsibility and can be tested independently.

When retrieval quality drops, we can inspect retrieval.

When generation quality drops, we can inspect prompts.

When ingest breaks, we can inspect indexing.

The separation becomes extremely valuable once the system grows beyond a proof of concept.

3. Following the data: the two flows as they run in the code

The diagram in section 2 shows the blocks. This section slows down: it takes one concrete document and one concrete question and follows them through each function, each SQL query, and each Bedrock call. Once you have read this section, the code in the later steps is just detail.

The sample data used throughout is data/in/warranty.md:

# Warranty Policy ## Warranty period All products are covered by a warranty of 12 months from the date of purchase. ## Returns Products with a manufacturing defect can be exchanged within the first 7 days. 

3.1 The ingest flow: from raw files to the index

This flow runs offline via npm run ingest, not over HTTP. Re-run it whenever documents change.

npm run ingest [-- <folder>] │ ▼ scripts/ingest.ts │ NestFactory.createApplicationContext(AppModule) (no HTTP port opened) ▼ IngestService.ingestDirectory(dir = INGEST_DIR) │ ▼ [1] LoaderService.loadDirectory ─────────► RawDocument[] │ recursive scan; .md/.txt = 1 document; .json = n documents; other extensions skipped │ │ for EACH RawDocument: ▼ [2] ChunkerService.chunk ────────────────► Chunk[] │ split by heading, split long sections (1200 chars, 120 overlap), prepend heading ▼ [3] BedrockService.embedDocuments ───────► one 1024-dim vector per chunk │ up to 96 texts per call, Cohere input_type = search_document ▼ [4] ChunkRepository.upsert (per chunk) ──► chunks table │ INSERT ... ON CONFLICT (chunk_id) DO UPDATE │ Postgres computes search_vector and updates HNSW + GIN indexes ▼ [5] ChunkRepository.deleteStale ─────────► delete leftover chunks of this document │ ▼ IngestReport { documents, chunks, deleted } → printed to console, pool closed 

[1] Read files. LoaderService walks the folder recursively. A markdown file becomes one RawDocument:

{ "source": "data/in/warranty.md", "title": "Warranty Policy", "text": "# Warranty Policy\n\n## Warranty period\nAll products are covered..." } 

source is the file path, and title comes from the first H1 (or the file name if there is none). For .json files, each element with a text field becomes its own document.

[2] Chunk. splitByHeading reads line by line and starts a new section whenever a line begins with #. The sample file produces 3 sections:

section A heading = "Warranty Policy" text = "" → no chunk section B heading = "Warranty period" text = "All products are..." section C heading = "Returns" text = "Products with a..." 

Section A has a heading but no body, so it is dropped. Sections B and C are shorter than 1200 characters, so they are kept whole. A longer section would be split further by splitLongText: paragraph boundaries first, then sentences, with 120 characters of overlap between consecutive pieces.

After prepending the heading (header injection), we get 2 chunks:

[ { "chunkId": "data/in/warranty.md#0", "title": "Warranty period", "content": "Warranty period\n\nAll products are covered by a warranty of 12 months from the date of purchase.", "metadata": { "heading": "Warranty period" } }, { "chunkId": "data/in/warranty.md#1", "title": "Returns", "content": "Returns\n\nProducts with a manufacturing defect can be exchanged within the first 7 days.", "metadata": { "heading": "Returns" } } ] 

chunkId has the form <source>#<position>. It is the key that lets the next ingest update the existing row instead of adding a new one.

[3] Embed. Both content strings go to Cohere in a single InvokeModel call:

{ "texts": ["Warranty period\n\nAll products are...", "Returns\n\nProducts with a..."], "input_type": "search_document", "truncate": "END" } 

The result is 2 vectors of 1024 floats each. Because the heading is part of content, chunk #0's vector carries the meaning "warranty period", not just the sentence about 12 months.

[4] Store. Each chunk is one INSERT ... ON CONFLICT (chunk_id) DO UPDATE. The application only writes content and embedding. Postgres does the rest:

• The search_vector column is recomputed from title + content with to_tsvector('simple', ...). Chunk #0 yields the tokens warranty, period, all, products, are, covered, by, a, of, 12, months, from, the, date, purchase.
• The HNSW index (on embedding) and the GIN index (on search_vector) are updated in the same statement.

[5] Clean up leftovers. deleteStale('data/in/warranty.md', ['...#0', '...#1']) deletes every row of this file whose chunk_id is not in the list just written. On the first run there is nothing to delete. If the file later shrinks from 5 chunks to 2, rows #2, #3 and #4 are deleted here.

At the end, the console prints a summary: number of documents, chunks, and stale chunks deleted.

Three things worth knowing about this flow:

• Every run re-embeds everything, even unchanged files. Fine for a small corpus; for a large one, store a content hash in metadata and skip chunks whose hash hasn't changed.
• source is the path exactly as scanned. Running npm run ingest (giving data/in/...) and later npm run ingest -- /absolute/path produces a different source, i.e. a duplicate set of chunks. Always use the same form of path.
• A file that yields no chunks is skipped, and its old chunks are not cleaned up. To remove a document from the index entirely, use ChunkRepository.deleteBySource.

3.2 The question flow: from question to answer

This flow runs online, once per request. One question costs 2 Bedrock calls (1 embed, 1 generate) and 2 SQL queries run in parallel.

Client │ POST /api/chat { "question": "How long is the product warranty?" } ▼ [1] ValidationPipe + AskDto ─────────────► 400 if empty, > 2000 chars, or unknown fields │ ▼ ChatController.ask → ChatService.ask │ ▼ [2] SearchService.hybridSearch(question) │ ├─ [2a] BedrockService.embedQuery ────► question vector (Cohere, input_type = search_query) │ ├─ [2b] in parallel (Promise.all): │ ├─ DenseSearch : ORDER BY embedding <=> $vec LIMIT 50 → HNSW index │ └─ KeywordSearch : search_vector @@ websearch_to_tsquery → GIN index │ └─ [2c] reciprocalRankFusion(k = 60) ─► top 5 chunks, with denseRank / keywordRank │ ▼ [3] PromptBuilder.build ─────────────────► { system: [...], user: [...] } │ system = SYSTEM_PROMPT + few-shot (cached) │ user = <context>[1]..[5]</context> + <question> (not cached) ▼ [4] BedrockService.generate ─────────────► InvokeModel Claude │ cache_control on the few-shot block; retry on throttling ▼ [5] ChatService assembles the result │ answer = all "text" blocks joined │ citations = the chunks that were sent, numbered to match [n] │ usage = input/output tokens + cache write/read tokens ▼ Client receives { answer, citations, usage } 

[1] Guard the input. ValidationPipe runs with whitelist and forbidNonWhitelisted, so the body may contain only the question field. AskDto requires question to be a non-empty string of at most 2000 characters. Invalid requests get a 400 immediately, before a single cent is spent on Bedrock.

[2a] Embed the question. The question goes to Cohere with input_type: "search_query". This mirrors step [3] of the ingest flow: documents use search_document, questions use search_query, same model. Get either one wrong and question vectors and document vectors no longer live on the same "map".

[2b] Two search branches run at the same time.

The dense branch compares the question vector with every chunk vector by cosine; thanks to HNSW it does not scan the whole table. With our 2-chunk sample corpus:

dense: 1. ...#0 "Warranty period" highest cosine 2. ...#1 "Returns" 

The keyword branch passes the raw question to websearch_to_tsquery('simple', ...):

'how' & 'long' & 'is' & 'the' & 'product' & 'warranty' 

This function joins words with AND. Chunk #0 contains none of how, long, is, and has products rather than product (the simple dictionary does not stem), so the keyword branch returns 0 results for this question. That is the real behavior of the code, and the reason the dense branch matters: for long natural-language questions the keyword branch usually stays silent. It shines on short queries containing codes or names, like E-4021 or warranty 12 months.

[2c] Merge with RRF. Each chunk gets 1 / (60 + rank) from every branch it appears in:

...#0 rrf = 1/61 = 0.01639 denseRank = 1 keywordRank = null ...#1 rrf = 1/62 = 0.01613 denseRank = 2 keywordRank = null 

If the query were warranty 12 months, chunk #0 would appear in both branches, score 1/61 + 1/61 = 0.03279, and pull far ahead of chunks found by only one branch. The list is cut to the top 5 (TOP_K).

[3] Build the prompt. PromptBuilder numbers the chunks from 1 and wraps them in tags (the repository uses Vietnamese prompt text and tag names; English equivalents are shown throughout this article, see section 12):

<context> [1] Warranty period Warranty period All products are covered by a warranty of 12 months from the date of purchase. [2] Returns Returns Products with a manufacturing defect can be exchanged within the first 7 days. </context> <question>How long is the product warranty?</question> 

The heading appears twice because title is printed first while content already contains the heading from header injection. It's harmless, but you can drop the title line when rendering to save tokens.

If neither branch finds anything, the context is (no relevant documents found) and the LLM is still called. The system prompt and few-shot example #2 teach the model to say it could not find the information instead of inventing an answer.

[4] Call Claude. The body sent via InvokeModel:

{ "anthropic_version": "bedrock-2023-05-31", "max_tokens": 2000, "system": [ { "type": "text", "text": "You are a Q&A assistant that answers from the provided documents..." }, { "type": "text", "text": "Examples of good answers: ...", "cache_control": { "type": "ephemeral" } } ], "messages": [ { "role": "user", "content": [ { "type": "text", "text": "<context>\n[1] Warranty period\n...\n</context>" }, { "type": "text", "text": "<question>How long is the product warranty?</question>" } ] } ] } 

Everything before cache_control (system prompt + few-shot) is identical on every request, so from the second request on it is read from cache. messages changes with every question and is always billed at full price. If Bedrock returns ThrottlingException or ServiceUnavailableException, the request is retried after 500ms and then 1s; any other error is thrown immediately.

[5] Return the result. Claude returns a content array. The code keeps only type: "text" blocks and joins them, ignoring other block types. citations is the list of chunks put into the prompt, numbered to match [n] in the context, so the client can turn [1] in the answer into a link to source:

{ "answer": "All products are covered by a 12-month warranty from the date of purchase [1].", "citations": [ { "index": 1, "chunkId": "data/in/warranty.md#0", "source": "data/in/warranty.md", "title": "Warranty period" }, { "index": 2, "chunkId": "data/in/warranty.md#1", "source": "data/in/warranty.md", "title": "Returns" } ], "usage": { "inputTokens": 180, "outputTokens": 25, "cacheWriteTokens": 0, "cacheReadTokens": 650 } } 

(The usage values above are illustrative.) Note that citations contains every chunk that was sent, not only the ones the model actually cited. The client should filter by the [n] numbers that appear in answer.

3.3 What makes answers accurate, and at which step

Accuracy doesn't come from one place. It accumulates across both flows:

• Ingest [2], header injection: chunks carry their section heading, so questions phrased like the heading still match the body text.
• Ingest [2], split by heading first, by length second: each chunk is about one idea, so its vector isn't "averaged out" across several topics.
• Ingest [3] + Question [2a], correct input_type: documents and questions are projected into the same vector space.
• Ingest [5], cleaning up leftovers: content deleted from a document can no longer leak into answers.
• Question [2b], two branches + over-fetching 50: the dense branch catches paraphrases, the keyword branch catches codes and names; over-fetching keeps chunks that do well in both branches from falling off the list.
• Question [2c], RRF: chunks found by both branches rise to the top.
• Question [3], system prompt rules + a refusal few-shot: the model answers only from context, cites [n], and knows how to say "not found".

What these two flows don't do yet, covered in the "Upgrade path" section at the end: Unicode NFC normalization for both documents and questions, rewriting questions using conversation history (the API is currently stateless), a confidence threshold to skip the LLM when results are too weak, and checking in code that the [n] the model cites actually exist.

4. Tech stack and why

One design goal guided every technology choice: keep the architecture understandable.

Every component should be replaceable without rewriting the rest of the system.

• NestJS 10 (TypeScript): modules + dependency injection make each layer replaceable on its own. Switching LLM providers is a one-line useClass change.
• Amazon Bedrock: one API and one AWS bill for both embeddings and generation. Credentials follow your organization's existing IAM.
• Cohere Embed Multilingual v3 (cohere.embed-multilingual-v3, 1024 dimensions): multilingual, which matters when documents mix a local language with English UI labels.
• Claude on Bedrock: generates answers and supports prompt caching.
• PostgreSQL 16 + pgvector: vector search (HNSW) and full-text search (GIN) in the same database. No extra service, no data sync between two systems.
• Plain pg, no ORM: SQL lives next to the module that uses it, so reading a query tells you which index it uses.
• @aws-sdk/client-bedrock-runtime: calls InvokeModel.
• Jest, ESLint/Prettier, multi-stage Docker, GitHub Actions.

That's the entire runtime dependency list:

"dependencies": { "@aws-sdk/client-bedrock-runtime": "^3.696.0", "@nestjs/common": "^10.4.0", "@nestjs/core": "^10.4.0", "@nestjs/platform-express": "^10.4.0", "class-transformer": "^0.5.1", "class-validator": "^0.14.1", "dotenv": "^16.4.5", "pg": "^8.12.0", "reflect-metadata": "^0.2.2", "rxjs": "^7.8.1" } 

5. Prerequisites

5.1 Local environment

• Node.js 20+
• Docker + Docker Compose
• An AWS account

5.2 Enable models on Amazon Bedrock

1. Open the Amazon Bedrock console and pick your region (this article uses ap-northeast-1).
2. Make sure your account has access to two models: Cohere Embed Multilingual v3 and a Claude model. The first time you use an Anthropic model, AWS may ask you to fill in a use-case form.
3. Open the Model catalog and note the exact model ID for Claude. Bedrock model IDs have the anthropic. prefix, for example anthropic.claude-opus-5-5. Some regions require calling through a cross-region inference profile; in that case use the profile ID the console shows.

5.3 Minimal IAM policy

The application only needs bedrock:InvokeModel:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["bedrock:InvokeModel"], "Resource": [ "arn:aws:bedrock:*::foundation-model/*", "arn:aws:bedrock:*:*:inference-profile/*" ] } ] } 

In production, narrow Resource down to the exact ARNs of the two models you use.

About credentials: static access keys work, but it's better to leave them empty and let the AWS SDK pick up AWS_PROFILE, SSO, or the EC2/ECS IAM role. The code below supports both.

6. Step 1: Create the project

npm i -g @nestjs/cli nest new rag-chatbot --package-manager npm --skip-git cd rag-chatbot npm i @aws-sdk/client-bedrock-runtime pg dotenv class-validator class-transformer npm i -D @types/pg ts-node 

Add these scripts to package.json:

"scripts": { "build": "nest build", "start:dev": "nest start --watch", "start:prod": "node dist/main", "db:init": "ts-node scripts/db-init.ts", "ingest": "ts-node scripts/ingest.ts", "search": "ts-node scripts/search.ts", "test": "jest", "lint": "eslint \"{src,scripts,test}/**/*.ts\"" } 

7. Step 2: Config in one place

Every setting is read from env, with sensible defaults so it runs out of the box. No per-project profile files.

.env.example:

# ── App PORT=8000 # ── PostgreSQL + pgvector DB_HOST=127.0.0.1 DB_PORT=5432 DB_NAME=ragdb DB_USER=postgres DB_PASSWORD=postgres # Must match the embedding model's dimensions and the vector(N) column in the schema. # cohere.embed-multilingual-v3 → 1024. Changing it = re-run db:init + re-embed everything. EMBEDDING_DIMENSIONS=1024 # ── AWS Bedrock AWS_REGION=ap-northeast-1 AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= AWS_SESSION_TOKEN= # Leave the keys above empty to let the SDK use AWS_PROFILE / SSO / instance role. BEDROCK_EMBEDDING_MODEL_ID=cohere.embed-multilingual-v3 BEDROCK_GENERATION_MODEL_ID=anthropic.claude-opus-5-5 # ── Ingest / chunking INGEST_DIR=./data/in CHUNK_MAX_CHARS=1200 CHUNK_OVERLAP_CHARS=120 EMBED_BATCH_SIZE=96 # ── Search TOP_K=5 # number of chunks finally placed in the prompt CANDIDATE_LIMIT=50 # candidates fetched by EACH branch before fusion RRF_K=60 # RRF constant k, 60 is the value from the original paper # ── Generation MAX_TOKENS=2000 ENABLE_PROMPT_CACHE=true 

src/config/config.ts (abridged; the full version has an AppConfig interface documenting each field):

const num = (v: string | undefined, fallback: number) => v === undefined || v === '' ? fallback : Number(v); export function loadConfig(): AppConfig { return { port: num(process.env.PORT, 8000), database: { host: process.env.DB_HOST || '127.0.0.1', port: num(process.env.DB_PORT, 5432), name: process.env.DB_NAME || 'ragdb', user: process.env.DB_USER || 'postgres', password: process.env.DB_PASSWORD || 'postgres', embeddingDimensions: num(process.env.EMBEDDING_DIMENSIONS, 1024), }, bedrock: { region: process.env.AWS_REGION || 'ap-northeast-1', accessKeyId: process.env.AWS_ACCESS_KEY_ID, secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY, sessionToken: process.env.AWS_SESSION_TOKEN, embeddingModelId: process.env.BEDROCK_EMBEDDING_MODEL_ID || 'cohere.embed-multilingual-v3', generationModelId: process.env.BEDROCK_GENERATION_MODEL_ID || 'anthropic.claude-opus-5-5', }, ingest: { dir: process.env.INGEST_DIR || './data/in', chunkMaxChars: num(process.env.CHUNK_MAX_CHARS, 1200), chunkOverlapChars: num(process.env.CHUNK_OVERLAP_CHARS, 120), embedBatchSize: num(process.env.EMBED_BATCH_SIZE, 96), }, search: { topK: num(process.env.TOP_K, 5), candidateLimit: num(process.env.CANDIDATE_LIMIT, 50), rrfK: num(process.env.RRF_K, 60), }, generation: { maxTokens<...