I was knee-deep in building my SaaS when users started asking questions I couldn't answer from the UI alone. My app had a solid backend, Lambda functions, Fastify, PostgreSQL on RDS. I wrote about that whole architecture in my previous post. But users wanted to search through documentation, find answers buried in help articles, and ask natural-language questions about the product. The UI just wasn't built for that.
So I started looking into RAG. Every tutorial I found showed the same setup: LangChain running locally, Pinecone for vectors, and a Python script tying it all together. Great for demos. Not great for a production SaaS already running on AWS.
Here's the thing. I didn't want to add another service to my stack. I was already paying for PostgreSQL on RDS. I was already managing Lambda functions. Adding Pinecone or Weaviate meant another bill, another SDK, another thing to monitor at 2 AM. Then I found pgvector, and everything clicked.
pgvector is a PostgreSQL extension. It adds vector storage and similarity search right inside the database I was already running. No new infrastructure. No new vendor. Just an extension. I paired it with Lambda for compute, Bedrock for the LLM, and RDS Proxy for connection pooling. The whole pipeline costs me about $54/month at 10K queries per day.
Let me walk you through every piece of how I built it.
Photo by Tara Winstead on Pexels
What Is RAG and Why Does It Work
RAG stands for Retrieval-Augmented Generation. The name sounds academic, but the idea is simple. And once I understood it, I realized it was exactly what my users were asking for without knowing the term.
You have your own data. Documents, help articles, product specs, whatever. You want an LLM to answer questions about that data. But the LLM was trained on the internet, not your private stuff. So you give it context at query time.
The flow looks like this:
- User asks a question. "How do I reset my API key?"
- Retrieval step. You search your own documents for the most relevant chunks. This is semantic search, not keyword matching. You're comparing the meaning of the question against the meaning of each chunk.
- Generation step. You stuff those relevant chunks into a prompt and send it to the LLM. "Based on the following context, answer this question."
- LLM responds with an answer grounded in your actual data.
That's it. No fine-tuning. No retraining. No waiting weeks for a model to learn your docs.
And that's exactly why RAG beats fine-tuning for most use cases. Fine-tuning is slow, expensive, and your model becomes stale the moment your data changes. With RAG, you just update the documents in your database. The next query picks up the new content automatically. Your data is always live.
The magic is in the retrieval step. If you retrieve the right chunks, the LLM gives great answers. If you retrieve garbage, the LLM confidently presents garbage. So the vector search layer matters a lot. That's where pgvector comes in. And for my SaaS, it meant I could keep everything inside the PostgreSQL instance I was already running.
Photo by cottonbro studio on Pexels
Setting Up pgvector in PostgreSQL (RDS)
When I first looked at Pinecone, I nearly added it. The docs are slick and the onboarding is smooth. Then I checked the pricing page and realized I was already paying for PostgreSQL. Why add another database when I could just enable an extension?
If you're already running PostgreSQL on RDS, adding vector search is surprisingly easy. You don't need Pinecone. You don't need Weaviate. You just enable an extension.
pgvector adds a vector column type to PostgreSQL. You store embeddings as vectors, then query them with distance operators. Cosine similarity, L2 distance, inner product. All the standard stuff.
Enable the Extension
First, make sure your RDS parameter group allows pgvector. Here's the CloudFormation snippet:
RDSParameterGroup: Type: AWS::RDS::DBParameterGroup Properties: Family: postgres16 Description: "PostgreSQL 16 with pgvector" Parameters: shared_preload_libraries: "pg_stat_statements,pgvector" Then run the SQL to create the extension:
CREATE EXTENSION IF NOT EXISTS vector; That's it. PostgreSQL now understands vectors.
Create the Embeddings Table
Here's the schema I use. The vector(1536) column size matches OpenAI's text-embedding-3-small model. If you're using Bedrock Titan, switch to vector(1024).
CREATE TABLE hrr_document_chunks ( id BIGSERIAL PRIMARY KEY, document_id TEXT NOT NULL, chunk_index INTEGER NOT NULL, content TEXT NOT NULL, metadata JSONB DEFAULT '{}', embedding vector(1536) NOT NULL, created_at TIMESTAMPTZ DEFAULT NOW(), updated_at TIMESTAMPTZ DEFAULT NOW(), UNIQUE (document_id, chunk_index) ); The metadata column is a JSONB field where I store things like the source URL, document title, section heading, and any other context that helps with filtering later. You'll want this when your dataset grows and you need to scope searches to specific document types.
Create an HNSW Index
Without an index, pgvector does a sequential scan. That's fine for a few thousand rows. But once you hit tens of thousands of embeddings, you need an index.
I recommend HNSW over IVFFlat. HNSW is slightly slower to build but gives better recall and works much better for Lambda workloads. More on that in the lessons learned section.
CREATE INDEX idx_chunks_embedding_hnsw ON hrr_document_chunks USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 200); The m and ef_construction parameters control the index quality. Higher values give better recall but slower inserts. These defaults work well for datasets up to about 500K embeddings. I landed on these numbers after a week of testing different configurations against my own dataset.
Photo by Rahul Pandit on Pexels
Generating and Storing Embeddings
Now you need to get your data into that table. This is the ingestion pipeline: take your documents, break them into chunks, generate embeddings for each chunk, and store everything in pgvector.
For my SaaS, the documents were help articles, product guides, and FAQ pages that users kept asking about. I run ingestion as a separate Lambda function. It gets triggered whenever new documents are added, either through an S3 event or a direct API call.
Chunking
Chunking strategy matters more than most people think. I split documents into chunks of about 500 tokens with a 50-token overlap between consecutive chunks. The overlap prevents information from getting lost at chunk boundaries.
function chunkDocument(text, maxTokens = 500, overlap = 50) { const sentences = text.split(/(?<=[.!?])\s+/); const chunks = []; let currentChunk = []; let currentLength = 0; for (const sentence of sentences) { const sentenceTokens = estimateTokens(sentence); if (currentLength + sentenceTokens > maxTokens && currentChunk.length > 0) { chunks.push(currentChunk.join(" ")); // Keep last few sentences for overlap const overlapSentences = []; let overlapLength = 0; for (let i = currentChunk.length - 1; i >= 0; i--) { overlapLength += estimateTokens(currentChunk[i]); if (overlapLength > overlap) break; overlapSentences.unshift(currentChunk[i]); } currentChunk = overlapSentences; currentLength = overlapLength; } currentChunk.push(sentence); currentLength += sentenceTokens; } if (currentChunk.length > 0) { chunks.push(currentChunk.join(" ")); } return chunks; } function estimateTokens(text) { return Math.ceil(text.split(/\s+/).length * 1.3); } Generating Embeddings with Bedrock Titan
Here's how to call Bedrock's Titan Embeddings model. This uses the AWS SDK v3:
import { BedrockRuntimeClient, InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime"; const bedrockClient = new BedrockRuntimeClient({ region: "us-east-1" }); async function generateEmbedding(text) { const response = await bedrockClient.send( new InvokeModelCommand({ modelId: "amazon.titan-embed-text-v2:0", contentType: "application/json", accept: "application/json", body: JSON.stringify({ inputText: text, dimensions: 1024, normalize: true, }), }) ); const result = JSON.parse(new TextDecoder().decode(response.body)); return result.embedding; } If you prefer OpenAI embeddings instead, the call is straightforward:
async function generateEmbeddingOpenAI(text) { const response = await fetch("https://api.openai.com/v1/embeddings", { method: "POST", headers: { "Authorization": `Bearer ${process.env.OPENAI_API_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify({ model: "text-embedding-3-small", input: text, }), }); const data = await response.json(); return data.data[0].embedding; } Upserting Chunks
Now bring it all together. Chunk the document, embed each chunk, and upsert into pgvector:
async function ingestDocument(pool, documentId, text, metadata = {}) { const chunks = chunkDocument(text); for (let i = 0; i < chunks.length; i++) { const embedding = await generateEmbedding(chunks[i]); await pool.query( `INSERT INTO hrr_document_chunks (document_id, chunk_index, content, metadata, embedding) VALUES ($1, $2, $3, $4, $5::vector) ON CONFLICT (document_id, chunk_index) DO UPDATE SET content = $3, metadata = $4, embedding = $5::vector, updated_at = NOW()`, [documentId, i, chunks[i], JSON.stringify(metadata), JSON.stringify(embedding)] ); } // Clean up old chunks if document got shorter await pool.query( `DELETE FROM hrr_document_chunks WHERE document_id = $1 AND chunk_index >= $2`, [documentId, chunks.length] ); return chunks.length; } The ON CONFLICT clause handles re-ingestion cleanly. If a document gets updated, you just run ingest again and it overwrites the old chunks. The cleanup query at the end removes stale chunks if the document got shorter after an edit.
One thing to watch out for: Bedrock Titan has a rate limit. I hit this on day one when I tried to ingest my entire knowledge base at once. If you're ingesting thousands of documents, add a small delay between embedding calls or use batch processing. I throttle to about 10 requests per second and that's been reliable.
Photo by Media Dung on Pexels
Retrieval: Semantic Search on Lambda
This is the part that made it all feel real. A user sends a question, and you need to find the most relevant chunks from your database. Fast.
The first time I ran a semantic search against my own docs and got back exactly the right paragraph, I knew this was going to work. The retrieval Lambda takes a query string, embeds it using the same model you used for ingestion, then runs a cosine similarity search against pgvector. The whole thing takes about 100-200ms on a warm Lambda invocation.
async function retrieveChunks(pool, query, topK = 5) { // Embed the user's query with the same model used for ingestion const queryEmbedding = await generateEmbedding(query); // Cosine similarity search using pgvector's <=> operator const result = await pool.query( `SELECT id, document_id, content, metadata, 1 - (embedding <=> $1::vector) AS similarity FROM hrr_document_chunks WHERE 1 - (embedding <=> $1::vector) > 0.3 ORDER BY embedding <=> $1::vector LIMIT $2`, [JSON.stringify(queryEmbedding), topK] ); return result.rows; } A few things to notice here.
The <=> operator is pgvector's cosine distance. Lower distance means higher similarity. I compute 1 - distance to get a similarity score between 0 and 1. The WHERE clause filters out anything below 0.3 similarity, which cuts out the noise. You don't want the LLM seeing irrelevant chunks.
The topK parameter controls how many chunks you return. I default to 5. Going higher gives the LLM more context but costs more tokens. Going lower risks missing relevant information. Five is a good starting point for most use cases.
Connection Pooling with RDS Proxy
If you read my previous post, you know this pattern. Lambda functions can spin up hundreds of concurrent instances. Each one opens a database connection. Without connection pooling, you'll exhaust PostgreSQL's connection limit in seconds.
RDS Proxy handles this. Your Lambda points to the proxy endpoint, and the proxy multiplexes connections to the actual database. The code is identical to a normal pg.Pool setup:
import pg from "pg"; const pool = new pg.Pool({ host: process.env.DB_HOST, // RDS Proxy endpoint port: 5432, database: process.env.DB_NAME, user: process.env.DB_USER, password: process.env.DB_PASSWORD, max: 5, idleTimeoutMillis: 30000, connectionTimeoutMillis: 5000, ssl: { rejectUnauthorized: true }, }); Set max: 5 on the Lambda side. The proxy handles the actual pooling across all Lambda instances. Keeping the per-function pool small prevents any single Lambda from hogging connections.
Photo by Nataliya Vaitkevich on Pexels
Generation: Calling the LLM with Context
You've retrieved the relevant chunks. Now you need to turn them into an answer. This is where the LLM comes in.
I went back and forth on this part. The idea is simple: build a prompt that includes the retrieved chunks as context, along with the user's question. Then call the LLM and return whatever it generates. But getting the prompt right took more iterations than I expected.
The Prompt Template
Here's the prompt template I use. It's deliberately strict about staying grounded in the context:
function buildPrompt(question, chunks) { const context = chunks .map((chunk, i) => `[Source ${i + 1}] ${chunk.content}`) .join("\n\n"); return { system: `You are a helpful assistant that answers questions based on the provided context. Rules: - Only use information from the provided context to answer. - If the context does not contain enough information, say so honestly. - Keep answers concise and direct. - Reference which source(s) you used when possible.`, user: `Context: ${context} Question: ${question} Answer based only on the context above.`, }; } That system prompt is important. I learned this after my first demo went sideways. Without explicit instructions to stay grounded, the LLM will happily make things up using its training data. One of my test users asked about a feature we didn't have, and the LLM invented a whole tutorial for it. The "say so honestly" part is key. I'd rather get an "I don't have enough information" response than a confident hallucination.
Calling Bedrock Claude
Here's the Bedrock call using Claude Haiku. I use Haiku instead of Sonnet for RAG because it's 10x cheaper and the quality difference is minimal when you're giving it good context:
import { BedrockRuntimeClient, InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime"; const bedrockClient = new BedrockRuntimeClient({ region: "us-east-1" }); async function generateAnswer(question, chunks) { const prompt = buildPrompt(question, chunks); const response = await bedrockClient.send( new InvokeModelCommand({ modelId: "anthropic.claude-3-haiku-20240307-v1:0", contentType: "application/json", accept: "application/json", body: JSON.stringify({ anthropic_version: "bedrock-2023-05-31", max_tokens: 1024, system: prompt.system, messages: [{ role: "user", content: prompt.user }], }), }) ); const result = JSON.parse(new TextDecoder().decode(response.body)); return result.content[0].text; } Handling Token Limits
Here's a practical concern. Each chunk might be 500 tokens. If you retrieve 5 chunks, that's 2,500 tokens of context. Add the system prompt, the question, and the response, and you're looking at maybe 4,000 tokens total. That fits easily within Haiku's context window.
But what if your chunks are longer, or you need more of them? I cap the total context at 3,000 tokens and truncate the lowest-similarity chunks first:
function truncateContext(chunks, maxTokens = 3000) { let totalTokens = 0; const selected = []; for (const chunk of chunks) { const tokens = estimateTokens(chunk.content); if (totalTokens + tokens > maxTokens) break; selected.push(chunk); totalTokens += tokens; } return selected; } Since the chunks are already sorted by similarity (highest first), this naturally keeps the most relevant ones and drops the least relevant. Simple and effective.
Photo by Matheus Bertelli on Pexels
Wiring It Together: The API Endpoint
Now let's connect everything. This is where my existing SaaS architecture paid off. The API endpoint receives a question, runs retrieval, generates an answer, and returns it. I use Fastify on Lambda (same setup as my previous post), and adding a new route was straightforward. The pattern works with any framework.
async function ragHandler(request, reply) { const { question } = request.body; if (!question || question.trim().length === 0) { return reply.status(400).send({ error: "Question is required" }); } // Step 1: Retrieve relevant chunks const chunks = await retrieveChunks(request.server.pg, question, 5); if (chunks.length === 0) { return reply.send({ answer: "I couldn't find any relevant information to answer your question.", sources: [], }); } // Step 2: Truncate context to fit token limits const trimmedChunks = truncateContext(chunks); // Step 3: Generate answer const answer = await generateAnswer(question, trimmedChunks); // Step 4: Return answer with source metadata return reply.send({ answer, sources: trimmedChunks.map((c) => ({ documentId: Source：https://dev.to/harunrrayhan/building-a-production-rag-pipeline-on-aws-lambda-and-pgvector-11d2Related posts
Turn Today’s Messy Notes Into Tomorrow’s Checklist With AI
A 763B MoE at "2 Bits": Where the Bytes Really Go in the GSQ-RCO GGUFs, and Why 100% Recovery Sits Inside the Noise
How I ship fixes to open-source projects maintainers actually merge (with AI in the loop)
Prompt Injection Through Documents: How to Harden an LLM Extraction Pipeline
Keeping noindex, the sitemap and the image sitemap in agreement for programmatic pages
