November 2026 | ~18 min read

The Question Nobody Could Answer

Every engineering org has the same problem. New engineer joins, asks "how does the deployment pipeline handle database migrations?" The answer exists — buried in a Confluence page from 2023, a Slack thread from last April, and a post-mortem that three people remember reading. Nobody can find it. So the question gets asked again. And again.

I built a production-ready RAG pipeline on AWS to solve this. Not a notebook demo. Not a "just call the API" tutorial. A real pipeline — document ingestion from S3, vector embeddings via Amazon Titan, retrieval from OpenSearch Serverless, generation via Claude 3.5 Sonnet through Bedrock, a Lambda query API behind API Gateway, and CloudWatch observability across the whole stack. All deployed with Terraform. Zero console clicks.

The goal: any engineer can hit an API endpoint (or a Slack bot wired to it) with a natural language question and get an accurate answer with citations back to the source documents.

The Numbers 

┌──────────────────────────────────────────────────────────────────┐ │ PRODUCTION RAG PIPELINE │ ├──────────────────────────────────┬───────────────────────────────┤ │ Metric │ Value │ ├──────────────────────────────────┼───────────────────────────────┤ │ Documents indexed │ 2,400+ (Confluence, runbooks, │ │ │ post-mortems, ADRs) │ │ Avg query latency (p50) │ 1.8 seconds │ │ Avg query latency (p99) │ 4.2 seconds │ │ Monthly AWS cost │ ~$380 (at ~500 queries/day) │ │ Embedding model │ Amazon Titan Text Embed v2 │ │ Generation model │ Claude 3.5 Sonnet via Bedrock │ │ Infrastructure │ 100% Terraform, zero console │ └──────────────────────────────────┴───────────────────────────────┘

Table of Contents

  • Why RAG Over Fine-Tuning
  • Architecture Overview
  • OpenSearch Serverless Collection
  • Bedrock Knowledge Base
  • Query API with Lambda
  • Observability and Cost Control
  • Document Ingestion Pipeline
  • Lessons Learned
  • Action Items Checklist

Why RAG Over Fine-Tuning

Before writing any Terraform, I had to answer the first question: why RAG instead of fine-tuning a model on our internal docs?

┌──────────────────────────┬───────────────────────┬───────────────────────┐ │ Factor │ RAG │ Fine-Tuning │ ├──────────────────────────┼───────────────────────┼───────────────────────┤ │ Data freshness │ Real-time (re-ingest) │ Stale (retrain cycle) │ │ Citation support │ Native (source docs) │ None │ │ Cost to update │ ~$0 (S3 sync) │ $100s–$1000s/retrain │ │ Hallucination control │ Grounded in sources │ No guarantees │ │ Setup complexity │ Medium │ High │ │ Latency │ Higher (~2s) │ Lower (~500ms) │ │ Domain-specific language │ Depends on retrieval │ Strong │ │ Data privacy │ Data stays in your VPC│ Sent to training API │ └──────────────────────────┴───────────────────────┴───────────────────────┘

RAG wins when:

  • Documents change frequently. Internal runbooks, ADRs, and post-mortems update weekly. Fine-tuning means retraining every time a doc changes. RAG means dropping a new file in S3.
  • You need citations. "According to the post-mortem from March 2026..." is the difference between a useful answer and a hallucination you can't verify.
  • Cost matters. Fine-tuning Claude or Llama on 2,400 documents costs real money per training run. RAG costs pennies per ingestion.

Fine-tuning wins when you need specialized domain language (medical terminology, legal jargon) or sub-second latency. For an internal knowledge base where accuracy and freshness matter more than speed, RAG is the clear choice.

Architecture Overview 

┌─────────────────────────────────────────────────────────────────────────────┐ │ PRODUCTION RAG PIPELINE │ │ │ │ ┌──────────┐ ┌─────────────────┐ ┌──────────────┐ │ │ │ │ │ Bedrock Data │ │ Amazon │ │ │ │ S3 │───▶│ Source │───▶│ Titan │ │ │ │ (docs) │ │ (chunking) │ │ Embed v2 │ │ │ └──────────┘ └─────────────────┘ └──────┬───────┘ │ │ │ │ │ ▼ │ │ ┌────────────────────┐ │ │ │ OpenSearch │ │ │ │ Serverless │ │ │ │ (vector store) │ │ │ └────────┬───────────┘ │ │ │ │ │ ┌──────────┐ ┌─────────────────┐ │ │ │ │ API │ │ Lambda │ │ │ │ │ Gateway │───▶│ (query API) │───▶ Bedrock KB ───▶ Claude 3.5 │ │ └──────────┘ └─────────────────┘ (retrieve + Sonnet │ │ generate) (answer) │ │ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ CloudWatch: latency, tokens, errors, cost anomaly alarms │ │ │ └─────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────────────┘

Why OpenSearch Serverless over the alternatives?

  • Pinecone: Great product, but adds a third-party dependency and egress costs. Bedrock Knowledge Bases have native AOSS integration — no custom retrieval code needed.
  • pgvector (Aurora/RDS): Solid for small datasets. Falls behind at scale — no built-in ANN indexing tuning, and you're paying for a full database instance.
  • S3 Vectors: New and promising, but Bedrock Knowledge Bases don't support it as a storage backend yet. When they do, it'll be the cheapest option.
  • OpenSearch Serverless: Native Bedrock integration, managed scaling, no cluster to babysit. The minimum cost stings ($350/month for 2 OCUs), but you're trading ops burden for dollars.

Component cost breakdown:

┌──────────────────────────────────┬───────────────────────────────┐ │ Component │ Monthly Cost │ ├──────────────────────────────────┼───────────────────────────────┤ │ OpenSearch Serverless (2 OCUs) │ ~$350 │ │ Bedrock - Titan Embed v2 │ ~$4 (ingestion) │ │ Bedrock - Claude 3.5 Sonnet │ ~$18 (500 queries/day) │ │ Lambda │ ~$2 │ │ API Gateway │ ~$2 │ │ S3 (document storage) │ <$1 │ │ CloudWatch │ ~$3 │ ├──────────────────────────────────┼───────────────────────────────┤ │ Total │ ~$380/month │ └──────────────────────────────────┴───────────────────────────────┘

Yes, OpenSearch Serverless is 92% of the bill. We'll talk about that.

OpenSearch Serverless Collection

OpenSearch Serverless for vector search requires three things: an encryption policy, a network policy, and a data access policy. Without all three, the collection won't create. This trips up everyone the first time.

Terraform Configuration 

# opensearch-serverless.tf resource "aws_opensearchserverless_security_policy" "encryption" { name = "rag-kb-encryption" type = "encryption" policy = jsonencode({ Rules = [ { Resource = ["collection/rag-knowledge-base"] ResourceType = "collection" } ] AWSOwnedKey = true }) } resource "aws_opensearchserverless_security_policy" "network" { name = "rag-kb-network" type = "network" policy = jsonencode([ { Rules = [ { Resource = ["collection/rag-knowledge-base"] ResourceType = "collection" }, { Resource = ["collection/rag-knowledge-base"] ResourceType = "dashboard" } ] AllowFromPublic = true } ]) } resource "aws_opensearchserverless_collection" "kb" { name = "rag-knowledge-base" description = "Vector store for RAG knowledge base" type = "VECTORSEARCH" depends_on = [ aws_opensearchserverless_security_policy.encryption, aws_opensearchserverless_security_policy.network, ] } resource "aws_opensearchserverless_access_policy" "kb" { name = "rag-kb-access" type = "data" policy = jsonencode([ { Rules = [ { Resource = ["collection/rag-knowledge-base"] ResourceType = "collection" Permission = [ "aoss:CreateCollectionItems", "aoss:DeleteCollectionItems", "aoss:UpdateCollectionItems", "aoss:DescribeCollectionItems" ] }, { Resource = ["index/rag-knowledge-base/*"] ResourceType = "index" Permission = [ "aoss:CreateIndex", "aoss:DeleteIndex", "aoss:UpdateIndex", "aoss:DescribeIndex", "aoss:ReadDocument", "aoss:WriteDocument" ] } ] Principal = [ aws_iam_role.bedrock_kb.arn, data.aws_caller_identity.current.arn ] } ]) }

A few things to note:

  • AllowFromPublic = true on the network policy: For a production deployment, you'd scope this to a VPC endpoint. I'm keeping it public here for simplicity — the data access policy controls who can actually read/write.
  • The depends_on block is required. Without it, Terraform tries to create the collection before the encryption policy exists, and the API returns a cryptic error about missing encryption configuration.
  • Both the Bedrock role and your own IAM identity need access. The Bedrock role writes embeddings during ingestion. Your identity is needed to create the vector index.

Creating the Vector Index

Bedrock Knowledge Bases require a specific index schema. You can't create this index through Terraform — the OpenSearch Serverless API doesn't expose index management. Use a Python script:

# scripts/create_vector_index.py  import boto3 import json from opensearchpy import OpenSearch, RequestsAWSV4SignerAuth, AWSV4SignerAuth from requests_aws4auth import AWS4Auth COLLECTION_ENDPOINT = "https://your-collection-id.us-east-1.aoss.amazonaws.com" INDEX_NAME = "bedrock-knowledge-base-default-index" REGION = "us-east-1" credentials = boto3.Session().get_credentials() auth = AWS4Auth( credentials.access_key, credentials.secret_key, REGION, "aoss", session_token=credentials.token, ) client = OpenSearch( hosts=[{"host": COLLECTION_ENDPOINT.replace("https://", ""), "port": 443}], http_auth=auth, use_ssl=True, verify_certs=True, connection_class="RequestsHttpConnection", ) index_body = { "settings": { "index": { "knn": True, "knn.algo_param.ef_search": 512, } }, "mappings": { "properties": { "bedrock-knowledge-base-default-vector": { "type": "knn_vector", "dimension": 1024, "method": { "engine": "faiss", "space_type": "l2", "name": "hnsw", "parameters": { "ef_construction": 512, "m": 16, }, }, }, "AMAZON_BEDROCK_METADATA": { "type": "text", "index": False, }, "AMAZON_BEDROCK_TEXT_CHUNK": { "type": "text", }, } }, } response = client.indices.create(index=INDEX_NAME, body=index_body) print(json.dumps(response, indent=2))

Why the field names matter: Bedrock Knowledge Bases expect specific field names — bedrock-knowledge-base-default-vector, AMAZON_BEDROCK_TEXT_CHUNK, and AMAZON_BEDROCK_METADATA. If you use custom field names, you'll need to specify the mapping in the knowledge base configuration. Using the defaults saves a layer of configuration.

Dimension 1024 matches Amazon Titan Text Embeddings v2. If you switch to a different embedding model, this number changes (Cohere Embed = 1024, Titan v1 = 1536).

Bedrock Knowledge Base

The knowledge base ties everything together: it connects S3 (documents) to the embedding model (Titan) to the vector store (OpenSearch Serverless).

IAM Role

The Bedrock service needs permissions to read from S3, generate embeddings, and write to OpenSearch Serverless:

# iam.tf data "aws_caller_identity" "current" {} resource "aws_iam_role" "bedrock_kb" { name = "bedrock-knowledge-base-role" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [ { Action = "sts:AssumeRole" Effect = "Allow" Principal = { Service = "bedrock.amazonaws.com" } Condition = { StringEquals = { "aws:SourceAccount" = data.aws_caller_identity.current.account_id } } } ] }) } resource "aws_iam_role_policy" "bedrock_kb" { name = "bedrock-knowledge-base-policy" role = aws_iam_role.bedrock_kb.id policy = jsonencode({ Version = "2012-10-17" Statement = [ { Sid = "BedrockInvokeModel" Effect = "Allow" Action = ["bedrock:InvokeModel"] Resource = [ "arn:aws:bedrock:us-east-1::foundation-model/amazon.titan-embed-text-v2:0" ] }, { Sid = "S3ReadDocuments" Effect = "Allow" Action = [ "s3:GetObject", "s3:ListBucket" ] Resource = [ aws_s3_bucket.documents.arn, "${aws_s3_bucket.documents.arn}/*" ] }, { Sid = "OpenSearchServerlessAccess" Effect = "Allow" Action = ["aoss:APIAccessAll"] Resource = [ aws_opensearchserverless_collection.kb.arn ] } ] }) }

The aws:SourceAccount condition in the trust policy prevents confused deputy attacks — without it, any AWS account could potentially assume your Bedrock role.

Knowledge Base and Data Source 

# knowledge-base.tf resource "aws_s3_bucket" "documents" { bucket = "rag-knowledge-base-documents-${data.aws_caller_identity.current.account_id}" } resource "aws_s3_bucket_versioning" "documents" { bucket = aws_s3_bucket.documents.id versioning_configuration { status = "Enabled" } } resource "aws_bedrockagent_knowledge_base" "main" { name = "internal-knowledge-base" role_arn = aws_iam_role.bedrock_kb.arn knowledge_base_configuration { type = "VECTOR" vector_knowledge_base_configuration { embedding_model_arn = "arn:aws:bedrock:us-east-1::foundation-model/amazon.titan-embed-text-v2:0" } } storage_configuration { type = "OPENSEARCH_SERVERLESS" opensearch_serverless_configuration { collection_arn = aws_opensearchserverless_collection.kb.arn vector_index_name = "bedrock-knowledge-base-default-index" field_mapping { vector_field = "bedrock-knowledge-base-default-vector" text_field = "AMAZON_BEDROCK_TEXT_CHUNK" metadata_field = "AMAZON_BEDROCK_METADATA" } } } } resource "aws_bedrockagent_data_source" "s3" { name = "s3-documents" knowledge_base_id = aws_bedrockagent_knowledge_base.main.id data_deletion_policy = "RETAIN" data_source_configuration { type = "S3" s3_configuration { bucket_arn = aws_s3_bucket.documents.arn } } vector_ingestion_configuration { chunking_configuration { chunking_strategy = "FIXED_SIZE" fixed_size_chunking_configuration { max_tokens = 512 overlap_percentage = 20 } } } }

Chunking Strategy: Why Fixed-Size at 512 Tokens

Bedrock Knowledge Bases support three chunking strategies:

  • Default chunking: Bedrock decides. Uses ~300 token chunks with automatic overlap. Fine for getting started, terrible for control.
  • Fixed-size chunking: You set the chunk size and overlap. Predictable, debuggable, and works well for mixed document types.
  • Hierarchical chunking: Parent and child chunks. Better semantic coherence, but more complex to tune and debug.
  • Semantic chunking: Splits on topic boundaries using embedding similarity. Best results in theory, but slower ingestion and harder to predict costs.

I chose fixed-size at 512 tokens with 20% overlap for three reasons:

  1. Predictable cost. 2,400 documents at ~2,000 tokens each = ~4,800 chunks. Easy to estimate embedding costs.
  2. Debuggable retrieval. When an answer is wrong, I can trace it to a specific chunk. With semantic chunking, chunk boundaries are non-deterministic.
  3. Good enough. After testing all four strategies, fixed-size at 512 tokens produced the best balance of answer quality and retrieval precision for our document types (markdown, HTML, PDF).

The 20% overlap (102 tokens) ensures that if a concept spans a chunk boundary, at least part of it appears in both chunks. Without overlap, you get answers that start mid-sentence.

Triggering Ingestion

After deploying the infrastructure and uploading documents to S3, trigger the initial sync:

#!/bin/bash # scripts/trigger_ingestion.sh KNOWLEDGE_BASE_ID=$(terraform output -raw knowledge_base_id) DATA_SOURCE_ID=$(terraform output -raw data_source_id) echo "Starting ingestion job..." INGESTION_JOB=$(aws bedrock-agent start-ingestion-job \ --knowledge-base-id "$KNOWLEDGE_BASE_ID" \ --data-source-id "$DATA_SOURCE_ID" \ --output json) JOB_ID=$(echo "$INGESTION_JOB" | jq -r '.ingestionJob.ingestionJobId') echo "Job ID: $JOB_ID" # Poll until complete while true; do STATUS=$(aws bedrock-agent get-ingestion-job \ --knowledge-base-id "$KNOWLEDGE_BASE_ID" \ --data-source-id "$DATA_SOURCE_ID" \ --ingestion-job-id "$JOB_ID" \ --query 'ingestionJob.status' \ --output text) echo "Status: $STATUS" if [ "$STATUS" = "COMPLETE" ] || [ "$STATUS" = "FAILED" ]; then break fi sleep 10 done # Show statistics aws bedrock-agent get-ingestion-job \ --knowledge-base-id "$KNOWLEDGE_BASE_ID" \ --data-source-id "$DATA_SOURCE_ID" \ --ingestion-job-id "$JOB_ID" \ --query 'ingestionJob.statistics'

For 2,400 documents, initial ingestion takes about 15 minutes. Subsequent incremental syncs process only changed/new files and finish in under a minute.

Query API with Lambda

The query layer is a Lambda function behind API Gateway. It takes a natural language question, calls Bedrock's RetrieveAndGenerate API, and returns a structured response with citations.

Lambda Function 

# lambda/query_handler.py  import json import os import logging import boto3 logger = logging.getLogger() logger.setLevel(logging.INFO) bedrock_agent = boto3.client("bedrock-agent-runtime") KNOWLEDGE_BASE_ID = os.environ["KNOWLEDGE_BASE_ID"] MODEL_ARN = os.environ.get( "MODEL_ARN", "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0", ) SYSTEM_PROMPT = """You are an internal knowledge assistant for an engineering team. Answer questions using ONLY the provided search results. If the search results don't contain enough information to answer the question, say so clearly — do not make up information. Rules: - Cite your sources by referencing the document title and section - If multiple documents provide relevant information, synthesize them - For code examples, include the full snippet from the source document - If the question is ambiguous, ask for clarification - Keep answers concise but complete""" def handler(event, context): try: body = json.loads(event.get("body", "{}")) question = body.get("question", "").strip() if not question: return response(400, {"error": "Missing 'question' in request body"}) if len(question) > 1000: return response(400, {"error": "Question exceeds 1000 character limit"}) logger.info(f"Query: {question[:100]}...") result = bedrock_agent.retrieve_and_generate( input={"text": question}, retrieveAndGenerateConfiguration={ "type": "KNOWLEDGE_BASE", "knowledgeBaseConfiguration": { "knowledgeBaseId": KNOWLEDGE_BASE_ID, "modelArn": MODEL_ARN, "generationConfiguration": { "promptTemplate": { "textPromptTemplate": ( f"{SYSTEM_PROMPT}\n\n" "Search results:\n$search_results$\n\n" "Question: $query$\n\n" "Answer:" ) } }, "retrievalConfiguration": { "vectorSearchConfiguration": { "numberOfResults": 5, "overrideSearchType": "HYBRID", } }, }, }, ) answer = result["output"]["text"] citations = [] for citation in result.get("citations", []): for ref in citation.get("retrievedReferences", []): location = ref.get("location", {}) s3_uri = location.get("s3Location", {}).get("uri", "") content_text = ref.get("content", {}).get("text", "")[:200] citations.append( { "source": s3_uri.split("/")[-1] if s3_uri Source：https://dev.to/garrett_yan/building-a-production-rag-pipeline-on-aws-bedrock-knowledge-bases-opensearch-serverless-and-1gpd
My favorites
PreviousHow to Select a Chatbot API for Invoice Quality Tradeoffs — Portable Cost AnalysisNextA Demand-Driven Decision Support Platform for EcoFit Smart Bottle

Related posts

Convert PNG to JPEG or WebP in the Browser: What Happens to Transparency

Convert PNG to JPEG or WebP in the Browser: What Happens to Transparency

dev.to
How I Built a Privacy-Focused Loan Calculator with Pure JavaScript and Chart.js

How I Built a Privacy-Focused Loan Calculator with Pure JavaScript and Chart.js

dev.to
Game Development for Startups: From MVP to Launch

Game Development for Startups: From MVP to Launch

dev.to
Integrate App Event Logging Without Advanced Infrastructure — A Rollback-Safe Checkout API

Integrate App Event Logging Without Advanced Infrastructure — A Rollback-Safe Checkout API

dev.to
Password Reset Deliveries: 3 Receipt Template Controls for Email Domain Reputation

Password Reset Deliveries: 3 Receipt Template Controls for Email Domain Reputation

dev.to
Base64 in the Browser: UTF-8 Bytes, Padding and a Round Trip You Can Check

Base64 in the Browser: UTF-8 Bytes, Padding and a Round Trip You Can Check

dev.to