The proof of concept worked in my account. The real test was whether the customer's team could build it again in theirs.

A customer evaluating Amazon OpenSearch Service as a vector store asked for two things: proof that it could serve 300 queries per second against 4,096-dimension vectors with server-side latency under 50 milliseconds, and an easy path to load those vectors. I built a proof of concept in my own account that loaded 10 million synthetic 4,096-dimension vectors from Amazon S3 through Amazon OpenSearch Ingestion and benchmarked it. The numbers came in well under the target. Then I hit the harder problem, which every team that builds a POC eventually hits: the people who take it to production weren't in the room while I built it. This post covers what I built, what the load taught me, and how a coding agent turned the whole thing into two markdown files the customer could use to rebuild it on their side.

The ask: throughput, latency, and a loading path

Most vector search POCs start with a latency number. This customer needed 300 queries per second at k=500 with server-side latency under 50 milliseconds. Large k matters here, because every shard returns its own 500 candidates and the coordinator merges them. The cost of a query grows with the number of shards it touches, so the shard layout is part of the latency answer.

The loading path mattered just as much. Their embeddings already lived in files, and they didn't want to write and run a custom bulk loader. I proposed S3 to OpenSearch Ingestion to OpenSearch Service. An ingestion pipeline scans the bucket once, parses each line into a document, and writes it to the domain over HTTPS with SigV4. Nobody has to operate loader hosts, and the pipeline scales its OpenSearch Compute Units (OCUs) on its own.

What I built

The whole stack is a single self-contained AWS CloudFormation template, synthesized from AWS CDK in Python. Anyone can deploy it directly without CDK, bootstrapping, or an asset bucket. It creates an OpenSearch Service domain on OpenSearch 3.7 with r8gd.8xlarge.search data nodes, three dedicated cluster manager nodes, and zone awareness across three Availability Zones. The domain runs inside a VPC with fine-grained access control and encryption in transit and at rest.

The vector index uses Faiss HNSW in on_disk mode at 8x compression. Four-bit quantized vectors and the graph live in memory, and full-precision vectors stay on local NVMe for rescoring if recall needs it. GPU acceleration builds the HNSW graphs for large segments, including the post-load force merge, on service-managed GPUs. The data nodes keep their CPU for indexing and search.

The ingestion pipeline creates its own index template on start, so the index comes up with the right mapping on the first bulk write. Failed documents land in a dead-letter bucket. A small post-load script stops the pipeline, force merges to one segment per shard, adds the replica, waits for green health, and warms the graphs.

The coding agent and I worked out the sizing from three rules. Shard size came first: each primary shard should hold 50 to 100 GB, because recovery and relocation slow down above that. At more than 100 million 4,096-dimension vectors, an index copy runs to a couple of terabytes, which allows anywhere from about 23 to 45 primaries. We picked the fewest that stayed comfortably under 100 GB, 30 primaries. At k=500, every primary returns 500 candidates per query, so each extra shard adds work to every search. Node count came next. A query touches one copy of every primary, so each data node handles query rate × primaries ÷ nodes shard searches per second. With 15 data nodes, that's 300 × 30 ÷ 15, or 600 shard searches per second per node. If each shard search takes about 10 ms, that keeps a 32-vCPU node at roughly 19 percent busy at peak, which leaves room for p99 spikes and for losing a node. The node count also had to be a multiple of three, one set per Availability Zone, and divide the 60 shard copies evenly. Memory and disk fit easily at that size, so CPU and latency set the node count, not capacity.

For the benchmark I generated 10 million synthetic 4,096-dimension vectors and loaded them into three primary shards on six data nodes. That made each shard about the same size as a production shard. To put the same load on each node that production would, the benchmark has to scale the query rate by the shards-per-node ratio: 300 × (30 ÷ 15) ÷ (3 ÷ 6) works out to 1,200 queries per second. An open-loop load generator ramped from 50 up to 1,400 queries per second, past that mark. At k=500 with rescoring off, server-side latency held at a p50 of 9 ms and a p99 of 15 ms across the ramp, with zero errors. Production fans out to more shards, so I tell teams to expect production p50 to land near the benchmark's p90. Even then, the result clears the 50 ms target with room to spare.

What the load taught me

The first load hit several problems. Each one taught me something the customer's team would need. Load throughput peaked when the OpenSearch Ingestion pipeline reached about 26 OCUs, then fell as the pipeline kept scaling. Bulk latency climbed past 15 seconds, and the domain started rejecting writes. With zero replicas during the load, only the nodes holding primaries take writes, and their indexing memory filled up. The fix was to cap the pipeline at two to three OCUs per write node and cut the bulk size from 20 MiB to 5 MiB.

Under that back-pressure, the pipeline also re-read about 40 percent of the S3 objects. Large files didn't finish before another OCU claimed them. Setting a document ID field turned those re-reads into harmless overwrites instead of duplicates. Keeping each source file in S3 under about 50 MB stopped most of the re-reads. That limit applies to the files the pipeline scans, separate from the 5 MiB size of each bulk request it sends to the domain. Heap pressure that stayed high after the load came down with a single refresh, which became the first step of the post-load script.

None of this was visible in the design on paper. All of it would bite the customer's team on their first load if nobody wrote it down.

Why POC handoffs are hard

The person who builds a POC knows why every setting is there. The team that takes it to production, often SREs or a platform group, sees the settings without the reasons. They also work against different constraints: their own VPC and subnets, their own IAM model, their own S3 bucket and encryption keys, and data that may not match the synthetic set. A template alone doesn't tell them which values they can change, which ones are fixed once the index exists, or what to check before they deploy.

The usual handoff is a meeting, a slide deck, and a repository link. The reasons fade within a week. When something breaks during the production load, the team ends up rediscovering the same lessons I already paid for.

README and RECREATE: two files, two readers

I had built the POC with a coding agent, so I asked that agent to write two markdown files from the work we had done together. Each file has a different reader.

The README is for people. It explains the architecture, every CloudFormation parameter with its default and a sensible range, the sizing math behind the node and shard counts, the options I considered and rejected, and the lessons from the test load. It closes with a section titled "What we still don't know, and how to adjust." Each row there is an assumption, how to check it, what to change if it's wrong, and which file or parameter to edit.

The RECREATE file is for a coding agent. It tells the agent what to build and in what order: the project layout, every parameter with its constraints, the verbatim assets, the validation to run after each change, and the done criteria. It also sets rules. The agent must collect every input before writing code, must not deploy or delete AWS resources without explicit human approval for that specific action, and must not put account IDs or credentials into files that will be shared.

The most useful part of RECREATE is a set of fourteen decision points. Each one names when it applies, the question, how to determine the answer, the options with their consequences, and a default. Is the source data NDJSON or Parquet? Is there a unique ID field? Are metadata fields used for filtering? Can the subnets reach S3? Is the document count acceptable after the load? At each gate, the agent stops, reports what it found, and asks the human to choose. Those choices are recorded in a decisions file that later changes can cite. The unknowns that would have derailed a handoff become explicit questions with a way to answer them.

Handing the agent the keys, and the brakes

This approach works because the reader on the other side is now often an agent too. The customer's engineer opens the repository with a coding agent, points it at RECREATE, and works through the decision points with their own account and data. The agent does the mechanical rebuilding, and the engineer makes the calls that depend on their environment. The README keeps the reasoning available for anyone who wants to know why.

The same pattern fits any POC you plan to hand off. Build it with an agent, and while the context is still fresh, have the agent write down what it built for a human and how to rebuild it for another agent. Put the unknowns in a table with checks and defaults. Make the approval gates explicit, because the next team's account is not yours.

Bringing it together

The customer wanted proof that Amazon OpenSearch Service could serve 300 queries per second under 50 milliseconds, and an easy path to load their vectors. S3 to OpenSearch Ingestion to OpenSearch Service, with on_disk 8x compression and GPU-accelerated graph builds, delivered a 9 ms p50 on a production-sized shard layout. The benchmark answered the customer's question. The README and RECREATE files are what let their team build the same thing in their own account. If you build POCs for a living, try ending the next one by asking your agent to write both files.