Python vs Go vs Rust for a Vector Search API: the Language Explained 5% of the Throughput Variance

Most "Python vs Go vs Rust" API benchmarks I have seen measure the HTTP layer with a trivial handler. A semantic-search API spends much of its time somewhere else: inside the vector store. So I built the same /search endpoint nine times, three languages times three stores, and ran all of it on tight hardware: one 8 GiB Mac mini, closer to an edge box or a small VM than to a database server. The question was not only which stack is fastest, but which one degrades first when memory and threads run short, and why.

Thesis: for a semantic-search API, the store you pick decides most of your throughput. The language decides much less.

Antithesis I expected to find: Python's JSON parsing on the same cores as the search should drag FastAPI down at high load, so the language should matter most exactly where it hurts.

What the data says, for this design on this machine: the store explains about two thirds of the variance in throughput, the language about 5% (method in Result 1). Which store was fastest depended on data size and k, not on the framework. And the two most consequential things I found were not speed rankings at all: a Go binding that crashes at a kernel thread limit, and an 8 GiB machine whose free memory moved pgvector's throughput by 4 to 7x.

The setup

The 3 x 3 factorial. One API contract, implemented three times:

Each one is backed by three stores:

All three indexes use cosine distance and the same build parameters, HNSW M = 16 and ef_construction = 128. Only the search-time knob (ef) differs per store, and it is set by recall calibration (below), not by hand.

DuckDB is pinned to 1.5.6 in all three languages because the Rust duckdb crate has no 2.0 build yet, and one engine version across languages matters more than the newest one.

Deployment per language, fixed before any data. Go and Rust run as one process each. Python runs 6 uvicorn workers in front of pgvector (one per core the server was meant to use) and 1 worker in front of the embedded stores, because every extra worker would load its own copy of the index into 8 GiB. The Python handler parses the raw body with orjson and checks the vector length, like the Go and Rust handlers; it does not use a Pydantic model. pgvector connection pools hold 24 connections in total per server, and DuckDB gets 6 threads. An ablation swapped the Python process model (1 vs 6 workers for pgvector, 1 vs 2 for the embedded stores; 100k, recall 0.95, k 10, 3 runs each): the median MST was the same ladder step in both arms for all three stores (3,200, 1,600 and 400 rps). Postgres runs with shared_buffers 1 GB and effective_cache_size 3 GB, sized for 8 GiB.

Host. One Mac mini: Apple M2 (4 performance + 4 efficiency cores), 8 GiB RAM, macOS 26.5.2, on AC power, kept awake with caffeinate. That constraint is what this study is about. No CPU pinning, because macOS has no affinity API on Apple silicon. The API server, Postgres and the load generator share all 8 cores. Everything is installed natively, no Docker.

Data. Cohere Wikipedia embeddings (768 dimensions) as packaged by VectorDBBench: 100k vectors, and a seeded 300k subset of the 1M set. Exact top-100 ground truth by brute force.

Equal recall, not equal parameters. Comparing stores at the same ef is misleading, because the same number buys different recall in different engines. Each store's search knob was calibrated on 200 queries to hit a target recall of 0.90, 0.95 or 0.99, and recall@k was then scored on held-out queries from the actual HTTP responses. Held-out recall landed within -0.011 to +0.044 of target in all 36 store, scale, k and target combinations (the +0.044 is pgvector at k 100, where the calibration grid starts at ef 100 so that every query can return 100 hits); the median deviation per store is +0.002 for DuckDB, +0.004 for pgvector and +0.013 for LanceDB (for example 0.971 at a 0.95 target at 300k). Overshooting costs throughput, so LanceDB's overshoot works slightly against it. The one undershoot worth naming is pgvector at 100k, k 10, target 0.90: ef 16 reached 0.901 on the calibration queries and 0.889 on the held-out ones. Those two configurations (ids only and ids plus text) hold the study's highest MST, 6,400 rps, so read that number as recall 0.89.

The metric. Max sustainable throughput (MST): the highest offered rate at which a 30 second open-loop step (vegeta) sends at least 98% of requests with at most 0.1% errors and p99 at or below 100 ms. Each run starts a fresh server process, warms up for 20 s at 50 rps (discarded), then climbs a doubling ladder on the grid 25, 50, 100 ... rps, with 3 s between steps, and stops at the first step that fails. The centre cell starts at 25 and adds three bisection steps. Elsewhere the ladder starts about four doublings below the centre-cell MST of the same language and store, and walks down if that first step already fails, so MST has 2x resolution and "the same" means "the same ladder step". A cell's value is the median MST of its 5 runs.

The design. 24 configurations (2 scales x 3 recall targets x top-k 10 or 100 x ids only or ids plus ~1 KB of text per hit) x 9 cells x 5 replicates, plus calibration, index builds and ablations: 1,543 runs over 88 hours. Run order follows a Williams design drawn at random per block (seed 20261022), so no cell always runs after the same neighbour. Warm cache everywhere except a cold-start ablation.

Result 1: the store dominates

Method. For each of the 24 configurations separately, I take the 45 runs (9 cells x 5 runs; a few cells have 4 after exclusions) and split the total sum of squares of log MST into store, language, store x language interaction and residual, the classic two-way ANOVA decomposition. A share is that term's sum of squares divided by the total. The table shows the median of each share across the 24 configurations. Medians are taken term by term, so the four rows do not add up to 100%.

Fact. The median variance shares are:

The store explains more than the language in 23 of 24 configurations. The exception is a tie: 100k vectors, recall 0.99, k 10, ids only, where each explains 50%. Across configurations the store share ranges from 33% to 100% and the language share from 0% to 50%, with 22 of 24 configurations at 14% or less. Because MST sits on a doubling ladder, log MST takes few distinct values, so these shares are coarse. They describe this design (these stores, scales, recall targets and k values on this host), not every possible one.

At low load the result is even more one-sided. For p50 latency at about a quarter of each cell's MST, the store share is 79% and the language share 5% (main run).

Interpretation. Look at either panel of the chart and ask what moves a bar more: changing its colour (the store) or moving it sideways (the language). Within a language, the best store beats the worst by a median of 4x for Python and Rust and 2x for Go. Within a store, the best language beats the worst by a median of 2x for DuckDB and pgvector and 1x for LanceDB. On a 2x ladder, that is one step for the language and one to two steps for the store, and the store's steps show up in almost every configuration.

Why the language mattered so little here: the HTTP layer was not the bottleneck. With a stub store that returns fixed ids, every language sustains at least 12,800 rps (median of 5 runs, JSON in, k 100 plus text out included). No run with a real store in the whole study got past 6,400. FastAPI parsing a 768-float JSON body is not what limits a semantic-search service on this machine.

What the language layer actually does. Here is the pgvector /search path in each of the three APIs, copied from the benchmarked source and trimmed only with .... Each handler parses the JSON body, checks the vector length, makes one store call and encodes the result as JSON. The full sources are in the repo: https://github.com/firfircelik/vector-api-bench/tree/main/api

FastAPI (api/python/app.py):

@app.post("/search") async def search(req: Request): body = orjson.loads(await req.body()) vec = np.asarray(body["vector"], dtype=np.float32) if vec.shape != (DIM,): return Response(status_code=400) return jresp(await state["store"].search(vec, int(body.get("k", 10)), bool(body.get("doc", False)))) class PgVectorStore: ... async def search(self, vec, k, doc): async with self.pool.connection() as c: rows = await (await c.execute(self.sql_doc if doc else self.sql, (vec, k))).fetchall() r = {"ids": [x[0] for x in rows], "scores": [float(x[1]) for x in rows]} ... 

Go net/http (api/go/main.go; write sets the content type and calls json.NewEncoder(w).Encode):

mux.HandleFunc("POST /search", func(w http.ResponseWriter, r *http.Request) { var q struct { ... } if err := json.NewDecoder(r.Body).Decode(&q); err != nil || len(q.Vector) != dim { http.Error(w, "bad request", 400) return } ... res, err := st.Search(r.Context(), q.Vector, q.K, q.Doc) ... write(w, res) }) func (s *pgStore) Search(ctx context.Context, v []float32, k int, doc bool) (*resp, error) { ... rows, err := s.pool.Query(ctx, q, pgv.NewVector(v), k) ... 

Rust axum (api/rust/src/main.rs):

async fn search(State(a): S, Json(q): Json<Req>) -> Result<Json<Resp>, (StatusCode, String)> { if q.vector.len() != a.dim { return Err((StatusCode::BAD_REQUEST, "dim".into())); } a.search(q.vector, q.k, q.doc).await.map(Json).map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e)) } ... Store::Pg { pool, sql, sql_doc } => { let c = pool.get().await.map_err(|e| e.to_string())?; let st = c.prepare_cached(if doc { sql_doc } else { sql }).await.map_err(|e| e.to_string())?; let rows = c.query(&st, &[&pgvector::Vector::from(v), &k]).await.map_err(|e| e.to_string())?; Ok(Resp { ids: rows.iter().map(|r| r.get(0)).collect(), scores: rows.iter().map(|r| r.get(1)).collect(), docs: if doc { Some(rows.iter().map(|r| r.get(2)).collect()) } else { None } }) } 

That is the whole language layer: parse, one store call, encode. A wrapper this thin is consistent with the language explaining about 5% of the throughput variance, and with the stub-store floor of at least 12,800 rps in every language. Where the language does show up (Result 3), it is in the store client underneath that one call, not in these handlers.

Result 2: which store was fastest depended on the data, not the framework

Fact. Geometric mean of the cell medians across the 23 non-centre configurations:

Counting the fastest store per configuration and language (69 cells), with ties: LanceDB 51, pgvector 48, DuckDB 2. Without ties: LanceDB 20, pgvector 18, DuckDB 0. Ties are common (31 of 69 cells) because the ladder doubles.

The map splits cleanly:

• 100k vectors, k 10: pgvector is fastest or tied in 18 of 18 cells (geo-mean 2,743 vs LanceDB 1,796 rps), up to 6,400 rps at the 0.90 target (held-out recall 0.89, see the setup).
• 300k vectors, k 100: LanceDB is fastest or tied in 18 of 18 cells (geo-mean 544 rps vs DuckDB 224 and pgvector 121).
• 300k vectors, k 10: close, LanceDB 1,330 vs pgvector 1,158 rps.
• DuckDB VSS never wins a cell on its own. Its two shared wins are 400 rps ties at 300k, k 100.

Interpretation. pgvector is the fastest store here at 100k. At 300k it falls behind LanceDB, and at 300k with k 100 it is last (121 rps vs DuckDB 224). My reading is memory: at 300k the pgvector index is 1.23 GB, larger than the 1 GB shared_buffers I could afford on an 8 GiB machine, so part of it has to come from the OS page cache, which competes with everything else on the machine, and at k 100 each query plausibly touches much more of the graph and the heap. The memory-pressure section supports this: at 300k, pgvector's latency tracked free memory (r = -0.80), DuckDB's and LanceDB's much less. LanceDB lost 37% of its geo-mean from 100k to 300k, pgvector 82%. This is one host with 8 GiB, so I would not carry the 300k ranking to a server with 64 GiB without measuring it there.

Result 3: where the language does show up

The language effect is real in two places, and in both it is the store client, not the web framework.

DuckDB: the binding's cost per query. Without HTTP at all (6 concurrent direct client calls, 300k, recall 0.95), DuckDB answers 844 queries per second from Go, 615 from Rust and 573 from Python, at 7.0, 11.5 and 11.6 ms of client CPU per query. Through the API at the centre cell the CPU per request repeats the order: Go 3.7, Rust 5.6, Python 6.6 ms. Same DuckDB 1.5.6 underneath, same thread setting. The data does not tell me why the Go binding is cheaper.

pgvector: Python is one step lower in the fastest configurations. At 100k, recall 0.90, k 10, Python/pgvector reaches 3,200 rps where Go and Rust reach 6,400. My reading: that is where the API process does the most work per second and pgvector's server-side cost per query is lowest, so the API layer's own cost becomes visible.

LanceDB: the language makes no difference to the median (best/worst ratio 1.0x), until Go hits the wall in the next section.

The Go + LanceDB thread ceiling: the stack that broke

This one is not a throughput ranking. It is a failure mode, and it is the finding I would most want to know before shipping.

Fact. In the clean centre rerun (5 runs, 300k, recall 0.95, k 10):

• Through 1,600 rps the Go/LanceDB server is healthy: p99 about 2 ms, 0 errors, and a flat 216 to 328 OS threads.
• In the 3,200 rps step the process dies within about 4 seconds, in all 5 runs. In the four runs where the 2 second sampler caught the climb, the thread count reached 1,133 to 2,048 within about 2 seconds; the fifth died between samples, and its crash dump shows Go thread ids up to 1,501.
• server.log shows the same sequence 5 times: runtime/cgo: pthread_create failed: Resource temporarily unavailable, then SIGABRT. Four of the Go crash dumps are complete. They list 1,517 to 1,765 goroutines, and all but one goroutine per dump is in state syscall inside the same cgo function, _Cfunc_simple_lancedb_table_select_query_ipc. The fifth dump is interleaved with other output but shows the same function. The highest Go thread id in the dumps is between 1,501 and 1,766.
• The Mac's per-process thread limit is kern.num_taskthreads = 2048. Two of the five runs were sampled at exactly 2,048 threads.
• The three bisection steps after the crash (2,262, 1,902 and 1,744 rps) measured a dead port: 100% instant errors. The Go/LanceDB MST of 1,600 is therefore a lower bound. The true ceiling is somewhere between 1,600 and 3,200 rps.
• In the main run, 86% of Go/LanceDB runs (n = 117) ended their first saturated step with more than 90% errors. Every other cell had 0% such runs; they degraded with rising latency and timeouts instead.

The mechanism. The Go runtime documentation states it directly: GOMAXPROCS limits the threads executing Go code, but "there is no limit to the number of threads that can be blocked in system calls on behalf of Go code". A cgo call counts as one. The Go server calls lancedb-go's synchronous Execute once per request, and net/http starts a goroutine per request with no admission control. While a search runs inside the Rust core, its goroutine holds an OS thread, and the scheduler starts new threads for the goroutines that are still runnable.

That is harmless while the store keeps up. Measured without HTTP (300k, recall 0.95, 6 concurrent callers), LanceDB answers about 2,500 to 2,800 queries per second from any language. Offer 3,200 rps and the backlog grows. By Little's law, in-flight requests equal arrival rate times time in system; at 3,200 rps it takes only about half a second of queueing to have 1,500 calls in flight, and in Go each one is an OS thread. Go's own guard, debug.SetMaxThreads, defaults to 10,000 threads, far above the macOS limit of 2,048, so the kernel says no first and the runtime aborts the process.

Rust and Python use LanceDB's async API. A waiting request there is a future, not a thread, so they queue and slow down instead of crashing. Rust/LanceDB sustains 2,466 rps at the same cell, with bisection.

Interpretation. This is not "Go is slow". Through 1,600 rps the Go/LanceDB server was healthy, with p99 about 2 ms and 216 to 328 OS threads. During the 3,200 rps step the thread count climbed to the macOS limit of kern.num_taskthreads = 2048, pthread_create failed with EAGAIN and the process aborted. What is missing is a concurrency bound: unbounded goroutines making blocking cgo calls into the store turn every queued request into an OS thread, and on this host the thread limit runs out before anything else does. That makes MST 1,600 a lower bound; the true ceiling is somewhere between 1,600 and 3,200 rps, and on this host it was set by the thread limit, reached through this binding's blocking calls, rather than by the speed of Go code. The usual remedy is to cap in-flight store calls and let the excess wait or be rejected. I did not implement or measure that here, so I make no claim about what it would buy.

Two caveats. This is the lancedb-go cgo binding on Darwin; on a host with a higher per-process thread limit I would expect the same mechanism at a different threshold, which I did not measure. And the binding is young (its last tag is v0.1.2; I vendored a 2026 commit and rebuilt its core against LanceDB 0.40.0).

The memory-pressure lesson: how 8 GiB nearly wrote the wrong article

My first centre-cell numbers said pgvector was the slowest store in all three languages. They were low by a factor of 4 to 7.

Fact. Centre cell (300k, recall 0.95, k 10, ids), pgvector, median MST of 5 runs (rerun Python: 4, see below):

How I noticed. Three things did not fit.

1. In the first run, one replicate per language was already fast (Python 2,262, Rust 2,690, Go 1,037 rps) while the other four sat between 128 and 672 rps. A store that is bimodal on identical inputs is telling you about its environment.
2. Across 45 pgvector 300k runs, p50 latency at 100 rps correlated with the memory available before the step at r = -0.80. For DuckDB it was -0.14 and for LanceDB -0.35.
3. An A/A control (the same Rust/pgvector cell, 58 times between Oct 23 and Oct 27) split into a fast mode (26 runs, p50 1.9 ms at 388 rps) and a slow mode (32 runs, p50 8.6 ms).

In the main run, available memory before a step had a median of 2.7 GiB and a minimum of 0.7 GiB, and swap peaked at 10.9 GiB, while I was also using the Mac during the day.

How I tested it. I reran only the centre pgvector cells, plus the Go/LanceDB cell as a control, on the evening of Oct 27 (runs started between 18:46 and 20:50 TR), with a monitor sampling free pages, compressor pages, page-ins, page-outs, swap and per-process threads every 2 seconds. Nineteen of the 20 runs started with at most 0.34 GiB of swap, and all 14 pgvector runs among them landed in the fast mode. The remaining run, the very first one (Python/pgvector), started with 2.6 GiB of swap left over from the day and measured 258 rps. I excluded it, so the Python rerun median is over 4 runs. The Go/LanceDB control cell in the same rerun reproduced its original 1,600 rps.

What it changed. pgvector went from last to roughly level with LanceDB at the centre cell: Rust 2,933 vs 2,466 rps, Python 2,466 vs 2,466, and Go 2,690 against a Go/LanceDB value of 1,600 that is only the crash-bounded lower bound from the previous section. The clean pgvector rerun ran on a different, quieter evening than the LanceDB values, so this comparison crosses days, and I read it as level, not as a pgvector win. None of the main conclusions rests on the centre cell: they rest on the variance shares and on the scale flip in Result 2.

Then I checked the rest of pgvector. A one-replicate screen of all 69 pgvector cells moved 21 of them by more than 25%, 20 of them at 300k. I reran those 21 at 5 replicates. Their medians replace the originals in every table above. The pgvector geo-mean went from 1,039 to 816 rps; LanceDB and DuckDB did not change because they were not rerun.

Interpretation. On a machine where the index does not fit in the database's own buffer, throughput depends on how much page cache the rest of the system leaves you. That is not a benchmark artefact to hide; it is the production question "how much RAM does this index need" showing up in the numbers. But it means a single afternoon's run can be off by 5x without anything looking broken.

What I could not pin down

• pgvector at 300k with k 100 is noisy at the bottom of the ladder. These cells now sit at 25 to 400 rps. The first run had several at 400 to 800 that I could not reproduce. I tested one obvious explanation, that the reruns started their ladder at 100 rps instead of 25 and so hit a cold first step. Four configurations, 3 runs per arm, starts alternated: the per-configuration medians were 200 vs 100, 200 vs 200, 100 vs 100 and 200 vs 400 rps (start at 25 vs start at 100). The geo-mean of those medians is 168 rps either way (150 vs 168 over all 12 runs per arm). The hypothesis is refuted, and the higher first-run values remain unexplained. I report the rerun values.
• Some reruns overlapped daytime use. During the 5-replicate pgvector rerun, 10 of 105 runs showed swap-outs or page-in rates more than 4x their overnight level. Excluding them moves the pgvector geo-mean from 816 to 823 rps. With a stricter alignment of monitor samples to load steps, 14 runs are flagged and the geo-mean is 805. None of these changes a conclusion.
• The centre comparison crosses days. The clean pgvector rerun ran on Oct 27; the LanceDB and DuckDB centre values come from Oct 23. The Go/LanceDB control in the rerun had the same MST but lower latency (p50 at 100 rps 2.6 vs 3.9 ms), which suggests the host was quieter on the evening of Oct 27 than on Oct 23. That is why I call the centre cell level rather than a pgvector win.

What this tells you about constrained hosts, and what it doesn't

This study ran on one 8 GiB Mac mini whose 8 cores were shared by the API, the store and the load generator. That is closer to an edge box or a small VM than to a database server, and that is the point: on tight hardware the interesting question is which stack degrades first, and how.

Specific to this machine, do not carry over:

• The absolute MST values. Every rps figure above belongs to this host. On a bigger server with more cores, more memory and a higher thread limit I would expect them to be higher, but I have not measured that.
• The exact Go/LanceDB ceiling. 1,600 rps is the last healthy step before the macOS limit of 2,048 threads per process stopped the process in the 3,200 rps step. It is a property of this host and this binding, not a speed of Go or of LanceDB.
• The 300k pgvector numbers. A 1.23 GB index against 1 GB of shared_buffers on 8 GiB is a memory-bound setup. With room to cache the index, the 300k picture may look different, and I have not measured that.

What I expect to hold on other constrained hosts (expected, not shown here):

• The store dominates the language. Store 69%, language 5%, store ahead in 23 of 24 configurations, and a stub store at 12,800 rps or more in every language: the HTTP layer was far from the limit.
• The ranking flips with scale. pgvector led at 100k vectors (1,831 vs 1,295 rps), LanceDB at 300k (817 vs 338). Which side of the flip you are on depends on whether the index fits the memory you have.
• Failure modes come from the host's limits. Thread exhaustion killed Go/LanceDB, and memory pressure moved pgvector by 4 to 7x. A larger machine moves these limits further out, but the mechanisms stay: unbounded blocking calls consume OS threads, and an index that outgrows its buffer depends on whatever page cache is left.

These are expectations from one host. Treat them as hypotheses to check on your own hardware, not as results.

Threats to validity, and what this does not show

Each point below is a limit of the design, with what the data says about it.

• One host. Every number comes from one 8 GiB Mac mini (Apple M2, 4 performance + 4 efficiency cores, macOS 26.5.2). Nothing is pooled across machines. Absolute rps values belong to this host.
• macOS, not Linux. Most production APIs run on Linux. The Go/LanceDB ceiling is tied to the macOS limit kern.num_taskthreads = 2048, and memory behaviour differs (macOS compresses and swaps memory before anything is killed). I did not measure Linux, so I make no Linux claim.
• Load generator on the same host. vegeta, the API server and Postgres share all 8 cores, unpinned, because macOS has no CPU affinity API on Apple silicon. Load-generator CPU is recorded per step. No real-store step and no rerun step pushed it above 1.5 cores; only the stub-store steps at 25,600 rps did, which is why the stub floor is reported as "at least 12,800". The client still competes with the server for CPU, which is part of what a constrained host means.
• Index parameters. All three stores build HNSW with the same M = 16 and ef_construction = 128 (LanceDB: IVF_HNSW_FLAT with one partition, its closest equivalent). The search knob is calibrated per store to the same recall target, not set equal. Other build parameters, quantization or per-store tuning beyond this were not explored.
• Recall matching. Held-out recall from the actual HTTP responses landed within -0.011 to +0.044 of target everywhere; the largest undershoot (pgvector, 100k, k 10, 0.90 target, 0.889) is named in the setup.
• Warm-up and cache state. Every run starts a fresh server process and discards 20 s at 50 rps. All reported numbers are warm. The cold-start ablation could not drop the macOS page cache without root, so it measured a fresh process, not a cold disk, and I do not report it as cold.
• Repetitions. 5 runs per cell in a randomised Williams order (seed 20261022), cell value = median. Run-to-run residual is reported as its own variance share (median 12%). An A/A control of one cell repeated 58 times between Oct 23 and Oct 27 is one of the three signals that exposed the memory-pressure modes.
• Ladder resolution. Outside the centre cell MST is a doubling ladder, so most ratios are 1x, 2x or 4x by construction, and a 1.5x real difference can show up as 1x or 2x.
• Python deployment. Python ran 6 workers for pgvector and 1 for the embedded stores, decided before the data. Swapping that choice did not change the median MST in the ablation (see the setup).
• Memory pressure. The main run had a median of 2.7 GiB available before a step, and pgvector at 300k was sensitive to it (r = -0.80 between free memory before the step and p50 at 100 rps, 45 runs). The affected cells were rerun: the centre cell on a quieter evening (judged by the Go/LanceDB control's lower latency), and the 21 pgvector cells that moved by more than 25% at 5 replicates. Flagging daytime-contaminated reruns moves the pgvector geo-mean between 805 and 823 rps against 816 reported.
• Exclusions. A run counts as an outlier when its MST is at most a quarter of its configuration's median (or its first-step p99 is 5x worse, for short and cold modes). It is excluded only with a host-side cause: a host-gate timeout, swap of 6 GiB or more, swap growth of 0.5 GiB or more within a step, a crash, or the window when I was actively using the Mac. That gave 63 outliers and 45 exclusions (24 of 1,035 CORE runs, 2 of 45 centre runs). The rule was written after seeing the data, so here is its effect: exclusions change 3 of 216 cell medians and none of the store-level numbers above.
• Invalid steps. The runner flagged 190 of 7,859 steps as invalid. All 190 were overload drains (a 10 s client timeout stretched the step), so they count as saturated. Zero steps were host stalls.
• Statistics. The variance shares are descriptive. The protocol also planned permutation tests and bootstrap confidence intervals; I have not run them, so I make no significance claims.
• Protocol deviations. The repo's protocol lists every deviation from the base plan (no CPU pinning, no 1M scale because a 1M DuckDB process would need about 10 GiB, a 2-worker instead of 6-worker Python arm for embedded stores in the ablation) and the runner settings as actually run.
• A runner bug. The runner records a step's started timestamp when the step ends. Anything that aligns host-monitor samples to steps has to shift by one step. The thread chart and the stricter contamination check use the corrected alignment.
• Reproduction. Versions are pinned in the repo, with the scripts, per-run summaries, the rerun's raw data and host monitor, the crash markers, and the commands that recompute every number and regenerate every chart in this article (README, section "Reproduce the article numbers and charts"), and a claim-by-claim map from each number to its source file (analysis/claims.md): https://github.com/firfircelik/vector-api-bench

The takeaway

If you are choosing a stack for a semantic-search API, my reading of this data is: choose the store first and size...