Backend
Per-Query Recall Guarantees for Approximate Nearest Neighbor Search in Dynamic Vector Databases
arXiv cs.DB0 views
arXiv:2611.07579v1 Announce Type: new Abstract: Vector search runs as an operator in database systems, where a missed neighbor can make a filter, join, or aggregate wrong, yet approximate nearest neighbor (ANN) indexes are tuned only for an average recall target. In our measurements, indexes tuned to an average recall of 0.99 left 5-19% of in-distribution queries and, tuned without a query log, up to 99.6% of real out-of-distribution queries below this target, and the misses grew under updates. We present PQR-Search, a k-ANN search method that guarantees $\Pr[\mathrm{Recall}(q)<\tau]\le\delta$ for every query. The probability is only over a random orthogonal transformation inside the index, so no distributional assumption is needed, only that queries and updates do not depend on it; the guarantee holds after insertions and deletions without retuning. PQR-Search stores 1-bit codes of rotated residuals with respect to IVF centroids and prunes with exact quantiles of their error distribution. A formula predicting the fraction of vectors that pass the code test, with no data-fitted constants, chooses index parameters and maintenance operations without affecting the guarantee. On 20 datasets of up to 100 million vectors, failure rates stayed at most $\delta$ in every audit, including all 33 steps of the official NeurIPS'23 streaming runbook, whereas CTR, a conformal guarantee averaged over queries, left up to 16.4% of queries below the target. Where HNSW tuned to an average recall of 0.99 met that target, PQR-Search had 2.5x its single-query latency and 1.3x lower batch throughput, yet that HNSW still left 3-29% of queries below 0.99. PQR-Search was faster than FAISS exhaustive search in every static setting (26.4x geometric mean in batch; 69-154x at 100M scale), faster than IVF with nprobe chosen afterwards for every test query in all 52 main settings, and 27-76x faster at one thread than PUFFINN, which also gives per-query guarantees.
Source:https://arxiv.org/abs/2611.07579PreviousSelective Retrieval Routing for Multi-Violation Knowledge Graph RepairNextCleaning or Recalling? Knowledge Dependence in LLM-based Data Cleaning [Experiment, Analysis & Benchmark]
Related posts
Stochastic Derivative Estimation for Discontinuous Sample Performances: A Leibniz Integration Perspective
0arXiv stat.CO
Efficiently Estimating Cycle Statistics on Graphs via Spanning Trees
0arXiv stat.CO
I Audited 26 Episodes of a 59,000-Episode Open Robot Dataset. The Data Is Fine — the Metadata Isn't.
0dev.to Data
Is Google Maps accurate for closed businesses? We checked 400 places
0dev.to Data
WARN Act and Your FSA: What Happens to Your Flexible Spending Account After a Layoff
0dev.to News
The Week of November 27: 5,991 Workers, a Thanksgiving-Week Layoff Wave
0dev.to News