AI & ML
Beyond Dimensionality: Domain Shift and Catastrophic Forgetting in Single- vs. Multi-Vector Retrieval
arXiv cs.IR0 views
arXiv:2603.29519v2 Announce Type: replace Abstract: Recent work by Weller et al. introduced the naturalistic benchmark LIMIT and showed that widely used single-vector embedding models suffer substantial degradation in retrieval quality, attributing this primarily to limited embedding dimensionality. In our work, we revisit this explanation and argue that dimensionality alone is unlikely to account for the observed failures; in fact, results of Alon et al. imply that $2k+1$ dimensions suffice for top-$k$ retrieval over a fixed set of queries and documents. Motivated by this, we investigate alternative drivers of poor performance, including tokenization, linguistic structure, and domain shift. Our empirical analysis shows that tokenization artifacts and linguistic similarity yield only modest improvements, whereas domain shift and misalignment between embedding similarity and task-specific relevance are the dominant factors. While finetuning partially mitigates these issues and improves recall, it introduces catastrophic forgetting: finetuning single-vector models on LIMIT-like datasets reduces their MS MARCO Recall@10 by more than half, whereas multi-vector models forget much less. We observe the same pattern when finetuning on the real-world HotpotQA dataset, and between the single-vector and multi-vector heads of Jina-v4, which share one backbone.
Source:https://arxiv.org/abs/2603.29519PreviousCan Explanations Improve Recommendations? Evidence from Prediction-Informed ExplanationsNextThe Missing Complement: State-Conditioned Minimal Sufficient Evidence for Coding Agents
Related posts
Reading project health off a download page — faster than star count
0dev.to
Best time trackers for developers, for code and for AI
0dev.to

Serving Qwen3.5-0.8B with a 0.016% weight edit: measure the loops, pin the sampling on the server
0dev.to
Design Tokens for AI: Extract a Website's Real Colours, Fonts, Shadows and Icons So an LLM Builds On Brand
0dev.to
Trying to get our Arms around AI efficiency
0Stack Overflow Blog
Why an Embedded Game Fails on Some Phones, and How to Debug It
0dev.to