AI & ML
Probabilistic Relevance Decisions for Semantic Retrieval
arXiv cs.IR0 views
arXiv:2611.08014v1 Announce Type: new Abstract: Dense similarity supplies reusable corpus representations, while probabilistic relevance decisions offer query-dependent document scoring. We compare dense-only BGE-M3 with TypeSafe Jev Noul and OpenAI Decisions Predicate scores on the official BEIR SciFact test benchmark. The study evaluates 300 official test queries using identical positive-enriched 100-document pools and full-corpus dense retrieval followed by reranking at candidate depths of 50 and 100. Unlike the controlled pools, neither dense prefix receives positive injection. Jev has the highest observed Recall@5 and nDCG@10 under the evaluated conditions. At depth 50, full-corpus nDCG@10 is 0.6413 for BGE-M3, 0.7545 with OpenAI and 0.7805 with Jev. Extending depth from 50 to 100 increases known-positive candidate recall but produces small, uncertain final-effectiveness changes. All four paired-bootstrap depth intervals include zero. Unique-condition API scoring cost approximately doubles, while compatible cached decisions reduce incremental expenditure. OpenAI returns many exact-zero scores, and secondary exact-tie randomization reveals ranking sensitivity without replacing deterministic primary results. Top-100 latency mixes cached and new execution. Historical hybrid latency is a saved-component sum under different execution environments and timer boundaries. These exploratory, unadjusted analyses support relevance decisions as a complement to dense retrieval within this protocol, without establishing general superiority, probability calibration or equivalence between depths.
Source:https://arxiv.org/abs/2611.08014PreviousMEMOIR: A Training-Free Memory Lifecycle for Long-Horizon LLM AgentsNextDomain-specific information retrieval
Related posts
Hypergraph Generation via Structured Stochastic Diffusion
0arXiv stat.CO
Sparse Probabilistic Richardson Extrapolation
0arXiv stat.CO
Dynamic Bayesian Predictive Stacking via Markovian Spatiotemporal Propagation
0arXiv stat.CO
Coherent forecast combination for linearly constrained multiple time series
0arXiv stat.CO
New optimisations for exact, scalable 2D Gaussian process inference, with applications to astronomical time-series
0arXiv stat.CO
A tractable transition function for the Wright-Fisher diffusion with selection via gamma duality
0arXiv stat.CO