AI & ML
System One models Are Not Better Rerankers
arXiv cs.IR0 views
arXiv:2610.32318v1 Announce Type: cross Abstract: System One models, or decision models, answer a question about an input with a calibrated probability instead of generated text. Within weeks of the first release, in September 2026, they were used as rerankers. We test whether the decision fine-tuning behind them makes a better reranker than plain reranker fine-tuning of the same model. For six Qwen backbones from 0.6B to 27B parameters and ten TREC DL and BEIR collections, we compare the untuned model, asked zero-shot, with its decision fine-tunes and its reranker fine-tunes, two of which we train ourselves. In our work, we draw the conclusion that decision fine-tuning never makes the better reranker. From 0.6B to 9B, reranker fine-tuning wins on every backbone by 5 to 29 points nDCG@10, also on zero-shot collections, and at 27B the two tie. Up to 9B, decision fine-tuning adds nothing over simply asking the untuned model. Moreover, we find that the calibration advantage of decision models depends on the question. Asked whether a passage answers the query, they are better calibrated than the published reranker fine-tunes from 4B on. Asked a slightly different question from prior work, they lose this advantage. Lastly, calibration does not make up for a worse ranking: up to 9B, a reranker fine-tune keeps the better passages at a fixed threshold. If you need a reranker, fine-tune a reranker.
Source:https://arxiv.org/abs/2610.32318PreviousMining Frames: Association Rule Mining as an Unsupervised Framework for News Framing DetectionNextEvery Edge Needs a Quote: Building a Quote-Grounded Knowledge Graph of Healthcare Software Adoption with Language-Model Agents
Related posts
Most voted crypto projects on Coinvote, Oct 26 to Nov 1, 2026
0dev.to Data
Bluesky This Week — trending topics & AI brand mentions (2026-11-02)
0dev.to Data
Gemini 4 Carbon e DeepSeek V4.1 Flash Cyber: a Guerra Cibernética Autônoma e o Topo do Artificial Analysis
0dev.to Machine Learning
What can an AI agent do for you every day?
0dev.to Machine Learning
Mon modèle de paris semblait excellent sur 3 671 prédictions. Il n'a jamais battu le bookmaker.
0dev.to Machine Learning
HR manager LinkedIn About section examples you can adapt
0dev.to Career