Backend
A Jury, Not an Electorate: Label-Free Reliability Estimation for Ensembles of Language Models
arXiv cs.MA0 views
arXiv:2610.21085v1 Announce Type: cross Abstract: An ensemble of language models is a jury, not an electorate. The answers are testimony from witnesses of unknown reliability, and the question is not what the majority said but who is credible. Most ensemble methods assign each expert a fixed reputation, often learned from labels. We ask whether it can instead be inferred from the pattern of agreement and disagreement among the experts. We model the experts as a jury: each question has a latent true answer, each expert reports it with unknown reliability, and those reliabilities are estimated by Expectation Maximisation (EM) from unlabelled answer configurations alone. We show that its decision rule reduces to a log-odds weighted linear score over the experts' answer distributions. What changes is not how the jury votes, but how it learns whom to trust. On a 1560-question benchmark, the jury, with nine experts of up to 3B parameters, performs comparably to the same fusion rule with label-fitted weights, showing no statistically significant advantage from supervision. Instead, it outperforms the best single model with matched self-consistency and the strongest label-free alternative on the full benchmark.
Source:https://arxiv.org/abs/2610.21085PreviousAthanor: An Agent-Native System for Autonomous Post-Training ResearchNextDiviner: Agentic RCA with Human-Governed Adaptation for Large-Scale Infrastructure
Related posts
I Had a Perfect Sprint Board. 18 AI Subagents Still Wrecked the Repo.
0dev.to GitHub
What Is Attribution? AI Music Needs a Trail Back to the People Who Made the Work
0dev.to News
A Free Google Alerts Alternative: Monitoring Global News With the GDELT DOC API
0dev.to News
Butter Scraped Over Too Much Bread
0dev.to Discuss
From Six Seven to Six Zeros to What Next?
0dev.to Discuss
When a document assistant refuses: quota, context, and the missing model call
0dev.to C#