AI & ML
The Dice Roll Method: A Standardized Protocol for Repeated-Query Auditing of Large Language Model Brand Recommendations
arXiv cs.IR0 views
arXiv:2609.04047v2 Announce Type: replace Abstract: Background: Researchers increasingly use repeated identical prompts to audit stochastic variation in large language model (LLM) brand recommendations, yet no standardized protocol exists for setting iteration counts, selecting stability metrics, or establishing reliability thresholds. Objective: We formalize the Dice Roll Method as a reusable protocol for repeated-query auditing of LLM brand recommendations, grounded in a generative model of temperature-scaled nucleus sampling. Methods: Total response variance is decomposed into sampling, prompt-phrasing, run-to-run, and model-version components. The stack: a negative-binomial mixed model with iterations as repeated measures; Cliff's delta as the distribution-free effect size; dependence-preserving bootstrap; simulation-based power; a generalizability-theory decomposition; drift diagnostics on pinned snapshots. We reanalyse five brand-recommendation auditing studies: approximately 190,000 observations, 270+ brands, 6 languages, iteration counts 5 to 40. Results: The recomputed D-study supports no fixed tiers. Across 250 distinct questions (S5), G is 0.96 at five runs and 1 to 2 runs reach 0.80, also within each model. For five versions of one question (S1 Valentine's subset, Gemini) it is 0.27 (95% CI 0.00 to 0.52), which fixes no run count. Runs follow from a pilot: n = 4 x residual / cell variance. Simulated single-cell power for a large Cliff's delta is 0.15 to 0.19 at five runs and 0.8 at 15 to 25. The four metric families (count, set, embedding, PASOR) are complementary. A pre-registered external validation on three independent corpora reproduces the D-study prediction in 37 of 39 cells with no failures; fixed tiers do not transfer. Conclusion: The protocol gives repeated-query auditing of LLM brand recommendations a statistically principled footing under the conditional, non-Gaussian structure of real autoregressive generation.
Source:https://arxiv.org/abs/2609.04047PreviousQASP: Query-Adaptive Robust Vector Search PolicyNextWhen RAG Doesn't Help: LLMs for Plausible Material Prediction of Vehicle Components
Related posts
Lavoro su piattaforma e algoritmi: cosa cambia con la direttiva europea per chi progetta il software
0dev.to
LifeVal Benchmark Targets Everyday Personal Assistant Tasks Shaped by ChatGPT User Feedback
0dev.to
Oracle files to sell $20B in new stock, bonds to finance AI buildout
0Hacker News
7.9% to 48.1%: LLM rewrote Qwen weights, no training
0dev.to
Spousal Open Work Permits in Canada: Who Qualifies in 2026 and What to Prepare
0dev.to
Reading ingredient lists from phone photos: a crop bug, a model bench, and why the AI never gives the verdict
0dev.to