When RAG Doesn't Help: LLMs for Plausible Material Prediction of Vehicle Components

arXiv:2609.18437v2 Announce Type: replace Abstract: In this work, we explore whether LLMs can predict and explain plausible materials for vehicle components such as brake discs or fuel injectors without extensive fine-tuning. We compare a standard generative LLM baseline, single-pass Retrieval-Augmented Generation (RAG), and an iterative Chain-of-Verification (CoVe) variant across two LLMs, using a domain-filtered Wikipedia corpus that proved helpful in prior work. As no gold standard exists, we develop and release a reusable web-based annotation tool supporting crucial functions for structured domain-expert judgments, such as highlighting, ratings and ranking. Expert agreement is reasonable for plausibility, lower for ratings. LLM-based generation compares favorably to prior work. RAG and CoVe do not improve on it and are partly rated worse by experts. Contrary to prior work, Wikipedia-based retrieval does not help in our setup.