AI & ML
Learning from Failures: A Failure-Driven Prompt Refinement for LLM-Based Vulnerability Analysis
arXiv cs.SE0 views
arXiv:2610.08405v1 Announce Type: new Abstract: Large Language Models have emerged as promising tools for software vulnerability analysis, but their effectiveness depends heavily on prompt design. Existing research primarily compares prompting strategies using aggregate performance metrics, providing limited insight into why models fail or how prompts can be improved systematically. We propose Failure-Driven Prompt Refinement (FDPR), a methodology that analyzes recurring model failures to guide evidence-based prompt refinement. Using the Damn Vulnerable Java Application (DVJA), we identify recurring failure modes, including false positives, false negatives, unsupported reasoning, and CWE misclassification, and translate them into targeted prompt refinements. We then evaluate the resulting prompt on the Juliet Test Suite and perform cross-model validation to assess generalizability. The results show that failure-driven refinement improves the reliability of LLM-based vulnerability analysis while yielding reusable prompt design principles. More broadly, this work demonstrates that recurring model failures provide a principled foundation for prompt engineering, enabling the systematic development of more reliable LLM-based vulnerability analysis systems.
Source:https://arxiv.org/abs/2610.08405PreviousNewer and Bigger, but Safer? A Longitudinal Study of the Functionality-Security Gap in LLM-Generated CodeNextRAPO-Sol: Retrieval-Augmented Preference Optimization for Repository-Level Solidity Code Generation
Related posts
‘A perfect scientist, a perfect mum’: researchers mourn two biologists’ deaths in Spanish train crash
0Nature News

Apple says memory costs and Siri AI will weigh on holiday quarter profit margins
09to5Mac

Making Local LLMs Usable on 8GB GPUs
0HackerNoon

Educational Byte: What Are Synthetic Tokens in Crypto Networks?
0HackerNoon
Your AI Assistant Is Waiting Around - An AI Agent Goes to Work
0HackerNoon
Your AI Agent's Memory Needs Two Clocks
0HackerNoon