AI & ML
Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks
arXiv cs.CR0 views
arXiv:2611.06257v1 Announce Type: new Abstract: Open-weight language models that fit on a single consumer GPU are becoming capable on a growing range of tasks, and they matter because they can run locally without sending the code to a hosted service. Vulnerability reproduction, turning a vulnerability description into an input that crashes the vulnerable build, is a natural target for them, since it is an automated analysis task closer to test generation than to code comprehension and it runs against a build that must be executed rather than read. In order to assess whether current models can perform it, we evaluate twelve configurations of local models, spanning three generations, two architectures and three quantization levels, on ten CyberGym vulnerability-reproduction tasks, three runs each under one harness, and release the full artefact. We show that every configuration, down to the 4B model, solves most of the six easier tasks, and configurations separate only on the remaining four. For the harder tasks, the 27B model is required. Quantization did not change what the models solved in any matched pair, but it changed how fast they got there, and the mixture of experts reached the same scores in less wall clock. We also examine the agent trajectories. Almost every run we examined, including those that fail, names the vulnerable function and states the correct hypothesis, and what separates a solved task from an unsolved one is whether that hypothesis becomes a working input.
Source:https://arxiv.org/abs/2611.06257PreviousExplainable Anomaly Detection in 5G Networks through Continuous Time GraphsNextBeyond Identity Creation: Bilinear Sybil Cost in Identities and Time
Related posts
TROPT: An Open Framework for Unifying and Advancing Discrete Text Optimization
0arXiv cs.CR
UpSafe$^\circ$C: Upcycling for Controllable Safety in Large Language Models
0arXiv cs.CR
Systematic Evaluation of Adversarial Jailbreaking Attacks and Defenses Across Seven Large Language Models
0arXiv cs.CR
Practice Makes (Im)Perfect: A Look Back at Benchmarking Practices for Microarchitectural Side-Channel Attacks
0arXiv cs.CR
Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Traces
0arXiv cs.CR
Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code
0arXiv cs.CR