Last Secure Code Benchmark: Is the Code You Vibe Secure?
Authors: Tianyi Wu, Chao Ge, Peng Chen, Fengji Zhang, Yongxing Wu, Yuchong Xie, Jian Li, Chaoyuan Peng, Kaikai Zhang, Dong Huang, Mingzhe Du, Yukai Huang, Jiaheng Zhang, Terry Yue Zhuo, See-Kiong Ng (Qwen)
Abstract
AI coding agents now write much of the code that ships, yet whether that code is secure remains unclear. We introduce the Last Secure Code Benchmark (LSCBench), a comprehensive evaluation of secure code generation by coding agents. Built from 150 real vulnerabilities, its 450 tasks span seven widely used languages (Java, C, C++, JavaScript, Python, PHP, and Go), 24 CWEs including 21 of the 2025 CWE Top 25 and 8 of the Top 10 KEV weaknesses, more than any benchmark we compare with, and three vibe-coding scenarios: filling in a function, writing a file, and building a repository’s source tree from scratch. Every task asks only for functionality, through a specification that never mentions security, and hidden functional and security tests grade the result. Across six frontier agents, 73.6% to 91.3% of tasks pass functionally, yet even the strongest, Claude Fable 5, produces code that is both functional and secure for only 25.1% of tasks, and the others for 16.4% to 24.7%. The best model only passes 12% of the vulnerability scenarios across all vibe coding setting, posing strong concern on the security of the code written by vibe coding.
1 Introduction
Coding agents now write a growing share of everyday software. In this workflow, often called vibe coding, the user states what the software should do and leaves the implementation to the agent. Adoption is broad: in one survey 75% of respondents use it and 90% of those are satisfied (Sarkar and Drosos, 2025), beginners in coding are its keenest users (Fawzy et al., 2025), and Anthropic reports using it in production (Anthropic, 2025). The implementation also involves security decisions that the user often does not state, such as how the code handles untrusted input, protects sensitive data, and enforces access boundaries. The agent makes these decisions implicitly, and users, who check whether the functionality works and rarely inspect the code, leave them unexamined. Reports already describe agent-written code that stored API keys in plaintext or shipped broken authentication, some of it exploited (Archibald and Kaplan, 2025). A functional check cannot catch these flaws: a document importer can handle ordinary archives correctly while letting a crafted archive write outside its directory. As agents take on more development work, whether their code is both functional and secure becomes an essential question.
Development requests also differ in scope: an agent may be asked to add a function to an existing codebase, implement a feature that needs a whole file, or build application code from scratch, with different context, amounts of code to write, and integration decisions in each case. Several benchmarks evaluate the security of generated code (Peng et al., 2025; Yang et al., 2024; Vero et al., 2025; Zhao et al., 2025), but each falls short on at least one of three dimensions: the scope of generated code, the languages covered, or the vulnerability categories reached.
How can we determine whether coding agents consistently generate secure code across all of these vibe coding settings?
| Benchmark | Tasks | Task context | Multi-file | Avg. Gen. lines | Languages | Top 25 (2025) | KEV 10 (2025) |
|---|---|---|---|---|---|---|---|
| CWEval | 119 | File | × | 10.0 | C/C++/Go/JS/Py | 10/25 | 5/10 |
| SALLM | 100 | File | × | 12.9 | Py | 11/25 | 6/10 |
| SecCodePLT | 1,345 | Function | × | 8.1 | Py | 13/25 | 5/10 |
| Asleep | 71 | File | × | 19.6 | C/Py | 13/25 | 7/10 |
| BaxBench | 392 | None | ✓ | — | Go/JS/PHP/Py/Ruby/Rust | 9/25 | 4/10 |
| A.S.E | 120 | Repo | ✓ | 35.8 | PHP/Py/Go/JS/Java | 4/25 | 3/10 |
| SecureAgentBench | 105 | Repo | ✓ | 42.5 | C/C++ | 7/25 | 3/10 |
| SUSVIBES | 186 | Repo | ✓ | 175.0 | Py | 16/25 | 6/10 |
| LSCBench | 450 | Func./File/Repo | ✓ | 155/1782/4120 | Java/C/C++/JS/Py/PHP/Go | 21/25 | 8/10 |
Table 1: Comparison of secure code generation benchmarks. LSCBench’s 450 tasks are its 150 scenarios at three granularities, and its three generation-line averages correspond to Function/File/Repository tasks.
We introduce the Last Secure Code Benchmark (LSCBench) to fill this final piece. LSCBench consists of 150 real-world vulnerability scenarios, each branched into tasks at three granularities of code generation, function, file, and repository, for 450 tasks in all. Every task states functional requirements only and stays neutral about the vulnerability and its mitigation, matching the vibe-coding setting in which the developer says what the software should do and the agent must recognize and handle the security risk on its own. Implementations are graded by separate functional and security test suites, validated on both the vulnerable and the fixed code of the scenario. The 450 tasks span seven languages and 24 Common Weakness Enumeration (CWE) categories, including 21 of the 2025 MITRE CWE Top 25 (MITRE, 2025b) and 8 of the 2025 CWE Top 10 KEV Weaknesses (MITRE, 2025a). LSCBench thus evaluates secure code generation across multiple languages, crucial vulnerability categories, and scopes of code generation at once, which no previous benchmark does; Table 1 compares them.
We evaluated six frontier models and found that the code they write is rarely secure, even though it is usually functionally correct. Functional pass rates range from 73.6% to 91.3%, but the best rate of passing both the functional and the security suite is 25.1%, and the best model passes both at all three granularities on only 12% of the scenarios. Our contributions are threefold:
- A comprehensive agentic benchmark evaluating secure code generation. LSCBench consists of 150 vulnerability scenarios and 450 tasks at three granularities across seven languages and the top vulnerability categories, with separate functional and security tests.
- A detailed construction cookbook to support crafting hack-free executable secure code generation tasks. We document every step, from cutting a scenario at three granularities to specification drafting, review, environment sweeps, and control runs, so that each task is well-defined, executable end-to-end, and hack-free.
- Extensive experiments and in-depth analysis of current SOTA models’ capabilities. We evaluate six frontier agents on all 450 tasks. They complete most tasks but secure few of them; a larger scope lowers functional completion but not security; more turns do not buy security; weaknesses that hinge on an unstated policy are almost never secured; and securing a vulnerability at one granularity rarely carries over to the other two.
2 Related Work
Coding agents and code functionality benchmarks.
Code generation is usually judged by whether the code works, from function-level suites such as HumanEval, MBPP, and APPS (Chen et al., 2021; Austin et al., 2021; Hendrycks et al., 2021) to repository-level tasks such as SWE-bench and NL2Repo-Bench (Jimenez et al., 2023; Ding et al., 2025), and agents improve quickly on them (Pan et al., 2024; Xie et al., 2025). None asks whether the code is secure, although early studies found about 40% of security-relevant completions vulnerable (Pearce et al., 2021).
Secure code generation.
Early security benchmarks prompt a model for a single snippet with a known weakness class: SecurityEval, CodeLMSec, SALLM, CWEval, SeCodePLT, CODEGUARD+, and SafeGenBench (Siddiq and Santos, 2022; Hajipour et al., 2023; Siddiq and Santos, 2023; Peng et al., 2025; Yang et al., 2024). Recent ones work at the repository level: BaxBench builds backend applications graded by expert exploits, A.S.E gives models repository context from projects with documented CVEs, and SecureAgentBench and SUSVIBES turn vulnerability-fixing commits into agent tasks with functional and security tests (Vero et al., 2025; Lian et al., 2025; Dilgren et al., 2025; Zhao et al., 2025). Training methods range from fine-tuning on curated commits to reinforcement learning with security rewards (He and Vechev, 2023; He et al., 2024; Xu et al., 2024; Liu et al., 2025; Wu et al., 2026; Yang et al., 2026). Because manual review does not scale, LLM judges can be fooled, and static analyzers match patterns rather than behavior (Bhatt et al., 2023; Lian et al., 2025), LSCBench runs security tests against the running program. Recently attention has shifted to the offensive side, with CyberGym and ExploitGym testing whether an agent can reproduce a vulnerability or write an exploit (Wang et al., 2025; Wang et al., 2026), while whether an agent writes secure code has received less attention. LSCBench takes up this question and differs from the benchmarks above in three ways (Table 1): it evaluates agents on realistic projects across seven languages, covers more of the crucial CWEs, and spans every vibe-coding setting.
Figure 1: Construction pipeline of LSCBench. Scenarios come from CVE-Factory and expert-crafted cases; tests are written and both code states validated. Stage 1 builds Function, File, and Repo task environments from the vulnerable repository. Stage 2 drafts the specifications for the three granularities in one multi-turn session without the reference patch. Stage 3 has a fix-visible review agent and five security experts correct the drafted materials, gates every task with the reference and empty solutions, yielding 150 scenarios and 450 tasks.
3 Last Secure Code Benchmark
LSCBench is a benchmark of functional implementation tasks derived from real vulnerabilities. Each scenario comes from a real-world project with a known vulnerability, its reference fix, functional tests, and a security test that replays a proof-of-concept attack against the running program and checks whether the vulnerable behavior occurs. Each scenario is cut at three granularities: Function tasks fill the masked code regions where the vulnerability lived, File tasks implement the source files that contained them, and Repository tasks rebuild the declared source tree from scratch. Every task gives the agent functional specifications that are neutral about security, which measures whether an agent writes secure code in the ordinary vibe-coding setting.
3.1 Benchmark Construction
Construction has four steps (Figure 1): scenario sourcing and selection, then three stages that turn each scenario into tasks (Stage 1), draft their specifications (Stage 2), and review, gate, and select them (Stage 3). Appendix B gives the details.
Scenario sourcing and selection.
Vulnerability scenarios come from two sources. CVE-Factory (Luo et al., 2026) provides reproducible CVE task environments in Python, JavaScript, Go, PHP, C, and C++, each with a vulnerable environment, a reference patch, and tests. Java is under 5% of CVE-Factory, so five security experts with at least three years of industry experience hand-crafted 100 Java application scenarios from vulnerabilities in their companies’ codebases, each with a vulnerable project, a reference patch, functional tests, and a security test. From CVE-Factory we keep the tasks whose CWE is on the MITRE CWE Top 25 and that can be cut at all three granularities, and rank them by a difficulty score, the mean of three percentile (pct) ranks over the eligible pool P,
D(c)=\tfrac{1}{3}\bigl[\mathrm{pct}_{P}(\mathrm{files}(c))+\mathrm{pct}_{P}(\mathrm{lines}(c))+\mathrm{pct}_{P}(\mathrm{repo}(c))\bigr], where files(c), lines(c), and repo(c) are the files the fix touches, the lines it changes, and the repository size in lines of code. The score favors scenarios hard in two ways: a fix spanning more files and lines is harder to get right, and a larger repository means more code to reconstruct around it, testing whether the agent loses sight of security while getting the functionality to work. We then draw the hardest CVEs per language while covering every CWE class, 226 in all. The five experts found the upstream CVE-Factory tests unfit as graders: many inspect source text rather than behavior, so sometimes a candidate solution can pass their security tests by adding specific comments. Experts therefore rewrote both suites from scratch for every selected CVE using domain knowledge with the help of an agent (Qwen3.8-Max), so that every security test drives a proof-of-concept attack through the external surface and observes behavior. A rewritten suite is accepted only if the functional suite passes and the security test fails on the vulnerable code, and both pass on the fixed code. 315/326 scenarios with validated tests enter Stage 1.
Stage 1: Task construction and validation.
Each scenario becomes three executable task environments with different implementation scopes. The Function granularity masks, for every changed range of the patch, the smallest complete function that contains it, so that a hole is a unit of the program rather than the shape of the diff and can be described end-to-end. The File granularity removes the files the reference patch touches. The Repository granularity empties a declared source scope: many real-world projects are too large to rebuild from nothing, so we remove the whole repository when it is small enough (<8,000 lines of code) and otherwise the files under the same subtree as the patched files, until 8,000 lines are removed. The rest of the repository stays as substrate, so the task keeps the long-range structure of a real codebase, its callers, build system, and integration constraints, while the amount to write stays feasible. An environment sweep then removes everything that would hand a solver the answer, from compiled outputs, installed copies, and package caches to the Git history, the hidden suites, the patch, and revealing comments, and the task image is flattened before release. Finally each granularity is graded by the real task runner with two controls: the reference solution must pass both suites, and a narrated candidate, the vulnerable code with the patch’s lines pasted in as comments, must fail the security suite, which catches a suite that matches text instead of observing behavior. 267/315 scenarios leave Stage 1 with all three granularities.
Stage 2: Specification drafting.
A drafting agent (Qwen3.8-Max) then writes the specification for each task in an isolated container that holds the vulnerable project and both test suites but no reference patch. It first writes an interface summary that states how the tests and retained callers reach the vulnerable project, without stating expected outcomes, and then, in the same multi-turn session, drafts one specification per removed unit, widest first: the repository-level task, then one per removed file, then one per masked region, so that every narrower document is written with the wider context established and stays consistent with it; we examined a sample of 30 scenarios to confirm that the specifications are consistent across granularities. The writer must ground every statement in the code it reads, may not copy a test’s inputs or expected outputs, and may give no security guidance in either direction.
Stage 3: Review, fix, and final selection.
A review agent (Qwen3.8-Max), given the reference patch, checks every clause of the drafts for two defects: functional descriptions requiring the vulnerable implementation or leaking the fix, and rewrites them to hold for both the vulnerable and the fixed code, so they describe only functionality. The five experts then review every clause against both suites and the patch, checking that the required interfaces are complete and reachable and nothing leaks the fix or pushes toward the vulnerable design. 254 scenarios with all three granularities reach the final gate, where their 762 tasks run under Harbor (Harbor Framework Team, 2026) with two controls: the oracle (reference patch applied) must pass both suites and the nop (nothing changed) must pass neither. 247/254 scenarios (164 CVE and 83 Java) pass, and the experts select the final 150 by balancing languages, covering the 2025 CWE Top 25 and Top 10 KEV lists, and prioritizing each language’s most severe weakness classes (Table 7); all 450 tasks pass both controls.
3.2 Benchmark Overview
LSCBench contains 150 vulnerability scenarios, 115 from public CVEs and 35 expert-crafted Java application scenarios, in Java, C, C++, JavaScript, Python, PHP, and Go. We cover 24 CWE categories, including 21 of the 2025 MITRE CWE Top 25 and 8 of the 2025 CWE Top 10 KEV Weaknesses (Figure 2), and each yields three tasks, one per granularity, for 450 in total (Table 2).
Figure 2: Composition of the 150 scenarios. (a) Share of scenarios by CWE class: the ten most frequent classes are shown and the remaining 14 are grouped. (b) Share of scenarios by language, with C and C++ combined.
| Mean | Median | Max | ||
|---|---|---|---|---|
| Context | # Lines | 650K | 70K | 37,060K |
| # Files | 3,812 | 507 | 81,768 | |
| Function tasks | # Lines | 155 | 94 | 1,686 |
| # Files | 1.8 | 1 | 10 | |
| # Regions | 2.8 | 2 | 18 | |
| File tasks | # Lines | 1,782 | 480 | 46,423 |
| # Files | 1.8 | 1 | 10 | |
| Repo tasks | # Lines | 4,120 | 2,480 | 46,423 |
| # Files | 13.7 | 10 | 68 | |
| Patch-related | # Lines | 31.3 | 18 | 199 |
| # Files | 1.8 | 1 | 10 | |
| Test cases | # Functional | 11.2 | 11 | 33 |
| # Security | 6.8 | 5 | 43 |
Table 2: Task statistics over the 150 scenarios. Context covers the whole vulnerable repository; the task rows count the hand-written reference code that each granularity withholds, excluding build products that the retained build regenerates.
Figure 3: Agent-visible inputs, reconstruction scopes, and separate verification. Dashed outlines mark withheld code; each granularity is an independent task. Only editable-scope files enter the verifier, where hidden functional and security suites determine joint success.
Every task hands the agent a container with the project in its vulnerable state, the code masked or emptied according to its granularity, its dependencies and toolchain, and scripts that build the project, prepare its environment, and start its service. The container holds neither the tests nor the reference patch, and everything that could reveal the removed code is swept (Section 3.1). Every task carries the same contract: the editable scope, the target tree, the functional contract, the build and run commands, and an interface reachability contract that states where existing callers and the tests reach the code without stating expected outcomes. The granularities differ in what is withheld and in where, and how finely, the functional contract is written. For example, one scenario is built from a command injection in a mount service: the Function task fills the masked method that builds and runs the mount command, the File task rewrites the whole service class, and the Repo task rebuilds the service package from its specification, and in all three the specification says what the command must do and nothing about how the process is spawned. Figure 3 shows the agent-visible inputs at each granularity.
Function.
The target files stay in place and each designated region, a function, a class fragment, or statements from the patch context, is replaced by a <MASKED> placeholder; a task can span several regions across files. The functional contract is inline in a statement of about 250 lines and is the most detailed of the three: for every region it gives the interface, observable behavior, inputs, outputs and errors, side effects, sequencing, and boundary. The agent may edit only the placeholders.
File.
The target files are removed in full, and the functional contract points to one specification per file under /file_specs/, which gives the file’s scope, interface symbol by symbol, observable requirements, dependencies, and the implementer’s freedom; the rest of the project stays immutable.
Repository.
The declared source scope is emptied, and the task statement points to a repository specification at /app/spec.md, the coarsest functional description of the three: the goals, what to build, the environment, the role of each file, and a usage guide for every exported symbol, followed by the same recorded facts as the other granularities.
At every granularity the agent reads the specification, implements it, and may build, run, and test the project, but the functional and security suites stay hidden and run only in the verifier (Section 4).
4 Experiment Setup
Agents and models.
We evaluate six frontier agentic LLMs: Claude Fable 5 (Anthropic, 2026), GPT-5.6-Sol (OpenAI, 2026), GLM-5.3 (GLM-5-Team, 2026), Qwen3.8-Max (Qwen Team, 2026), DeepSeek-V4-Pro (DeepSeek-AI, 2026), and Kimi-K3 (Team, 2026). Five run under Claude Code (v2.1.143) and GPT-5.6-Sol under its native scaffold Codex (v0.154.0), each with the framework’s default system prompt and tool set, its interactive tools disabled so that no human is in the loop, a cap of 500 turns, and no network access, so the agent cannot search for answers. Every model runs at its highest available reasoning effort (Appendix D). Qwen3.8-Max also served as the construction agent that wrote the tests and specifications (Section 3.1); its results sit in the middle of the field on both axes, so we see no sign that authoring the materials advantaged it.
Grading.
We run tasks with Harbor (Harbor Framework Team, 2026) in its separate-verifier mode: when the agent stops, only the editable scope is copied into a verifier container built from the same base image, which rebuilds the project with the frozen script, starts its services, and runs both suites.
Metrics.
For model m, scenario s, and granularity g, let Fm,s,g and Sm,s,g be the Boolean functional and security outcomes of the attempt, counted as failures when missing. Over a task set T we report FuncPass, SecPass, and Func∧Sec, the fractions of tasks whose functional suite passes, whose security suite passes, and that pass both,
\displaystyle\textsc{FuncPass}_{m}(T) \displaystyle=\frac{1}{|T|}\sum_{(s,g)\in T}\mathbf{1}\{F_{m,s,g}=1\},\quad\textsc{SecPass}_{m}(T)=\frac{1}{|T|}\sum_{(s,g)\in T}\mathbf{1}\{S_{m,s,g}=1\}, \displaystyle\textsc{Func}{\wedge}\textsc{Sec}_{m}(T) \displaystyle=\frac{1}{|T|}\sum_{(s,g)\in T}\mathbf{1}\{F_{m,s,g}=1\land S_{m,s,g}=1\}. All three are pass@1: every task is attempted once, as in vibe coding, where the user expects working code from the first attempt. Security alone is not a success measure, since an attempt that implements nothing passes every security test; SecPass only separates the two axes and Func∧Sec is the headline number. Because every scenario appears at all three granularities, we also report each rate at the scenario level, where a scenario counts only if it passes at all three; these rates ask whether a model handles a vulnerability consistently however much of the project it must write.
5 Results and Findings
Figure 4: Main results of six agentic LLMs on LSCBench. Left: share of the 450 tasks passed. Right: share of the 150 scenarios passed at all three granularities. Green is Func∧Sec; the gray part extends to FuncPass, so its visible share is functional but vulnerable; the light track to 100 is not functional. Bold marks the best Func∧Sec per panel; SecPass and the conditional rates are in Table 10 (Appendix E).
Agents Complete Most Tasks but Secure Few of Them.
Figure 4 summarizes the six models on the 450 tasks. Functional completion is high and nearly uniform: five of the six models pass the functional suite on about 90% of tasks, and only DeepSeek-V4-Pro falls clearly behind. Security does not follow: the best model, Claude Fable 5, is both functional and secure on only 25% of tasks, and the others cluster just below. The gap between the two rates is one outcome, an implementation that works and keeps the vulnerability. A user who accepts an implementation once its tests pass would therefore ship the vulnerability most of the time, and nothing in the workflow would warn them, since the functional requirements are met and the code appears usable.
Securing the Same Vulnerability at Every Granularity Is Rare.
The right panel of Figure 4 asks a stricter question: does a model handle a vulnerability at every granularity? Most models complete all three tasks of a scenario more than half of the time, but even the best secures all three for only about 12% of the scenarios. Securing a vulnerability at one granularity is therefore no evidence that the model has understood it completely.
Different Models Secure Complementary Sets of Tasks.
Pooling the six models, at least one secures 198 of the 450 tasks (44.0%), against 113 (25.1%) for the best single model, while only 19 tasks are secured by all six and 62 by exactly one. The two leaders, Claude Fable 5 and GPT-5.6-Sol, share 62 secure tasks but each secures 19 and 18 that no other model does, and the other four together add 36 that neither leader secures. Which vulnerabilities a model avoids thus depends on its own knowledge and behavior, so a second model reviewing the first, or an ensemble, would cover more than any single agent. The rest is headroom: 252 tasks (56.0%), and 57 of the 150 scenarios at every granularity, are secured by no model at all. Additional results are available in Appendix E.
5.1 Results by Granularity
Figure 5: FuncPass, SecPass, and Func∧Sec by granularity. Each line follows one model over the same 150 scenarios; the logo marks the Repo value.
Figure 6: Average turns per task against Func∧Sec over the 450 tasks. A turn is one step of the agent’s trajectory; GPT-5.6-Sol runs under Codex, which counts steps differently.
Figure 5 shows the pass rates at the three granularities. Functional completion falls with scope for every model, by a few points for the strongest and by more than twenty for DeepSeek-V4-Pro. Interestingly, security does not fall with it: Func∧Sec is flat or rising from Function to Repo for every model and rises most for GPT-5.6-Sol, whose Repo implementations are secure nearly a third of the time, and for three models the File granularity is a local minimum, where the agent has neither the original function’s structure to follow nor the freedom to design the module. A possible reason is that an agent building a module from a specification chooses the data flow and the checks itself, whereas one filling a hole inside existing code is invited by the surrounding structure to reproduce the original vulnerable implementation. Larger scope thus trades functional completion for security, and for the strongest models the trade is favorable.
5.2 Turns versus Func∧Sec
More Turns Do Not Buy Security.
Figure 6 places each model by its average turns per task against its Func∧Sec. The model that uses the fewest turns, Claude Fable 5, secures the most tasks, while Qwen3.8-Max, GLM-5.3, and Kimi-K3 spend about twice as many turns and secure fewer. Within a model and a granularity, implementations that pass both suites take about as many turns as those that work but stay vulnerable; only Qwen3.8-Max and GLM-5.3 spend markedly more on their secure Repo solutions. Security, where it appears, is thus a choice made while writing rather than an extra phase of work or verification, which suggests training secure coding agents end to end: implement, verify security with self-written tests, then patch.
5.3 Weakness Types and Languages
Models Secure Local Checks More Often Than Unstated Policies.
We group the 24 CWEs into eight families by what the defense has to do (Table 11 in Appendix E); Figure 8 shows Func∧Sec for each family and each language, per model and pooled over the six, with the families ordered by the pooled rate; the top and the bottom of the order are the same for every model. Memory safety is secured most often: its defenses, bounds, null, and lifetime checks, are local to the code being written, and careful code includes them anyway. Injection and path handling follow, with a wide spread inside injection between OS command injection, which about a third of attempts avoid by passing arguments as a list, and code injection, which almost none avoid. At the bottom are the families whose defense is a policy the specification never states: who may act, which inputs to reject, which data to withhold, and which destinations to refuse; information exposure and server-side request forgery (SSRF) are almost never secured by any model. The pattern points to the ability at work: the vulnerabilities that are often secured are part of writing the code well, such as a bounds check or an escaped shell argument, which a good agent includes unasked; the ones rarely secured are policies the code cannot see, such as which data a response may reveal, and closing them requires reasoning about how the code will be used and attacked once deployed. Current agents thus secure code as a side effect of writing it well, not by anticipating its deployment, and this could be a consequence of the community’s current focus on training agents to pass tests; closing the gap will take supervision that scores the threat model as well as the behavior.
Working PHP and Go Code Is Most Often Vulnerable.
Across languages, C, Python, and JavaScript code is secured most often, at 27% to 30% of tasks, and Java, Go, and PHP least often, at 10% to 18% (Figure 8). Much of this difference comes from which weakness types each language’s scenarios contain: C’s scenarios are almost all memory safety, the family models secure best, whereas the Java, Go, and PHP scenarios are dominated by injection, access control, and file handling, and Java carries most of the SSRF and deserialization scenarios, which are rarely secured in any language.
5.4 Failure Patterns
Figure 7: A functional but vulnerable attempt, end to end. Kimi-K3 rebuilds the removed ImportExportService.java (File granularity, CWE-502, expert-crafted Java scenario) from a specification that requires each archive entry to be “parsed as YAML into a mapping” and leaves the parsing helper to the implementer. It reads eleven context files, writes the 207-line class with new Yaml().load(text), compiles the whole source tree offline, and writes its own smoke test against stubbed mappers, growing it from 27 to 33 checks after finding and fixing a corrupt-archive edge case. All 14 hidden functional tests pass; all four hidden security tests fail, because the default SnakeYAML constructor honors every !!class tag in the uploaded YAML. Claude Fable 5 and GPT-5.6-Sol parse the same entries with SafeConstructor and pass both suites. Quotes are verbatim; “…” marks elisions.
Each Model Fails in Its Own Way.
Our security expert team audited 20 failed trajectories per model, 120 in all, and attributed each failure to its cause. The failures are not random: each model’s failures follow a behavioral pattern, and the patterns differ sharply. Kimi-K3 treats the functional contract as exhaustive: what the contract does not spell out, it does not do, and when its own tests surface a security anomaly it argues itself out of fixing it (“the test is too strict”), so its submissions are functionally thorough with the defenses missing. Figure 7 follows one such submission end to end. GPT-5.6-Sol often reproduces the upstream implementation from memory, replaying the vulnerability along with the rest and occasionally “optimizing” a strict check into a lenient one. DeepSeek-V4-Pro writes its own code but has the largest gaps in defensive basics, such as wrong API versions and validation in name only, and often closes with “it is an environment issue” instead of running the project’s build scripts to verify. Claude Fable 5 reaches for the upstream code shapes it knows best, with the most verbatim replays of the six and the vulnerability preserved with the same fidelity, and it loses further attempts to Anthropic’s content-safety rejection. GLM-5.3 takes the default-allow shortcut whenever the specification is ambiguous or the component complex, shipping engines without class filters and checks that are declared but never called. Qwen3.8-Max writes first and runs later, and its failures are dominated by execution: deadlocks, regex backtracking, and quadratic render paths. Appendix F gives detailed case studies.
5.5 Security-Reminder Ablation
| Model | FuncPass | Func∧Sec |
|---|---|---|
| GPT-5.6-Sol | 87.6 (−1.5) | 46.2 (+21.5) |
| Qwen3.8-Max | 91.8 (+2.5) | 37.6 (+15.6) |
| GLM-5.3 | 87.6 (+0.5) | 38.2 (+16.9) |
| DeepSeek-V4-Pro | 70.4 (−3.2) | 19.1 (+1.1) |
Figure 8: Func∧Sec (%) by weakness family and language, per model and pooled (All); parentheses give scenario counts.
The main results measure what agents produce when the user does not mention that security needs to be considered, the usual vibe-coding setting. A natural follow-up question is whether the vulnerabilities disappear once the user adds a security reminder. We rerun GPT-5.6-Sol, Qwen3.8-Max, GLM-5.3, and DeepSeek-V4-Pro on all 450 tasks with one sentence appended to every task statement: “Please make sure your implementation satisfies the functional requirements while also being secure and free of any plausible vulnerabilities.” Table 8 reports the result: for all but DeepSeek-V4-Pro, the one sentence lifts Func∧Sec by 15 to 21 points while FuncPass barely moves, yet most tasks stay insecure even with the reminder, which suggests that current models still lack secure-coding knowledge even when reminded. Deserialization, path handling, and memory safety reach about 50% Func∧Sec for the three models, but information exposure and server-side request forgery stay near the bottom: a reminder to be secure does not make the agent think about which data to withhold or which destinations to refuse.
6 Conclusion
We introduced the Last Secure Code Benchmark (LSCBench), which, unlike any prior benchmark, evaluates secure code generation by coding agents across seven languages, 24 CWE categories including 21 of the CWE Top 25, and three granularities, from filling a function to rebuilding a source tree, on 450 tasks built from 150 real vulnerabilities, graded by hidden functional and security tests, and documented an executable, hack-free construction cookbook for secure code generation tasks. Six frontier agents write code that is largely functional, yet even the best model produces code that is both functional and secure on only 25% of the tasks, which raises a serious concern about whether their code can be deployed in the real world without careful review. We hope LSCBench helps close this gap by making security visible where vibe coding hides it.
Acknowledgments
We would like to thank the Qwen Team at Alibaba Token Hub (ATH), Alibaba Group, for providing the computational resources and foundation models (Qwen) used in this research.
References
Anthropic (2025) Anthropic Vibe coding in prod — code w/ claude. External Links: Link Cited by: §1.
Anthropic (2026) Anthropic Claude Fable 5 and Claude Mythos 5. External Links: Link Cited by: §4.
Archibald and Kaplan (2025) N. Archibald and C. Kaplan Passing the Security Vibe Check: The Dangers of vibe coding. External Links: Link Cited by: §1.
Austin et al. (2021) J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. J. Cai, M. Terry, Q. V. Le, and C. Sutton Program synthesis with large language models. ArXiv abs/2108.07732. External Links: Link Cited by: §2.
Bhatt et al. (2023) M. Bhatt, S. Chennabasappa, C. Nikolaidis, S. Wan, I. Evtimov, D. Gabi, D. Song, F. Ahmad, C. Aschermann, L. Fontana, S. Frolov, R. P. Giri, D. Kapil, Y. Kozyrakis, D. J. LeBlanc, J. Milazzo, A. Straumann, G. Synnaeve, V. Vontimitta, S. Whitman, and J. Saxe Purple llama cyberseceval: a secure coding benchmark for language models. ArXiv abs/2312.04724. External Links: Link Cited by: §2.
Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. S. Winter, P. Tillet, F. P. Such, D. W. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba Evaluating large language models trained on code. ArXiv abs/2107.03374. External Links: Link Cited by: §2.
DeepSeek-AI (2026) DeepSeek-AI DeepSeek-v4: towards highly efficient million-token context intelligence. Cited by: §4.
Dilgren et al. (2025) C. Dilgren, P. Chiniya, L. Griffith, Y. Ding, and Y. Chen SecRepoBench: benchmarking llms for secure code generation in real-world repositories. ArXiv abs/2504.21205. External Links: Link Cited by: §2.
Ding et al. (2025) J. Ding, S. Long, C. Pu, H. Zhou, H. Gao, X. Gao, C. He, Y. Hou, F. Hu, Z. Li, W. Shi, Z. Wang, D. Zan, C. Zhang, X. Zhang, Q. Chen, X. Cheng, B. Deng, Q. Gu, K. Hua, J. Lin, P. Liu, M. Li, X. Pan, Z. Peng, Y. Qin, Y. Shan, Z. Tan, W. Xie, Z. Wang, Y. Yuan, J. Zhang, E. Zhao, Y. Zhao, H. Zhu, C. Zou, M. Ding, J. Jiao, J. Liu, M. Liu, Q. Liu, C. Tao, J. Yang, T. Yang, Z. Zhang, X. Chen, W. Huang, and G. Zhang NL2Repo-bench: towards long-horizon repository generation evaluation of coding agents. ArXiv abs/2512.12730. External Links: Link Cited by: §2.
Fawzy et al. (2025) A. Fawzy, A. Tahir, and K. Blincoe Vibe coding in practice: motivations, challenges, and a future outlook – a grey literature review. Proceedings of the IEEE/ACM 48th International Conference on Software Engineering: Software Engineering in Practice. External Links: Link Cited by: §1.
GLM-5-Team (2026) GLM-5-Team GLM-5: from vibe coding to agentic engineering. External Links: 2602.15763, Link Cited by: §4.
Hajipour et al. (2023) H. Hajipour, K. Hassler, T. Holz, L. Schönherr, and M. Fritz CodeLMSec benchmark: systematically evaluating and finding security vulnerabilities in black-box code language models. 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp. 684–709. External Links: Link Cited by: §2.
Harbor Framework Team (2026) Harbor Framework Team Harbor: A framework for evaluating and optimizing agents and models in container environments. External Links: Document, Link Cited by: §3.1, §4.
He and Vechev (2023) J. He and M. T. Vechev Large language models for code: security hardening and adversarial testing. Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security. External Links: Link Cited by: §2.
He et al. (2024) J. He, M. Vero, G. Krasnopolska, and M. T. Vechev Instruction tuning for secure code generation. ArXiv abs/2402.09497. External Links: Link Cited by: §2.
Hendrycks et al. (2021) D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. X. Song, and J. Steinhardt Measuring coding challenge competence with apps. ArXiv abs/2105.09938. External Links: Link Cited by: §2.
Jimenez et al. (2023) C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan SWE-bench: can language models resolve real-world github issues?. ArXiv abs/2310.06770. External Links: Link Cited by: §2.
Lian et al. (2025) K. Lian, B. Wang, L. Zhang, L. Chen, J. Wang, Z. Zhao, Y. Yang, H. Duan, H. Zhao, S. Liao, M. Guo, J. Quan, Y. Zhong, C. He, Z. Chen, J. Wu, H. Li, Z. Li, J. Yu, H. Li, and D. Zhang A.s.e: a repository-level benchmark for evaluating security in ai-generated code. In Annual Meeting of the Association for Computational Linguistics, External Links: Link Cited by: §2.
Liu et al. (2025) J. Liu, N. Diwan, Z. Wang, H. Zhai, X. Zhou, K. A. Nguyen, T. Yu, M. Wahed, Y. Deng, H. Benkraouda, Y. Wei, L. Zhang, I. Lourentzou, and G. Wang PurpCode: reasoning for safer code generation. ArXiv abs/2507.19060. External Links: Link Cited by: §2.
Luo et al. (2026) X. Luo, J. Zhang, S. Zhou, R. Huang, C. Xiao, Q. Zhu, Z. Ma, X. Yue, Y. Yue, W. Zeng, and W. Che CVE-factory: scaling expert-level agentic tasks for code security vulnerability. ArXiv abs/2602.03012. External Links: Link Cited by: §3.1.
MITRE (2025a) MITRE CWE - 2025 CWE Top 10 KEV Weaknesses. External Links: Link Cited by: §1.
MITRE (2025b) MITRE CWE - 2025 CWE Top 25 most dangerous software weaknesses. External Links: Link Cited by: §1.
OpenAI (2026) OpenAI GPT-5.6 System Card - OpenAI Deployment Safety Hub. External Links: Link Cited by: §4.
Pan et al. (2024) J. Pan, X. Wang, G. Neubig, N. Jaitly, H. Ji, A. Suhr, and Y. Zhang Training software engineering agents and verifiers with swe-gym. ArXiv abs/2412.21139. External Links: Link Cited by: §2.
Pearce et al. (2021) H. A. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri Asleep at the keyboard? assessing the security of github copilot’s code contributions. 2022 IEEE Symposium on Security and Privacy (SP), pp. 754–768. External Links: Link Cited by: §2.
Peng et al. (2025) J. Peng, L. Cui, K. Huang, J. Yang, and B. Ray CWEval: outcome-driven evaluation on functionality and security of llm code generation. 2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code), pp. 33–40. External Links: Link Cited by: §1, §2.
Qwen Team (2026) Qwen Team Qwen3.8-Max: a new bar for coding and cowork. External Links: Link Cited by: §4.
Sarkar and Drosos (2025) A. Sarkar and I. Drosos Vibe coding: programming through conversation with artificial intelligence. ArXiv abs/2506.23253. External Links: Link Cited by: §1.
Siddiq and Santos (2022) M. L. Siddiq and J. C. S. Santos SecurityEval dataset: mining vulnerability examples to evaluate machine learning-based code generation techniques. Proceedings of the 1st International Workshop on Mining Software Repositories Applications f...
