Security reports written by agents have a specific smell. The findings are confident, the file references are approximate, and some of them do not survive a ten-minute check. I have thrown away more of these than I have used, mostly because disproving a fabricated vulnerability costs more than finding a real one.

So when cloudflare/security-audit-skill showed up third on a third-party GitHub weekly ranking the week I was looking at it (+9,047 stars, aggregator numbers rather than an official GitHub list), I read past the pitch. The README opens with "A coding-agent skill that turns your agent into a security auditor." Fine. The part I cared about was two scripts with no dependencies that validate the artifacts the agent writes: validate-coverage-ledger.cjs and validate-findings.cjs.

I installed it, planted three vulnerabilities in a two-file demo repo of my own, and ran a restricted audit. The validators rejected my output seven times before anything counted. This is the show-and-tell version.

The setup

The skill is agent-agnostic. Its one requirement is quoted here exactly as written: "A coding agent with a model that supports tool use and parallel sub-agents".

Mine is Qwen Code with the model set to qwen3.8-max, baseUrl pointing at https://dashscope.aliyuncs.com/compatible-mode/v1 (Alibaba Cloud Model Studio, also known as Bailian). qwen3.8-max supports function calling, so tool use is covered. Keys come from the console API key page, region China Beijing.

Install, copied from the README:

npx skills add https://github.com/cloudflare/security-audit-skill --skill security-audit

I ended up dropping the skill directory into my workspace's agent skills folder, which is the same thing on disk. Use the command above.

The repo contains SKILL.md, thirteen attack-surface knowledge files, a report-schema.json, the two validators, and tests. Six phases run in order: Reconnaissance, Coverage-led hunting, Candidate validation, Structured output, Independent record verification, Target-neutral reporting. Three profiles control how much work each phase does. I picked quick, the restricted path for small targets: coarse-grained coverage units, one hunter wave, one critic pass.

The target was mine, and the bugs were mine

Two files. Three vulnerabilities I wrote in deliberately:

  • a query parameter concatenated into eval()
  • a hardcoded key with a live-looking prefix sitting in source
  • an upload filename joined into a path with no sanitization

Worth saying out loud: nothing real was audited. The findings below describe my own throwaway code and nothing else.

Three rejections on the ledger

Phase 1 writes coverage-ledger.json. My first version was an object wrapping an array, because that is how I would design it. The validator disagreed immediately:

ERROR: $: expected a top-level array

Second version, top-level array, and it listed fourteen required fields I had left out: coverage_id, canonical_refs, subsystem, starting_paths, ordinary_attack_class_block, among others.

Third version was rejected on semantics rather than missing fields. coverage_id is not a label you make up. It is derived: take the unit's surface, boundary, subsystem, and attack class, percent-encode each, join with ::. unresolved must be an array. And a unit still in planned state cannot have an agent assigned, so agent_id stays null until something is actually working on it.

Fourth version:

PASS: 3 coverage units valid

The derived ID makes sense once you notice that the README describes repeat runs as additive. The skill reads the previous ledger and findings, re-verifies units whose source changed, and carries unchanged evidence forward with its fingerprint. Identity that comes from the unit's coordinates survives a re-run. Identity that comes from whoever wrote the file does not.

Four rejections on the findings

Hunting and falsification went as you would hope, and all three planted bugs reproduced:

1. eval injection input: ' || '1'=='1 -> returned rows: 1 (expected 0) 2. path traversal resolve('/tmp/uploads/' + '../../etc/passwd') -> /etc/passwd 3. hardcoded key grep -n sk-live src/server.js -> 3:const API_KEY = 'sk-live-...'

Writing them into findings.json is where the skill got picky, across four failed versions:

  • every trace entry needs a kind of entrypoint, propagation, or sink, plus file, line, scope, description
  • evidence anchors to file and line; quoting a chunk of code is not evidence
  • conditions is an array
  • the execution block needs attacker_perspective, payloads, instructions, observed_result
  • severity and confidence are objects with a score and a reason, and severity additionally needs an overall_severity from a string enum
  • the array sorts lexicographically by fingerprint

Version five passed: PASS: 3 findings valid.

Read that list as a set of questions instead of a schema and it is a decent audit methodology. Where does attacker input enter? How does it reach the sink? What did you run, with what payload, and what came back? How severe, and why do you say so? Prose answers let you skip any of them. Structured fields do not.

What I would tell you before installing it

The seven rejections were never about the model being weak. Every one was container type, missing field, wrong enum, unsorted array, or an ID that had to be derived. That is the point of the design: fabricating a finding means fabricating a trace anchored to a real file and line, an execution block that reproduces, and a severity object that validates. Making things up gets more expensive than checking.

Also be clear about what my run does not cover. Quick profile on a two-file repo, one time through. Phases 5 and 6 never ran, so there was no independent re-verification of my records and no target-neutral report. I ran hunter and verifier serially in a single agent, which means the parallel sub-agent orchestration the README asks for is untested by me. And the star count is a third-party aggregator's weekly figure.

If you want the same experience, the loop is short: install the skill, plant two or three bugs in a repo you own, run the quick profile, and count your rejections. Mine was seven, and the number is the useful part, because each one points at a field you thought you could hand-wave.

Model side needs nothing exotic. The requirement is a coding agent whose model supports tool use and parallel sub-agents; mine runs on qwen3.8-max through Bailian, and I drove the whole audit from one agent. The Bailian CLI (bl) page covers the CLI and the model list on Alibaba Cloud Model Studio.

Has a schema ever talked you out of a finding you were sure about? I am curious whether other people's agents fail on the same fields, or whether overall_severity being a string enum was just my week.