Andrej Karpathy coined “vibe coding” in a post on X on February 2, 2025: you describe what you want, accept what the model writes, and “fully give in to the vibes.” By November, Collins had picked it as its Word of the Year. Adoption followed the hype: in the 2025 Stack Overflow Developer Survey, 84% of respondents said they use or plan to use AI tools, yet more of them distrust the accuracy of those tools (46%) than trust it (33%).
So is vibe coding bad? The research on AI generated code vulnerabilities gives a consistent answer: the code is not bad by default, but it is not safe by default either.
- In “Asleep at the Keyboard?” (IEEE S&P 2022), Pearce et al. generated 1,689 programs with GitHub Copilot across 89 security-relevant scenarios. About 40% were vulnerable.
- Veracode’s GenAI Code Security Report tested code from more than 100 LLMs in Java, JavaScript, Python and C#. AI-generated code introduced risky security flaws in 45% of tests, and larger, newer models did not do better.
- Perry et al. (“Do Users Write More Insecure Code with AI Assistants?”) found that participants with an AI assistant wrote significantly less secure code, and were more likely to believe it was secure.
- Spracklen et al. (USENIX Security 2025) analyzed 576,000 generated code samples from 16 models: on average at least 5.2% of the packages suggested by commercial models and 21.7% by open-source models did not exist.
The fix is not to stop using the assistant, but to make every AI-written change pass the checks you would demand from a new hire. In this post we take one small, realistic “vibe-coded” Flask module, run a guardrail pipeline over it, look at what each tool caught and what it missed, fix the code, and wire everything into pre-commit and GitHub Actions.
What we will do:
- Look at the vibe-coded module and its two happy-path tests.
- Run ruff, mypy (strict), bandit, detect-secrets, vulture and pip-audit on it.
- Write the tests the assistant did not: API edge cases plus Hypothesis property-based tests, behind a 90% coverage gate.
- Fix the module and get a fully green run.
- Lock it in with pre-commit and a GitHub Actions workflow, and watch the hooks block a bad commit.
Versions (pinned in requirements-dev.txt): Python 3.13.18, Flask 3.1.3, ruff 0.17.3, mypy 2.4.1, bandit 1.9.4, detect-secrets 1.5.0, vulture 2.16, pip-audit 2.10.1, pytest 9.2.0, pytest-cov 7.1.0, Hypothesis 6.170.13, pre-commit 4.7.0. Every output block below is copied from a real run.
The specimen: a vibe-coded orders API
The prompt was the kind you type in a hurry: “Flask API for orders: list a customer’s orders with pagination, apply a percentage discount, notify a webhook.” I wrote the result by hand to reproduce the patterns the studies above keep finding, so every issue here is deliberate, but none of it is exotic. It runs and it demos well:
import json import os import sqlite3 import requests from flask import Flask, jsonify, request app = Flask(__name__) DB_PATH = "shop.db" PAYMENTS_API_KEY = "sk_live_51NzQ8vKp2ExampleKey9fT3" WEBHOOK_SECRET = "whsec_3f9a1c7e2b" WEBHOOK_URL = os.getenv("WEBHOOK_URL") def get_db(): conn = sqlite3.connect(DB_PATH) conn.row_factory = sqlite3.Row return conn def apply_discount(total, percent): return round(total - total * percent / 100, 2) def paginate(items, page, per_page): start = (page - 1) * per_page return items[start:start + per_page] def legacy_total(items): total = 0 for i in items: total += i["price"] * i["qty"] return total @app.route("/orders") def list_orders(): customer = request.args.get("customer") page = request.args.get("page", 1) db = get_db() rows = db.execute( f"SELECT id, customer, total, status FROM orders WHERE customer = '{customer}'" ).fetchall() orders = [dict(r) for r in rows] return jsonify(paginate(orders, page, 20)) @app.route("/orders/<int:order_id>/discount", methods=["POST"]) def discount(order_id): data = request.get_json() percent = data["percent"] db = get_db() row = db.execute("SELECT total FROM orders WHERE id = ?", (order_id,)).fetchone() new_total = apply_discount(row["total"], percent) db.execute("UPDATE orders SET total = ? WHERE id = ?", (new_total, order_id)) db.commit() notify(order_id, new_total) return jsonify({"id": order_id, "total": new_total}) def notify(order_id, total): try: requests.post( WEBHOOK_URL, json={"id": order_id, "total": total}, headers={"X-Signature": WEBHOOK_SECRET}, verify=False, ) except: pass if __name__ == "__main__": app.run(host="0.0.0.0", debug=True) The assistant also wrote tests. They pass:
from shop.app import apply_discount, paginate def test_apply_discount(): assert apply_discount(100, 10) == 90 def test_paginate(): assert paginate([1, 2, 3, 4, 5], 1, 2) == [1, 2] Try to count the problems before the tools do. The matrix near the end lists sixteen, and no tool in this post flags three of them.
The guardrail pipeline at a glance
Each tool answers one narrow question. None of them replaces the others, and none of them replaces a reviewer.
| Tool | Question it answers | Where it runs |
|---|---|---|
ruff | Is this code idiomatic, without obvious bugs (unused imports, bare except)? | pre-commit, CI |
mypy --strict | Do the types line up across function boundaries? | pre-commit, CI |
bandit | Does the AST contain known insecure patterns (CWE-mapped)? | pre-commit, CI |
detect-secrets | Does a staged file contain something that looks like a credential? | pre-commit, CI |
vulture | Is there code nothing calls? | pre-commit, CI |
pip-audit | Do pinned dependencies have published advisories? | pre-commit (on requirements change), CI |
pytest + Hypothesis + coverage | Does the code do the right thing on inputs nobody thought of? | pre-push, CI |
Pipeline diagram: commit-time hooks (ruff, bandit, detect-secrets, vulture, mypy, pip-audit on requirements change), pre-push tests with a 90% coverage gate, and the CI job that reruns everything with 1,000 Hypothesis examples
The project configuration that all tools read lives in one pyproject.toml:
[project] name = "shop" version = "0.1.0" requires-python = ">=3.12" [tool.ruff] line-length = 100 target-version = "py312" # vulture whitelists are bare expressions by design; force-exclude also applies to pre-commit extend-exclude = ["vulture_whitelist.py"] force-exclude = true [tool.ruff.lint] select = ["E", "F", "B", "BLE", "UP", "SIM"] [tool.mypy] strict = true packages = ["shop"] [tool.bandit] exclude_dirs = ["tests", ".venv"] [tool.pytest.ini_options] addopts = "--cov=shop --cov-branch --cov-report=term-missing --cov-fail-under=90" [tool.coverage.run] omit = ["tests/*"] Step 1. ruff: fast, but only the surface
$ ruff check --output-format concise . shop/app.py:1:8: F401 [*] `json` imported but unused shop/app.py:64:5: SIM105 Use `contextlib.suppress(BaseException)` instead of `try`-`except`-`pass` shop/app.py:71:5: E722 Do not use bare `except` Found 3 errors. [*] 1 fixable with `--fix` (1 unsafe fix available with `--fix --unsafe-fixes`). [exit 1] ruff caught the unused json import and the bare except. That is useful hygiene, but notice SIM105: it proposes contextlib.suppress(BaseException). Accepting that “unsafe fix” (ruff labels it that way itself) keeps the real bug, a webhook failure that silently disappears, and just makes it shorter. The right fix is to catch requests.RequestException and log it.
ruff also ships rules derived from flake8-bandit under the S prefix (the f-string query would be S608), and you can enable them. I keep bandit separate here so you can see exactly what each tool finds; in a large repo, ruff --select S is the faster option.
Step 2. mypy –strict: noise first, then the real bug
On the unannotated vibe code, strict mode returns twelve errors, and every one of them says the same thing:
$ mypy shop/app.py:16: error: Function is missing a return type annotation [no-untyped-def] shop/app.py:22: error: Function is missing a type annotation [no-untyped-def] shop/app.py:26: error: Function is missing a type annotation [no-untyped-def] shop/app.py:31: error: Function is missing a type annotation [no-untyped-def] shop/app.py:39: error: Function is missing a return type annotation [no-untyped-def] shop/app.py:42: error: Call to untyped function "get_db" in typed context [no-untyped-call] shop/app.py:47: error: Call to untyped function "paginate" in typed context [no-untyped-call] shop/app.py:51: error: Function is missing a type annotation [no-untyped-def] shop/app.py:54: error: Call to untyped function "get_db" in typed context [no-untyped-call] shop/app.py:56: error: Call to untyped function "apply_discount" in typed context [no-untyped-call] shop/app.py:59: error: Call to untyped function "notify" in typed context [no-untyped-call] shop/app.py:63: error: Function is missing a type annotation [no-untyped-def] shop/app.py:66: error: Argument 1 to "post" has incompatible type "str | None"; expected "str | bytes" [arg-type] Found 13 errors in 1 file (checked 2 source files) [exit 1] By default, mypy does not type-check the bodies of unannotated functions, and AI assistants often skip annotations in quick scripts. So mypy has nothing to check until you add signatures. After adding only the signatures (no logic changes), the noise disappears and a real bug appears:
$ mypy shop/app.py:48: error: Argument 2 to "paginate" has incompatible type "str | int"; expected "int" [arg-type] shop/app.py:67: error: Argument 1 to "post" has incompatible type "str | None"; expected "str | bytes" [arg-type] Found 2 errors in 1 file (checked 2 source files) [exit 1] request.args.get("page", 1) returns the string "2" when the parameter is present and the integer 1 when it is not. The happy-path test never sends page, so the code passed. In production, GET /orders?customer=alice&page=2 crashes with TypeError and returns a 500.
The second error is a deployment bug: os.getenv("WEBHOOK_URL") returns None when the variable is not set, and the code passes it straight to requests.post. Because of the bare except, that failure would also be silent.
Step 3. bandit and detect-secrets: security patterns and leaked keys
bandit -q -c pyproject.toml -r shop -f custom \ --msg-template "{relpath}:{line}: {test_id}[{severity}/{confidence}] {msg}" shop/app.py:12: B105[LOW/MEDIUM] Possible hardcoded password: 'whsec_3f9a1c7e2b' shop/app.py:44: B608[MEDIUM/MEDIUM] Possible SQL injection vector through string-based query construction. shop/app.py:69: B501[HIGH/HIGH] Call to requests with verify=False disabling SSL certificate checks, security issue. shop/app.py:65: B113[MEDIUM/LOW] Call to requests without timeout shop/app.py:71: B110[LOW/HIGH] Try, Except, Pass detected. shop/app.py:76: B201[HIGH/MEDIUM] A Flask app appears to be run with debug=True, which exposes the Werkzeug debugger and allows the execution of arbitrary code. shop/app.py:76: B104[MEDIUM/MEDIUM] Possible binding to all interfaces. [exit 1] Seven findings, all real: the f-string SQL (B608, CWE-89), verify=False (B501), no timeout (B113), the swallowed exception (B110), debug=True that exposes the Werkzeug debugger (B201), and binding to all interfaces (B104).
Now look at what is missing. WEBHOOK_SECRET was flagged, but PAYMENTS_API_KEY = "sk_live_..." was not. B105 is name-based: it looks for variables whose names resemble password, pass, passwd, pwd, secret, token or secrete. “API key” is not on that list, so bandit let a live-looking payment key through.
This is why secure AI generated code needs a dedicated secret scanner. detect-secrets combines provider regexes, entropy checks and keyword rules:
$ detect-secrets-hook shop/app.py ERROR: Potential secrets about to be committed to git repo! Secret Type: Base64 High Entropy String Location: shop/app.py:11 Secret Type: Secret Keyword Location: shop/app.py:11 Secret Type: Stripe Access Key Location: shop/app.py:11 Secret Type: Secret Keyword Location: shop/app.py:12 [exit 1] # (help text with links removed) It flagged the Stripe-style key three ways (entropy, keyword, Stripe detector) and also caught the webhook secret.
Step 4. pip-audit: the dependency the model remembered
The assistant pinned requests==2.31.0, an older release of the kind models reproduce from their training data. pip-audit resolves requirements.txt and checks every package against an advisory database:
$ pip-audit -r requirements.txt --aliases Found 6 known vulnerabilities in 1 package Name Version ID Fix Versions Aliases -------- ------- --------------- ------------ ----------------------------------- requests 2.31.0 PYSEC-2026-1873 2.32.0 CVE-2024-35195, GHSA-9wx4-h78v-vm56 requests 2.31.0 PYSEC-2026-1872 2.32.4 CVE-2024-47081, GHSA-9hjg-9r4m-mvj7 requests 2.31.0 PYSEC-2026-2275 2.33.0 CVE-2026-25645, GHSA-gc5v-m9x4-r6x2 requests 2.31.0 PYSEC-2026-1873 2.32.0 CVE-2024-35195, GHSA-9wx4-h78v-vm56 requests 2.31.0 PYSEC-2026-1872 2.32.4 CVE-2024-47081, GHSA-9hjg-9r4m-mvj7 requests 2.31.0 PYSEC-2026-2275 2.33.0 CVE-2026-25645, GHSA-gc5v-m9x4-r6x2 [exit 1] Three advisories, each printed twice in this run (the JSON output shows the same duplication; the unique set is three). The first one is directly relevant: CVE-2024-35195 means that once a Session makes its first request to a host with verify=False, later requests to that host keep skipping certificate verification for the life of the pooled connection. Our module disables verification on purpose, which makes this worse, not better. The fix is requests==2.34.2 and removing verify=False.
If the advisory service is unreachable. pip-audit queries PyPI by default and also supports OSV (--vulnerability-service osv). In my sandbox, the PyPI service worked, but the OSV endpoint was blocked by the network proxy (api.osv.dev ... Tunnel connection failed: 403 Forbidden). If your build agents cannot reach either service, do not drop the check. Run pip-audit in a CI job that does have network access, or rely on GitHub’s Dependabot alerts on the repository, and keep --strict so a dependency that cannot be resolved fails the job instead of being skipped.
One more caution from the pip-audit README: pip-audit -r is functionally equivalent to pip install -r, and its isolation is not a security boundary. Do not run it on requirements files you do not trust.
Step 5. vulture: dead code and its false positives
$ vulture shop --min-confidence 60 shop/app.py:1: unused import 'json' (90% confidence) shop/app.py:11: unused variable 'PAYMENTS_API_KEY' (60% confidence) shop/app.py:18: unused attribute 'row_factory' (60% confidence) shop/app.py:31: unused function 'legacy_total' (60% confidence) shop/app.py:38: unused function 'list_orders' (60% confidence) shop/app.py:50: unused function 'discount' (60% confidence) [exit 3] vulture found legacy_total, which nothing calls, and the unused PAYMENTS_API_KEY. It also reported list_orders and discount as unused, which is wrong: Flask calls them through @app.route. The vulture README says it directly: code that is only called implicitly may be reported as unused, and it suggests --ignore-decorators "@app.route" or a whitelist. With 60% confidence for functions, you cannot simply raise the threshold without also hiding legacy_total. In the fixed version I generated a whitelist with vulture shop --make-whitelist.
Step 6. Tests the assistant did not write: edge cases, properties and a coverage gate
The two vibe-coded tests pass, and the coverage gate (--cov-fail-under=90 with branch coverage) already says something is off:
$ pytest -q -p no:cacheprovider tests/test_app.py .. ERROR: Coverage failure: total of 44 is less than fail-under=90 shop/app.py 50 27 4 1 44% 17-19, 32-35, 40-47, 52-60, 64-72, 76 FAIL Required test coverage of 90% not reached. Total coverage: 44.44% 2 passed in 0.20s # (output filtered to the relevant lines) Coverage is a weak signal for AI code: an assistant can produce 90% coverage with assertions that check nothing. The real value is in tests that state what must always be true. First, API tests that describe the contract the prompt never mentioned:
from decimal import Decimal from flask.testing import FlaskClient def test_customer_filter_is_not_injectable(client: FlaskClient) -> None: resp = client.get("/orders", query_string={"customer": "x' OR '1'='1"}) assert resp.status_code == 200 assert resp.get_json() == [] def test_page_query_param_is_parsed(client: FlaskClient) -> None: resp = client.get("/orders", query_string={"customer": "alice", "page": "1"}) assert resp.status_code == 200 assert [o["id"] for o in resp.get_json()] == [1] def test_bad_page_is_a_client_error(client: FlaskClient) -> None: resp = client.get("/orders", query_string={"customer": "alice", "page": "abc"}) assert resp.status_code == 400 def test_discount_applies(client: FlaskClient) -> None: resp = client.post("/orders/1/discount", json={"percent": 10}) assert resp.status_code == 200 assert Decimal(str(resp.get_json()["total"])) == Decimal("90.00") def test_discount_over_100_percent_is_rejected(client: FlaskClient) -> None: resp = client.post("/orders/1/discount", json={"percent": 150}) assert resp.status_code == 400 def test_non_numeric_percent_is_rejected(client: FlaskClient) -> None: resp = client.post("/orders/1/discount", json={"percent": "ten"}) assert resp.status_code == 400 def test_missing_percent_is_rejected(client: FlaskClient) -> None: resp = client.post("/orders/1/discount", json={"pct": 10}) assert resp.status_code == 400 def test_unknown_order_is_404(client: FlaskClient) -> None: resp = client.post("/orders/999/discount", json={"percent": 10}) assert resp.status_code == 404 Second, property-based tests with Hypothesis. Instead of picking examples, we describe the input space and an invariant, and Hypothesis searches for a counterexample and then shrinks it to the smallest one it can find:
from decimal import Decimal from hypothesis import given from hypothesis import strategies as st from shop.app import apply_discount, paginate money = st.decimals(min_value=0, max_value=1_000_000, places=2) percent = st.decimals(min_value=-1000, max_value=1000, places=2) @given(total=money, pct=percent) def test_discount_never_increases_total_or_goes_negative(total: Decimal, pct: Decimal) -> None: try: result = Decimal(str(apply_discount(total, pct))) except ValueError: return # rejecting bad input is allowed; returning garbage is not assert 0 <= result <= total @given(items=st.lists(st.integers(), max_size=60), per_page=st.integers(1, 25)) def test_pages_cover_every_item_exactly_once(items: list[int], per_page: int) -> None: pages = [paginate(items, p, per_page) for p in range(1, len(items) // per_page + 2)] assert [x for page in pages for x in page] == items @given(items=st.lists(st.integers(), min_size=1), page=st.integers(max_value=0)) def test_page_below_one_is_rejected(items: list[int], page: int) -> None: try: result = paginate(items, page, 10) except ValueError: return raise AssertionError(f"page={page} returned {result!r} instead of raising") Against the vibe-coded module:
$ pytest -q -p no:cacheprovider --tb=short TypeError: unsupported operand type(s) for -: 'str' and 'int' TypeError: unsupported operand type(s) for -: 'str' and 'int' TypeError: can't multiply sequence by non-int of type 'float' KeyError: 'percent' TypeError: 'NoneType' object is not subscriptable E Failing test case: test_discount_never_increases_total_or_goes_negative( E total=Decimal('0.01'), E pct=Decimal('-50.00'), E AssertionError: page=0 returned [] instead of raising E Failing test case: test_page_below_one_is_rejected( E items=[0], E page=0, FAIL Required test coverage of 90% not reached. Total coverage: 77.78% FAILED tests/test_api.py::test_customer_filter_is_not_injectable - AssertionE... FAILED tests/test_api.py::tes...
