Most engineering teams rely on CI/CD pipelines to catch leaked API keys, tokens, and database passwords.

When a secret triggers an alert in GitHub Actions or GitLab CI, the secret is already compromised:

  1. The commit is recorded in git DAG history across local developer machines.
  2. The payload was pushed over TLS to remote remotes and webhook listeners.
  3. Bot crawlers monitoring the public firehose often scrape live commits within 30 seconds of push.

At that point, "removing" the commit with git rebase or git revert is purely cosmetic—the credential must be revoked and rotated immediately across production infrastructure.

The Problem With Regex-Only Pre-Commit Hooks

To stop secrets before they leave the workstation, teams deploy git pre-commit hooks. But regex-heavy rulesets suffer from two catastrophic flaws:

  1. High False Positive Rates: Regexes matching high-entropy strings trip on UUIDs, test hashes, Base64 fixtures, and lockfile checksums, training developers to bypass hooks with --no-verify.
  2. Format Blindness: A regex targeting sk_live_[0-9a-zA-Z]{24} catches known provider prefixes, but completely misses private RSA keys, raw JWTs, custom session tokens, or un-prefixed bearer credentials assigned in local config files.

Shannon Entropy Combined with AST Context

A resilient local defense combines abstract syntax tree (AST) traversal with Shannon entropy calculations:

import math def shannon_entropy(data: str) -> float: if not data: return 0.0 entropy = 0.0 for x in set(data): p_x = float(data.count(x)) / len(data) if p_x > 0: entropy += - p_x * math.log2(p_x) return entropy

In plain text, English sentences and variable names have an entropy score between 2.5 and 3.5. Cryptographically random strings (Base64 keys, hex digests, high-entropy tokens) typically score > 4.5.

By parsing code files into AST nodes before evaluating entropy:

  • String Literals vs Variable Names: Entropy analysis is restricted strictly to string constants and assignment values, completely ignoring function names and type definitions.
  • Assignment Context: Variable assignments where the target identifier contains words like key, token, secret, auth, or password have their entropy threshold lowered to 3.8.
  • Entropy Whitelisting: Well-known deterministic strings (such as Git commit SHAs, SHA256 checksums, and hex color codes) are whitelisted by character set and length.

Shift Left to Zero-Leak Commits

Continuous monitoring of repositories is essential, but the ultimate security perimeter is the local developer staging area. Catching uncommitted secrets before they form git commits eliminates emergency credential rotation fire drills.

Run an instant conversion, SEO & leak audit on your website free: https://leaksift.pages.dev/?src=devto