Every platform shift in enterprise computing has followed the same script. Mainframes, client-server, the web, mobile, cloud and SaaS each arrived through the business first and reached security second, and each time the controls were bolted on after the first serious incident. LLMs are following the script closely, with one new feature: the component you are trying to secure can be talked into misbehaving.

OWASP’s 2026 Top 10 for LLM Applications, released in August, is built around exactly that premise. Prompt injection stays at number one, and Excessive Agency jumped from sixth to third as agents gained tools and memory. Meanwhile, IBM’s 2026 breach report says more than 20% of organizations reported a breach targeting an AI model or application directly.

The engineering takeaway is that you should stop trying to build a model that can’t be fooled and design the system around the assumption that it will be.

Why prompt-level defenses aren’t enough

Models process trusted instructions and untrusted data in the same context window. A system prompt that says “never reveal secrets” is a request, not a control. Filters and guardrails raise the cost of an attack, but a motivated attacker only needs to succeed once. In June 2025, EchoLeak (CVE-2025-32711) showed a zero-click indirect injection against Microsoft 365 Copilot: no user mistake, only a crafted email the assistant later read.

Real controls live outside the model. The most valuable one is a gateway between the model and everything it can touch.

Pattern: a policy-enforcing tool gateway

The model never calls tools directly. It proposes a call, and deterministic code decides whether to execute it. This version enforces five things: an allowlist, role scoping, per-session call budgets, human approval for side-effecting tools once untrusted content has entered the context, and an audit trail.

from dataclasses import dataclass
from typing import Callable

@`dataclass(frozen=True)
class Tool:
name: str
handler: Callable[..., str]
allowed_roles: frozenset
side_effects: bool = False
max_calls: int = 3

class ToolGateway:
def init(self, tools, approve, audit):
self.tools = {t.name: t for t in tools}
self.approve = approve # (session_id, tool_name, args) -> bool
self.audit = audit # (dict) -> None
self.calls = {}

def invoke(self, session_id, role, tool_name, args, tainted): tool = self.tools.get(tool_name) key = (session_id, tool_name) decision, reason = "allow", "ok" if tool is None: decision, reason = "deny", "unknown tool" elif role not in tool.allowed_roles: decision, reason = "deny", "role not permitted" elif self.calls.get(key, 0) >= tool.max_calls: decision, reason = "deny", "call budget exceeded" elif tool.side_effects and tainted and not self.approve(session_id, tool_name, args): decision, reason = "deny", "approval required after untrusted input" self.audit({ "session": session_id, "role": role, "tool": tool_name, "arg_keys": sorted(args), "tainted": tainted, "decision": decision, "reason": reason, }) if decision == "deny": return f"Denied: {reason}" self.calls[key] = self.calls.get(key, 0) + 1 return tool.handler(**args)`

The tainted flag is the important part. Set it to True the moment a session ingests anything the user didn’t type: a web page, an email, a PDF, a RAG chunk. From that point, anything with side effects (sending mail, writing records, issuing refunds) needs explicit approval. An injected instruction can still make the model ask, but it can’t make the system act.

Five more controls that belong in the design

Scope credentials per user, not per application. If the agent holds a service account that can read everything, any injection inherits everything. Pass the end user’s identity and permissions through to retrieval and tools.
**
Treat model output as untrusted input.**
Never pass it unvalidated into a shell, a SQL query, or a rendered HTML page. Improper Output Handling stays on the OWASP list for a reason.

Secure the AI supply chain
In IBM’s 2025 data, the most common AI incidents occurred through compromised apps, APIs or plug-ins. Pin and review models, plugins, MCP servers and prompt templates the way you review dependencies.

Log what the model saw and did
Prompts, retrieved sources, tool proposals and decisions. Without these, incident response is guesswork, and IBM puts the average time to identify and contain a breach at 247 days.

Inventory your non-human identities.
Agents, API keys and service accounts multiply quietly. IBM’s 2026 findings indicate fewer than half of organizations are actively securing them.

The part code doesn’t fix
Engineering controls only cover the systems you know about. In IBM’s 2025 report, 20% of organizations traced a breach to shadow AI, adding roughly $670,000 to breach costs where shadow AI was heavy, and among organizations that had AI-related incidents, 97% lacked proper AI access controls. Those are governance and inventory failures: nobody knew what was running, what data it touched, or who could use it.

That’s why we published the Symosis AI Security Assessment as a PDF: a structured way for engineering, security and risk teams to benchmark where they stand across the areas code can’t reach, and to see the gaps before an attacker or auditor does.

Download the Symosis AI Security Assessment (PDF)

Disclosure: the author works at Symosis, which publishes the assessment.