Running AI Agents on Kubernetes Without Handing Them the Cluster: Agent Sandbox + gVisor, Step by Step

TL;DR

• An AI agent is not a normal workload. It executes code that was generated seconds ago and never reviewed, it takes instructions from content it reads (web pages, tickets, documents — prompt injection is a design condition, not an edge case), and it holds real credentials: LLM API keys, tool tokens, sometimes cloud access. Your isolation has to assume the code inside is hostile.
• This is already a production question. The CNCF and SlashData State of Cloud Native Development Q3 2026 report (released 10 November 2026) found that 49% of developers integrating AI into their services already have agentic systems in production — and KubeCon + CloudNativeCon North America 2026 ran a dedicated AI Inference + Agentic track for the first time.
• The stack in this guide: a dedicated namespace with quotas → an agent identity with zero Kubernetes permissions → gVisor kernel isolation via RuntimeClass → the Kubernetes Agent Sandbox CRD (a SIG Apps project) for lifecycle, templates and warm pools → default-deny networking with named egress holes → short-lived, scoped credentials → audit on everything.
• One honesty note up front: Agent Sandbox is young — its APIs are at v1beta1 and the current release is v1.0.10. Treat the CRD as the lifecycle convenience it is. The load-bearing security layers are the ones underneath it: namespace, RBAC, RuntimeClass, NetworkPolicy. Those work with or without the CRD.

Every platform team has had some version of the same meeting this year. A product team wants to ship an agent: it reads tickets, writes code, calls internal tools, maybe opens pull requests. It needs to run somewhere, it needs credentials to be useful, and it will execute — by design — whatever the model decides to execute. The question lands on the platform team: where does this thing live?

The default answer, "just deploy it like a service," quietly assumes the code in the pod is trusted because a human wrote and reviewed it. For an agent, that assumption is false on day one. This guide builds the other answer: a sandbox stack on Kubernetes, layer by layer, with working manifests. The pattern is the same one you would use for any untrusted workload — CI runners executing strangers' code are the closest classical analogue — tuned for what makes agents different.

Why an agent is a different threat model

Four properties separate an agent runtime from an ordinary microservice:

1. It runs unreviewed code. The whole point of a coding or tool-using agent is that it generates commands, scripts and tool calls at runtime. There is no code review gate between the model and the shell.
2. Its instructions arrive over untrusted channels. An agent that reads a web page, a support ticket or a repository README is reading text an attacker may control. A crafted document can try to redirect the agent — read this file, call that tool, send this data somewhere. You cannot patch this at the model layer with confidence, so you bound what a redirected agent can reach.
3. It is credentialed by design. An agent with no keys can do nothing. An agent with an LLM API key, a Git token and a database password is useful — and is a standing, machine-speed confused deputy holding all of them.
4. It is long-lived and stateful. Agent sessions run for minutes to hours, accumulate files and context, and often persist a workspace. That is a different shape from both stateless services and batch jobs, and it is exactly the shape Kubernetes has historically been awkward at — one reason the Sandbox abstraction exists.

Put together, the design question changes. For a normal workload you ask, "what if this pod is compromised?" For an agent you ask, "this pod will eventually do something it should not — what can it reach when it does?" Every step below answers that question at a different layer. If you are newer to cluster hardening in general, our Kubernetes security basics guide covers the foundations this builds on.

The isolation stack at a glance

No single layer is the security story. The story is that an attacker (or a confused model) has to defeat all of them in sequence, and each one is independently testable.

Step 1 — A namespace that assumes the worst

Agents get their own namespace. Not a shared tools namespace, not the namespace of the product they serve — their own, so that quotas, policies and network rules apply to agents and nothing else.

apiVersion: v1 kind: Namespace metadata: name: agents labels: pod-security.kubernetes.io/enforce: restricted pod-security.kubernetes.io/audit: restricted pod-security.kubernetes.io/warn: restricted --- apiVersion: v1 kind: ResourceQuota metadata: name: agents-quota namespace: agents spec: hard: pods: "20" requests.cpu: "4" requests.memory: 8Gi limits.cpu: "8" limits.memory: 16Gi requests.ephemeral-storage: 10Gi limits.ephemeral-storage: 20Gi count/persistentvolumeclaims: "20" --- apiVersion: v1 kind: LimitRange metadata: name: agents-defaults namespace: agents spec: limits: - type: Container default: cpu: "1" memory: 1Gi ephemeral-storage: 2Gi defaultRequest: cpu: 250m memory: 512Mi 

Three decisions in this manifest are worth more than the rest:

• Pod Security Standards at restricted means the API server itself rejects pods that ask for privilege escalation, host namespaces, or root-by-default. You get a baseline even from manifests you did not write — including ones a controller generates.
• The quota caps the fleet, not just the pod. An agent stuck in a reasoning loop is a cost and availability incident before it is a security one. Twenty pods and 8 CPU of limits is a starting point for a pilot, sized so a runaway can hurt the agents namespace and nothing else. (Our resource requests and limits guide covers the sizing discipline in depth.)
• Ephemeral storage is quota'd. Agents write files — checkouts, build artifacts, downloads. Unbounded ephemeral storage is how one session fills a node disk and evicts its neighbours.

One step further, on clusters that can afford it: taint a dedicated node pool for agent workloads and give the sandbox template the matching toleration. Kernel isolation (Step 3) reduces what a hostile pod can do to a node; physical separation removes the shared fate entirely. For a first deployment, the namespace fence plus gVisor is a defensible start; note the node pool as the scale-up move.

Step 2 — Identity: the agent gets zero Kubernetes permissions

This is the step teams most often get backwards, so be deliberate. There are two identities in this design, and only one of them ever touches the API server:

• The orchestrator — your service that receives a request, creates a SandboxClaim, and hands back a session. It needs a small, explicit Role.
• The agent runtime — the pod that runs generated code. It needs no Kubernetes permissions at all, and its ServiceAccount token should not even be mounted.

apiVersion: v1 kind: ServiceAccount metadata: name: agent-runtime namespace: agents automountServiceAccountToken: false --- apiVersion: v1 kind: ServiceAccount metadata: name: agent-orchestrator namespace: agents --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: agent-orchestrator namespace: agents rules: - apiGroups: ["extensions.agents.x-k8s.io"] resources: ["sandboxclaims"] verbs: ["create", "get", "list", "watch", "delete"] - apiGroups: ["agents.x-k8s.io"] resources: ["sandboxes"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: agent-orchestrator namespace: agents roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: agent-orchestrator subjects: - kind: ServiceAccount name: agent-orchestrator namespace: agents 

Read the asymmetry: the orchestrator can create claims and read sandboxes in this namespace only — no secrets, no pods/exec, no other namespaces. The agent runtime's ServiceAccount exists so the pod has a named identity, but with automountServiceAccountToken: false and no RoleBindings, code inside the sandbox that goes looking for the usual /var/run/secrets/kubernetes.io/serviceaccount/token finds nothing to steal. If your agent framework insists it needs to "call Kubernetes tools," that is the moment to stop and redesign: give the agent an application-level API with the three operations it actually needs, mediated by your orchestrator — never raw cluster credentials. Our Kubernetes RBAC guide covers the audit pattern for keeping Roles this small in practice.

Step 3 — Kernel isolation: gVisor via RuntimeClass

Namespaces and RBAC govern the API plane. They do nothing about the kernel plane: a container under the standard runc runtime shares the host's kernel, so a kernel exploit or an overly broad syscall path is a node compromise. An agent executing generated code is precisely the workload that should not share a kernel with your other tenants.

gVisor inserts a user-space kernel (the Sentry) between the container and the host. The container's syscalls are intercepted and serviced by gVisor, which makes a much smaller, carefully chosen set of calls to the host on the workload's behalf. A bug in the workload's code now has to escape a second, purpose-built boundary before it sees the real kernel.

On self-managed nodes with containerd, register the runsc runtime (the shim is containerd-shim-runsc-v1; the runtime type is io.containerd.runsc.v1):

# /etc/containerd/config.toml (excerpt) — then restart containerd [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc] runtime_type = "io.containerd.runsc.v1" 

Then expose it to the scheduler as a RuntimeClass — the handler name must match the runtime name you registered in containerd:

apiVersion: node.k8s.io/v1 kind: RuntimeClass metadata: name: gvisor handler: runsc 

On GKE, you do not edit containerd configs: GKE Sandbox is the managed form of the same idea — a node pool where pods declaring runtimeClassName: gvisor run under gVisor. Either way, from the pod spec's point of view the contract is identical, which is why the template in Step 4 carries a single line: runtimeClassName: gvisor.

Verify with a throwaway pod before you build anything on top — apply a test pod with the runtime class, confirm it reaches Running, and check that kubectl get pod <name> -o yaml shows the runtimeClassName you set. A pod stuck in ContainerCreating with runtime errors is a node-side installation problem, and it is much cheaper to meet it now than after the templates land.

gVisor or Kata? They are the two realistic choices, and the decision is mostly about your nodes:

Agent runtimes — shells, interpreters, file tools, HTTP calls — are a good gVisor fit. If your agents will run heavy builds or unusual low-level tooling, benchmark both against your real workload before standardising; the RuntimeClass abstraction means the choice stays a one-line change per template.

Step 4 — Agent Sandbox: lifecycle as a CRD

With the fence (Step 1), the identities (Step 2) and the kernel boundary (Step 3) in place, you still need to manage the sandboxes themselves: create one per session, keep it warm enough to start fast, give it a stable identity and workspace, and — most importantly — make it die on schedule. That lifecycle is what the Kubernetes Agent Sandbox project provides.

Agent Sandbox is a SIG Apps project that adds a Sandbox custom resource: a single, stateful pod with a stable hostname, optional persistent storage, and lifecycle management (creation, scheduled deletion, pause and resume) — the "lightweight, single-container VM" shape that agent sessions actually have, instead of forcing them into Deployments or one-member StatefulSets.

Be precise about what it is not: the project describes itself as a sandbox orchestrator. It deliberately delegates low-level isolation to secure runtimes like gVisor or Kata, by creating pods configured with the RuntimeClass you choose. The CRD gives you lifecycle and consistency; the security still comes from Steps 1–3 and 5–6. Designs that treat "we installed the Sandbox CRD" as "we sandboxed the agent" have it exactly backwards.

Install it pinned — never from latest — and verify the CRDs and controller afterwards:

export VERSION="v1.0.10" kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${VERSION}/sandbox-with-extensions.yaml kubectl get crd sandboxes.agents.x-k8s.io kubectl get deploy agent-sandbox-controller -n agent-sandbox-system 

(v1.0.10 is the current release as of this writing, November 2026; the project asks for Kubernetes 1.25 or later, with 1.29+ recommended so CRD CEL validation rules are fully enforced.) The extensions manifest adds the three resources that make the project practical: SandboxTemplate, SandboxWarmPool and SandboxClaim.

A minimal Sandbox shows the shape — one pod template, nothing more:

apiVersion: agents.x-k8s.io/v1beta1 kind: Sandbox metadata: name: agent-session-demo namespace: agents spec: podTemplate: spec: runtimeClassName: gvisor restartPolicy: Never containers: - name: agent # Stand-in: use your own agent runtime image and entrypoint  image: python:3.12-slim command: ["sleep", "infinity"] 

In production you would rarely create bare Sandbox objects by hand. You define one hardened SandboxTemplate — owned by the platform team, reviewed like code — and let everything else instantiate from it. Here is the centrepiece of this guide; every field earns its place:

apiVersion: extensions.agents.x-k8s.io/v1beta1 kind: SandboxTemplate metadata: name: agent-sandbox-template namespace: agents spec: # The controller manages a NetworkPolicy for sandboxes from this template.  networkPolicyManagement: Managed networkPolicy: egress: # DNS only — name resolution is not general network access  - ports: - protocol: UDP port: 53 - protocol: TCP port: 53 # TLS out — tightened further in Step 5  - ports: - protocol: TCP port: 443 volumeClaimTemplates: - metadata: name: workspace spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 5Gi podTemplate: spec: serviceAccountName: agent-runtime automountServiceAccountToken: false runtimeClassName: gvisor restartPolicy: Never securityContext: runAsNonRoot: true runAsUser: 10001 fsGroup: 10001 seccompProfile: type: RuntimeDefault containers: - name: agent # Stand-in: your agent runtime image belongs here  image: python:3.12-slim command: ["sleep", "infinity"] securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: ["ALL"] resources: requests: cpu: 250m memory: 512Mi limits: cpu: "1" memory: 1Gi ephemeral-storage: 2Gi env: - name: LLM_API_KEY valueFrom: secretKeyRef: name: llm-credentials key: api-key # Fail closed: no key, no sandbox  optional: false volumeMounts: - name: workspace mountPath: /workspace - name: tmp mountPath: /tmp volumes: - name: tmp emptyDir: sizeLimit: 512Mi 

Details that matter:

• The template is the security policy. Runtime class, non-root, dropped capabilities, read-only root filesystem, token automount off, per-container limits — all fixed here, once, by the platform team. Application teams get sandboxes; they do not get to renegotiate the isolation line by line. That separation is the entire point of the template/claim split.
• Managed network policy means the controller maintains a NetworkPolicy for sandboxes created from this template, with the DNS-and-443 egress shape above as the starting point. Step 5 hardens networking at the namespace level as well — belt and braces, deliberately.
• The workspace volume is per-sandbox. Each sandbox gets its own PVC from the template, so one session's files are never another session's files, and storage dies with the sandbox.
• optional: false on the secret. If the LLM credentials Secret is missing, the pod refuses to start. Fail closed, find out in staging — not a runtime that silently runs unauthenticated, and not a fallback to some shared default key.

Claims and warm pools are how sessions actually start. A SandboxWarmPool keeps a small number of template-conformant sandboxes pre-created; a SandboxClaim checks one out:

apiVersion: extensions.agents.x-k8s.io/v1beta1 kind: SandboxWarmPool metadata: name: agent-warmpool namespace: agents spec: replicas: 2 sandboxTemplateRef: name: agent-sandbox-template --- apiVersion: extensions.agents.x-k8s.io/v1beta1 kind: SandboxClaim metadata: name: session-0001 namespace: agents spec: warmPoolRef: name: agent-warmpool lifecycle: # Reap the claim (and its sandbox) 10 minutes after the session finishes  ttlSecondsAfterFinished: 600 shutdownPolicy: Delete 

Two operational rules for pools. First, a pool is a trust boundary: never serve two teams — and certainly never two tenants — from the same warm pool, because pool members are interchangeable by design. One pool per template per trust level. Second, a claimed sandbox is spent: it serves exactly one session, and when the claim ends the sandbox is deleted, not wiped and returned. The pool tops itself back up to spec.replicas with fresh sandboxes — that refill behaviour is what makes warm starts safe to offer. And the claim's lifecycle fields give every session a hard expiry even if your orchestrator crashes mid-cleanup: expiry is enforced by the claim controller, not by your code remembering to tidy up.

Step 5 — Network: default-deny, then named holes

Network policy is where "sandbox" stops being a metaphor. The template's managed policy is a start; the namespace gets its own explicit posture so the rules survive template edits and apply to anything else that ever lands in agents:

apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all namespace: agents spec: podSelector: {} policyTypes: - Ingress - Egress --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: agent-egress namespace: agents spec: podSelector: {} policyTypes: - Egress egress: # DNS to the cluster resolver — nothing else in kube-system  - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: kube-system ports: - protocol: UDP port: 53 - protocol: TCP port: 53 # TLS to the outside world — deliberately nothing on port 80,  # which keeps the cloud metadata endpoint (169.254.169.254)  # unreachable even before the rules below are added  - ports: - protocol: TCP port: 443 

Three points of craft here. First, ingress stays closed: there is no rule allowing anything into the sandboxes, because agents should be reached through your orchestrator (or the project's sandbox router), never directly. Second, note what is absent — port 80 egress. The cloud instance metadata service lives on a link-local address over plain HTTP, and it is the single highest-value target inside any pod. Default-deny plus a 443-only allow means the metadata endpoint is unreachable by construction. Third, be honest about the limit of the 443 rule: NetworkPolicy works on IPs and ports, not names, so "443 anywhere" still reaches anything that listens on 443, including services you did not intend. It is the pragmatic minimum, not the finished state.

The finished state is a named destination list. If your CNI supports FQDN policy (Cilium is the common case), express the LLM egress as the exact host it should be:

apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: agent-egress-llm namespace: agents spec: endpointSelector: {} egress: - toFQDNs: - matchName: api.example-llm-provider.com toPorts: - ports: - port: "443" protocol: TCP - toEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: kube-system k8s-app: kube-dns toPorts: - ports: - port: "53" protocol: UDP 

Swap in your provider's real API hostname. The alternative shape — used when the CNI cannot do FQDNs — is an egress proxy: deny all direct egress, allow exactly one proxy service, and let the proxy enforce the allowlist (and hold the real provider credentials, which Step 6 picks up). Either pattern beats "443 to the world" by a wide margin. Our Kubernetes network policies guide walks through policy composition and the usual debugging traps if you are building these for the first time.

One more warning that experience earns the hard way: NetworkPolicy only constrains traffic if your CNI enforces it. Confirm enforcement on a test pod (try to reach a disallowed destination and watch it fail) before you claim the control exists. An unenforced policy is a comment, not a boundary.

Step 6 — Secrets and credentials: short-lived, scoped, or not at all

Every layer so far assumed the agent might go wrong. Credentials decide how much that costs. The rules, in descending order of preference:

1. The sandbox holds nothing it does not need for this session. No kubeconfigs, no cloud CLI credential files, no shared "team" tokens baked into the image. If a credential is in the sandbox and the session is manipulated, the credential is gone — design so that "gone" is a small event.
2. Scope what it must hold. The LLM key in the template is a dedicated key for the agent platform — not the organisation's primary provider account — with a spend limit set at the provider. A leaked agent key should be a billing alert, not an incident. Keep provider keys and rotation in a namespace-scoped Secret, managed like any other workload secret (our ConfigMaps and Secrets guide covers the handling patterns).
3. Prefer per-session, execution-scoped credentials. The strongest pattern: the sandbox holds no standing secret at all. Your orchestrator (or an egress proxy) holds the real credential, and the agent receives a token minted for one session — or even one tool execution — that is revoked when the run ends. The Agent Sandbox project's own examples include exactly this pattern: execution-scoped credentials delivered to a reused sandbox, with egress pinned to a credential proxy and the token revoked when the process exits. Even if you do not adopt that example wholesale, steal its shape.
4. Cloud access goes through an identity, never a file. If the agent legitimately needs cloud APIs, attach a per-namespace workload identity whose role is three verbs on one resource — the same least-privilege discipline as on-prem RBAC, applied to the cloud side (see our AWS IAM least privilege guide). Long-lived access keys inside a sandbox are a finding, full stop.

The rare exception: something in the sandbox genuinely must authenticate to the Kubernetes API — for example, a tool gateway you run in-pod. Do not fall back to the mounted default token. Issue a projected, audience-bound token with a one-hour life instead:

# Pod-spec fragment — add to the template's containers/volumeMounts and volumes volumeMounts: - name: tool-token mountPath: /var/run/secrets/tokens readOnly: true # ...and under spec.volumes: volumes: - name: tool-token projected: sources: - serviceAccountToken: audience: tool-gateway expirationSeconds: 3600 path: token 

The audience means the token is only acceptable to your tool gateway; the expiry means a stolen copy is a one-hour problem; the projection means it is minted per pod, not stored in a Secret that outlives the session.

Step 7 — Audit and observability: assume you will need the tape

When an agent does something expensive or strange — and the interesting incidents are usually "expensive" before they are "malicious" — the first questions are forensic: which session, created by whom, ran what, talked to whom, for how long? Wire the answers up before the first real session:

• API audit logs. Sandbox and claim creation/deletion are the control-plane events that matter. Your audit trail should answer "who or what created this SandboxClaim, from which identity, at what time" without archaeology. An orchestrator identity (Step 2) makes this attribution clean: every sandbox traces to one claim, every claim to one orchestrator request.
• Pod logs, shipped out of the sandbox. Collect sandbox stdout/stderr into your central logging with the sandbox and claim names as labels. Logs that live only inside the sandbox are destroyed with it — by your own TTL, at the worst possible moment. Agent frameworks should also emit structured tool-call logs at the application layer (which tool, what arguments, what came back); ship those through the orchestrator, not from inside the sandbox, so a manipulated agent cannot quietly stop reporting itself.
• Lifecycle and resource signals. Alert on: sandboxes living far longer than the session norm, claim TTLs being hit repeatedly (sessions dying mid-work), ResourceQuota saturation in the agents namespace (the signature of a runaway loop), and — where your CNI reports them — NetworkPolicy drop spikes from sandbox pods, which are often the first visible sign of an agent trying to reach something it should not.
• Events and controller logs. The agent-sandbox controller's events on claims and pools (adoption failures, refill problems) are your early warning that the warm pool is empty and sessions are cold-starting — an availability problem your users will report as "the agent product is slow."

Our Kubernetes observability guide covers the logging/metrics plumbing; the agent-specific work is the labelling discipline above, so sandbox telemetry joins up with session telemetry from your application.

What I would not run yet

A guide that only sells the stack is not trustworthy, so the reservations, plainly:

• Do not let the CRD's youth become your risk. Agent Sandbox's APIs are v1beta1 and the project is releasing fast (v1.0.10 at this writing). Pin the version, read the release notes before upgrading — recent releases have changed router NetworkPolicy behaviour in ways that need action — and keep your GitOps manifests as the source of truth so a rollback is a revert. The pattern in this guide does not depend on the CRD; if the project stalled tomorrow, plain Pods plus Steps 1–3 and 5–7 would still stand.
• Do not give an agent write access to anything production-shaped in v1 of your rollout. Read-only tools first. The sandbox limits blast radius; it does not make a bad write un-happen. Irreversible actions (sends, deletes, merges, payments) belong behind human approval at the application layer — no kernel feature gives you that.
• Do not share warm pools across t...