An Overview of Agent Substrate

A Kubernetes Pod is a useful unit of deployment. An agent session has a different lifetime: it may run briefly, wait for minutes, then need its process and filesystem state back for the next turn. Keeping a dedicated environment resident through every wait can consume capacity that other sessions could use.

Agent Substrate separates those lifetimes. Kubernetes runs a pool of worker Pods; Substrate places sandboxed agent environments onto them, persists their state, and restores them when work arrives. This article follows one request through that system, then examines the components and a Counter demo you can try locally.

TL;DR

• Kubernetes manages Workers; Substrate manages Actors. Actor state and assignments live behind a separate API backed by PostgreSQL.
• Workers can serve many Actors over time. Capacity-based concurrency is combined with suspend/resume to reuse compute.
• atelet and ateom save and restore sandbox state. They use gVisor or microVMs; the snapshot fidelity determines what is preserved.
• Credentials can stay outside the Actor. An egress gateway and credential provider can inject API credentials into outgoing requests.
• Check the implemented behavior, not just the design. Authorization enforcement is experimental and disabled by default for ordinary Control RPCs. Idle detection and performance claims also require context.

Implementation details target Agent Substrate v0.5.0. See Components and versions for the full scope.

This walkthrough assumes familiarity with Pods, Deployments and basic Kubernetes networking. Start with the request lifecycle, jump to the kind demo, or use the evaluation questions to assess the fit for your platform.

The design decision: keep the workers, move the sessions

An agent may execute code briefly, wait for inference or an external tool, then wait indefinitely for another user turn. Running untrusted code also creates a need for isolated environments, often with one tenant per sandbox.

The combination matters: a large fleet can contain many more waiting environments than actively executing ones. Reserving CPU and memory for every waiting agent wastes capacity. Snapshot-based multiplexing is intended to let those environments retain state without remaining assigned to running compute.

Source: docs/architecture.md#L17-L23.

What Kubernetes still does—and what Substrate moves elsewhere

The architecture document identifies three pressures behind its design:

These are the project's design judgments for its intended scale, rather than universal Kubernetes limits or a claim that every Pod takes seconds to start.

Kubernetes still handles infrastructure provisioning and worker Pod lifecycles. Substrate introduces a separate control plane for Actor state, assignment and resumption, so an activation does not need to wait for a fresh Pod lifecycle.

Sources: docs/architecture.md#L153, docs/architecture.md#L160-L172.

The important boundary is the activation path: a new turn can restore an Actor onto an existing Worker without provisioning another Pod. Database write throughput, reconciliation and environment preparation all contribute to that choice.

Follow a request into a suspended Actor

The execution layer combines routing, state persistence and sandbox isolation:

Source: Agent Substrate on GKE, Google Cloud Blog (September 16, 2026).

Two lifecycles, two APIs

Sources: docs/glossary.md#L10-L23, docs/glossary.md#L38-L52.

An Actor is the stateful environment that belongs to a session. A Worker is the reusable execution capacity. Suspending an Actor releases its assignment; it does not require destroying and recreating the worker Pod. Worker capacity can also permit several Actors to run concurrently.

Restore first, forward second

For an HTTP request targeting a suspended Actor, the activation path has four steps:

1. The router identifies the Actor from ate-target-actor.
2. The control plane assigns a Worker and coordinates restoration of the Actor's snapshot.
3. Once the assignment is ready, the router forwards the request through the Worker's tunnel to the Actor.
4. The Actor handles the request and returns its response.

For an Actor that is already running, ResumeActor returns the existing assignment. The call is idempotent.

Source: workflow_resume.go.

Returning a response does not automatically suspend the Actor. The application or higher-level system must choose a safe suspension boundary, taking account of outstanding work and network calls.

Suspend and resume

Source: Agent Substrate on GKE, Google Cloud Blog (September 16, 2026).

Here, “idle” means state has been persisted and the environment removed from running compute, rather than a process simply sleeping. The illustration is conceptual: it does not imply a fixed one-to-one relationship between Workers and Actors.

When a request arrives, atenet-router asks the control plane to resume the target Actor and resolve its Worker assignment. A suspended Actor is restored from its latest snapshot before traffic is forwarded. It may resume on a different Worker.

The router itself does not decide whether the Actor is running; that decision belongs to the control plane. Suspending the Actor saves state and releases its assignment for reuse.

Source: docs/architecture.md#L193-L204.

What runs where

The threat-model diagram shows how these components connect:

Source: docs/threat-model.md#L30-L40.

The repository also includes supporting programs:

The README's tour is not an exhaustive inventory; the sidecar, provider, dataplane and setup entries are supplemented from their own source and documentation.

Sources: README.md#L260-L272, cmd/objectstore-plugin/main.go#L15-L19, cmd/ate-setup/README.md#L3, cmd/dataplane/envoy/README.md.

Resource ownership and assignment

Source: Bringing you Agent Sandbox on GKE and Agent Substrate, Google Cloud Blog (May 20, 2026).

The green sandbox-isolation block is the foundation for the checkpoint and restore operations. Agent Substrate incorporates Agent Sandbox's secure runtime and snapshot capabilities into a separate execution layer.

classDiagram namespace kube-apiserver { class WorkerPool { <<CRD>> } class Deployment class WorkerPod { ateom runsc } } namespace ate-api-server { class ActorTemplate { <<record>> } class Actor { <<record>> status snapshotRefs } class Worker { <<record>> actorAssignments podIP } } ActorTemplate ..> WorkerPool : worker_selector (labels) WorkerPool ..> Deployment : reconciled by atecontroller Deployment "1" *-- "*" WorkerPod : manages Actor "*" --> "1" ActorTemplate : actor_template Actor "0..*" --> "0..1" Worker : runs on Worker "1" --> "1" WorkerPod : maps to

Adapted from docs/architecture.md#L268-L303; the multiplicity reflects the implementation's support for multiple Actors per Worker.

The split between API servers is central. Infrastructure resources stay in Kubernetes, while frequent Actor/Worker assignment changes go through ate-api-server and its state store.

Two resource layers

The architecture gives three reasons for this split:

• Keep high-frequency Actor updates away from the Kubernetes control plane.
• Use low-latency lookups and atomic assignments for resumption. The document's 100ms figure is a design target, not a result measured here.
• Retain Kubernetes RBAC, auditing and policies for infrastructure resources such as WorkerPools and SandboxConfigs.

One detail in that document is stale: it says authorization has not been implemented. The same v0.5.0 tag contains OpenFGA-based authorization. --experimental-enable-authz defaults to false and controls enforcement for ordinary Control RPCs. AccessPolicy RPCs are checked regardless of that flag.

Sources: docs/architecture.md#L247-L261, cmd/ateapi/main.go#L84-L88, cmd/ateapi/internal/authz/interceptor.go#L28-L60.

ate-api-server: the Actor control plane

ateapi owns Actor lifecycles, schedules them onto Workers and coordinates snapshots. Its gRPC server connects to PostgreSQL, creates Kubernetes clients and configures its mTLS credentials.

Sources: docs/glossary.md#L56-L58, cmd/ateapi/main.go#L509-L521, cmd/ateapi/main.go#L560-L586.

templateReconciler := controlapi.NewActorTemplateReconciler(persistence, controlSrv, *templateResyncInterval) templateReconciler.Start(shutdownCtx) workerAssignmentReconciler := controlapi.NewWorkerAssignmentReconciler(persistence, workerCache) workerAssignmentReconciler.Start(shutdownCtx) 

Source: cmd/ateapi/main.go#L326-L330.

Despite their names, these reconcilers are internal Substrate loops rather than controller-runtime reconcilers watching custom resources. ActorTemplateReconciler prepares golden snapshots; WorkerAssignmentReconciler handles Actors lost when ateom restarts.

This component does not map one-to-one to kube-apiserver: it combines Actor-specific API handling, scheduling and workflows.

atecontroller: conventional Kubernetes reconciliation

atecontroller uses ctrl.NewManager and registers four reconcilers, including WorkerPoolReconciler. Its core structure is familiar:

func (r *WorkerPoolReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { log := log.FromContext(ctx) wp := &atev1alpha1.WorkerPool{} if err := r.Get(ctx, req.NamespacedName, wp); err != nil { if k8errors.IsNotFound(err) { return ctrl.Result{}, nil } return ctrl.Result{}, fmt.Errorf("failed to get worker pool %q: %w", req.NamespacedName, err) } if !wp.GetDeletionTimestamp().IsZero() { log.Info("WorkerPool is being deleted") return ctrl.Result{}, nil } if err := r.reconcileWorkerPool(ctx, wp); err != nil { log.Error(err, "Failed to reconcile worker pool") return ctrl.Result{}, err } return ctrl.Result{}, nil } 

Sources: cmd/atecontroller/main.go#L233-L276, cmd/atecontroller/internal/controllers/workerpool_controller.go#L105-L129.

Reconciliation does more than produce a Deployment. The WorkerPool reconciler protects referenced SandboxConfigs with finalizers; a separate SandboxConfigProtectionReconciler releases protection. Updating the recorded SandboxConfig uses two status writes so that status never references a configuration whose protection has already been released.

Owns(&Deployment{}) and Watches(&SandboxConfig{}, ...) connect these resources through normal controller-runtime mechanisms.

Sources: cmd/atecontroller/internal/controllers/workerpool_controller.go#L134-L136, cmd/atecontroller/internal/controllers/workerpool_controller.go#L416-L425.

atelet and ateom: checkpoint and restore

atelet runs on each node as a DaemonSet. ateom runs inside worker Pods and exposes operations such as RunWorkload, CheckpointWorkload and RestoreWorkload.

atelet coordinates worker supervision, image caching and CSI volumes. It also reuses kubelet's image credential-provider mechanism through --image-credential-provider-config.

Sources: docs/architecture.md#L324-L326, cmd/atelet/main.go#L103-L104.

Snapshot upload and download are delegated to objectstore-plugin sidecars over a Unix-socket gRPC API:

The main processes do not directly transfer snapshots to object storage. This does not mean they hold no credentials at all: mTLS and database authentication remain separate concerns.

Source: docs/architecture.md#L330.

For gVisor, cmd/ateom-gvisor/runsc.go invokes runsc checkpoint and runsc restore. The microVM implementation uses Kata integration and Cloud Hypervisor, with userfaultfd demand paging when restoring memory.

Sources: docs/architecture.md#L328-L338.

Kubelet and the container runtime still run the worker Pod itself. Substrate manages sandboxed Actors inside it. Kubernetes already has a Kubelet Checkpoint API; Substrate's distinction is coordinating persistence, assignment, restore and reconnection as an Actor lifecycle.

atenet: routing that can resume its destination

Ingress requests identify an Actor with the ate-target-actor header, formatted as <atespace>/<actor>. In the Envoy configuration described here, an ext_proc processor calls the control plane's ResumeActor RPC before resolving the destination.

Once the assignment is available, the router opens an mTLS CONNECT tunnel to the Worker's atunnel listener on port 8443. atunnel, hosted by ateom, forwards traffic over the Actor's private veth interface. Worker Pod port 80 is not a direct Actor ingress endpoint.

Sources: docs/architecture.md#L344-L354.

The Kubernetes analogy is Service/Ingress/Gateway routing with an extra orchestration step: restore the destination if necessary before forwarding to it.

podcertcontroller: Pod certificate signers

podcertcontroller provides three signers:

Source: cmd/podcertcontroller/main.go#L15-L22.

It watches certificates.k8s.io/v1 PodCertificateRequest objects. The API became GA in Kubernetes v1.37:

PodCertificateRequest: { {Version: version.MustParse("1.34"), Default: false, PreRelease: featuregate.Alpha}, {Version: version.MustParse("1.35"), Default: false, PreRelease: featuregate.Beta}, {Version: version.MustParse("1.37"), Default: true, PreRelease: featuregate.GA}, // LockToDefault in 1.38 }, 

Source: kube_features.go.

The API lets kubelet request certificates and project them into Pods. A signer must still process those requests. The well-known kubernetes.io/kube-apiserver-client-pod signer described in the API comments is unimplemented in this version.

Substrate fills its own needs with the three signers above. Its README calls the controller a “polyfill” for the missing signers.

Sources: certificate API types, README.md#L268-L269.

Keep session state and API credentials separate

Isolation, identity and credentials are separate controls

Source: Agent Substrate on GKE, Google Cloud Blog (September 16, 2026).

Kernel isolation does not by itself protect external API credentials. That also requires the gateway and credential-provider configuration described below.

The security document distinguishes Met, In Progress, Future and Out of Scope. Selected principles in v0.5.0 are:

Sources: docs/security.md#L30-L33, docs/security.md#L41, docs/security.md#L59, docs/security.md#L75-L77.

Warning: The same document lists Atespace authorization as In Progress, and node blast-radius containment and storage encryption as Future. The security architecture is not evidence that every control is fully implemented. Ordinary Control RPC authorization still requires the experimental flag.

Inject credentials outside the Actor

With egress credential injection, an Actor sends an HTTPS request containing a placeholder header. The gateway obtains the real credential and substitutes it outside the sandbox:

1. The Actor sends the request with a placeholder value.
2. The egress gateway terminates TLS and matches an EgressPolicy header-replacement rule.
3. The credential provider authorizes access and reads the secret.
4. The gateway replaces the header and makes a separate TLS connection to the destination.

The Actor must trust the gateway's CA. If the request does not contain the header to replace, credential lookup and replacement do not occur.

This keeps the API key outside Actor memory and snapshots. It does not prevent misuse of an allowed API; destination restrictions and application authorization still matter.

Sources: docs/egress-credential-injection.md#L1-L15, docs/egress-credential-injection.md#L41-L54.

The reference kubernetes-secrets provider starts with an empty deny-all policy. Readable namespaces must be explicitly allowed for each Atespace. Actors do not read Kubernetes Secrets themselves.

Source: README.md#L132-L136.

EgressPolicy defines destinations and header replacement

An EgressPolicy specifies allowed traffic and header replacement rules. The credential provider separately decides whether the Actor can use the referenced secret. The API defines the policy as an unordered set of allows: traffic matching no rule is denied.

Source: pkg/proto/ateapipb/ateapi.proto#L609-L610.

For example, allow HTTPS to api.example.com and replace its Authorization header:

rules: - https: hostnames: ["api.example.com"] effects: replaceHeaders: - header: Authorization prefix: "Bearer " credentialUri: ate-secret://k8s.io/default/ns1/example-api/token 

Source: docs/egress-credential-injection.md#L69-L76.

The policy is associated with an Actor rather than embedded in its ActorTemplate:

# Create an egress policy. kubectl ate create egress-policy <actor-name> -a <atespace> -f policy.yaml 

Source: cmd/kubectl-ate/README.md#L247-L253.

Note: The Claude Code demo discussed later directly places ANTHROPIC_API_KEY in the ActorTemplate environment. That differs from gateway injection. In either case, the Actor needs an egress policy permitting its HTTPS destination, such as api.anthropic.com.

The complete request path

With routing and credential injection in place, an HTTP-triggered agent can resume and call an external LLM as follows. This combines the documented flows; Counter itself does not call an LLM.

sequenceDiagram actor User as User / higher-level system participant Router as atenet-router participant API as ate-api-server participant Atelet as atelet participant Tunnel as ateom / atunnel participant Agent as Actor participant Egress as atenet-egress participant Cred as credential provider participant LLM as External LLM API User->>Router: HTTP request (ate-target-actor) Router->>API: ResumeActor alt Actor is suspended API->>Atelet: Restore request Atelet->>Tunnel: RestoreWorkload Tunnel-->>Atelet: Restore complete Atelet-->>API: Complete else Actor is running Note over API: Return existing assignment end API-->>Router: Worker assignment Router->>Tunnel: Forward through mTLS CONNECT Tunnel->>Agent: Forward request Agent->>Egress: HTTPS (placeholder header) Egress->>Cred: Request credential Cred-->>Egress: Return API key after authorization Egress->>LLM: Replace header and send over a separate TLS connection LLM-->>Egress: Response Egress-->>Agent: Response Agent-->>Tunnel: Result Tunnel-->>Router: Response Router-->>User: Response Note over User,API: A higher-level system can request SuspendActor to save state and release the assignment

Sources: Resume sequence, credential injection. Cold boot without a snapshot and other branches are omitted.

The Actor-to-gateway and gateway-to-provider connections use separate TLS sessions. The gateway must terminate the first to replace headers in the second.

Try the Counter demo with kind

The Counter workload makes state preservation visible: each request increments counters in process memory, a file and the root filesystem. The example below uses gVisor and includes recorded output from a local kind run.

Set up the cluster and demo

Building v0.5.0 requires Go 1.27.0 and a working Docker-based development environment. The microVM option additionally requires /dev/kvm and suitable virtualization support.

Sources: go.mod#L1-L5, hack/create-kind-cluster.sh#L74-L87.

Use a dedicated checkout and cluster name. The creation script deletes and recreates an existing cluster with the same name, so choose one that does not overlap with an environment you want to keep.

git clone --branch v0.5.0 --depth 1 https://github.com/agent-substrate/substrate.git substrate-v0.5.0 cd substrate-v0.5.0 export KIND_CLUSTER_NAME=ate-overview # create cluster and local registry (IPv4; IP_FAMILY=dual|ipv6 overrides) hack/create-kind-cluster.sh # install ate, PostgreSQL, rustfs hack/install-ate-kind.sh --deploy-ate-system --credential-provider='{"name":"k8s.io"}' # install counter demo hack/install-ate-kind.sh --deploy-demo-counter # install kubectl-ate go install ./cmd/kubectl-ate # create a counter actor in the demo's atespace (--template names the # actor template, resolved in the actor's atespace) kubectl ate create actor my-counter-1 -a ate-demo-counter --template counter # port-forward the network router to bind to local port `8000` kubectl port-forward -n ate-system svc/atenet-router 8000:80 

Source: README.md#L111-L129.

The creation script sets up a local registry and a kind cluster with kindest/node:v1.37.0. It explicitly enables certificate-related feature gates, including ClusterTrustBundleProjection; some comments still describe v1.36 requirements.

Source: hack/create-kind-cluster.sh#L104-L113.

The system installation uses ko to build and push its Go components and deploys PostgreSQL, RustFS, CRDs and SandboxConfig resources. The Counter installation adds its WorkerPool and ActorTemplate and waits for a golden snapshot.

WorkerPool and ActorTemplate

The demo's WorkerPool includes the following settings. The installer resolves template placeholders and ko:// references before applying the configuration.

apiVersion: ate.dev/v1alpha1 kind: WorkerPool metadata: name: counter namespace: ate-demo-counter labels: workload: counter spec: replicas: 3 workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor sandboxClasses: - name: gvisor template: resources: limits: cpu: "1" memory: 1Gi requests: cpu: 250m memory: 1Gi 

Source: demos/counter/counter.yaml.tmpl#L26-L63.

An ActorTemplate is not a Kubernetes object. This YAML maps to Substrate's protobuf fields and is passed through kubectl ate create actor-template:

metadata: atespace: ate-demo-counter name: counter workerSelector: matchLabels: workload: counter containers: - name: counter image: ko://github.com/agent-substrate/substrate/demos/counter wakeupProbe: httpGet: Source：https://dev.to/yosshi_/an-overview-of-agent-substrate-501d
My favorites

Related posts

Five Things That Make Dependabot Painful for Terraform

A Terraform block in dependabot.yml looks identical to an npm or uv one: - package-ecosystem : terraform directory : /te

0dev.to GitHub

Tracking Roblox player counts: what the public data actually gives you

Roblox does not publish a public API for per-game concurrent player counts, so anything you see is a sample. I have been

0dev.to Data

Telegram gold signals this week: results to Oct 30

From October 24 to October 30, we executed 3,629 signals from 405 Telegram gold channels at the broker. More channels ma

0dev.to Data

Google Trending Searches Today in 10 Countries (Nov 02, 2026)

What people are searching on Google right now: the top Google Trends trending searches in 10 countries on November 02, 2

0dev.to Data

Custom-Made Furniture in India: Sizes, Budget, Timeline and What to Check Before Ordering

Custom made furniture India buyers order is built around your room's measurements, your choice of wood and finish, and t

0dev.to Design

Mobile App Onboarding: A UX and Development Checklist for 2026

Mobile app onboarding sits at the intersection of product design, authentication, permissions and application performanc

0dev.to Design