An enterprise AI agent isn’t a bigger chatbot. A chatbot answers one message at a time. An agent keeps working toward a goal: it reads what the user is asking for, breaks the work into steps, pulls in the knowledge it needs, weighs the evidence, calls external tools, and checks the results before it decides whether to keep going or hand back an answer. Every one of those steps touches real systems and real data, so governance, observability, and trust controls must live at each step. Most chatbot designs don’t provide them.

This post lays out a reference architecture for that kind of agent, organized as five layers: an API gateway that handles identity and policy, a five-phase agent runtime (Understand, Plan, Retrieve, Reason, and Act and Observe), a knowledge and state tier that gives the agent memory, a platform-services layer for models and tools, and a trust and control framework that runs across all of them. Each layer maps to AWS managed services. We also cover the design patterns that work and the mistakes we have seen teams make.

The rise of enterprise AI agents

Enterprises are moving past single-turn question-and-answer interfaces for business reasons. They want to automate more of the routine work so their people can spend time elsewhere, ship products and customer experiences faster, and get more done without adding headcount. Those goals justify the investment, but reaching them requires moving from single-turn chatbots to agents that can carry out multi-step work on their own. That jump brings new problems to solve in orchestration, memory, and governance, and the rest of this section is about what those problems demand of the architecture.

From chatbots to agents

Enterprise AI is shifting from systems that answer a question and stop to agents that plan, use tools, and come back with an outcome you can verify. The difference is the loop. Rather than returning one response, an agent cycles through understanding, planning, retrieval, reasoning, and action, and it keeps cycling until it either reaches the user’s goal or decides a human needs to step in.

At AWS re:Invent 2025, Werner Vogels said that the developers who succeed think in systems and build with precision. The teams pulling ahead with agents aren’t the teams that moved first or chose the newest model. They are the ones who treat the whole thing as a system and understand that a choice made in one layer shows up in every layer around it.

What is an AI agent? An AI agent uses a foundation model (FM) as its reasoning core and adds tools, memory, and guardrails so it can take on work that takes more than one step. Where a single-turn chatbot answers and forgets, an agent holds onto state, reasons across several steps, and acts in systems outside itself.

The enterprise imperative: Governed, observable, trustworthy, cost-aware

A capable model is the more straightforward part. Running agents in an enterprise means putting governance, observability, trust controls, policy enforcement, and cost controls around every step they take. Cost matters as much as capability: many enterprises find token spend growing faster than the productivity gains it delivers, so an agent must also be designed to spend inference where it adds value. That is what this architecture is built to do:

• Governed – Every action clears a policy gate, an approval workflow, or a guardrail before it runs.
• Observable – Each step emits a trace (latency, tokens, tool status, and execution path) for later audit.
• Trustworthy – Confidence scoring, evaluations, and human checkpoints sit between the agent and the user.
• Iterative – The loop sharpens its understanding on each pass as tool results come back.

What this post covers

What follows is a reference architecture for enterprise AI agents on AWS, walked through layer by layer from the first request to the verified result, with diagrams, the AWS services that implement each piece, and the patterns worth copying and avoiding. The aim is to give architects and engineering leaders a shared vocabulary and a mental model for designing these systems.

Architecture overview: The governed agent loop

The AWS Well-Architected Framework Generative AI Lens organizes generative AI workloads around six pillars (operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability), each with its own practices and controls. We split the agent into layers that line up with those pillars, so every layer owns a clear responsibility that traces back to a recognized standard instead of a one-off choice. Teams can change, secure, and scale one layer without disturbing the others while keeping the end-to-end governance the Lens requires.

Layered architecture design

The design is layered and loop-based, with five tiers that each do one job. The agent runtime sits in the middle. It’s a five-phase loop that turns a user’s goal into a verified outcome. Around it sit the layers for knowledge, platform services, trust controls, and the governance that cuts across everything.

Reference architecture diagram

Figure 1 shows the five layers together. Read it top to bottom: the API gateway takes the request and enforces identity and policy, then the flow moves down through the agent runtime, the knowledge and state tier, and the platform services, with the trust and control framework running alongside on the right.

Design principles

Five principles shaped this design, and each one left a mark on the layers you will see later in concrete ways. Zero trust is why identity context travels through every phase of the loop. Observable by default is why there is a dedicated step-level tracing tier instead of leaning on application logs.

These principles aren’t ad-hoc. They come directly from the AWS Well-Architected Framework Generative AI Lens. The Lens’s security guidance maps closely to the following five principles. Least-privilege access across models, data stores, endpoints, and workflows is what zero trust and defense in depth rest on. Securing the data and the traffic between components is separation of concerns in practice. Watching and enforcing the boundaries between the control and data planes is observable by default doing its job. And constraining what the agent is allowed to do, through guardrails and clear limits, is where fail-safe defaults come from. Because it builds on the Lens, this design follows recognized AWS guidance rather than a single team’s convention. It also reflects what AWS Professional Services has implemented and run in production with enterprise customers across industries such as retail and technology: lessons learned in the field that we apply on every engagement, not a set of theoretical best practices. For the full security guidance, see the Generative AI Lens security pillar in the AWS Well-Architected documentation.

Principle	Description
Zero trust	Every request is authenticated and authorized before reaching the runtime. User identity flows through every phase.
Defense in depth	Multiple layers of controls (API gateway, policy gate, trust and control, governance) facilitate no single point of failure.
Separation of concerns	Each architectural layer has clear boundaries and responsibilities. Knowledge is separated from compute. Trust is separated from execution.
Observable by default	Every step produces telemetry. Tracing, metrics, and logs are first-class architectural concerns, not afterthoughts.
Fail-safe defaults	When uncertain, the agent defaults to the safest action: clarify with the user or escalate to a human.

Entry point: User goal and API gateway, identity and policy

Layer 1 of 5 · Entry point: API gateway, identity, and policy (see Figure 1)

Every interaction starts with a user goal: a request in plain language that kicks off the loop. Before it reaches the runtime, it goes through the API gateway and the identity and policy layer, where the zero-trust posture is enforced.

User goal

The user goal starts with a business objective that drives the top-level intent, stated in natural language. It might be as small as “What was our Q3 revenue?” or as involved as “Book a meeting with Sarah, check her calendar availability, and send a summary of our last project discussion.” The agent’s first job is to read that intent and work out the most efficient way to deliver it.

API gateway

The API gateway is the front door for every interaction. It terminates TLS, routes requests (REST, plus WebSocket for streaming responses), applies rate limits, and validates input. On AWS, this is usually Amazon API Gateway with WebSocket APIs for real-time streaming.

Identity and authentication

No request reaches the runtime unauthenticated. The identity layer checks OAuth 2.0 / OpenID Connect (OIDC) tokens, resolves who the caller is, and attaches their context (role, permissions, organization, and session ID) to the request. That context rides along through every later phase, so each tool call and data lookup stays within the caller’s permissions.

Policy enforcement

The policy layer handles role-based access control (RBAC), tool-level permissions, data scoping, and usage quotas. It decides which tools the agent might call for a given user, which data it might touch, and how many tokens it might use in a session.

Component	Responsibility	AWS service
API gateway	TLS termination, request routing, WebSocket streaming, rate limiting	Amazon API Gateway, Application Load Balancer (ALB)
Identity	Token validation, OAuth 2.0/OIDC, user context injection, session management	Amazon Cognito, AWS Identity and Access Management (IAM)
Policy	RBAC, tool permissions, data scoping, usage quotas	IAM policies, AWS Organizations

Design principle: Zero trust. The gateway authenticates and authorizes every request before it reaches the runtime. The user’s identity, role, and permissions ride on the session context and stay with the request through every phase of the loop.

The agent runtime: Five-phase cognitive loop

Layer 2 of 5 · Agent runtime: the five-phase cognitive loop (see Figure 1)

The runtime performs the core work of the agent. It runs a continuous loop of five phases that track how a careful professional handles a hard task: understand what is being asked, plan an approach, gather the information you need, reason through the choices, then act and watch what comes back before going around again. (see Figure 2)

A note on the model. We aren’t claiming a new theory of agent cognition here. These five phases combine proven ideas under one vocabulary. They draw on the sense, plan, act loop from robotics, the reasoning and acting pattern from ReAct prompting, and the self-reflection step from approaches such as Reflexion (a technique in which the model critiques its own previous attempt and uses that feedback to improve the next one). The enterprise contribution is naming the phases out loud, so you have a clear place to attach governance, observability, and trust controls at each one. If you already think in three or four steps, say plan, act, observe, these phases map directly onto that model.

Passes through the loop can end in a final answer, or they can go around again with new evidence from a tool result, an error to handle, or extra context. This ability to refine its work separates an agent from a single large language model (LLM) call.

Phase 1: Understand (intent, context, and constraints)

Understand is the agent’s perception step. It reads the user’s input and pulls out three things:

• Intent – What the user is actually trying to do. The agent classifies the request type (retrieve information, perform an action, or generate content) and identifies the domain.
• Context – A working picture built from several sources: the conversation so far (short-term memory), the user’s profile and permissions from the gateway, session state, and relevant history from earlier interactions (long-term memory).
• Constraints – The boundaries it has to respect: time limits, data access tied to the IAM role, token budgets, which tools are available, and organization-specific policies.

On AWS. Understand usually runs on Amazon Bedrock with a foundation model such as Anthropic Claude or Amazon Nova doing the intent classification. For model availability by AWS Region, refer to Supported models by AWS Region in Amazon Bedrock. The system prompt carries the agent’s persona, its tools, and its behavioral guardrails, and Amazon Bedrock Guardrails filter the input before the model ever sees it.

Phase 2: Plan (decompose, select tools, and define success criteria)

With the intent clear, the agent moves to Plan, where the foundation model breaks the goal into steps.

Sub-step	Description	Output
Decompose	Break the user’s goal into a sequence of sub-tasks. Complex goals might require multiple tool calls and intermediate reasoning steps.	Ordered task graph (directed acyclic graph, or DAG)
Select tools	For each sub-task, select the appropriate tool(s) from the tool registry using tool descriptions and few-shot examples.	Tool-to-task mapping
Success criteria	Define what “done” looks like for each sub-task. The Reason phase uses this to evaluate whether a tool call succeeded.	Evaluation rubric

No tool runs straight from the plan. Each call first passes through the tool gateway and policy enforcement layer, which checks permissions, inspects the input parameters, applies governance policy, and confirms the request is within budget.

Re-planning. Plan is not a one-shot step. When Reason sees that a sub-task failed or came back wrong, the agent can revise the plan by reordering the task graph, choosing a different tool, or splitting a stubborn step into smaller ones.

Phase 3: Retrieve (knowledge, memory, and session state)

In Retrieve, the agent grounds its reasoning in facts from three sources:

• Knowledge base (Retrieval Augmented Generation, or RAG) – Semantic search over enterprise documents using embeddings and vector retrieval, with reranking for precision and access filtering for security.
• Memory – Short-term (the conversation buffer), long-term (preferences and facts it has learned), and episodic (how past tasks turned out).
• Session state – The current interaction’s tool-result cache, intermediate computations, and external API responses.

The RAG pipeline runs in four stages:

1. Query rewriting: Optimize the natural language query for search, optionally generating multiple queries for different facets.
2. Embedding and search: Embed the query and match against the vector index using approximate nearest neighbor (ANN) search.
3. Reranking: A cross-encoder model reranks initial results for precision, filtering out false positives.
4. Access filtering: Results are filtered based on the user’s identity and data access policies.

On AWS. Amazon Bedrock Knowledge Bases is the fully managed RAG capability. It handles chunking, embedding with Amazon Titan Embeddings, and vector storage in Amazon OpenSearch Serverless. For memory, Amazon DynamoDB holds session state and Amazon ElastiCache caches recent results for low-latency reads.

Phase 4: Reason (model routing, decide, and reflect)

Reason is where the model turns evidence into a decision. With the retrieved context in hand, it does three things:

• Model routing – Pick the right model for this step. Simple work goes to smaller, faster models. Complex reasoning goes to a larger, more capable model. That keeps cost and latency down, and it’s one of the most effective cost-optimization levers in the architecture.
• Decision – Combine what was retrieved with the plan to decide the next tool call, the parameters to pass, whether the plan needs changing, or whether to ask a clarifying question.
• Reflection – Look at the result of each action. Did the call succeed? Does the output match what was expected? Does the plan need to change?

Anti-pattern: Infinite loops. Knowing when to stop is a core design problem. With no safeguards, an agent can get stuck reasoning in circles. The usual defenses are a cap on iterations, a timeout budget, classifying errors as transient or permanent, and a path to escalate to a human. Getting this wrong costs more than one unfinished task. A looping agent keeps calling the model and its tools, so token spend and compute climb with every wasted pass and nothing gets done. The effects compound. Latency rises and service level agreements (SLAs) slip. Downstream APIs and model endpoints absorb redundant load, which can trigger throttling for other workloads. The user waits on a request that never completes, which erodes the trust this architecture exists to protect. Caps on iterations, time, and tokens, plus an escalation path, keep a confused agent from turning into a runaway bill.

Phase 5: Act and observe (APIs, code, data, and MCP tools)

Act and Observe is where the agent reaches into the outside world. It carries the most governance risk because it changes real systems.

• Tool execution – The agent calls external tools through standard interfaces (REST APIs, code sandboxes, data connectors, and Model Context Protocol (MCP) tools) and logs every call with its full request and response.
• Observation – Each result is parsed, validated, and fed back into context: structured returns, error messages, status codes, side effects.
• Approval gate – Before a destructive action runs (a write, delete, or send), it goes through the policy and approval gate. Depending on the risk and the organization’s policy, that is an automated check, a human sign-off, or both.

The feedback loop

When a tool returns success, an error, or partial data, the result flows back into Understand and becomes part of the context for the next pass. Each pass through the loop uses this new context to move closer to the right answer. What flows back includes:

• Tool results: Structured data, API responses, and computation outputs.
• Errors: Error messages, timeouts, and permission denials that inform replanning.
• New evidence: Information discovered during tool execution that changes the understanding of the task.

Tool gateway and policy enforcement

The tool gateway sits between the runtime and the platform services, and it’s the single checkpoint every tool call goes through. It runs the RBAC checks, validates input, applies governance policy, and enforces rate limits and cost budgets. On AWS, that maps to Amazon Bedrock Agent action groups with IAM-based authorization, and you can add AWS Lambda authorizers when you need custom policy logic.

Knowledge and state layer

Layer 3 of 5 · Knowledge and state (see Figure 1)

The knowledge and state layer gives the agent two things it cannot work without: factual grounding and memory that survives across sessions. It’s organized as four tiers:

Tier	Purpose	AWS service
Enterprise search	Full-text and semantic search across organizational knowledge bases, documentation, and wikis.	Amazon Kendra, Amazon OpenSearch Service
Vector index	Embeddings store for RAG retrieval. Supports approximate nearest neighbor search for semantic similarity.	Amazon OpenSearch Serverless, Amazon Aurora PostgreSQL (pgvector), Amazon S3 Vectors
Short-term memory	Conversation buffer and session context window. Provides continuity within a single interaction.	Amazon ElastiCache, Amazon DynamoDB (time to live, or TTL)
Long-term memory	User preferences, learned patterns, facts, and knowledge graph. Persists across sessions.	Amazon DynamoDB, Amazon Neptune

Platform services layer

Layer 4 of 5 · Platform services (see Figure 1)

The platform services layer is the shared infrastructure the runtime depends on. These managed services keep operational complexity out of the agent.

Model gateway

The model gateway is one interface in front of many foundation models. It routes each task to the right model, falls back to another when one fails, and handles rate limiting, token metering, and response caching. Because it hides each provider’s API behind a common schema, you can swap models without touching agent code.

On AWS, Amazon Bedrock is that gateway, with serverless access to a growing lineup of foundation models including Anthropic Claude, Amazon Nova, Meta Llama, and Mistral. For model availability by Region, refer to Supported models by AWS Region in Amazon Bedrock. To run custom models, use Amazon SageMaker endpoints for managed inference with automatic scaling.

Tool registry

The tool registry is the catalog of every API, function, and connector the agent can call, along with their schemas, permissions, descriptions, and versions. Plan reads this registry to choose tools for each sub-task. Each tool is registered with an OpenAPI schema that spells out its inputs, outputs, and side effects.

On AWS, Amazon Bedrock Agent action groups hold those schemas and wire them to AWS Lambda functions or API endpoints, giving you a governed, versioned catalog the agent can discover and call.

Secrets and IAM

This is where API keys, database credentials, and OAuth tokens live and get rotated. Fine-grained IAM policies decide which agents can reach which secrets, so credentials don’t leak into a trace or a log. AWS Secrets Manager handles the rotation, and IAM enforces least privilege.

Event bus

The event bus moves agent lifecycle events (start, tool call, completion, and error) asynchronously, which lets monitoring, alerting, and downstream integrations stay decoupled from the agent. Amazon EventBridge is the backbone, with Amazon Simple Notification Service (Amazon SNS) and Amazon Simple Queue Service (Amazon SQS) for fan-out and queuing.

Observability: Step-level tracing

Watching an agent isn’t like watching a normal application. An agent runs a non-deterministic loop. The number of steps varies, tool calls branch, and plans change mid-execution. Step-level tracing must capture the full execution graph, not only the request and response.

At every step, the observability layer captures four kinds of signal:

Signal	What it captures	Why it matters
Step-level tracing	Parent-child span relationships, phase transitions, decision branching points	Debug non-deterministic behavior. Trace root cause of incorrect outputs
Latency	Per-step wall clock time, model inference time, tool round-trip time	Identify bottlenecks. SLA compliance. Performance optimization
Tokens	Input and output tokens per model call, context window utilization, cumulative usage	Cost attribution and optimization (per user, tenant, and workflow). Context overflow prevention. Prompt optimization
Tool status	Tool call success/failure rates, response codes, retry counts, payload sizes	Tool reliability scoring. Automatic fallback decisions. SLA monitoring

On AWS – Use AWS X-Ray with custom segments for the agent’s steps, Amazon CloudWatch for metrics and alarms, and Amazon CloudWatch Logs Insights to query trace data. Amazon Bedrock logs model invocations natively, with the input and output payloads and token counts.

Trust and control framework

Layer 5 of 5 · Trust and control, with cross-cutting governance (see Figure 1)

A lack of trust is the main barrier to deploying enterprise agents. This layer sits between the runtime and the final output and runs each action through a series of gates to confirm it meets the organization’s standards before it ships. These controls also help organizations meet the obligations of emerging regulatory frameworks such as the EU AI Act, helping reduce exposure to penalties and keep AI use compliance-aware and responsible. (see Figure 3)

Grounded in recognized standards. The four gates aren’t an arbitrary sequence. Each one maps to a responsible-AI control that already exists. They track the responsible-AI dimensions of the AWS Well-Architected Generative AI Lens (safety, veracity and robustness, controllability, transparency, and governance) and the risk functions in the NIST AI Risk Management Framework (govern, map, measure, manage). The mapping is direct. Gate 1, guardrails, is the safety gate: it filters harmful content and the output-handling risks in the OWASP Top 10 for LLM Applications. Gate 2, evaluations, is where veracity and robustness get checked, through groundedness and hallucination tests. Gate 3, confidence scoring, is the transparency gate. It surfaces uncertainty and shows the source of each claim. Gate 4, human approval, is what keeps a person in control as the system scales. Measured against standards like these, the pipeline is auditable on terms reviewers and regulators already know, rather than an internal checklist.

Gate 1: Guardrails

Content filtering, personally identifiable information (PII) detection, toxicity checks, and topic deny-lists. Guardrails work on the output, scanning the agent’s response before the user sees it. On AWS, Amazon Bedrock Guardrails gives you configurable filters for hate, insults, sexual content, violence, and misconduct, plus your own denied topics and PII redaction.

Gate 2: Evaluations

This gate checks whether the retrieved content supports the response. This gate covers groundedness, hallucination detection, and format validation, and it can hand the output to a separate “judge” model to score the primary model’s work.

Gate 3: Confidence scoring

The agent rates its own confidence, attributes its key claims to sources, flags where it is unsure, and checks that confidence against a threshold. Anything below the line goes to a human instead of straight to the user.

Gate 4: Human approval (HITL)

For high-risk actions, such as moving money, modifying data, or sending external communications, the agent routes through a human-in-the-loop (HITL) approval step. The reviewer sees the proposed action, the agent’s reasoning, and the confidence score, and can approve it, change it, or reject it. Every decision is logged.

Verified outcomes

After an action clears the gates, it ends up as one of four outcome types, each with its own governance bar:

Outcome	Description	Risk level	Governance
Respond	Generate a natural language response to the user. Lowest risk: guardrails and content filters are sufficient.	Low	Guardrails only
Write	Create or modify data in an external system (database record, document, configuration).	Medium	Policy gate + logging
Trigger workflow	Initiate an automated workflow or pipeline (deployment, data pipeline, notification chain).	High	Policy + HITL approval
Escalate / HITL	Agent determines that human intervention is needed: confidence too low, action too risky, or request out of scope.	Critical	Full approval chain

Cross-cutting governance

Governance is the layer underneath everything else. It helps the organization operate within its legal, regulatory, and ethical requirements at every level, and it rests on five pillars (see Figure 4):

• Security: Encryption at rest and in transit, virtual private cloud (VPC) isolation, AWS WAF and AWS Shield protection, data classification, and network segmentation.
• Observability: Distributed tracing, metrics dashboards, anomaly detection, alert routing, and SLA monitoring across all agent operations.
• Audit: Immutable audit logs, decision trails, tool call records, data lineage tracking, and automated compliance reports.
• Cost: Token metering per user and team, model tiering for cost optimization, budget alerts, per-user quotas, and cost attribution dashboards.
• Compliance: Considerations such as data residency, retention policies, and mechanisms to support data subject requests (for example, access and deletion) that your organization might need to address under regulations like GDPR or CCPA. If your workload processes protected health information, use HIPAA eligible services and review your responsibilities under the AWS Shared Responsibility Model. AWS compliance programs such as SOC 2 cover the underlying AWS services, not your application. See AWS Services in Scope by Compliance Program.

Design patterns and best practices

The following patterns and anti-patterns come from AWS Professional Services engagements in which we built and ran enterprise agents in production with customers, not from a whiteboard.

Teams already run architectures like this in production on AWS. One of these is Druva, a data security provider, which built DruAI, a multi-agent system on Amazon Bedrock AgentCore where specialized agents handle telemetry analysis, investigation guidance, and recovery workflows. According to the Druva case study, 68 percent of customer issues are now resolved through agent workflows, support-assisted resolution is 58 percent faster, and cyber investigations that once took 30–60 days now finish in minutes. Separately, Cox Automotive used Amazon Bedrock AgentCore to move 17 agentic solutions into production in under a year, with seven more in development. This architecture reflects patterns from these deployments.

Recommended patterns

These patterns answer design questions that keep coming up when teams build enterprise agents. Each pattern stands on its own. Adopt the ones that fit your workload and leave the rest.

Pattern: Graduated autonomy

Start agents read-only and widen their write access a step at a time as they earn trust. Use feature flags or IAM role escalation to set the autonomy level per user, team, or use case. That way the organization builds confidence in the system before handing it broader permissions.

Pattern: Fail-safe defaults

When the agent encounters a missing tool, an API timeout, or an ambiguous request, it should take the safe route: ask a clarifying question instead of guessing, or escalate to a human instead of risking a bad action. Encode this in the system prompt and back it up in the trust and control layer.

Pattern: Multi-model strategy

Match the model to the phase. Small, fast models can handle intent classification and simple routing (Phase 1). Larger models justify their cost in planning and reasoning (Phases 2 and 4). Specialized models do the evaluations in the trust layer. Teams that route this way usually cut cost while keeping quality where it counts, though the actual savings depend on your mix of work, your model choices, and your traffic. Benchmark against your own use cases before you commit to a target.

Pattern: Structured tool interfaces

Give every tool an explicit OpenAPI schema that defines input and output types, error codes, rate limits, and declared side effects. That is what lets Plan pick tools intelligently and lets Observe check what came back against what was promised.

Anti-patterns to avoid

These are the traps we have watched teams fall into early. Recognizing them early reduces rework.

Anti-pattern: Monolithic agent

Resist the urge to build one agent that does everything. Split the work into specialized agents (for example, calendar, code, and data analysis) coordinated by a supervisor agent. You get better reliability, straightforward testing, and the freedom to scale and deploy each one on its own.

Anti-pattern: Unbounded loops

Put a hard ceiling on loop iterations. A common production setting is 3 to 5 iterations against a total token budget. Past that, the agent should summarize where it got to and either escalate or ask the user what to do. Without that ceiling, a confused agent can spend tokens indefinitely and still get nowhere.

Anti-pattern: Trust-later architecture

Don’t file implementing governance, guardrails, and observability under “later.” They must be primitives from day one. Bolting trust controls onto an agent that is already in production costs more and introduces more defects than building them in from the start.

Bringing this architecture to life on AWS

This section takes the Figure 1 reference architecture and grounds it in concrete AWS services, mapping each layer of the five-layer loop to the managed services that implement it. Treat it as a set of sensible starting points, not a fixed bill of materials. Your scale, data residency, latency, and cost requirements will push some of these choices around, so use the mappings as a baseline, check them against the AWS Well-Architected Generative AI Lens, and tune for your own workload.

Mapping architecture layers to AWS services

Walking Figure 1 from the entry point down to verified outcomes, here is how each layer lands on AWS.

Layer 1: Entry point.

Configure Amazon API Gateway with a WebSocket API for streaming agent responses. Attach an Amazon Cognito user pool for OAuth 2.0/OIDC authentication. Define IAM policies that scope tool access by user role.

Layer 2: Agent runtime.

Create an Amazon Bedrock Agent with a system prompt that encodes your agent’s persona, behavioral guardrails, and the list of available tools. Define action groups that map to AWS Lambda functions or API endpoints. Enable Amazon Bedrock Guardrails for input and output filtering.

Layer 3: Knowledge and state.

Create an Amazon Bedrock Knowledge Base backed by Amazon Simple Storage Service (Amazon S3) as the data source and Amazon OpenSearch Serverless as the vector store. Configure Amazon DynamoDB tables for session state (with TTL for automatic expiry) and long-term memory. Optionally, add Amazon ElastiCache for low-latency result caching.

Layer 4: Platform services.

Amazon Bedrock acts as the model gateway. Register tools as Amazon Bedrock Agent action groups with OpenAPI schemas. Store API keys and credentials in AWS Secrets Manager with automatic rotation enabled. Configure Amazon EventBridge rules for agent lifecycle events.

Layer 5: Observability and governance.

Turn on Amazon Bedrock model invocation logging. Configure AWS X-Ray tracing for AWS Lambda functions. Set up Amazon CloudWatch dashboards for token usage, latency percentiles, and tool success rates. Turn on AWS CloudTrail for audit logging and AWS Config rules for compliance checks.

From baseline to production. These mappings give you a governed starting point, but most of the engineering effort goes into hardening this baseline into a production-grade agent tuned to your security, compliance, cost, and scale requirements. If you want to accelerate that journey, AWS Professional Services and AWS Partners can help you design, build, and operate this architecture for your own workloads. Learn more about engaging AWS Professional Services or finding an AWS Partner. Work with your security and legal teams to meet your organizational security, regulatory, and compliance requirements before deployment. Under the AWS Shared Responsibility Model, you are responsible for configuring and operating these services to meet those requirements.

Conclusion

The hard part of an enterprise agent was never the model. It’s everything around the model. That includes the identity that follows a request, the memory that persists, the gates an action clears, and the traces that show what happened. Get those right and the agent becomes something a business can actually rely on. Skip them and you have a clever demo that no one will put in front of a customer.

The governed, observable loop in this post is a reference point for teams moving past basic chatbots toward agents they can trust in production.

A few things worth holding onto:

1. Governance isn’t a feature. It belongs in every layer, not bolted on at the end.
2. You can’t govern what you cannot see. Step-level tracing is what makes reliable agent operations possible.
3. The loop is the point. The five-phase loop, with feedback on every pass, is what distinguishes an agent from a chatbot.
4. Trust is earned in steps. Start narrow and read-only, then widen as the system proves itself.
5. The pieces are already on AWS. Amazon Bedrock for models and guardrails, Amazon EventBridge for event-driven observability, and AWS IAM for access control. Every component maps to a managed service.

Next steps

• Explore Amazon Bedrock on the AWS Management Console to create your first agent with a knowledge base and action groups. See the Amazon Bedrock documentation for the latest agent capabilities.
• Read the documentation. The Amazon Bedrock documentation covers agent creation, tool configuration, and guardrails setup.
• Review the Well-Architected guidance. The AWS Well-Architected Framework includes best practices for AI/ML workloads.
• Browse reference architectures. The AWS Architecture Center contains additional patterns for AI/ML systems.
• Dive deeper into agentic AI. The Foundations of Agentic AI on AWS prescriptive guidance covers conceptual foundations, and the Enterprise Multi-Account Architecture for Agentic AI covers production infrastructure.
• Build it with AWS. Want to bring this architecture to your business? Engage AWS Professional Services or an AWS Partner to design, build, and operate a governed, observable enterprise AI agent tailored to your environment.

Further reading

For a service-by-service reference that maps these architectural layers to AWS offerings, refer to the AWS Prescriptive Guidance on agentic AI: Foundations of Agentic AI on AWS covers the conceptual foundations, and Enterprise Multi-Account Architecture for Agentic AI covers production infrastructure. Because managed services evolve, these guides are kept current with the recommended services for each layer.

---

About the authors