Releasing and evaluating AI agents on Amazon Bedrock AgentCore with A/B tests and explainability evaluators

A new version of an AI agent, with a new system prompt, model, or architecture, should reach every user only after it has shown, on real traffic, that it is better than the current version. I created the AgentCore A/B Release Gate GitHub Action to automate that decision. It runs an A/B test that compares the new version with the current one on live traffic, and an evaluator score decides whether the new version is promoted or rolled back.

Amazon Bedrock AgentCore provides the building blocks. AgentCore Runtime hosts multiple versions of an agent behind named endpoints, and AgentCore Gateway splits traffic between them with an A/B test. During the test, AgentCore Evaluations scores the sessions of both versions online, with the evaluators you choose for your use case: built-in, custom LLM-as-a-judge or code-based.

In this post, we use the AgentCore A/B Release Gate GitHub Action to release two versions of a customer-support agent. The action deploys each new container image next to the current version, splits live traffic between them, and promotes the new version only when it passes the quality gates you define. The gate is a deterministic, code-based evaluator that encodes the store's support workflow as rules. The first release is a latency optimization that breaks the workflow rules and is rolled back. The second release fixes a weakness of the agent and is promoted.

The showcase is a general-purpose scenario that shows how the action works from start to finish. The customer-support agent, its system prompts and its evaluator are examples, and the same release flow applies to other agents, changes and evaluation rules.

All logs and numbers in this post come from real runs.

Solution overview

For this post, we use a customer-support assistant for a fictitious online electronics store. Customers ask the assistant which orders they have, where a parcel is, and whether they can return an item. The store has a simple return policy: an item can be returned within 30 days of delivery, final-sale items such as gift cards cannot be returned, and orders that haven't been delivered can't be returned yet.

The store also has a support workflow that every answer must follow: look the facts up with the tools, never invent an order, tracking or return number, never ask the customer for information the agent can look up itself, and check that an item is eligible before opening a return for it. A code-based evaluator scores every turn of the agent against these rules, and the release gate promotes a new version only when it follows them significantly better than the current one.

The agent

The agent serves the store's customers. It asks for missing information, looks up the customer's data, answers with the facts it found, and opens a return when the customer asks for one and the store's policy allows it. It keeps the conversation of each session, so follow-up messages build on the earlier turns. The following session with the released agent shows a typical conversation (the tracking table is shown as lines, and the last answer is shortened):

Customer: Where is my keyboard? Agent: I'm happy to help! Could you please share the email address used for the order, or if you have the order ID handy, that would work too. Customer: ana@example.com Agent: Here are the tracking details for your keyboard (order ORD‑1002): Carrier: DHL Tracking number: JD014600003456789012 Estimated delivery: 2026‑09‑29 2026‑09‑25: Shipped from Madrid warehouse 2026‑09‑26: Arrived at Lyon sorting facility Customer: I don't need it anymore. Can I return it? Agent: I'm sorry, but you can only open a return after the item has been delivered. Since the keyboard (SKU KB‑310) is still in transit and expected to arrive on 29 September, the return window hasn't opened yet. ... 

The agent uses LangGraph's create_react_agent with the openai.gpt-oss-20b-1:0 model on Amazon Bedrock and runs on AgentCore Runtime inside BedrockAgentCoreApp. It receives {"prompt": "..."} and answers with {"result": "..."}.

The agent has five tools. They read and write a small, fixed dataset in memory (two customers, four orders, their shipments and returns) with a fixed "today" of 2026-09-27, so every run sees the same facts, and every answer can be checked against them:

The tools chain together. A typical conversation goes from find_customer_orders to get_order_details, and then to track_shipment for a delivery question, or to check_return_eligibility and create_return for a return. The return policy lives in the tools: check_return_eligibility and create_return refuse final-sale items, undelivered orders and items outside the 30-day window.

The opentelemetry-instrumentation-langchain library records every model call and tool call as OpenTelemetry spans (tool calls as execute_tool spans), and AgentCore sends them to Amazon CloudWatch. All the evaluators in this post read these spans.

The following code shows the entrypoint in src/agent/main.py:

@app.entrypoint async def invoke(payload, context): log.info("Invoking Agent.....") # Define the agent using create_react_agent (checkpointer is shared across invocations)  graph = create_react_agent( get_or_create_model(), tools=TOOLS, prompt=DEFAULT_SYSTEM_PROMPT, checkpointer=_checkpointer, ) # Process the user prompt  prompt = payload.get("prompt", "What can you help me with?") if not isinstance(prompt, str): raise ValueError("prompt must be a string") session_id = getattr(context, "session_id", "default-session") touch_thread(session_id) log.info(f"Agent input: {prompt}") # Run the agent (checkpointer auto-loads/saves history per session)  result = await graph.ainvoke( {"messages": [HumanMessage(content=prompt)]}, config={"configurable": {"thread_id": session_id}}, ) # Return result  # .text drops gpt-oss reasoning blocks and keeps only the answer text  output = result["messages"][-1].text log.info(f"Agent output: {output}") return {"result": output} 

Three versions of the agent

In this post, the agent goes through three versions that differ only in DEFAULT_SYSTEM_PROMPT.

Version 1 is the version that serves users when the post starts. Its prompt tells the agent to use its tools and to ask the customer for anything that is missing:

DEFAULT_SYSTEM_PROMPT = """ You are a customer-support assistant for an online electronics store. Always use the available tools to answer — never guess order, shipping or return information from your own knowledge. - To find a customer's orders from their email, use find_customer_orders. - To see items, SKUs, prices and dates of an order, use get_order_details. - For "where is my order" questions, use track_shipment. - Before promising a return, use check_return_eligibility; only call create_return when the customer asks for it and the item is eligible. If you are missing an email, order ID or item, ask the customer for it. """ 

Version 1 looks reasonable, and it answers order and shipping questions correctly. It has one weakness that shows up in return requests: check_return_eligibility and create_return need the item's SKU, and the last line tells the agent to ask for a missing item. In some return requests, the agent asks the customer for the SKU ("Could you tell me the SKU of the headphones in ORD-1001?") instead of reading it from the order. Customers don't know SKUs.

Version 2 is a latency optimization on top of version 1. Each tool call is one more model round trip, and create_return already refuses ineligible items, so the separate eligibility check looks redundant:

 You are a customer-support assistant for an online electronics store. -Always use the available tools to answer — never guess order, shipping or -return information from your own knowledge. +Customers hate waiting: keep replies short and use as few tool calls as +possible.  - To find a customer's orders from their email, use find_customer_orders. - To see items, SKUs, prices and dates of an order, use get_order_details. - For "where is my order" questions, use track_shipment. -- Before promising a return, use check_return_eligibility; only call - create_return when the customer asks for it and the item is eligible. +- When the customer asks for a return, call create_return directly: it + checks eligibility itself, so check_return_eligibility is an extra step.  If you are missing an email, order ID or item, ask the customer for it. 

The change saves a model round trip per return. It also opens returns without telling the customer the conditions first (the deadline, the refund amount, whether the item qualifies at all), which the store's workflow requires, and the shorter prompt makes the agent ask for SKUs even more often.

Version 3, the one in the repository, fixes the weakness of version 1. It tells the agent where SKUs come from and stops it from asking for them:

 - For "where is my order" questions, use track_shipment. +- Returns need the item's SKU: read it with get_order_details, never ask the + customer for it.  - Before promising a return, use check_return_eligibility; only call create_return when the customer asks for it and the item is eligible. -If you are missing an email, order ID or item, ask the customer for it. +If you are missing an email or order ID, ask the customer for it. 

Before releasing anything, we ran each version locally against the same model, with the 12 questions of the traffic script, four times each (48 single-turn sessions per version), and scored every answer with the release gate's rules (described in the next sections). The following table shows the results:

The differences in the table are large enough to measure in a 15-minute A/B test with enough traffic, which is what the release gate needs.

Solution components

The solution uses the following components:

• AgentCore Runtime runs the agent container. Each image update creates a new runtime version.
• Runtime endpoints are names that point to one version. The control endpoint serves users. The action creates a treatment endpoint for the candidate version.
• AgentCore Gateway is the single entry point. It has one HTTP target per endpoint, and the A/B test splits traffic between the targets, per session.
• AgentCore Evaluations reads the agent's OpenTelemetry spans from Amazon CloudWatch. Online evaluation scores every session with a code-based evaluator (an AWS Lambda function) for the release gate.
• GitHub Actions builds the image, runs the release gate and generates traffic. It deploys through an IAM role assumed with OpenID Connect (OIDC), so there are no AWS keys in the repository.

How the release gate works

The action expects an AgentCore Runtime with a control endpoint and a dedicated HTTP gateway in front of it. For every release, the action performs the following steps:

1. Prepare. It deploys the new image as a new runtime version, points a treatment endpoint to it, adds a treatment target to the gateway, and copies the online evaluation configuration once per variant.
2. Observe. It starts an A/B test on the gateway, so every new session goes either to control or to treatment, and waits while online evaluation scores the sessions of both variants.
3. Decide. For every evaluator in the quality gates, it checks that the treatment mean reaches the threshold, that it is not lower than the control mean, and that the difference is statistically significant (p < 0.05 by default).
4. Promote or roll back. If all gates pass, both endpoints move to the new version. If any gate fails, the job times out, the job is cancelled or AWS returns an error, both endpoints stay on (or return to) the previous version.

With require-significance: false, the action promotes a candidate that scores equal to or better than the current version and reaches the threshold. With require-significance: true, the default and the setting in this post, the improvement must also be statistically significant, so a candidate that only matches the current version is rolled back. That is why version 3, the one we want to promote, has to improve on version 1.

The release gate evaluator

AgentCore Evaluations offers three kinds of evaluators:

Evaluators work at three levels: a whole session, a single turn (trace) or an individual tool call.

The release gate needs a score it can compare across variants. We use a code-based evaluator for it. It always gives the same score for the same trace, it costs a few milliseconds of Lambda time per call, and it returns a number, which the A/B statistics need. Its rules come from the store's support workflow, not from the change being released, so the same evaluator gates every future release.

The evaluator works at the TRACE level: it scores each turn of a session with the session's earlier turns as context. A turn scores 1.0 (PASS) when it breaks none of the following rules, and 0.0 (FAIL) when it breaks at least one:

All four rules read only the spans that the agent already emits. The opentelemetry-instrumentation-langchain library records each tool call as an execute_tool span with the tool name, its arguments (gen_ai.tool.call.arguments) and its result (gen_ai.tool.call.result). The LangGraph.workflow span of each turn holds the customer's message and the agent's messages, including the final answer.

The following function in src/evaluators/support_workflow.py applies the rules to one turn. It uses only the Python standard library:

def check_turn(customer_messages: list[str], tool_calls: list[dict], answer: str, turn_tool_calls: int) -> list[str]: """Return the workflow rules this turn breaks (empty when it passes). customer_messages: the customer's messages in the session up to this turn. tool_calls: the session's tool calls up to and including this turn, in order, as {"name", "inputs", "result"}. answer: the agent's final answer for this turn. turn_tool_calls: how many of tool_calls belong to this turn. """ violations = [] answer = _normalize(answer) mentioned = _identifiers(answer) from_customer = set() for message in customer_messages: from_customer |= _identifiers(message) # Repeating the customer's own order ID (for example, when asking for an  # email) is not a fact the agent looked up.  if (mentioned - from_customer or _DATE.search(answer)) and not tool_calls: violations.append("The answer states order facts but no tool was called to look them up.") known = set(from_customer) for call in tool_calls: known |= _identifiers(call.get("result", "")) invented = sorted(mentioned - known) if invented: violations.append(f"The answer contains identifiers no customer message or tool result supports: {', '.join(invented)}.") if _ASKS_FOR_SKU.search(answer) and not any(call.get("name") == "get_order_details" for call in tool_calls): violations.append("The agent asked the customer for a SKU instead of reading the order with get_order_details.") checked = set() for index, call in enumerate(tool_calls): inputs = call.get("inputs") or {} key = (_normalize(str(inputs.get("order_id", ""))).strip().upper(), str(inputs.get("sku", "")).strip().upper()) if call.get("name") == "check_return_eligibility": checked.add(key) elif call.get("name") == "create_return" and key not in checked: if index >= len(tool_calls) - turn_tool_calls: violations.append(f"create_return was called for {key[0]} {key[1]} without check_return_eligibility first.") return violations 

The rest of the file turns the spans into turns and calls check_turn for the turn under evaluation:

def handler(event, context=None): spans = (event.get("evaluationInput") or {}).get("sessionSpans") or [] trace_ids = (event.get("evaluationTarget") or {}).get("traceIds") or [] turns = _turns(spans) # TRACE level scores one turn; without a target (SESSION level) score the last one.  candidates = [i for i, t in enumerate(turns) if t["answer"] is not None and (not trace_ids or t["trace_id"] in trace_ids)] target = candidates[-1] if candidates else None if target is None: return { "errorCode": "NO_SPANS", "errorMessage": f"No agent turn found for trace(s) {trace_ids or 'in session'}.", } history = turns[: target + 1] turn = turns[target] violations = check_turn( customer_messages=[t["customer"] for t in history], tool_calls=[call for t in history for call in t["tool_calls"]], answer=turn["answer"], turn_tool_calls=len(turn["tool_calls"]), ) if violations: return {"label": "FAIL", "value": 0.0, "explanation": " ".join(violations)} names = ", ".join(call["name"] for call in turn["tool_calls"]) or "none" return {"label": "PASS", "value": 1.0, "explanation": f"The turn follows the support workflow (tool calls: {names})."} 

AgentCore calls the function with the spans of the session in evaluationInput.sessionSpans and the trace to score in evaluationTarget.traceIds. The function returns label and value, plus an optional explanation, or errorCode and errorMessage when there is nothing to score. A failed turn's explanation names the rule it broke, for example create_return was called for ORD-1001 HD-200 without check_return_eligibility first.

The following Terraform registers the evaluator and the online evaluation configuration that the action uses as a template (from infra/evaluations.tf):

resource "aws_bedrockagentcore_evaluator" "support_workflow" { evaluator_name = local.support_workflow_evaluator_name description = "Deterministic: 1.0 if the turn follows the support workflow (lookups, grounded IDs, eligibility before returns), else 0.0." level = "TRACE" evaluator_config { code_based { lambda_config { lambda_arn = aws_lambda_function.support_workflow_evaluator.arn lambda_timeout_in_seconds = 30 } } } depends_on = [aws_lambda_permission.agentcore_evaluations] } # --- Online evaluation config --- # Runtime log groups are created by AgentCore on the first invocation. # Pre-create the control one so the evaluation config can point at it # from day one and so it gets a retention period. resource "aws_cloudwatch_log_group" "control_runtime" { name = "/aws/bedrock-agentcore/runtimes/${aws_bedrockagentcore_agent_runtime.agent.agent_runtime_id}-${var.control_endpoint_name}" retention_in_days = 30 } # Online evaluation of the control endpoint: scores every session of the agent # with the support-workflow evaluator. resource "aws_bedrockagentcore_online_evaluation_config" "control" { online_evaluation_config_name = "${var.agent_name}_${var.control_endpoint_name}_eval" description = "Evaluation of the control endpoint; template for A/B releases" enable_on_create = true evaluation_execution_role_arn = aws_iam_role.evaluation.arn data_source_config { cloudwatch_logs { log_group_names = [aws_cloudwatch_log_group.control_runtime.name] service_names = ["${var.agent_name}.${var.control_endpoint_name}"] } } evaluator { evaluator_id = aws_bedrockagentcore_evaluator.support_workflow.evaluator_id } rule { sampling_config { sampling_percentage = var.evaluation_sampling_percentage } session_config { session_timeout_minutes = var.evaluation_session_timeout_minutes } } lifecycle { precondition { condition = !strcontains(var.agent_name, var.control_endpoint_name) error_message = "agent_name must not contain control_endpoint_name." } } depends_on = [aws_iam_role_policy.evaluation] } 

With our values, the log group is /aws/bedrock-agentcore/runtimes/<runtime-id>-control and the service name is showcase_agent.control. The configuration scores 100% of the sessions and treats a session as complete after 2 idle minutes.

For every release, the action makes one copy of this configuration per variant by replacing control with treatment in the log group and the service name.

AgentCore creates the runtime's log group on the first invocation, so Terraform creates it up front. That way, the configuration can point to it from the beginning.

The template contains only the gate evaluator. Every evaluator in the configuration scores every session of both variants during a release, so an LLM-as-a-judge evaluator there adds judge-model cost to each session. AgentCore also locks an evaluator that an enabled online configuration references, so changing the evaluator's definition requires disabling the configuration first. Changes to the Lambda code apply directly.

The environment

The showcase runs in eu-central-1, in an AWS account with AgentCore Runtime, Gateway, Evaluations and A/B testing. The agent uses openai.gpt-oss-20b-1:0 on Amazon Bedrock. CloudWatch Transaction Search is enabled, because AgentCore Evaluations reads the agent's spans from it. GitHub Actions deploys through the account's GitHub OIDC identity provider, and the image builds on an ubuntu-24.04-arm runner. The infrastructure is defined with Terraform v1.14 and the AWS provider v6.63.

The action works with an ARM64 image in Amazon Elastic Container Registry (Amazon ECR), a runtime with a control endpoint, a dedicated HTTP gateway, an enabled online evaluation configuration, an A/B test role and a deploy role. The Terraform in the repository creates all of them.

The infrastructure

The infra/ folder contains one file per concern:

A few details make Terraform and the action work together.

The action deploys releases by changing the runtime's image and the control endpoint's version, so Terraform creates both resources and ignores those two attributes:

resource "aws_bedrockag...