Designing a Simulation-First Safety Gate for Strands Robots

What happens when an AI agent moves beyond generating text and starts controlling a physical robot?

The definition of failure changes completely.

For a chatbot, a bad answer might waste a few seconds.

For a robot, a bad action can cause a collision, damage hardware, drop an object, or create an unsafe situation.

That led me to a different question:

What if robot behavior had to pass a safety gate before it was allowed to reach real hardware?

Instead of:

AI Agent → Robot 

`

I propose:

text
AI Agent
↓
Simulation
↓
Episode Recording
↓
Safety Evaluation
↓
PASS / FAIL / UNKNOWN
↓
Human Approval
↓
Real Hardware


This article explores that architecture using Strands Robots, simulation, episode recording, deterministic safety checks, and Amazon Bedrock AgentCore Evaluations.

The Problem: "It Worked" Is Not Enough

Imagine the task is:

text
Pick up the red cube and place it in the box.


A robot policy might successfully complete the task.

But during the episode, it could also:

• Move a joint too quickly
• Leave the permitted workspace
• Collide with an object
• Oscillate between two actions
• Take too long to complete the task
• Behave unexpectedly near an obstacle
• Complete the task while violating a safety constraint

So I would separate two questions:

Task Evaluation

Did the robot accomplish the requested task?

Safety Evaluation

Did the robot accomplish the task without violating safety constraints?

These should not be treated as the same metric.

A robot can have:

text
Task Success = TRUE
Safety = FALSE


And that episode should not automatically reach physical hardware.

Why Simulation Should Come First

A simulation environment gives us a place to test behavior before connecting the policy to a physical robot.

The proposed workflow is:

`text
DEVELOPMENT

 ┌───────────────┐ │ Strands Agent │ └───────┬───────┘ │ ▼ ┌───────────────┐ │ Robot Tool │ └───────┬───────┘ │ ▼ ┌───────────────┐ │ Simulation │ │ MuJoCo │ └───────┬───────┘ │ ▼ Record Episode │ ▼ Safety Evaluation │ ┌─────────┴─────────┐ │ │ FAIL PASS │ │ ▼ ▼ Reject Hardware Test 

`

The important design decision is:

Simulation is not only a development environment. It becomes a quality gate.

Strands Robots provides simulation and recording capabilities, including recording policy runs into LeRobot datasets. This gives us a practical way to capture episodes that can later be inspected and evaluated.

Architecture

The complete proposed architecture looks like this:

text
USER
│
▼
┌─────────────┐
│ Strands │
│ Agent │
└──────┬──────┘
│
▼
Amazon Bedrock
│
▼
Robot Tool
│
▼
┌──────────────────┐
│ Simulation │
│ Environment │
│ │
│ • Robot State │
│ • Actions │
│ • Cameras │
│ • Telemetry │
└────────┬─────────┘
│
▼
Episode Recorder
│
▼
Safety Evaluation
│
┌────────────┼────────────┐
│ │ │
▼ ▼ ▼
PASS UNKNOWN FAIL
│ │ │
▼ ▼ ▼
Hardware Human Reject
Test Review


The system intentionally separates decision-making, simulation, evaluation, and hardware execution.

Recording Robot Episodes

An important part of this architecture is recording the behavior of the robot.

A simplified example looks like this:

`python
from strands_robots import Robot

robot = Robot("so100")

robot.start_recording(
repo_id="user/safety-test",
task="pick up the cube",
fps=50
)

robot.run_policy(
robot_name="so100",
instruction="pick up the cube",
policy_provider="mock",
duration=10.0
)

robot.stop_recording()
`

The exact policy and robot configuration will depend on your environment.

The important concept is that the execution becomes a reusable episode.

Instead of only seeing:

text
SUCCESS


we can inspect:

text
Observation
Action
Robot State
Camera Data
Timing
Task
Episode Outcome


That gives us something measurable.

What Should the Safety Gate Check?

I would start with five categories.

1. Workspace Safety

Did the robot remain inside the permitted workspace?

For example:

text
x_min ≤ x ≤ x_max
y_min ≤ y ≤ y_max
z_min ≤ z ≤ z_max


If the end effector leaves the permitted region:

text
SAFETY = FAIL


This is a deterministic check.

There is no reason to ask an LLM to decide whether:

text
0.75 <= 0.50


A normal program can do that more reliably.

2. Joint Velocity

The policy might complete the task successfully while producing unsafe movement.

For each joint:

text
velocity[joint] <= maximum_allowed_velocity


Example:

`python
MAX_VELOCITY = 2.0

if observed_velocity > MAX_VELOCITY:
safety_status = "FAIL"
`

This is another example where deterministic code is preferable to an LLM.

3. Collision and Contact

Collision detection is one of the most important signals.

The evaluator should look for events such as:

text
unexpected_contact
collision
self_collision
unsafe_contact


For example:

json
{
"task_success": true,
"collision_free": false
}


That should produce:

text
REJECT


not:

text
SUCCESS


4. Execution Time

A policy that normally completes a task in five seconds but suddenly requires sixty seconds should be investigated.

A simple rule could be:

`python
MAX_EPISODE_SECONDS = 20

if episode_duration > MAX_EPISODE_SECONDS:
return "REJECT"
`

This can also help identify policies that become trapped in repetitive behavior.

5. Action Stability

Consider an agent producing:

text
LEFT
RIGHT
LEFT
RIGHT
LEFT
RIGHT
...


for a long period.

The robot is technically moving, but the policy is not converging.

We can calculate a simple oscillation metric:

text
oscillation_score =
direction_changes / total_actions


If the score becomes unusually high, the episode can be flagged for review.

A Simple Safety Evaluator

A simplified implementation could look like this:

`python
def evaluate_episode(episode):

violations = [] if not workspace_is_safe(episode): violations.append("WORKSPACE_VIOLATION") if not joint_velocity_is_safe(episode): violations.append("JOINT_VELOCITY_LIMIT") if has_collision(episode): violations.append("COLLISION") if episode.duration > 20: violations.append("TIMEOUT") if action_oscillation_is_high(episode): violations.append("ACTION_OSCILLATION") return { "approved": len(violations) == 0, "violations": violations } 

`

The most important principle here is:

Fail closed.

If the system cannot establish that an episode is safe, it should not automatically approve it for hardware execution.

Adding an UNKNOWN State

Most evaluation systems naturally produce:

text
PASS
FAIL


For physical AI, I think we need:

text
PASS
FAIL
UNKNOWN


For example:

json
{
"decision": "UNKNOWN",
"reason": "Insufficient evidence to verify collision-free execution"
}


This creates a safer workflow:

text
Episode
│
▼
Safety Evaluator
│
┌───────────┼───────────┐
│ │ │
▼ ▼ ▼
PASS FAIL UNKNOWN
│ │ │
▼ ▼ ▼
Hardware Reject Human
Test Review


The important idea is:

UNKNOWN should not mean PASS.

If the system does not have enough evidence, the safe action is to stop and request review.

Where Amazon Bedrock Fits

I would not use an LLM as the only robot safety mechanism.

Instead, I would use two evaluation layers.

Layer 1 — Deterministic Evaluation

Use normal code for measurable constraints:

text
Workspace boundaries
Joint velocity
Collision flags
Episode duration
Distance thresholds
Action limits


Layer 2 — AI-Based Evaluation

Use an evaluator model for higher-level questions:

`text
Did the robot accomplish the requested task?

Was the behavior consistent with the task?

Was there unexpected behavior?

Does this episode require human review?
`

This division is important.

The LLM should not replace deterministic safety controls.

It should complement them.

Agent Evaluation Is Different From Robot Evaluation

There are actually two systems to evaluate.

The Agent

`text
Did the agent:

• Choose the correct tool?
• Generate the correct instruction?
• Follow the task?
• Recover from an error?
`

The Robot

`text
Did the robot:

• Stay inside the workspace?
• Avoid collisions?
• Respect velocity limits?
• Complete the movement safely?
`

That creates a two-layer evaluation architecture:

text
┌────────────────────────────────┐
│ STRANDS AGENT │
│ │
│ Tool Selection │
│ Instruction Quality │
│ Goal Completion │
└───────────────┬────────────────┘
│
▼
┌────────────────────────────────┐
│ ROBOT POLICY │
│ │
│ Workspace Safety │
│ Velocity Safety │
│ Collision Detection │
│ Action Stability │
│ Execution Time │
└───────────────┬────────────────┘
│
▼
HARDWARE APPROVAL


Amazon Bedrock AgentCore Evaluations supports built-in, third-party, and custom evaluators, including code-based evaluators for deterministic evaluation logic.

That makes it possible to combine AI-based evaluation with programmatic checks instead of depending entirely on an LLM judge.

Why Batch Evaluation Matters

One successful episode does not prove that a robot policy is reliable.

Imagine:

text
Episode 1
Cube at expected position
→ PASS


Then:

text
Episode 2
Cube moved 10 cm
→ FAIL


Or:

text
Episode 3
Cube rotated
→ FAIL


The real question is:

How consistently does the policy behave across different situations?

That is why I would build a reusable evaluation dataset.

Building a Safety Dataset

Example scenarios:

Then run every policy against the same scenarios.

This makes policy comparisons reproducible.

The Release Gate

The final release workflow becomes:

text
New Policy
│
▼
Simulation Tests
│
▼
Record Episodes
│
▼
Safety Evaluation
│
┌──────────┴──────────┐
│ │
FAIL PASS
│ │
▼ ▼
Reject Agent Evaluation
│
▼
Quality Threshold
│
┌─────────┴─────────┐
│ │
FAIL PASS
│ │
▼ ▼
Reject Hardware Test


This creates two quality gates.

Gate 1

Is the robot behavior physically safe?

Gate 2

Is the agent behavior correct?

Only after both pass should the system progress toward physical hardware.

Turning Failures Into Training Data

A failed episode should not simply disappear.

Instead, capture the failure:

json
{
"episode": "episode_042",
"status": "REJECTED",
"violations": [
{
"type": "JOINT_VELOCITY",
"joint": "joint_3",
"observed": 2.84,
"limit": 2.0
}
],
"recommended_action": "Review policy checkpoint"
}


Now the failure becomes useful data.

The development loop becomes:

text
Run
↓
Evaluate
↓
Find Failure
↓
Record Failure
↓
Improve Policy
↓
Run Again


This is much more useful than treating evaluation as the final step.

From Evaluation to Continuous Improvement

After collecting enough episodes, the dataset becomes a reusable engineering asset.

text
Successful Episodes
+
Failure Episodes
+
Human Reviews
↓
Evaluation Dataset
↓
Policy Improvement
↓
New Policy
↓
Simulation


This creates a continuous improvement loop.

Why Simulation Is Powerful

Simulation lets us test many variations without physically repeating every experiment.

For example:

text
100 episodes
1,000 episodes
10,000 episodes


can be evaluated before moving to a limited hardware test.

Strands Robots supports simulation and recording workflows, making it possible to capture robot episodes as datasets for later evaluation.

But there is an important limitation.

Simulation Is Not Reality

Passing thousands of simulated episodes does not automatically make a policy safe on a physical robot.

The real world introduces:

• Sensor noise
• Calibration errors
• Communication latency
• Mechanical variation
• Unexpected obstacles
• Lighting changes
• Hardware wear
• Model mismatch

Therefore:

Simulation should reduce risk, not claim to eliminate it.

The first physical execution should still be controlled and supervised.

Proposed Production Architecture

For a production-oriented system, I would use:

text
USER
│
▼
STRANDS AGENT
│
Amazon Bedrock
│
▼
ROBOT TOOL
│
┌─────────┴─────────┐
│ │
SIMULATION HARDWARE
│
▼
EPISODE RECORDER
│
▼
SAFETY EVALUATOR
│
┌────────┼────────┐
│ │ │
PASS FAIL UNKNOWN
│ │ │
│ │ └── Human Review
│ │
│ └─────────── Policy Improvement
│
▼
Hardware Approval
│
▼
Controlled Execution
│
▼
AgentCore / CloudWatch
Evaluation & Monitoring


AgentCore Evaluations supports online, on-demand, and batch evaluation workflows. Batch evaluation is particularly useful for baseline measurement, regression testing, and comparing agent behavior before and after changes.

What I Would Build Next

If I were taking this from architecture to implementation, I would build the following pipeline:

Phase 1 — Simulation

Run the same task across multiple scenarios.

Phase 2 — Recording

Save observations, actions, camera data, and robot state.

Phase 3 — Deterministic Safety

Calculate:

text
Collision Rate
Workspace Violations
Maximum Joint Velocity
Episode Duration
Oscillation Score


Phase 4 — Agent Evaluation

Evaluate:

text
Tool Selection
Task Completion
Instruction Quality
Recovery Behavior


Phase 5 — Decision

Return:

text
PASS
FAIL
UNKNOWN


Phase 6 — Hardware

Only approved policies reach controlled physical execution.

The Bigger Idea

The interesting question is no longer simply:

Can an AI agent control a robot?

The more important engineering question is:

Can we build a measurable system that knows when robot behavior is safe enough to progress toward the real world?

That changes the architecture.

Instead of:

text
Agent → Robot


we get:

text
Agent
↓
Simulation
↓
Recording
↓
Evaluation
↓
Safety Gate
↓
Human Review
↓
Hardware
↓
Continuous Monitoring


The robot gets autonomy gradually.

Final Takeaway

Physical AI needs more than capable models.

It needs controlled progression from simulation to reality.

My proposed approach is:

1. Start in simulation.
2. Record complete episodes.
3. Apply deterministic safety checks.
4. Apply AI-based evaluation where it makes sense.
5. Introduce an explicit UNKNOWN state.
6. Reject unsafe episodes.
7. Send uncertain episodes for human review.
8. Promote only approved behavior toward hardware.
9. Keep collecting failures and use them to improve the system.

The goal isn't to build an agent that simply acts.

The goal is to build an agent whose actions have to earn the right to reach the real world.

References

• Strands Robots
• Amazon Bedrock AgentCore Evaluations
• AgentCore Custom Evaluators
• AgentCore Batch Evaluations
• Strands Robots Recording and Datasets