Ai2's ToyBlocks shows why robot tests need a small world you control. Here is a toy version in pandas: mix the data, hold out a task, sort the failures.
A note on how this was written: an AI writing assistant drafted it from the source listed at the end, and every claim was checked against that source before it went out.
At a glance
- ToyBlocks, from Ai2, is a small simulated tabletop with a robot arm and blocks. You can change the training data, run the same test again, and trace why something failed.
- I build a toy version of that idea in plain Python: pick a mix of demonstrations, keep one task out, make block sizes vary, and sort every failure by its cause.
- The point is the habit: one pooled success rate hides what the policy never saw.
Robots fold laundry, but nobody can say why they fail
When I read Ai2's ToyBlocks post, one line stayed with me: robot demos are often narrow and don't explain what the robot learned or why it fails. I run into the same thing with AI agents. A demo shows that something worked once. It doesn't tell you what breaks it.
Their answer is a small, controlled world. Scientists study complex systems in a petri dish or a terrarium, where they can change one thing and watch what happens. ToyBlocks does the same for robots: a robot arm, some blocks on a table, and tools that make tasks and training demonstrations automatically. Ai2 calls this kind of setup a FoundationWorld, and ToyBlocks is the first one.
What ToyBlocks actually controls
The world is kept simple on purpose, but it isn't trivial. The robot pushes, flips and stacks blocks, and the blocks have pictures and letters on their faces, so there are also tasks like searching, sorting and arranging. Sometimes the robot has to turn a block over to see a hidden face before it knows where the block goes.
The design has four requirements: realistic physics and appearance, tasks and demonstrations made automatically at scale, challenges that need both reasoning and dexterity, and broad coverage of the tasks the domain allows. The post says an open-ended domain can't meet all four. A narrow one can.
Two parts matter most for testing. First, domain randomization: block size, table height and other physical settings change from scene to scene, which shows how well a model holds up when the geometry shifts. Second, the training mix is in your hands. Their own example: make more pushing or flipping demonstrations, then test on tasks that were kept out of training.
Under the hood, short skills are trained with reinforcement learning and motion planning in Isaac Lab, Nvidia's open-source framework for robot learning in simulation. While making skills, the simulator can hand over privileged information, like a block's exact position and orientation, which a real robot would have to estimate from camera images. A task-and-motion planner then strings the skills together into full demonstrations, with no person needed to steer the robot remotely.
Build the toy world
I'm not going to rebuild a physics simulator. What I can copy is the testing pattern: a world where I set every knob, so any failure can be traced back to a knob. First, the scenes. Each one gets a task, a block size and a table height.
import numpy as np import pandas as pd rng = np.random.default_rng(7) TASKS = ["push", "flip", "stack", "sort"] def make_scenes(n, size_range, rng): return pd.DataFrame({ "task": rng.choice(TASKS, size=n), "block_cm": rng.uniform(*size_range, size=n).round(1), "table_cm": rng.uniform(70, 80, size=n).round(1), }) scenes = make_scenes(8, (3.0, 5.0), rng) print(scenes) Output:
task block_cm table_cm 0 sort 3.6 72.5 1 stack 4.7 74.5 2 stack 3.0 75.0 3 sort 4.6 75.5 4 stack 4.6 80.0 5 sort 3.9 77.9 6 sort 3.6 76.2 7 push 3.6 79.9 This prints a handful of random scenes. Check that every row has a task and a size inside the range you asked for. Table height is only there to show the idea; the toy policy below ignores it.
Choose the training mix, and keep one task out
This is the knob ToyBlocks makes cheap to turn. I make demonstrations with a mix I pick, and I leave "sort" out entirely so I can test on it later.
def make_demos(mix, n, size_range, rng): tasks = rng.choice(list(mix), size=n, p=list(mix.values())) sizes = rng.uniform(*size_range, size=n) return pd.DataFrame({"task": tasks, "block_cm": sizes}) mix = {"push": 0.5, "flip": 0.2, "stack": 0.3} # "sort" kept out on purpose demos = make_demos(mix, 600, (3.0, 5.0), rng) print(demos["task"].value_counts()) print("sizes seen:", demos["block_cm"].min().round(2), "to", demos["block_cm"].max().round(2)) Output:
task push 291 stack 183 flip 126 Name: count, dtype: int64 sizes seen: 3.0 to 4.99 Look at the counts: they should roughly follow the mix. The size line shows the only block sizes the policy will ever see while training.
A fake policy whose weak spots I know
Here's the part a real lab can't do: I write down exactly how the policy fails. Skill on a task grows with the number of demos for that task, and success drops off the further a block's size is from anything seen in training. Since I know the truth, I can check whether my evaluation finds it.
def train(demos): counts = demos["task"].value_counts().to_dict() lo, hi = demos["block_cm"].min(), demos["block_cm"].max() return {"counts": counts, "lo": lo, "hi": hi} def success_prob(policy, task, size): n = policy["counts"].get(task, 0) skill = 1 - np.exp(-n / 100) # more demos, better skill gap = max(policy["lo"] - size, size - policy["hi"], 0) return skill * np.exp(-gap / 0.5) # worse the further from sizes seen policy = train(demos) for t in TASKS: print(t, round(success_prob(policy, t, 4.0), 3), round(success_prob(policy, t, 6.0), 3)) Output:
push 0.946 0.127 flip 0.716 0.096 stack 0.84 0.112 sort 0.0 0.0 Each line shows one task's chance of success at a block size inside the training range and at one outside it. Compare push with flip, and look at what sort gets.
Randomize the test, then sort failures by cause
Now the test scenes use a wider size range, my small version of domain randomization. Each result gets a cause label. One rule I set before running: a scene that is both an unseen task and an unseen size goes in its own "both" bucket. If I counted it under whichever label I checked first, the split would depend on the order of my if statements, not on the policy.
def evaluate(policy, scenes, rng): rows = [] for s in scenes.itertuples(): ok = rng.random() < success_prob(policy, s.task, s.block_cm) unseen = s.task not in policy["counts"] off = not (policy["lo"] <= s.block_cm <= policy["hi"]) cause = "both" if unseen and off else "unseen task" if unseen else "size off" if off else "in range" rows.append({"task": s.task, "ok": ok, "cause": cause}) return pd.DataFrame(rows) test = make_scenes(2000, (2.0, 7.0), rng) # wider sizes than training res = evaluate(policy, test, rng) print(res.groupby("cause")["ok"].agg(["mean", "size"]).round(3)) Output:
mean size cause both 0.000 295 in range 0.808 600 size off 0.277 917 unseen task 0.000 188 Read both columns: success rate and how many scenes are in each bucket. A bucket with a terrible rate but only a few scenes is a weaker signal than a big one.
What the pooled number hides
A single success rate pools every task and every size into one number. That's the number that ends up in a demo video caption. Print it next to the in-range scenes split by task.
print("pooled success:", round(res["ok"].mean(), 3)) inside = res[res["cause"] == "in range"] print(inside.groupby("task")["ok"].agg(["mean", "size"]).round(3)) Output:
pooled success: 0.37 mean size task flip 0.660 215 push 0.950 200 stack 0.827 185 The pooled line mixes the held-out task, the odd sizes and the familiar scenes together. The table under it shows what's left once the unfamiliar scenes are taken out, and whether the task with fewer demos lags even on familiar sizes.
Change the mix and run it more than once
One run is one draw. Here I compare two training mixes over several seeds each, the same move as Ai2's "make more flipping demonstrations" example, and print the average and spread for every task.
def run(mix, seed): r = np.random.default_rng(seed) p = train(make_demos(mix, 600, (3.0, 5.0), r)) out = evaluate(p, make_scenes(2000, (2.0, 7.0), r), r) return out[out["cause"] == "in range"].groupby("task")["ok"].mean() mixes = {"push heavy": mix, "flip heavy": {"push": 0.2, "flip": 0.5, "stack": 0.3}} for name, m in mixes.items(): table = pd.DataFrame([run(m, s) for s in range(5)]) print(name); print(table.agg(["mean", "std"]).round(3)) Output:
push heavy task flip push stack mean 0.714 0.952 0.845 std 0.043 0.018 0.024 flip heavy task flip push stack mean 0.949 0.701 0.845 std 0.004 0.070 0.024 See whether flip moves when it gets more demos, and whether that gap is bigger than the std row. If the spread is about as large as the difference, one run would have misled you.
Check it yourself
In the cell that sets mix, add "sort" as a fourth task and lower the other weights so all of them still add up to one. Then run every cell again.
In the cause table, the "unseen task" and "both" buckets should empty out, and sort should show up in the in-range table with a success rate close to the other tasks that got a similar number of demos.
Then try the other knob: in the make_demos call, give training the same size range the test scenes use. The "size off" bucket should mostly disappear. If a bucket doesn't shrink after you fixed its cause, your labels are measuring something else.
Where my toy stops and the real thing starts
My policy is a formula I wrote, so of course the test finds its weak spots. A real one learns from simulated physics, and ToyBlocks also has to keep the simulated arm and blocks behaving like their physical counterparts, which is a much harder problem. What carries over is the habit: name the knobs, hold some tasks out, randomize the scenes, and never trust one pooled number.
Have you tested a model, robot or otherwise, on a task you deliberately kept out of training? What did it get wrong that the pooled score didn't show?
