arXiv:2610.02527v2 Announce Type: replace Abstract: Learned visual reward models score a robot's execution from camera images and an instruction, and are increasingly used to fine-tune robot policies. Two checks decide whether to trust one: the reward model separates successes from failures, and task success rises during training. We show that both checks can pass while the policy learns to act on the wrong object. When a diffusion policy is fine-tuned against the reward model Robometer to open a drawer, task success rises by 10.2 percentage points. Wrong-object failures, which open a different drawer, rise by 10.9 points, whereas the same training on the simulator's task-completion signal raises success alone. A second reward model, RoboReward-8B, behaves the same way, and the effect reappears on a physical robot. This happens because of how the reward scores the policy's own failures. Most failures make little progress and score low, while a wrong-object failure looks like progress and scores well above them. Optimization shifts a policy toward outcomes that score above the executions they compete with, the executions they can take the place of. It therefore turns low-scoring failures into both successes and wrong-object failures, even when the reward model scores successes higher. This rule needs only executions of the initial policy, so it can be tested before training. On a stacking task we predicted which of three object placements Robometer would exploit, and training raised wrong-object stacks from 20% to 69% in that placement and in neither of the others. The same rule gives a check to run before optimization and a repair that needs no labels: scoring each execution again on frames re-rendered with the objects the instruction does not name put back removes the amplification on both tasks and on the robot.