Open Source
RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer
arXiv cs.RO0 views
arXiv:2610.09254v1 Announce Type: new Abstract: Simulation enables large-scale, low-cost robot data generation, but policies trained in simulation often fail to transfer to the real world due to the sim-to-real visual discrepancies. Existing approaches often rely on intermediate representations, which can discard rich semantic information or require additional perception modules at deployment. We address this visual sim-to-real gap with RoboRender, a framework that converts simulated trajectories into photorealistic RGB videos for policy learning. RoboRender trains a robot-oriented video generation model conditioned on simulated depth videos, language instructions, and robot RGB mask videos, preserving simulator geometry, robot motion, and action labels while synthesizing realistic textures, backgrounds, and distractors. The generated RGB videos are paired with simulator-provided states and actions to train policies for zero-shot real-world deployment. On robot video test sets, our video model outperforms depth-conditioned video generation baselines in generation quality. In real-world experiments across pick-and-place, articulated-object manipulation, and mobile manipulation tasks, policies trained on RoboRender-generated data achieve a 71% average success rate, outperforming raw simulation renderings and conventional visual domain randomization by approximately 7.1x and 3.6x, respectively. We further show that policy performance improves with more generated videos per simulation trajectory, increasing opening-task success by 65 percentage points. These results demonstrate that generative video rendering mitigates the visual sim-to-real gap for zero-shot policy transfer. Project website: https://robo-render.github.io/.
Source:https://arxiv.org/abs/2610.09254PreviousCo-Evolving Robot Orchestrators and Policies through DeploymentNextEvaluating the Transfer of Co-Evolved Communication from 2D to 3D Simulation
Related posts
Distillation didn't help our small model until we took the labels away
0dev.to Machine Learning

462. Minimum Moves to Equal Array Elements II
0dev.to PHP
MUSBRIC: LEGO-Compatible Pneumatic Muscle Bricks for Reconfigurable Musculoskeletal Robots
0arXiv cs.RO
OpenFlyV2: An onboard vision-language-action model for aerial perception and navigation
0arXiv cs.RO
Show HN: Open Bot Avatars – a spec and renderer for interactive bot avatars
0Hacker News New
Show HN: Elections Archive: every election in the US, citable and open source
0Hacker News New