AI & ML
World-Ego Modeling for Embodied Video Generation in Long-Horizon Navigation-Manipulation Tasks
arXiv cs.RO0 views
arXiv:2605.19957v2 Announce Type: replace-cross Abstract: Embodied video world models typically capture both scene evolution and the robot's behavior, which we refer to as the \emph{world} and the \emph{ego}, respectively. The world and the ego exhibit different underlying dynamics: world prediction relies primarily on visual history and emphasizes scene stability, whereas ego prediction relies more strongly on the current instruction and emphasizes accurate instruction following. Modeling both components within a single generation stream can entangle these different dependencies, making it difficult to specialize the prediction of either component. Consequently, it becomes difficult to simultaneously maintain scene consistency and accurate instruction following, particularly in long-horizon navigation-manipulation tasks. In this paper, we propose to decompose an embodied video into the world and the ego and disentangle their generation processes. Specifically, we define the world as the background and currently unmanipulated objects, and the ego as the robot and currently manipulated objects. Based on this definition, we develop the \emph{World-Ego Model} (WEM), which combines a vision-language state predictor using role-conditioned attention (RCA) and asymmetric query budgets with a semantic-routed mixture-of-experts (SR-MoE) diffusion generator. To enable rigorous evaluation, we further construct HTEWorld, a dataset and benchmark for long-horizon embodied video generation with hybrid navigation-manipulation tasks, providing 125K training video clips comprising over 4.5M frames with fine-grained instructions, together with 300 multi-turn evaluation trajectories covering over 2K instructions. Extensive experiments show that WEM achieves state-of-the-art performance on HTEWorld while remaining competitive on existing manipulation-oriented evaluations.
Source:https://arxiv.org/abs/2605.19957PreviousPhysMoDPO: Physically-Plausible Humanoid Motion with Preference OptimizationNextTrust Region Q Adjoint Matching
Related posts

What Arconatese 3-4 Pontedera tells us about Serie D - Girone D
0dev.to Watercooler
Your AI agent got an HTTP error. Should it retry?
0dev.to Discuss
Is AI dangerous, or just misunderstood?
0dev.to Discuss
Why we price a developer tool per fix instead of per seat
0dev.to Productivity
My LLM-Powered Sidekick for Painless PR Descriptions
0dev.to Productivity
Reward-Adaptive Backtracking GFlowNet: A Technique Worth Understanding
0dev.to Productivity