AI & ML
Beyond Successful Robot Demonstrations: Learning from Human Interactions and Modeling Failure Dynamics
arXiv cs.RO0 views
arXiv:2610.13522v1 Announce Type: new Abstract: Action-conditioned world models offer a promising foundation for generalist robotic intelligence by predicting the consequences of actions. However, existing robotic datasets provide limited coverage of the diverse interaction dynamics required for generalizable world modeling. Large-scale egocentric human videos offer complementary interaction experience, yet the embodiment gap between human hands and robotic manipulators hinders effective cross-embodiment learning. Meanwhile, the predominance of successful demonstrations leaves failure dynamics substantially underrepresented, limiting the ability of world models to anticipate unsuccessful executions. To address these limitations, we introduce HF-WM, an action-conditioned world model that combines Human interaction learning with Failure-aware post-training. To bridge the embodiment gap, we develop a dual-branch framework that couples video generation with auxiliary pointflow prediction, using stabilized pointflow as a unified motion representation across human and robot interactions while suppressing egocentric camera motion. To improve failure modeling, we further perform Group Relative Policy Optimization (GRPO) post-training on perturbed actions derived from successful demonstrations, guided by action alignment and physical plausibility rewards, enabling the model to learn plausible failure dynamics without paired ground-truth videos. Extensive experiments demonstrate that HF-WM achieves state-of-the-art performance in action-conditioned video generation and exhibits strong utility for policy evaluation in downstream robotic tasks.
Source:https://arxiv.org/abs/2610.13522PreviousRESCUE: Is This Alarm Worth Acting On? Counterfactual Evaluation of VLA Runtime MonitorsNextATHLETE: Learning Reactive Humanoid Tennis via Trajectory-Guided Motion Matching
Related posts

Canva Brings Free All-in-One Affinity App to iPad
0MacRumors
GPT-Rosalind-6 System Card Addendum
0Hacker News New
Mediroute: An open-source AI platform exposing global drug price disparities
0Hacker News New
An intuition for how neural nets can understand text
0Hacker News New
An open LLM read 291 retracted papers: 7.7% of passages mention the retraction
0Hacker News New
Show HN: Maat – AI agents that can diagnose Ubuntu but can't change it
0Hacker News Show