AI & ML
From Egocentric Human Video to Robot Manipulation in the Era of Embodied Intelligence
arXiv cs.RO0 views
arXiv:2611.13610v1 Announce Type: new Abstract: Egocentric human video (EHV) records close-range hand-object interaction (HOI) and the temporal structure of everyday activities, offering a potentially scalable source of experience for robot manipulation. Using this experience for robot control requires recovering useful supervision without recorded robot actions and accounting for differences in morphology, viewpoint, and action space. We define Ego2Robot learning as the process of learning or adapting robot manipulation policies from EHV, organized into four learning paradigms: explicit embodiment transfer, explicit action and task supervision, egocentric video representation pretraining, and latent action and dynamics modeling. The four paradigms differ in what they derive from human video: robot-format demonstrations, interpretable supervision, visual representations, or predictive models of actions and dynamics. For each paradigm, we trace what EHV information is extracted, how robot policies use it, and when the additional data is required. Besides, we review the benchmark datasets that cover EHV, robot demonstrations, and synthetic and simulated resources. This paper links EHV data's contribution to the task information preserved and the robot-specific grounding used during training or deployment. Finally, we identify challenges in physical feasibility, agent coordination, in-context adaptation, controlled evaluation, and predictive control.
Source:https://arxiv.org/abs/2611.13610PreviousFew-Shot Residual Adaptation of Pretrained Latent World ModelsNextAn IoT Based Approach for Platform Independent Positioning Service
Related posts
How Many Demonstrations Does a VLA Need? On the Sample Efficiency of VLA Finetuning
0arXiv cs.RO
Altitude-Adaptive UAV-to-Map Image Matching for Aerial Geo-Localization
0arXiv cs.RO
OASIS: Optimization-Aware Subgoal Inference with Safety Constraints for Autonomous Mobile Robot Navigation
0arXiv cs.RO
Fade to... Red? Scheduling Coloured Noise Exploration Supports Efficient Robot Learning with Reduced Movement Jerk
0arXiv cs.RO
DexForesight: Grounding Dexterous Policies with Implicit Spatial Foresight
0arXiv cs.RO
Control-Aware Command Generation for Asynchronous Control and Action Inference with Vision-Based Policies towards Fast Execution
0arXiv cs.RO