AI & ML
DexForesight: Grounding Dexterous Policies with Implicit Spatial Foresight
arXiv cs.RO0 views
arXiv:2611.11166v2 Announce Type: replace Abstract: Dexterous manipulation requires rich spatial foresight over both the surrounding scene and the robot hand. Existing world-action models often acquire such predictive knowledge through explicit future prediction. We introduce DexForesight, a framework that learns spatial foresight implicitly by injecting dynamics-aware world representations into the latent space of a dexterous policy. During training, DexForesight learns an action-conditioned spatial world model from offline scene depth, kinematic hand geometry, and demonstration trajectories. The world model predicts 3D hand-point motion and visual feature changes, capturing both articulated hand dynamics and scene changes under demonstrated actions, including the motion of fingers occluded from the cameras. We then freeze the world model and align policy queries with its predictive world queries, transferring the learned spatial dynamics into policy latents and using them to condition action generation. At deployment, only the policy is retained, operating on RGB observations, language, and robot state without depth input or explicit future prediction. On 11 DexJoCo simulation tasks, DexForesight improves the overall success rate of the base policy from 49.4% to 60.5%. On three real-robot dexterous manipulation tasks, DexForesight achieves 58.3% success, compared with 50.0% for the strongest baseline. Project page https://dexforesight.github.io/
Source:https://arxiv.org/abs/2611.11166PreviousControl-Aware Command Generation for Asynchronous Control and Action Inference with Vision-Based Policies towards Fast ExecutionNextFade to... Red? Scheduling Coloured Noise Exploration Supports Efficient Robot Learning with Reduced Movement Jerk
Related posts
How Many Demonstrations Does a VLA Need? On the Sample Efficiency of VLA Finetuning
0arXiv cs.RO
Altitude-Adaptive UAV-to-Map Image Matching for Aerial Geo-Localization
0arXiv cs.RO
OASIS: Optimization-Aware Subgoal Inference with Safety Constraints for Autonomous Mobile Robot Navigation
0arXiv cs.RO
Fade to... Red? Scheduling Coloured Noise Exploration Supports Efficient Robot Learning with Reduced Movement Jerk
0arXiv cs.RO
Control-Aware Command Generation for Asynchronous Control and Action Inference with Vision-Based Policies towards Fast Execution
0arXiv cs.RO
An Integrated Open-Source Stack for Vision-Based Autonomous Navigation of Wheeled Rovers
0arXiv cs.RO