AI & ML
Seeing and Imagining as One: Unified Video Generation and 4D Perception for Embodied World Modeling
arXiv cs.RO0 views
arXiv:2610.39232v2 Announce Type: replace-cross Abstract: Embodied world models should not only imagine plausible futures, but also perceive the geometric, semantic, and motion structure of the evolving world. We present GenSight, a unified embodied world model that brings future video generation and dense perception into a single pretrained video diffusion transformer. GenSight uses a unified spatiotemporal representation for video, depth, embodiment segmentation, and first-frame-relative 2D flow. Built on a shared DiT backbone, GenSight uses lightweight task modulation to predict all modalities within the same architecture. All tasks are jointly optimized within a unified velocity prediction space. At inference, GenSight supports two complementary modes: multi-step future video generation with aligned dense predictions and single-step dense perception from an observed video. By unifying modality representation, model architecture, and prediction space, GenSight avoids separate modality-specific pipelines and couples appearance, geometry, semantics, and motion in a shared representation. It promotes cross-modal consistency and allows dense perception to benefit from the spatiotemporal priors learned through video generation. Trained on embodied manipulation data spanning multiple robot embodiments and human-hand interactions, GenSight demonstrates strong performance in both video generation and dense 4D perception, highlighting the effectiveness of unifying seeing and imagining for embodied world modeling.
Source:https://arxiv.org/abs/2610.39232PreviousSV-WAM: An Efficient Surround-View World-Action Model for End-to-End Autonomous DrivingNextMaking Kubernetes AI-Ready for NVIDIA Vera Rubin on Day One
Related posts

Making Local LLMs Usable on 8GB GPUs
0HackerNoon

Educational Byte: What Are Synthetic Tokens in Crypto Networks?
0HackerNoon
Your AI Assistant Is Waiting Around - An AI Agent Goes to Work
0HackerNoon
Your AI Agent's Memory Needs Two Clocks
0HackerNoon
The Online Right’s Closing Campaign Argument: Fascist Anakin Skywalker Memes
0Wired

Apple is ‘not able to predict’ when it’ll have enough iPhone Duos to go around
0The Verge