AI & ML
Seeing and Imagining as One: Unified Video Generation and 4D Perception for Embodied World Modeling
arXiv cs.RO0 views
arXiv:2610.39232v2 Announce Type: replace-cross Abstract: Embodied world models should not only imagine plausible futures, but also perceive the geometric, semantic, and motion structure of the evolving world. We present GenSight, a unified embodied world model that brings future video generation and dense perception into a single pretrained video diffusion transformer. GenSight uses a unified spatiotemporal representation for video, depth, embodiment segmentation, and first-frame-relative 2D flow. Built on a shared DiT backbone, GenSight uses lightweight task modulation to predict all modalities within the same architecture. All tasks are jointly optimized within a unified velocity prediction space. At inference, GenSight supports two complementary modes: multi-step future video generation with aligned dense predictions and single-step dense perception from an observed video. By unifying modality representation, model architecture, and prediction space, GenSight avoids separate modality-specific pipelines and couples appearance, geometry, semantics, and motion in a shared representation. It promotes cross-modal consistency and allows dense perception to benefit from the spatiotemporal priors learned through video generation. Trained on embodied manipulation data spanning multiple robot embodiments and human-hand interactions, GenSight demonstrates strong performance in both video generation and dense 4D perception, highlighting the effectiveness of unifying seeing and imagining for embodied world modeling.
Source:https://arxiv.org/abs/2610.39232PreviousSV-WAM: An Efficient Surround-View World-Action Model for End-to-End Autonomous DrivingNextMaking Kubernetes AI-Ready for NVIDIA Vera Rubin on Day One
Related posts
Stopping Is a Capability: What Two Governance Talks at Google DevFest Atlanta 2026 Had in Common
0dev.to
Criei uma plataforma para desenvolvedores brasileiros. Agora vem a parte mais difícil: fazer ela crescer.
0dev.to
Agentic backdoors via pretraining poisoning
0Hacker News
Anthropic reveals $900B AI spending commitments ahead of IPO
0Hacker News
What is AI?
0dev.to AI
Cara Mengelola Grup Telegram dengan Bot AI Gratis: Solusi Lengkap untuk Pengembang & Pemilik Grup
0dev.to AI