Open Source
Video2SwimFish: An Automated Pipeline for Reconstructing Controllable Fish Models and Biological Locomotion from Real Fish Videos
arXiv cs.RO0 views
arXiv:2609.38966v2 Announce Type: replace Abstract: We present Video2SwimFish, an automated pipeline and benchmark for building controllable fish assets from real-fish videos for underwater embodied AI. Given synchronized multi-view videos of an individual fish, the pipeline generates a metrically scaled deformable mesh from a VLM-selected canonical frame, builds internal articulation adapted to that individual's morphology through a VLM actor-critic loop, and extracts a Biological Locomotion Manifold (BLM). The BLM provides a low-dimensional action space bounded by real-fish motion. Compared with learning directly in joint space, it lets the individual swimming policy of each reconstructed fish converge faster and reach a higher task success rate. We release two paired datasets: synchronized top- and front-view recordings of 120 individual fish across 6 species, and the controllable assets and individual swimming policies derived from them. Because every asset is tied to the animal it came from, the dataset supports a benchmark that evaluates locomotion learning not only on task success but on fidelity to that individual's locomotion behavior. We evaluate on two tasks, 3D trajectory following and target reaching, and a case study in which a simulated BlueROV underwater robot captures the assets. We further validate biological realism with a human study in which fish experts rate the simulated fish, as a complement to the automated metrics. We find that task success and locomotion fidelity do not necessarily improve together, and we identify faithful reproduction of individual animal locomotion as an open challenge for the community. Project website: https://hangongchen.github.io/video2swimfish-web/
Source:https://arxiv.org/abs/2609.38966PreviousRAVEL: Asynchronous Rolling Inference for Flow-Based Vision-Language-Action ModelsNextPILOT: Guiding Long-Horizon World-Model Planning for Robot Manipulation with Imagined Subgoals and Policy Priors
Related posts
ia-sincer v1.1.0: Less Context, Better Decisions, More Focused AI Agents
0dev.to OpenSource
Blocking git push Inside Claude Code Takes One Hook and Nine Patterns
0dev.to Git
Kairos: A Unified PDDL Planner for Robots: Plans in Milliseconds, with a Guarantee for Every Answer
0arXiv cs.RO
Curvature-aware 3D length estimation of greenhouse cucumbers using RGB-D imaging and cubic spline arc-length integration
0arXiv cs.RO
AWS FinOps Field Notes: Purging Incomplete S3 Multipart Uploads Before They Snowball
0dev.to AWS
Block Anything on YouTube
0Hacker News Ask