AI & ML
RoboTalk: Learning Multi-Robot Communication and Coordination from Multimodal Demonstrations
arXiv cs.RO0 views
arXiv:2609.23997v2 Announce Type: replace Abstract: Multi-robot collaboration could enable more efficient and scalable solutions to complex robotic tasks, but collaboration under partial observability remains challenging. Natural-language communication offers a promising approach to coordinating robots under partial observability. However, in decentralized manipulation, jointly learning explicit inter-robot communication and skill-level action selection from multimodal demonstrations remains underexplored for small vision-language models (VLMs) intended for on-device deployment. To address this gap, we introduce RoboTalk, a synthetic data-generation pipeline and dataset of 7,950 multimodal trajectories spanning 53 mobile-manipulation kitchen tasks for training small VLMs to communicate and coordinate. The dataset includes a leader-follower planning protocol, tool calls (perception, manipulation, navigation, and communication), rationale traces, and diversified natural-language communication. Fine-tuning open-source models on our dataset can reach 77% success on novel held-out tasks, a significant improvement over the untuned open source models, which had a success rate of around ~2%.
Source:https://arxiv.org/abs/2609.23997PreviousHIL-UMI: Bringing Human-in-the-Loop Post-Training of Vision-Language-Action Models to Universal Manipulation InterfaceNextICYMI: What landed for AI builders in September 2026
Related posts
How to spend three slow days in a city you have never visited
0dev.to Productivity
Cursor usage limits: the weekly pool, legacy plans and when they reset
0dev.to Productivity
How to Turn a Process Document Into a Working App With AI
0dev.to Machine Learning
Which AI subscription should I buy?
0dev.to Machine Learning
Your Classifier Says 0.97. Six Ways That Number Lies in .NET 10
0dev.to Machine Learning
Which AI song generator fits a lyrics-first workflow?
0dev.to Tutorial