General
Talk, Render, Act: Integrating Social Gesture and Digital Face with Synchronized Speech for Conversational Humanoid Robot
arXiv cs.RO0 views
arXiv:2610.06153v2 Announce Type: replace Abstract: Expressive humanoid interaction requires speech, facial animation, and body gestures to form a coherent response. However, many full-body humanoid robots produce speech and gestures without a visually expressive face, while talking-face animation and robot gesture generation are typically developed separately. We present Talk, Render, Act (TRABot), an agent-based framework comprising specialized agents for motion-atom construction, dialogue generation, motion planning, and facial animation. First, to produce natural and semantically meaningful gestures, we construct Robot-Ready Semantic Motion Atoms by segmenting long-form, G1-retargeted BEAT2 motion into units with natural gesture boundaries, human-verified communicative functions, and feasible trajectories. Second, to preserve semantic order and coordinate body motion with the spoken response, we introduce a Semantic-Conditioned Compositional Planner. Given an ordered semantic function sequence and an estimated response duration, the planner selects approved atoms to realize the longest feasible action sequence while accounting for transitions and neutral recovery. Finally, we deploy a Streaming Face-Speech-Body Integration system on a physical G1 humanoid, combining streaming dialogue audio, audio-driven facial animation, and semantically planned body motion in a unified real-time interaction loop. Quantitative and qualitative experiments demonstrate that TRAbot achieves the best overall performance among all compared conditions in terms of naturalness, expressiveness, and multimodal coherence.
Source:https://arxiv.org/abs/2610.06153PreviousForeAct3D: Policy-Grounded Future World Modeling for VLA PoliciesNextDoes Knowledge About Perceptual Uncertainty Help an Agent in Automated Driving?
Related posts
More people should build browsers
0Hacker News New
Byesky: Wipe your Bluesky account clean
0Hacker News New
Electric trucks hit 50% of China's heavy truck sales, diesel demand falls
0Hacker News New
Poles and polars in projective space with homogeneous coordinates
0Hacker News New
Making Peace with the Projective Plane
0Hacker News New
Show HN: Hacker News Aggregators, Aggregated
0Hacker News Show