AI & ML
JAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents
arXiv eess.AS0 views
arXiv:2610.22499v1 Announce Type: cross Abstract: Human communication is inherently multimodal, conveying intent not only through linguistic and vocal cues but also through visual signals. However, existing conversational systems have mainly focused on text or speech, leaving out the visual feedback that is essential for natural interaction. In this paper, we propose JAVIS, an end-to-end streaming framework for real-time audio-visual conversational agents. Our key idea is to let the voice and the face speak from a single representation. On top of a shared LLM, a Joint Refined Head produces a unified audio-visual representation, which is fed into dedicated streaming branches for speech and face generation. This end-to-end design avoids intermediate bottlenecks, enabling real-time generation while keeping speech and face tightly synchronized. In addition, we present JAVIS-Syn, a 334-hour synthetic corpus designed to address the lack of expressive audio-visual dialogue data, covering emotionally rich multi-turn conversations and assistant-style exchanges. Experiments show that JAVIS achieves approximately 200$\times$ lower initial latency than a strong baseline, while outperforming baselines in lip synchronization and facial expression appropriateness.
Source:https://arxiv.org/abs/2610.22499PreviousAcousticAgent: A Floorplan-Centric Simulation Platform for Distributed Multi-Room Speech ProcessingNextTowards Prosodically Informed Mizo TTS without Explicit Tone Markings
Related posts

Here’s how to listen live to Apple’s Q4 2026 earnings call
09to5Mac
Can AI Write CAD? Testing LLM-Generated 3D Models for Manufacturability
0HackerNoon
Guardz Integrates Palo Alto Networks Prisma Browser Protection to Address Growing Workspace Security
0HackerNoon

LinearB Launches Agentic Studio to Steer AI Coding Agents in a $127 Billion Market by 2032
0HackerNoon
We Built a Test to Catch Contact Forms That Lie. It Lied to Us Five Times First
0HackerNoon

The TechBeat: Why Knowledge Graphs and Ontologies Are Making a Comeback in the Age of AI (11/2/2026)
0HackerNoon