AI & ML
Who Says What: Symbolic Trimodal Binding Mechanisms in Audio-Visual LLMs
arXiv cs.SD0 views
arXiv:2609.31193v2 Announce Type: replace-cross Abstract: Current Audio-Visual LLMs (AVLLMs) struggle with reasoning over videos featuring multi-speaker dialogues. In such videos, resolving "who says what" is crucial, which necessitates trimodal (text-audio-visual) binding. Motivated by these challenges, we systematically investigate how this trimodal binding is achieved in AVLLMs. Specifically, we identify emergent symbolic trimodal binding mechanisms in AVLLMs that utilize modality-specific symbolic variables. By encoding auditory and visual components into symbolic variables-capturing temporal utterance sequences and spatial entity coordinates, respectively-the model establishes cross-modal linking within this abstract space. Crucially, we reveal that when trimodal binding fails, the breakdown predominantly stems from misaligned audio-visual connections. To overcome this bottleneck, we introduce an audio-visual prompting method utilizing an off-the-shelf Active Speaker Detection (ASD) model. By simply overlaying visual bounding boxes on active speakers, this training-free approach yields immediate performance gains across four conversation-centric benchmarks. Moreover, lightweight fine-tuning of fewer than 300 steps on these ASD-prompted-videos extends these gains to three general AV benchmarks, suggesting the generalizability of our method. Our code is available at https://github.com/kaistmm/Who-Says-What.
Source:https://arxiv.org/abs/2609.31193PreviousWhy ML-based cough models do not generalize: a systematic cross-dataset evaluation for tuberculosis screeningNextCollecting Infrastructure-Level Measures for Energy Analysis in Modern Computing Infrastructures
Related posts
Voyager 1 at one light-day: Historian revisits Golden Record's forgotten details
0Ars Technica

Grand Theft Auto VI has arrived
0The Verge

Suno now lets you make AI videos for your AI music
0The Verge
India’s Humyn Labs raises $47M to feed U.S. robotics companies the data they need
0TechCrunch
Felicis leads $7.5M bet on Unsiloed AI to make enterprise documents usable by AI agents
0TechCrunch

GPT-6.2 Astra and Sol: Cheaper, Shorter Answers, and the Fine Print
0dev.to