AI & ML
Any Channel Count, One Token Stream: Incorporating Spatial Audio Understanding and Reasoning into Large Audio-Language Models
arXiv cs.SD0 views
arXiv:2611.18112v1 Announce Type: new Abstract: Audio phase relations encode source locations, yet existing large audio-language models (LALM) reduce audio to single-channel content to stabilise general audio understanding, leaving spatial information for direction, distance, and source-relation questions unused. We present SpheraAudio, which accepts recordings with any channel count -- mono, stereo, First-Order Ambisonics (FOA), or higher -- and maps them into a single token stream. A spatial encoder turns complex multichannel spectra into spatial tokens interleaved with audio tokens at corresponding temporal positions. Spatial Multimodal RoPE (SMRoPE) makes each token's modality identity and temporal slot visible to rotary attention without altering the language model's original RoPE. Training runs in three stages -- spatial-encoder pretraining, projector alignment, and joint supervised-and-reinforcement learning -- progressing from geometric supervision to language-based spatial decisions, with answers trained as explicit spatial chain-of-thought: step-by-step geometric reasoning over the heard scene. Current benchmarks tie questions to a fixed channel-count format and score answers without spatial reasoning evidence. We introduce SpheraQA, which spans mono, stereo, and FOA recordings and organises questions hierarchically from single-attribute perception through cross-source relations to multi-step reasoning, scored programmatically against simulated scene geometry and real-recording annotations with step-by-step solution references. Across SpheraQA and public spatial-audio benchmarks, SpheraAudio sets a new performance standard for spatial audio understanding. It supports high-precision localisation, tracking, and reasoning over static and moving sound sources while preserving strong general-audio capabilities. We will release code, SpheraQA, and evaluation metrics to support reproducibility.
Source:https://arxiv.org/abs/2611.18112PreviousCoTTAGE: An Explainable Judge for Text-to-Audio GenerationNexteDIAR: Training-Free Open-Speaker Streaming Diarization for Long Meetings and Large Archives via e-means
Related posts
How to Trim Audio for an Online Course Lesson — Free, in Your Browser
0dev.to WebDev
Chemical Anchor Fasteners: A Practical Guide to Stainless Steel Chemical Anchor Studs
0dev.to WebDev
Automating Notion Desktop App Installation on Arch Linux
0dev.to Linux
existence-trace: a journal your AI reads, and has to prove it used
0dev.to Python
A 30-question regression suite for your customer-service chatbot (with a tiny Python runner)
0dev.to Python
The boring-infrastructure approach to running AI agents: cron, env files, launchd, and a markdown decision log
0dev.to DevOps