AI & ML
Why Multi-Source Audio Fails to Compose: A Geometric Diagnosis and Its Remedy
arXiv cs.SD0 views
arXiv:2611.24410v1 Announce Type: cross Abstract: Multi-source audio-to-image generation exhibits consistent compositional failures when audio tokens occupy geometrically displaced, semantically unanchored positions in the CLIP text-token space, where the diffusion U-Net lacks the prior to reliably attend to them or ground them in distinct spatial regions. We formalize this as the geometric displacement phenomenon and expose it through a two-level clustering analysis comparing BEATs and CLAP embeddings at the raw encoder and projected CLIP token levels. Three geometric properties emerge as important: intra-class compactness, angular alignment with the CLIP text manifold, and inter-token spatial separation. We show that CLAP, whose contrastive language-audio pretraining aligns embeddings with natural language semantics, substantially outperforms BEATs on all three properties, improving the silhouette score by more than 3x and the Audio-Image Content score as well. Building on this analysis, we introduce a multi-token extension that requires no diffusion-backbone retraining and assigns each audio source to a distinct placeholder token in the CLIP tokenizer, complemented by a mixed inference-time objective that combines Attend-and-Excite with an A* spatial separation loss to resolve residual cross-token interference. For evaluation, we construct multi-audio compositional benchmarks from VGGSound (VGG2, VGG3, and VGGN) and extend six standard audio-image metrics to the multi-source setting. Across five benchmarks and four strong baselines, our method achieves strong results on the metrics most directly tied to compositional faithfulness, with AIC improvements of up to 2.25x over the strongest baseline and FID reduced by nearly 2x on two-source generation. We propose geometric displacement as a diagnostic framework for cross-modal generation systems that project learned embeddings into a pretrained contrastive space.
Source:https://arxiv.org/abs/2611.24410PreviousLocation Matters: Layer-wise Prompt Conditioning for Voice--Style Disentanglement in Text-to-SpeechNextPractical Assessment of Agentic Systems in Acoustics
Related posts
CiGS: Conduction in Gaussian Splats for Interior Temperature Field from Thermal Images
0arXiv cs.GR
Probing Physics in Motion Diffusion Models Against Their Own Estimate
0arXiv cs.GR
NIRJALA: Draining the Sea out of Gaussian Splatting with Three-Path Optics and Reversible Physics
0arXiv cs.GR
How Music LLMs Actually Work: From Tokens to Waves (MusicGen, DiT, and Building Your Own Engine)
0dev.to Machine Learning
蒸馏 Lora RHLF 都是什么意思
0dev.to Machine Learning

Teaching an offline plant doctor to spot rust
0dev.to Machine Learning