AI & ML
Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval
arXiv cs.SD0 views
arXiv:2604.18360v4 Announce Type: replace Abstract: Audio-text retrieval systems based on Contrastive Language-Audio Pretraining (CLAP) achieve strong performance on traditional benchmarks; however, these benchmarks rely on caption-style queries that differ substantially from real-world search behavior, limiting their assessment of practical retrieval robustness. We present Omni-Embed-Audio (OEA), a retrieval-oriented encoder leveraging multimodal LLMs with native audio understanding. To systematically evaluate robustness beyond caption-style queries, we introduce User-Intent Queries (UIQs) - five formulations reflecting natural search behaviors: questions, commands, keyword tags, paraphrases, and exclusion-based negative queries. For negative queries, we develop a hard negative mining pipeline and propose discrimination metrics (HNSR, TFR) assessing models' ability to suppress acoustically similar distractors. Experiments on AudioCaps, Clotho, and MECAT show that OEA achieves comparable text-to-audio retrieval performance to state-of-the-art M2D-CLAP, while demonstrating clear advantages in two critical areas: (1) dominant text-to-text retrieval (+22% relative improvement), and (2) substantially superior hard negative discrimination (+4.3%p HNSR@10, +34.7% relative TFR@10), revealing that LLM backbones provide superior semantic understanding of complex queries.
Source:https://arxiv.org/abs/2604.18360PreviousScore-Based Training for Energy-Based TTS ModelsNextRemix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems
Related posts
Why we price a developer tool per fix instead of per seat
0dev.to Productivity
My LLM-Powered Sidekick for Painless PR Descriptions
0dev.to Productivity
Reward-Adaptive Backtracking GFlowNet: A Technique Worth Understanding
0dev.to Productivity
We checked three LLM-written paper explainers against the PDFs. Two had numbers the papers never report.
0dev.to Machine Learning
Should You Rent AI or Run Your Own?
0dev.to Machine Learning
Who's hiring this week: only 1 in 30 new tech jobs is junior, and 1 in 6 'AI jobs' is AI-training work (Oct 26-Nov 1, 2026)
0dev.to Career