AI & ML
Beyond Mimicry: Constrained Exploration with GRPO for Joint Multi-Talker ASR and Diarization under Unknown Speaker Counts
arXiv cs.SD0 views
arXiv:2611.22627v1 Announce Type: new Abstract: Joint multi-talker ASR with speaker attribution and timestamps is particularly challenging when the speaker count is unknown and the output must follow a strict structure. We observe that Supervised Fine-Tuning (SFT)-only alignment of SpeechLLMs is brittle under dense overlaps, often leading to burst hallucinations and malformed speaker/timestamp tags. We propose a unified two-stage generative framework: (i) CoT-augmented SFT that infers the global speaker count before transcription, and (ii) GRPO-based constrained exploration guided by a Multi-dimensional Constraint-Aware Reward (MCAR) that directly optimizes permutation-invariant accuracy while enforcing counting, temporal, and structural constraints. On Libri3Mix and a Dynamic-Mix (2+3) set (speaker count unknown at test time), our model achieves 14.52%/9.24% cpWER and 1.95%/1.12% WDER, respectively, with GRPO yielding a 35%/54% relative cpWER reduction over SFT+CoT.
Source:https://arxiv.org/abs/2611.22627PreviousHigh-Fidelity Binarized Neural VocoderNextBack to Mel: Learning to Bridge Fidelity and Perception in Generative Speech Enhancement
Related posts
Microsoft’s new Licensed Weights lets businesses customize AI models with their own data
0ZDNet

“Collaboration over competition”: Hugging Face hands model inference traffic to OpenRouter
0The New Stack
Robots learn human movement patterns to find paths through dense crowds
0Phys.org Tech
Researchers find eight ways AI could lead to our downfall
0Phys.org Tech
Q&A: Why transportation is critical to rebuilding countries after war and terrorism
0Phys.org Tech
Wine Merges Remote OpenGL Swapchain Image Presentation, CEF Now Works With Wayland
0Phoronix