Open Source
Rhythm-TTS: High-Fidelity Shortcut Latent Modeling and Probabilistic Rhythm Generation for Zero-Shot Text-to-Speech
arXiv cs.SD0 views
arXiv:2510.02848v2 Announce Type: replace Abstract: Zero-shot text-to-speech (TTS) systems are commonly built on either discrete codec tokens or continuous acoustic representations. Discrete-token models benefit from stable sequence modeling and explicit speech factorization, but they ultimately synthesize speech through codebook lookup and codeword composition, which can accumulate errors and impose a quality ceiling on naturalness and speaker similarity. Continuous-representation models can generate fine-grained speech latents directly, but often require heavier in-context modeling and less explicit alignment. This paper presents \textbf{Rhythm-TTS}, a shortcut zero-shot TTS framework that combines the factorizability of discrete codec tokens with the fidelity of continuous latent generation. Instead of using predicted codec tokens as the final acoustic representation, Rhythm-TTS uses them as a semantic-acoustic prior and directly generates holistic continuous speech latents with flow matching, bypassing the discrete-to-latent reconstruction path at inference time. We further address an underexplored dimension of speech naturalness: rhythm diversity across multiple synthesis runs. Motivated by the hypothesis that rhythm variation is largely governed by phoneme duration changes and the occurrence of silent pauses, we introduce probabilistic Duration and Silence Generators to sample dynamic pacing patterns before acoustic generation. Experiments on zero-shot TTS benchmarks evaluate intelligibility, naturalness, speaker similarity, efficiency, and run-to-run rhythm diversity, showing that the proposed shortcut latent modeling and probabilistic rhythm generation improve the fidelity and temporal naturalness of non-autoregressive zero-shot TTS. Code and audio samples are available at our demo page https://rhythm-tts.github.io.
Source:https://arxiv.org/abs/2510.02848PreviousAudioTurbo: Fast Text-to-Audio Generation with Rectified DiffusionNextTELEVAL: A Benchmark Designed for Spoken Language Models in Chinese Interactive Scenarios
Related posts
Omni-modal decomposition autoencoders learn full-stack wearable disentangled representations
0arXiv eess.SP
Spectral Vision Transformer for Efficient Tokenization with Limited Data
0arXiv eess.SP
Gaussian Quasi-Likelihood Estimation for Nonsynchronously Observed L\'evy processes
0arXiv stat.TH
Triple Overlap in Pipelined Krylov Methods on Multi-GPU Systems
0arXiv cs.DC
Learning Integer Winding Evolution Times in Quantum Phase Estimation
0arXiv math.IT
Thirteen Improved Binary Constant-Weight Codes with Explicit Cyclic Orbit Certificates
0arXiv math.IT