This is a simplified guide to an AI model called Xtts-V2 maintained by Lucataco. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.
Overview
xtts-v2 is a multilingual text-to-speech model from lucataco that clones voices using only a 6-second audio sample and synthesizes speech in 17 different languages. Built on Coqui's XTTS-v2 architecture, this model powers production services including Coqui Studio and the Coqui API. The single most important thing to know is that you need a clean audio reference sample—ideally 6 seconds of clear speech—to achieve high-quality voice cloning; the model includes a denoising option to handle noisier microphone recordings, but input quality directly affects output quality.
Best use cases
Cross-language voice dubbing for video content. If you have a source video with an English speaker and want to create versions in Spanish, German, or French while preserving the original voice characteristics, xtts-v2 excels at this. The model maintains emotional tone and speech patterns from the reference clip while generating natural-sounding output in the target language. This is significantly faster and cheaper than hiring voice actors for each language variant.
Character voice generation for interactive fiction and games. Game developers and interactive fiction creators can use short character voice samples to generate dialogue in any of 17 supported languages. The model preserves personality and emotional qualities from the reference audio, creating consistent character voices across hundreds of dialogue lines without needing studio time for each utterance.
Accessibility features for user-generated content. Creators can record a brief introduction or voice signature, then have the system generate narration for long-form content—podcasts, educational videos, or documentation—all in their voice. This makes accessibility and personalization tractable for small teams where hiring a voice actor is impractical.
Rapid prototyping of voice-enabled applications. Chatbots, virtual assistants, and voice-interactive tools need consistent voices during development. xtts-v2 lets you create multiple distinct voices from brief samples, test them with end users, and iterate without licensing constraints or dependency on external voice talent.
Emotion and style transfer between speakers. If you have recordings of different speakers with distinct emotional deliveries (excited, calm, authoritative), you can use those as references to apply similar emotional coloring to synthesized text. This enables fine-grained control over the tone and presence of generated speech.
Limitations
The model requires a minimum quality audio reference sample; while the cleanup_voice parameter applies denoising to handle microphone recordings, heavily corrupted or very short audio clips produce noticeably degraded output. Background noise, clipping, or artifacts in the speaker sample translate into artifacts in generated speech.
Audio input must be provided in one of five formats: WAV, MP3, M4A, OGG, or FLV. No duration limit is specified in the schema, though best results cluster around the recommended 6-second reference sample. The output is a single URI pointing to an audio file, which requires additional processing if you need the raw audio bytes or want to batch-process multiple outputs.
The model generates audio at 24 kHz sampling rate. If you require higher fidelity (48 kHz or above), you would need to upsample separately or choose a different model. There is no documented latency guarantee, so streaming or real-time voice interactions may experience delays.
The computational requirements are not publicly documented on Replicate, but based on the model architecture, inference typically takes several seconds per prediction depending on text length. For production applications requiring sub-second latency, this may not be suitable.
The CPML license (Coqui Public Model License) permits commercial use but carries restrictions around redistributing the model weights and certain competitive uses. Review the full license terms before deploying in commercial products that might involve competing with Coqui's own services.
The model is frozen as of November 2023 (latest version created 2023-11-28) and may not reflect improvements made to XTTS-v2 after that date. If the Coqui team has released newer versions or fixes, they would not be available in this Replicate deployment without a new version being published.
How it compares
indextts-2 by lucataco adds emotional expressiveness and duration control on top of voice cloning, letting you specify how long certain words should be held. Pick xtts-v2 if you need simplicity and multilingual coverage; pick indextts-2 if you need fine-grained control over speech timing and emotional intensity.
whisperspeech-small by lucataco is a lighter, open-source text-to-speech model that does not require voice samples—it uses Whisper embeddings for conditioning instead. Choose xtts-v2 for voice cloning and emotional transfer from reference audio; choose whisperspeech-small if you want a faster, smaller model that does not need speaker samples and works offline.
XTTS-v2 by coqui is the base model that this Replicate implementation wraps. Running it on Replicate gives you managed infrastructure and simple API integration without local GPU setup, but offers less control over model parameters and inference options compared to the HuggingFace version. Use the Replicate version for quick integration into applications; use the HuggingFace version if you need custom preprocessing, batch inference, or want to run it locally on your own hardware.
step-audio-tts-3b by lucataco is trained on synthetic data using LLM-Chat paradigm and lacks voice cloning capabilities. Choose xtts-v2 if speaker cloning and multilingual cross-language transfer are core requirements; use step-audio-tts-3b if you need a model trained on synthetic data and do not need voice reference samples.
xtts-v1 by pagebrain is the predecessor version requiring only 3-second audio samples instead of 6 seconds, but with fewer languages (15 instead of 17) and less stable speaker conditioning. Pick xtts-v2 for its architecture improvements and stability gains; pick xtts-v1 only if you have extremely constrained audio samples and cannot improve them.
Technical specifications
xtts-v2 is a transformer-based text-to-speech model that performs voice cloning and multilingual synthesis. The architecture accepts speaker reference audio and synthesizes new speech by conditioning on speaker embeddings extracted from the reference clip. The model generates audio at 24 kHz sampling rate and supports 17 languages: English, Spanish, French, German, Italian, Portuguese, Polish, Dutch, Russian, Chinese, Japanese, Arabic, Hindi, Turkish, Korean, and Hungarian.
The implementation runs on Replicate's managed infrastructure. The model accepts speaker audio in multiple formats (WAV, MP3, M4A, OGG, FLV) and includes a cleanup_voice preprocessing step that applies denoising filters suitable for microphone recordings. The text input has no documented length limit, but practical synthesis times increase with text length.
Key architectural improvements over XTTS-v1 include:
- Multiple speaker reference support and interpolation between speakers
- Enhanced speaker conditioning for more stable voice cloning
- Better prosody and naturalness in generated speech
- Expanded language support (added Hungarian and Korean)
- Stability improvements reducing artifacts
The model output is always a single audio file URI. No intermediate parameters, speaker embeddings, or confidence scores are exposed through the API.
Model inputs and outputs
Inputs
- text (string, required): The text to synthesize. No length limit documented. Default example: "Hi there, I'm your new voice clone. Try your best to upload quality audio"
- speaker (URI string, required): Audio file containing the reference voice. Supported formats: WAV, MP3, M4A, OGG, FLV. Recommended length: 6 seconds. Longer or shorter samples may reduce quality.
- language (enum string, optional): Target language for synthesis. Default: "en". Supported languages: en, es, fr, de, it, pt, pl, nl, ru, zh, ja, ar, hi, tr, ko, hu
- cleanup_voice (boolean, optional): Apply denoising to the speaker audio. Default: false. Set to true for noisy microphone recordings.
Outputs
- output (URI string): A URL pointing to the synthesized audio file in WAV format at 24 kHz sampling rate. The file is ephemeral and suitable for immediate use or re-hosting on your own storage.
Getting started
import replicate client = replicate.Replicate() output = client.run( "lucataco/xtts-v2:684bc3855b37866c0c65add2ff39c78f3dea3f4ff103a436465326e0f438d55e", input={ "text": "Hello, this is a test of voice cloning in French.", "speaker": "https://example.com/my_voice_sample.wav", "language": "fr", "cleanup_voice": False } ) print(output) # Returns a URI string pointing to the synthesized audio To use a local audio file as the speaker reference, upload it to a public URL or use the Replicate file upload feature. The cleanup_voice flag is useful when your speaker sample contains background noise from a microphone recording—enable it if the reference audio is not clean studio-quality.
Frequently asked questions
Q: What happens if I provide a speaker audio clip shorter than 6 seconds?
A: The model can work with shorter samples, but voice cloning quality degrades as the reference becomes too brief. A 6-second sample is recommended; anything under 2 seconds will likely produce poor results.
Q: Can I use xtts-v2 to generate multiple voices or swap between different speakers in a single run?
A: Not in a single API call. Each prediction uses one speaker reference and generates speech in one voice. To create multi-speaker content, you must run separate predictions for each speaker and concatenate the results.
Q: What license applies to audio generated by this model, and can I use it commercially?
A: The CPML (Coqui Public Model License) permits commercial use of generated audio. However, you cannot redistribute the model weights or use it to create competing voice cloning services. Review the full license at https://coqui.ai/cpml for competitive use restrictions.
Q: Why is the output audio sometimes distorted or unnatural-sounding?
A: Poor quality or irrelevant speaker reference audio is the most common cause. Ensure your reference sample is clean speech without long silences, background noise, or music. Enable cleanup_voice if recording from a microphone. Very short text may also sound stilted; provide more context if the output seems choppy.
Q: How does this compare to using XTTS-v2 directly from HuggingFace or running it locally?
A: The Replicate version provides a managed API requiring no GPU setup or model downloads—you submit text and audio, get back a result. The HuggingFace version gives you complete control, allows batch processing, and runs on your own hardware. Use Replicate for quick integration; use HuggingFace for customization and cost control at large scale.
Q: Is cleanup_voice essential, or can I skip it for clean reference audio?
A: Skip it for clean reference audio. The denoising adds latency and is optimized for microphone recordings. For studio-quality or high-fidelity reference samples, leaving cleanup_voice set to false produces better results.
Q: What is the expected latency for a single synthesis prediction?
A: Latency is not publicly specified, but typical inference ranges from 2-10 seconds depending on text length and load. This model is not suitable for real-time or streaming voice applications requiring sub-second response times.
Q: Does the model preserve emotional tone and accent from the reference audio?
A: Yes. The model extracts speaker characteristics—including accent, emotional coloring, and speech patterns—from the reference sample and applies them to synthesized text. This enables emotion transfer and style transfer when you use reference audio from speakers with distinct delivery styles.
