Voice Cloning (XTTS)

The XTTS paper proposes the XTTS system, a method that builds upon the Tortoise model.

Verified

Definition

The XTTS paper says most zero-shot multi-speaker TTS (ZS-TTS) systems support only a single language. The paper's introduction says most TTS systems were tailored from a single speaker's voice, but there is current interest in synthesizing voices for new speakers (not seen during training) employing only a few seconds of speech. The paper proposes the XTTS system and makes it publicly available.

The paper says its method builds upon the Tortoise model and adds several novel modifications to enable multilingual training, improve voice cloning, and enable faster training and inference. The paper reports that XTTS was trained in 16 languages and achieved state-of-the-art (SOTA) results in most of them.

At a glance

The paper says XTTS builds upon Tortoise and includes several novel modifications to enable multilingual training, improve ZS-TTS, and enable faster training and inference.

  • The VQ-VAE has 13M parameters, receives a mel-spectrogram as input, and encodes each frame with 1 codebook of 8192 codes at a 21.53 Hz frame rate.
  • The Conditioning Encoder is composed of 6 16-head Scaled Dot-Product Attention layers followed by a Perceiver Resampler, which produces a fixed number of embeddings independently of the input audio length.
  • The GPT-2 encoder is a decoder-only transformer composed of 443M parameters.
  • The GPT-2 encoder receives text tokens obtained via a 6681-token custom Byte-Pair Encoding (BPE) tokenizer and predicts the VQ-VAE audio codes as output.
  • The decoder is based on the HiFi-GAN vocoder and has 26M parameters.
  • The decoder is also conditioned with speaker embedding from the H/ASP model.
  • The decoder's speaker embedding is added in each upsampling layer via linear projection.
  • To improve speaker similarity, the paper also added the Speaker Consistency Loss (SCL).

The source does not state the order of these points, so they are listed without arrows.

How it works

The paper states that XTTS is a massively multilingual ZS-TTS model supporting low/medium resource languages. The paper states that its model can perform cross-language ZS-TTS without needing a parallel training dataset. In its preliminary experiments, the paper noticed that in massive multilingual training, the use of a single embedding leads to a decrease in the model's speaker cloning capability.

In preliminary experiments, the paper verified that filtering the less frequent codes improved the model's expressiveness. The paper says that, due to the high compression rate of the VQ-VAE, reconstructing the audio directly from the VQ-VAE codes leads to pronunciation issues and artifacts. To speed up inference, the paper trained the VQ-VAE and the encoder using 22.5 kHz audio signals.

In its conclusion, the paper says the XTTS model is faster than VALL-E because its encoder produces tokens at a 21.53 Hz frame rate, compared with 75Hz for the VALL-E model. The XTTS dataset is composed of public and internal datasets, and for English the paper used 541.7 hours from LibriTTS-R and 1812.7 hours from LibriLight. XTTS and YourTTS were trained using NVIDIA A100 GPUs with 80 GB, and XTTS was trained on 4 GPUs.

The paper's Experiment 3 trained the XTTS model with the XTTS dataset for approximately 2.5M steps.

Example

For its model comparison, the paper used 240 sentences for each supported language from FLORES+. For each speaker, the paper randomly selected one audio segment between 3 and 8 seconds as a reference during test sentence generation. To compare the similarity between the synthesized voice and the original speaker, the paper computes the Speaker Encoder Cosine Similarity (SECS) using the ECAPA2 speaker encoder.

The paper computed the Character Error Rate (CER) using the Whisper Large v3 model to evaluate pronunciation accuracy. The paper also measures user preference scores by comparing XTTS with HierSpeech++ and Mega-TTS 2 models. For speaker adaptation, the paper selected samples of approximately 10 min of speech from well-known or unique-style voices (e.g. whispering voices) in different languages.

The paper chose 3 speakers of English, 3 speakers of Portuguese, 1 speaker of Chinese, and 1 speaker of Arabic. The paper reports that the fine-tuned model improves the SECS from 0.5852 to 0.7166 when cloning these voices in a cross-lingual way.

Sources

This page is written from the official sources below. Open them to check the original text.

Verified ·

This page is written by AI from the official sources above and checked automatically. This site has not tested it. Found a mistake? Tell us.