Abstract

Diffusion models have demonstrated significant potential in speech synthesis tasks, including text-to-speech (TTS) and voice cloning. However, their iterative denoising processes are inefficient and hinder the application of end-to-end optimization with perceptual metrics. In this paper, we propose a novel method of distilling TTS diffusion models with direct end-to-end evaluation metric optimization, achieving state-of-the-art performance. By incorporating Connectionist Temporal Classification (CTC) loss and Speaker Verification (SV) loss, our approach optimizes perceptual evaluation metrics, leading to notable improvements in word error rate and speaker similarity. Our experiments show that DMDSpeech consistently surpasses prior state-of-the-art models in both naturalness and speaker similarity while being significantly faster. Moreover, our synthetic speech has a higher level of voice similarity to the prompt than the ground truth in both human evaluation and objective speaker similarity metric. This work highlights the potential of direct metric optimization in speech synthesis, allowing models to better align with human auditory preferences.

Model Overview

The framework consists of inference and three main components for training: (1) Inference (upper left): A few-step distilled generator synthesizes speech directly from noise, conditioned on the text and speaker prompt (red arrow). (2) Distribution Matching Distillation (upper right): Gradient computation for DMD loss where the student score model matches the teacher score model to align the distribution of student generator with the teacher distribution (purple arrow). (3) Multi-Modal Adversarial Training (lower right): The discriminator uses stacked features from the student score model to distinguish between real and synthesized noisy latents conditioned on both text and prompt (yellow arrows). (4) Direct Metric Optimization (lower left): Direct metric optimization for word error rate (WER) via CTC loss (pink arrow) and speaker embedding cosine similarity (SIM) via SV loss (blue arrow).

Comparison to Previous SOTA

This section contain samples sourced from other SOTA model demo page for a comparison with our model. For each model, we compare different aspects. All samples are downsampled to 16 kHz for fair comparison.

Our model was only trained on LibriLight dataset and has not seen any of the prompt voice below.

---

Comparison to NaturalSpeech 3 and DiTTo-TTS (Emotions)

We compare our model to NaturalSpeech 3 and DiTTo-TTS for emotional speech. The samples of NaturalSpeech 3, DiTTo-TTS, prompts, and texts are sourced from DiTTo-TTS demo page. The text synthesized is "why fades the lotus of the water" from LibriSpeech test set.

---

Comparison to DiTTo-TTS (Voice Clone)

We compare our model to DiTTo-TTS for cloning the voice of vairous fictional characters. The samples of DiTTo-TTS, prompts, and texts are sourced from its official demo page.

---

Comparison to Vall-E 2 (Continuation)

We compare our model to Vall-E 2 for both cross-sentence synthesis and sentence continuation. Due to policy restriction, we were unable to obtain samples from the authors and conduct subjective evaluation in our study. The comparison here is for reference purpose only, and samples are taken from its official demo page. All samples of Vall-E 2 are with GroupSize=1 as it is shown to have the best performance in the paper.

---

Comparison to StyleTTS 2 (Long-Form Generation)

We compare our model to StyleTTS 2 for long-form generation. Our model with only 7 second of prompt speech can perform similarly well as SOTA single-speaker model trained on this particular speaker with 24 hours of data. The sample is taken from its official demo page.