Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with
  Tacotron
  We present an extension to the Tacotron speech synthesis architecture that
learns a latent embedding space of prosody, derived from a reference acoustic
representation containing the desired prosody. We show that conditioning
Tacotron on this learned embedding space results in synthesized audio that
matches the prosody of the reference signal with fine time detail even when the
reference and synthesis speakers are different. Additionally, we show that a
reference prosody embedding can be used to synthesize text that is different
from that of the reference utterance. We define several quantitative and
subjective metrics for evaluating prosody transfer, and report results with
accompanying audio samples from single-speaker and 44-speaker Tacotron models
on a prosody transfer task.
