Mellotron: Multispeaker expressive voice synthesis by conditioning on
  rhythm, pitch and global style tokens
  Mellotron is a multispeaker voice synthesis model based on Tacotron 2 GST
that can make a voice emote and sing without emotive or singing training data.
By explicitly conditioning on rhythm and continuous pitch contours from an
audio signal or music score, Mellotron is able to generate speech in a variety
of styles ranging from read speech to expressive speech, from slow drawls to
rap and from monotonous voice to singing voice. Unlike other methods, we train
Mellotron using only read speech data without alignments between text and
audio. We evaluate our models using the LJSpeech and LibriTTS datasets. We
provide F0 Frame Errors and synthesized samples that include style transfer
from other speakers, singers and styles not seen during training, procedural
manipulation of rhythm and pitch and choir synthesis.
