Prosodic Clustering for Phoneme-level Prosody Control in End-to-End
  Speech Synthesis
  This paper presents a method for controlling the prosody at the phoneme level
in an autoregressive attention-based text-to-speech system. Instead of learning
latent prosodic features with a variational framework as is commonly done, we
directly extract phoneme-level F0 and duration features from the speech data in
the training set. Each prosodic feature is discretized using unsupervised
clustering in order to produce a sequence of prosodic labels for each
utterance. This sequence is used in parallel to the phoneme sequence in order
to condition the decoder with the utilization of a prosodic encoder and a
corresponding attention module. Experimental results show that the proposed
method retains the high quality of generated speech, while allowing
phoneme-level control of F0 and duration. By replacing the F0 cluster centroids
with musical notes, the model can also provide control over the note and octave
within the range of the speaker.
