Robust and fine-grained prosody control of end-to-end speech synthesis
  We propose prosody embeddings for emotional and expressive speech synthesis
networks. The proposed methods introduce temporal structures in the embedding
networks, thus enabling fine-grained control of the speaking style of the
synthesized speech. The temporal structures can be designed either on the
speech side or the text side, leading to different control resolutions in time.
The prosody embedding networks are plugged into end-to-end speech synthesis
networks and trained without any other supervision except for the target speech
for synthesizing. It is demonstrated that the prosody embedding networks
learned to extract prosodic features. By adjusting the learned prosody
features, we could change the pitch and amplitude of the synthesized speech
both at the frame level and the phoneme level. We also introduce the temporal
normalization of prosody embeddings, which shows better robustness against
speaker perturbations during prosody transfer tasks.
