Textless Speech Emotion Conversion using Discrete and Decomposed
  Representations
  Speech emotion conversion is the task of modifying the perceived emotion of a
speech utterance while preserving the lexical content and speaker identity. In
this study, we cast the problem of emotion conversion as a spoken language
translation task. We use a decomposition of the speech signal into discrete
learned representations, consisting of phonetic-content units, prosodic
features, speaker, and emotion. First, we modify the speech content by
translating the phonetic-content units to a target emotion, and then predict
the prosodic features based on these units. Finally, the speech waveform is
generated by feeding the predicted representations into a neural vocoder. Such
a paradigm allows us to go beyond spectral and parametric changes of the
signal, and model non-verbal vocalizations, such as laughter insertion, yawning
removal, etc. We demonstrate objectively and subjectively that the proposed
method is vastly superior to current approaches and even beats text-based
systems in terms of perceived emotion and audio quality. We rigorously evaluate
all components of such a complex system and conclude with an extensive model
analysis and ablation study to better emphasize the architectural choices,
strengths and weaknesses of the proposed method. Samples are available under
the following link: https://speechbot.github.io/emotion.
