Generating coherent spontaneous speech and gesture from text
  Embodied human communication encompasses both verbal (speech) and non-verbal
information (e.g., gesture and head movements). Recent advances in machine
learning have substantially improved the technologies for generating synthetic
versions of both of these types of data: On the speech side, text-to-speech
systems are now able to generate highly convincing, spontaneous-sounding speech
using unscripted speech audio as the source material. On the motion side,
probabilistic motion-generation methods can now synthesise vivid and lifelike
speech-driven 3D gesticulation. In this paper, we put these two
state-of-the-art technologies together in a coherent fashion for the first
time. Concretely, we demonstrate a proof-of-concept system trained on a
single-speaker audio and motion-capture dataset, that is able to generate both
speech and full-body gestures together from text input. In contrast to previous
approaches for joint speech-and-gesture generation, we generate full-body
gestures from speech synthesis trained on recordings of spontaneous speech from
the same person as the motion-capture data. We illustrate our results by
visualising gesture spaces and text-speech-gesture alignments, and through a
demonstration video at https://simonalexanderson.github.io/IVA2020 .
