Prosodic Alignment for off-screen automatic dubbing
  The goal of automatic dubbing is to perform speech-to-speech translation
while achieving audiovisual coherence. This entails isochrony, i.e.,
translating the original speech by also matching its prosodic structure into
phrases and pauses, especially when the speaker's mouth is visible. In previous
work, we introduced a prosodic alignment model to address isochrone or
on-screen dubbing. In this work, we extend the prosodic alignment model to also
address off-screen dubbing that requires less stringent synchronization
constraints. We conduct experiments on four dubbing directions - English to
French, Italian, German and Spanish - on a publicly available collection of TED
Talks and on publicly available YouTube videos. Empirical results show that
compared to our previous work the extended prosodic alignment model provides
significantly better subjective viewing experience on videos in which on-screen
and off-screen automatic dubbing is applied for sentences with speakers mouth
visible and not visible, respectively.
