Neural Style-Preserving Visual Dubbing
  Dubbing is a technique for translating video content from one language to
another. However, state-of-the-art visual dubbing techniques directly copy
facial expressions from source to target actors without considering
identity-specific idiosyncrasies such as a unique type of smile. We present a
style-preserving visual dubbing approach from single video inputs, which
maintains the signature style of target actors when modifying facial
expressions, including mouth motions, to match foreign languages. At the heart
of our approach is the concept of motion style, in particular for facial
expressions, i.e., the person-specific expression change that is yet another
essential factor beyond visual accuracy in face editing applications. Our
method is based on a recurrent generative adversarial network that captures the
spatiotemporal co-activation of facial expressions, and enables generating and
modifying the facial expressions of the target actor while preserving their
style. We train our model with unsynchronized source and target videos in an
unsupervised manner using cycle-consistency and mouth expression losses, and
synthesize photorealistic video frames using a layered neural face renderer.
Our approach generates temporally coherent results, and handles dynamic
backgrounds. Our results show that our dubbing approach maintains the
idiosyncratic style of the target actor better than previous approaches, even
for widely differing source and target actors.
