Pose-Controllable Talking Face Generation by Implicitly Modularized
  Audio-Visual Representation
  While accurate lip synchronization has been achieved for arbitrary-subject
audio-driven talking face generation, the problem of how to efficiently drive
the head pose remains. Previous methods rely on pre-estimated structural
information such as landmarks and 3D parameters, aiming to generate
personalized rhythmic movements. However, the inaccuracy of such estimated
information under extreme conditions would lead to degradation problems. In
this paper, we propose a clean yet effective framework to generate
pose-controllable talking faces. We operate on raw face images, using only a
single photo as an identity reference. The key is to modularize audio-visual
representations by devising an implicit low-dimension pose code. Substantially,
both speech content and head pose information lie in a joint non-identity
embedding space. While speech content information can be defined by learning
the intrinsic synchronization between audio-visual modalities, we identify that
a pose code will be complementarily learned in a modulated convolution-based
reconstruction framework.
  Extensive experiments show that our method generates accurately lip-synced
talking faces whose poses are controllable by other videos. Moreover, our model
has multiple advanced capabilities including extreme view robustness and
talking face frontalization. Code, models, and demo videos are available at
https://hangz-nju-cuhk.github.io/projects/PC-AVS.
