Let's Face It: Probabilistic Multi-modal Interlocutor-aware Generation
  of Facial Gestures in Dyadic Settings
  To enable more natural face-to-face interactions, conversational agents need
to adapt their behavior to their interlocutors. One key aspect of this is
generation of appropriate non-verbal behavior for the agent, for example facial
gestures, here defined as facial expressions and head movements. Most existing
gesture-generating systems do not utilize multi-modal cues from the
interlocutor when synthesizing non-verbal behavior. Those that do, typically
use deterministic methods that risk producing repetitive and non-vivid motions.
In this paper, we introduce a probabilistic method to synthesize
interlocutor-aware facial gestures - represented by highly expressive FLAME
parameters - in dyadic conversations. Our contributions are: a) a method for
feature extraction from multi-party video and speech recordings, resulting in a
representation that allows for independent control and manipulation of
expression and speech articulation in a 3D avatar; b) an extension to MoGlow, a
recent motion-synthesis method based on normalizing flows, to also take
multi-modal signals from the interlocutor as input and subsequently output
interlocutor-aware facial gestures; and c) a subjective evaluation assessing
the use and relative importance of the input modalities. The results show that
the model successfully leverages the input from the interlocutor to generate
more appropriate behavior. Videos, data, and code available at:
https://jonepatr.github.io/lets_face_it.
