Emotions Don't Lie: An Audio-Visual Deepfake Detection Method Using
  Affective Cues
  We present a learning-based method for detecting real and fake deepfake
multimedia content. To maximize information for learning, we extract and
analyze the similarity between the two audio and visual modalities from within
the same video. Additionally, we extract and compare affective cues
corresponding to perceived emotion from the two modalities within a video to
infer whether the input video is "real" or "fake". We propose a deep learning
network, inspired by the Siamese network architecture and the triplet loss. To
validate our model, we report the AUC metric on two large-scale deepfake
detection datasets, DeepFake-TIMIT Dataset and DFDC. We compare our approach
with several SOTA deepfake detection methods and report per-video AUC of 84.4%
on the DFDC and 96.6% on the DF-TIMIT datasets, respectively. To the best of
our knowledge, ours is the first approach that simultaneously exploits audio
and video modalities and also perceived emotions from the two modalities for
deepfake detection.
