BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for
  Conversational Gestures Synthesis
  Achieving realistic, vivid, and human-like synthesized conversational
gestures conditioned on multi-modal data is still an unsolved problem, due to
the lack of available datasets, models and standard evaluation metrics. To
address this, we build Body-Expression-Audio-Text dataset, BEAT, which has i)
76 hours, high-quality, multi-modal data captured from 30 speakers talking with
eight different emotions and in four different languages, ii) 32 millions
frame-level emotion and semantic relevance annotations.Our statistical analysis
on BEAT demonstrates the correlation of conversational gestures with facial
expressions, emotions, and semantics, in addition to the known correlation with
audio, text, and speaker identity. Qualitative and quantitative experiments
demonstrate metrics' validness, ground truth data quality, and baseline's
state-of-the-art performance. To the best of our knowledge, BEAT is the largest
motion capture dataset for investigating the human gestures, which may
contribute to a number of different research fields including controllable
gesture synthesis, cross-modality analysis, emotional gesture recognition. The
data, code and model will be released for research.
