Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task
  Learning and ArcFace
  Affective computing has been largely limited in terms of available data
resources. The need to collect and annotate diverse in-the-wild datasets has
become apparent with the rise of deep learning models, as the default approach
to address any computer vision task. Some in-the-wild databases have been
recently proposed. However: i) their size is small, ii) they are not
audiovisual, iii) only a small part is manually annotated, iv) they contain a
small number of subjects, or v) they are not annotated for all main behavior
tasks (valence-arousal estimation, action unit detection and basic expression
classification). To address these, we substantially extend the largest
available in-the-wild database (Aff-Wild) to study continuous emotions such as
valence and arousal. Furthermore, we annotate parts of the database with basic
expressions and action units. As a consequence, for the first time, this allows
the joint study of all three types of behavior states. We call this database
Aff-Wild2. We conduct extensive experiments with CNN and CNN-RNN architectures
that use visual and audio modalities; these networks are trained on Aff-Wild2
and their performance is then evaluated on 10 publicly available emotion
databases. We show that the networks achieve state-of-the-art performance for
the emotion recognition tasks. Additionally, we adapt the ArcFace loss function
in the emotion recognition context and use it for training two new networks on
Aff-Wild2 and then re-train them in a variety of diverse expression recognition
databases. The networks are shown to improve the existing state-of-the-art. The
database, emotion recognition models and source code are available at
http://ibug.doc.ic.ac.uk/resources/aff-wild2.
