Controlling for Confounders in Multimodal Emotion Classification via
  Adversarial Learning
  Various psychological factors affect how individuals express emotions. Yet,
when we collect data intended for use in building emotion recognition systems,
we often try to do so by creating paradigms that are designed just with a focus
on eliciting emotional behavior. Algorithms trained with these types of data
are unlikely to function outside of controlled environments because our
emotions naturally change as a function of these other factors. In this work,
we study how the multimodal expressions of emotion change when an individual is
under varying levels of stress. We hypothesize that stress produces modulations
that can hide the true underlying emotions of individuals and that we can make
emotion recognition algorithms more generalizable by controlling for variations
in stress. To this end, we use adversarial networks to decorrelate stress
modulations from emotion representations. We study how stress alters acoustic
and lexical emotional predictions, paying special attention to how modulations
due to stress affect the transferability of learned emotion recognition models
across domains. Our results show that stress is indeed encoded in trained
emotion classifiers and that this encoding varies across levels of emotions and
across the lexical and acoustic modalities. Our results also show that emotion
recognition models that control for stress during training have better
generalizability when applied to new domains, compared to models that do not
control for stress during training. We conclude that is is necessary to
consider the effect of extraneous psychological factors when building and
testing emotion recognition models.
