End-to-end Multimodal Emotion and Gender Recognition with Dynamic Joint
  Loss Weights
  Multi-task learning is a method for improving the generalizability of
multiple tasks. In order to perform multiple classification tasks with one
neural network model, the losses of each task should be combined. Previous
studies have mostly focused on multiple prediction tasks using joint loss with
static weights for training models, choosing the weights between tasks without
making sufficient considerations by setting them uniformly or empirically. In
this study, we propose a method to calculate joint loss using dynamic weights
to improve the total performance, instead of the individual performance, of
tasks. We apply this method to design an end-to-end multimodal emotion and
gender recognition model using audio and video data. This approach provides
proper weights for the loss of each task when the training process ends. In our
experiments, emotion and gender recognition with the proposed method yielded a
lower joint loss, which is computed as the negative log-likelihood, than using
static weights for joint loss. Moreover, our proposed model has better
generalizability than other models. To the best of our knowledge, this research
is the first to demonstrate the strength of using dynamic weights for joint
loss for maximizing overall performance in emotion and gender recognition
tasks.
