Utilizing Deep Learning Towards Multi-modal Bio-sensing and Vision-based
  Affective Computing
  In recent years, the use of bio-sensing signals such as electroencephalogram
(EEG), electrocardiogram (ECG), etc. have garnered interest towards
applications in affective computing. The parallel trend of deep-learning has
led to a huge leap in performance towards solving various vision-based research
problems such as object detection. Yet, these advances in deep-learning have
not adequately translated into bio-sensing research. This work applies novel
deep-learning-based methods to various bio-sensing and video data of four
publicly available multi-modal emotion datasets. For each dataset, we first
individually evaluate the emotion-classification performance obtained by each
modality. We then evaluate the performance obtained by fusing the features from
these modalities. We show that our algorithms outperform the results reported
by other studies for emotion/valence/arousal/liking classification on DEAP and
MAHNOB-HCI datasets and set up benchmarks for the newer AMIGOS and DREAMER
datasets. We also evaluate the performance of our algorithms by combining the
datasets and by using transfer learning to show that the proposed method
overcomes the inconsistencies between the datasets. Hence, we do a thorough
analysis of multi-modal affective data from more than 120 subjects and 2,800
trials. Finally, utilizing a convolution-deconvolution network, we propose a
new technique towards identifying salient brain regions corresponding to
various affective states.
