Investigating EEG-Based Functional Connectivity Patterns for Multimodal
  Emotion Recognition
  Compared with the rich studies on the motor brain-computer interface (BCI),
the recently emerging affective BCI presents distinct challenges since the
brain functional connectivity networks involving emotion are not well
investigated. Previous studies on emotion recognition based on
electroencephalography (EEG) signals mainly rely on single-channel-based
feature extraction methods. In this paper, we propose a novel emotion-relevant
critical subnetwork selection algorithm and investigate three EEG functional
connectivity network features: strength, clustering coefficient, and
eigenvector centrality. The discrimination ability of the EEG connectivity
features in emotion recognition is evaluated on three public emotion EEG
datasets: SEED, SEED-V, and DEAP. The strength feature achieves the best
classification performance and outperforms the state-of-the-art differential
entropy feature based on single-channel analysis. The experimental results
reveal that distinct functional connectivity patterns are exhibited for the
five emotions of disgust, fear, sadness, happiness, and neutrality.
Furthermore, we construct a multimodal emotion recognition model by combining
the functional connectivity features from EEG and the features from eye
movements or physiological signals using deep canonical correlation analysis.
The classification accuracies of multimodal emotion recognition are 95.08/6.42%
on the SEED dataset, 84.51/5.11% on the SEED-V dataset, and 85.34/2.90% and
86.61/3.76% for arousal and valence on the DEAP dataset, respectively. The
results demonstrate the complementary representation properties of the EEG
connectivity features with eye movement data. In addition, we find that the
brain networks constructed with 18 channels achieve comparable performance with
that of the 62-channel network in multimodal emotion recognition and enable
easier setups for BCI systems in real scenarios.
