Multi-level Stress Assessment from ECG in a Virtual Reality Environment
  using Multimodal Fusion
  ECG is an attractive option to assess stress in serious Virtual Reality (VR)
applications due to its non-invasive nature. However, the existing Machine
Learning (ML) models perform poorly. Moreover, existing studies only perform a
binary stress assessment, while to develop a more engaging biofeedback-based
application, multi-level assessment is necessary. Existing studies annotate and
classify a single experience (e.g. watching a VR video) to a single stress
level, which again prevents design of dynamic experiences where real-time
in-game stress assessment can be utilized. In this paper, we report our
findings on a new study on VR stress assessment, where three stress levels are
assessed. ECG data was collected from 9 users experiencing a VR roller coaster.
The VR experience was then manually labeled in 10-seconds segments to three
stress levels by three raters. We then propose a novel multimodal deep fusion
model utilizing spectrogram and 1D ECG that can provide a stress prediction
from just a 1-second window. Experimental results demonstrate that the proposed
model outperforms the classical HRV-based ML models (9% increase in accuracy)
and baseline deep learning models (2.5% increase in accuracy). We also report
results on the benchmark WESAD dataset to show the supremacy of the model.
