Human Action Recognition Using Deep Multilevel Multimodal (M2) Fusion of
  Depth and Inertial Sensors
  Multimodal fusion frameworks for Human Action Recognition (HAR) using depth
and inertial sensor data have been proposed over the years. In most of the
existing works, fusion is performed at a single level (feature level or
decision level), missing the opportunity to fuse rich mid-level features
necessary for better classification. To address this shortcoming, in this
paper, we propose three novel deep multilevel multimodal fusion frameworks to
capitalize on different fusion strategies at various stages and to leverage the
superiority of multilevel fusion. At input, we transform the depth data into
depth images called sequential front view images (SFIs) and inertial sensor
data into signal images. Each input modality, depth and inertial, is further
made multimodal by taking convolution with the Prewitt filter. Creating
"modality within modality" enables further complementary and discriminative
feature extraction through Convolutional Neural Networks (CNNs). CNNs are
trained on input images of each modality to learn low-level, high-level and
complex features. Learned features are extracted and fused at different stages
of the proposed frameworks to combine discriminative and complementary
information. These highly informative features are served as input to a
multi-class Support Vector Machine (SVM). We evaluate the proposed frameworks
on three publicly available multimodal HAR datasets, namely, UTD Multimodal
Human Action Dataset (MHAD), Berkeley MHAD, and UTD-MHAD Kinect V2.
Experimental results show the supremacy of the proposed fusion frameworks over
existing methods.
