Geometric Multimodal Deep Learning with Multi-Scaled Graph Wavelet
  Convolutional Network
  Multimodal data provide complementary information of a natural phenomenon by
integrating data from various domains with very different statistical
properties. Capturing the intra-modality and cross-modality information of
multimodal data is the essential capability of multimodal learning methods. The
geometry-aware data analysis approaches provide these capabilities by
implicitly representing data in various modalities based on their geometric
underlying structures. Also, in many applications, data are explicitly defined
on an intrinsic geometric structure. Generalizing deep learning methods to the
non-Euclidean domains is an emerging research field, which has recently been
investigated in many studies. Most of those popular methods are developed for
unimodal data. In this paper, a multimodal multi-scaled graph wavelet
convolutional network (M-GWCN) is proposed as an end-to-end network. M-GWCN
simultaneously finds intra-modality representation by applying the multiscale
graph wavelet transform to provide helpful localization properties in the graph
domain of each modality, and cross-modality representation by learning
permutations that encode correlations among various modalities. M-GWCN is not
limited to either the homogeneous modalities with the same number of data, or
any prior knowledge indicating correspondences between modalities. Several
semi-supervised node classification experiments have been conducted on three
popular unimodal explicit graph-based datasets and five multimodal implicit
ones. The experimental results indicate the superiority and effectiveness of
the proposed methods compared with both spectral graph domain convolutional
neural networks and state-of-the-art multimodal methods.
