Multi-modal Deep Analysis for Multimedia
  With the rapid development of Internet and multimedia services in the past
decade, a huge amount of user-generated and service provider-generated
multimedia data become available. These data are heterogeneous and multi-modal
in nature, imposing great challenges for processing and analyzing them.
Multi-modal data consist of a mixture of various types of data from different
modalities such as texts, images, videos, audios etc. In this article, we
present a deep and comprehensive overview for multi-modal analysis in
multimedia. We introduce two scientific research problems, data-driven
correlational representation and knowledge-guided fusion for multimedia
analysis. To address the two scientific problems, we investigate them from the
following aspects: 1) multi-modal correlational representation: multi-modal
fusion of data across different modalities, and 2) multi-modal data and
knowledge fusion: multi-modal fusion of data with domain knowledge. More
specifically, on data-driven correlational representation, we highlight three
important categories of methods, such as multi-modal deep representation,
multi-modal transfer learning, and multi-modal hashing. On knowledge-guided
fusion, we discuss the approaches for fusing knowledge with data and four
exemplar applications that require various kinds of domain knowledge, including
multi-modal visual question answering, multi-modal video summarization,
multi-modal visual pattern mining and multi-modal recommendation. Finally, we
bring forward our insights and future research directions.
