Multimodal Transfer Deep Learning with Applications in Audio-Visual
  Recognition
  We propose a transfer deep learning (TDL) framework that can transfer the
knowledge obtained from a single-modal neural network to a network with a
different modality. Specifically, we show that we can leverage speech data to
fine-tune the network trained for video recognition, given an initial set of
audio-video parallel dataset within the same semantics. Our approach first
learns the analogy-preserving embeddings between the abstract representations
learned from intermediate layers of each network, allowing for semantics-level
transfer between the source and target modalities. We then apply our neural
network operation that fine-tunes the target network with the additional
knowledge transferred from the source network, while keeping the topology of
the target network unchanged. While we present an audio-visual recognition task
as an application of our approach, our framework is flexible and thus can work
with any multimodal dataset, or with any already-existing deep networks that
share the common underlying semantics. In this work in progress report, we aim
to provide comprehensive results of different configurations of the proposed
approach on two widely used audio-visual datasets, and we discuss potential
applications of the proposed approach.
