MSD: Saliency-aware Knowledge Distillation for Multimodal Understanding
  To reduce a model size but retain performance, we often rely on knowledge
distillation (KD) which transfers knowledge from a large "teacher" model to a
smaller "student" model. However, KD on multimodal datasets such as
vision-language tasks is relatively unexplored, and digesting multimodal
information is challenging since different modalities present different types
of information. In this paper, we perform a large-scale empirical study to
investigate the importance and effects of each modality in knowledge
distillation. Furthermore, we introduce a multimodal knowledge distillation
framework, modality-specific distillation (MSD), to transfer knowledge from a
teacher on multimodal tasks by learning the teacher's behavior within each
modality. The idea aims at mimicking a teacher's modality-specific predictions
by introducing auxiliary loss terms for each modality. Furthermore, because
each modality has different saliency for predictions, we define saliency scores
for each modality and investigate saliency-based weighting schemes for the
auxiliary losses. We further study a weight learning approach to learn the
optimal weights on these loss terms. In our empirical analysis, we examine the
saliency of each modality in KD, demonstrate the effectiveness of the weighting
scheme in MSD, and show that it achieves better performance than KD on four
multimodal datasets.
