Multimodal Image Super-resolution via Deep Unfolding with Side
  Information
  Deep learning methods have been successfully applied to various computer
vision tasks. However, existing neural network architectures do not per se
incorporate domain knowledge about the addressed problem, thus, understanding
what the model has learned is an open research topic. In this paper, we rely on
the unfolding of an iterative algorithm for sparse approximation with side
information, and design a deep learning architecture for multimodal image
super-resolution that incorporates sparse priors and effectively utilizes
information from another image modality. We develop two deep models performing
reconstruction of a high-resolution image of a target image modality from its
low-resolution variant with the aid of a high-resolution image from a second
modality. We apply the proposed models to super-resolve near-infrared images
using as side information high-resolution RGB\ images. Experimental results
demonstrate the superior performance of the proposed models against
state-of-the-art methods including unimodal and multimodal approaches.
