Hyperspectral Pansharpening Based on Improved Deep Image Prior and
  Residual Reconstruction
  Hyperspectral pansharpening aims to synthesize a low-resolution hyperspectral
image (LR-HSI) with a registered panchromatic image (PAN) to generate an
enhanced HSI with high spectral and spatial resolution. Recently proposed HS
pansharpening methods have obtained remarkable results using deep convolutional
networks (ConvNets), which typically consist of three steps: (1) up-sampling
the LR-HSI, (2) predicting the residual image via a ConvNet, and (3) obtaining
the final fused HSI by adding the outputs from first and second steps. Recent
methods have leveraged Deep Image Prior (DIP) to up-sample the LR-HSI due to
its excellent ability to preserve both spatial and spectral information,
without learning from large data sets. However, we observed that the quality of
up-sampled HSIs can be further improved by introducing an additional
spatial-domain constraint to the conventional spectral-domain energy function.
We define our spatial-domain constraint as the $L_1$ distance between the
predicted PAN image and the actual PAN image. To estimate the PAN image of the
up-sampled HSI, we also propose a learnable spectral response function (SRF).
Moreover, we noticed that the residual image between the up-sampled HSI and the
reference HSI mainly consists of edge information and very fine structures. In
order to accurately estimate fine information, we propose a novel over-complete
network, called HyperKite, which focuses on learning high-level features by
constraining the receptive from increasing in the deep layers. We perform
experiments on three HSI datasets to demonstrate the superiority of our
DIP-HyperKite over the state-of-the-art pansharpening methods. The deployment
codes, pre-trained models, and final fusion outputs of our DIP-HyperKite and
the methods used for the comparisons will be publicly made available at
https://github.com/wgcban/DIP-HyperKite.git.
