RGB-Depth Fusion GAN for Indoor Depth Completion
  The raw depth image captured by the indoor depth sensor usually has an
extensive range of missing depth values due to inherent limitations such as the
inability to perceive transparent objects and limited distance range. The
incomplete depth map burdens many downstream vision tasks, and a rising number
of depth completion methods have been proposed to alleviate this issue. While
most existing methods can generate accurate dense depth maps from sparse and
uniformly sampled depth maps, they are not suitable for complementing the large
contiguous regions of missing depth values, which is common and critical. In
this paper, we design a novel two-branch end-to-end fusion network, which takes
a pair of RGB and incomplete depth images as input to predict a dense and
completed depth map. The first branch employs an encoder-decoder structure to
regress the local dense depth values from the raw depth map, with the help of
local guidance information extracted from the RGB image. In the other branch,
we propose an RGB-depth fusion GAN to transfer the RGB image to the
fine-grained textured depth map. We adopt adaptive fusion modules named W-AdaIN
to propagate the features across the two branches, and we append a confidence
fusion head to fuse the two outputs of the branches for the final depth map.
Extensive experiments on NYU-Depth V2 and SUN RGB-D demonstrate that our
proposed method clearly improves the depth completion performance, especially
in a more realistic setting of indoor environments with the help of the pseudo
depth map.
