HeatNet: Bridging the Day-Night Domain Gap in Semantic Segmentation with
  Thermal Images
  The majority of learning-based semantic segmentation methods are optimized
for daytime scenarios and favorable lighting conditions. Real-world driving
scenarios, however, entail adverse environmental conditions such as nighttime
illumination or glare which remain a challenge for existing approaches. In this
work, we propose a multimodal semantic segmentation model that can be applied
during daytime and nighttime. To this end, besides RGB images, we leverage
thermal images, making our network significantly more robust. We avoid the
expensive annotation of nighttime images by leveraging an existing daytime
RGB-dataset and propose a teacher-student training approach that transfers the
dataset's knowledge to the nighttime domain. We further employ a domain
adaptation method to align the learned feature spaces across the domains and
propose a novel two-stage training scheme. Furthermore, due to a lack of
thermal data for autonomous driving, we present a new dataset comprising over
20,000 time-synchronized and aligned RGB-thermal image pairs. In this context,
we also present a novel target-less calibration method that allows for
automatic robust extrinsic and intrinsic thermal camera calibration. Among
others, we employ our new dataset to show state-of-the-art results for
nighttime semantic segmentation.
