Sim2Real for Self-Supervised Monocular Depth and Segmentation
  Image-based learning methods for autonomous vehicle perception tasks require
large quantities of labelled, real data in order to properly train without
overfitting, which can often be incredibly costly. While leveraging the power
of simulated data can potentially aid in mitigating these costs, networks
trained in the simulation domain usually fail to perform adequately when
applied to images in the real domain. Recent advances in domain adaptation have
indicated that a shared latent space assumption can help to bridge the gap
between the simulation and real domains, allowing the transference of the
predictive capabilities of a network from the simulation domain to the real
domain. We demonstrate that a twin VAE-based architecture with a shared latent
space and auxiliary decoders is able to bridge the sim2real gap without
requiring any paired, ground-truth data in the real domain. Using only paired,
ground-truth data in the simulation domain, this architecture has the potential
to generate perception tasks such as depth and segmentation maps. We compare
this method to networks trained in a supervised manner to indicate the merit of
these results.
