Autoencoder for Synthetic to Real Generalization: From Simple to More
  Complex Scenes
  Learning on synthetic data and transferring the resulting properties to their
real counterparts is an important challenge for reducing costs and increasing
safety in machine learning. In this work, we focus on autoencoder architectures
and aim at learning latent space representations that are invariant to
inductive biases caused by the domain shift between simulated and real images
showing the same scenario. We train on synthetic images only, present
approaches to increase generalizability and improve the preservation of the
semantics to real datasets of increasing visual complexity. We show that
pre-trained feature extractors (e.g. VGG) can be sufficient for generalization
on images of lower complexity, but additional improvements are required for
visually more complex scenes. To this end, we demonstrate a new sampling
technique, which matches semantically important parts of the image, while
randomizing the other parts, leads to salient feature extraction and a
neglection of unimportant parts. This helps the generalization to real data and
we further show that our approach outperforms fine-tuned classification models.
