Towards Optimal Strategies for Training Self-Driving Perception Models
  in Simulation
  Autonomous driving relies on a huge volume of real-world data to be labeled
to high precision. Alternative solutions seek to exploit driving simulators
that can generate large amounts of labeled data with a plethora of content
variations. However, the domain gap between the synthetic and real data
remains, raising the following important question: What are the best ways to
utilize a self-driving simulator for perception tasks? In this work, we build
on top of recent advances in domain-adaptation theory, and from this
perspective, propose ways to minimize the reality gap. We primarily focus on
the use of labels in the synthetic domain alone. Our approach introduces both a
principled way to learn neural-invariant representations and a theoretically
inspired view on how to sample the data from the simulator. Our method is easy
to implement in practice as it is agnostic of the network architecture and the
choice of the simulator. We showcase our approach on the bird's-eye-view
vehicle segmentation task with multi-sensor data (cameras, lidar) using an
open-source simulator (CARLA), and evaluate the entire framework on a
real-world dataset (nuScenes). Last but not least, we show what types of
variations (e.g. weather conditions, number of assets, map design, and color
diversity) matter to perception networks when trained with driving simulators,
and which ones can be compensated for with our domain adaptation technique.
