Learning by Watching
  When in a new situation or geographical location, human drivers have an
extraordinary ability to watch others and learn maneuvers that they themselves
may have never performed. In contrast, existing techniques for learning to
drive preclude such a possibility as they assume direct access to an
instrumented ego-vehicle with fully known observations and expert driver
actions. However, such measurements cannot be directly accessed for the non-ego
vehicles when learning by watching others. Therefore, in an application where
data is regarded as a highly valuable asset, current approaches completely
discard the vast portion of the training data that can be potentially obtained
through indirect observation of surrounding vehicles. Motivated by this key
insight, we propose the Learning by Watching (LbW) framework which enables
learning a driving policy without requiring full knowledge of neither the state
nor expert actions. To increase its data, i.e., with new perspectives and
maneuvers, LbW makes use of the demonstrations of other vehicles in a given
scene by (1) transforming the ego-vehicle's observations to their points of
view, and (2) inferring their expert actions. Our LbW agent learns more robust
driving policies while enabling data-efficient learning, including quick
adaptation of the policy to rare and novel scenarios. In particular, LbW drives
robustly even with a fraction of available driving data required by existing
methods, achieving an average success rate of 92% on the original CARLA
benchmark with only 30 minutes of total driving data and 82% with only 10
minutes.
