GridToPix: Training Embodied Agents with Minimal Supervision
  While deep reinforcement learning (RL) promises freedom from hand-labeled
data, great successes, especially for Embodied AI, require significant work to
create supervision via carefully shaped rewards. Indeed, without shaped
rewards, i.e., with only terminal rewards, present-day Embodied AI results
degrade significantly across Embodied AI problems from single-agent
Habitat-based PointGoal Navigation (SPL drops from 55 to 0) and two-agent
AI2-THOR-based Furniture Moving (success drops from 58% to 1%) to three-agent
Google Football-based 3 vs. 1 with Keeper (game score drops from 0.6 to 0.1).
As training from shaped rewards doesn't scale to more realistic tasks, the
community needs to improve the success of training with terminal rewards. For
this we propose GridToPix: 1) train agents with terminal rewards in gridworlds
that generically mirror Embodied AI environments, i.e., they are independent of
the task; 2) distill the learned policy into agents that reside in complex
visual worlds. Despite learning from only terminal rewards with identical
models and RL algorithms, GridToPix significantly improves results across
tasks: from PointGoal Navigation (SPL improves from 0 to 64) and Furniture
Moving (success improves from 1% to 25%) to football gameplay (game score
improves from 0.1 to 0.6). GridToPix even helps to improve the results of
shaped reward training.
