Pretraining & Reinforcement Learning: Sharpening the Axe Before Cutting
  the Tree
  Pretraining is a common technique in deep learning for increasing performance
and reducing training time, with promising experimental results in deep
reinforcement learning (RL). However, pretraining requires a relevant dataset
for training. In this work, we evaluate the effectiveness of pretraining for RL
tasks, with and without distracting backgrounds, using both large, publicly
available datasets with minimal relevance, as well as case-by-case generated
datasets labeled via self-supervision. Results suggest filters learned during
training on less relevant datasets render pretraining ineffective, while
filters learned during training on the in-distribution datasets reliably reduce
RL training time and improve performance after 80k RL training steps. We
further investigate, given a limited number of environment steps, how to
optimally divide the available steps into pretraining and RL training to
maximize RL performance. Our code is available on GitHub
