Learning Representations for Pixel-based Control: What Matters and Why?
  Learning representations for pixel-based control has garnered significant
attention recently in reinforcement learning. A wide range of methods have been
proposed to enable efficient learning, leading to sample complexities similar
to those in the full state setting. However, moving beyond carefully curated
pixel data sets (centered crop, appropriate lighting, clear background, etc.)
remains challenging. In this paper, we adopt a more difficult setting,
incorporating background distractors, as a first step towards addressing this
challenge. We present a simple baseline approach that can learn meaningful
representations with no metric-based learning, no data augmentations, no
world-model learning, and no contrastive learning. We then analyze when and why
previously proposed methods are likely to fail or reduce to the same
performance as the baseline in this harder setting and why we should think
carefully about extending such methods beyond the well curated environments.
Our results show that finer categorization of benchmarks on the basis of
characteristics like density of reward, planning horizon of the problem,
presence of task-irrelevant components, etc., is crucial in evaluating
algorithms. Based on these observations, we propose different metrics to
consider when evaluating an algorithm on benchmark tasks. We hope such a
data-centric view can motivate researchers to rethink representation learning
when investigating how to best apply RL to real-world tasks.
