Analyzing Reinforcement Learning Benchmarks with Random Weight Guessing
  We propose a novel method for analyzing and visualizing the complexity of
standard reinforcement learning (RL) benchmarks based on score distributions. A
large number of policy networks are generated by randomly guessing their
parameters, and then evaluated on the benchmark task; the study of their
aggregated results provide insights into the benchmark complexity. Our method
guarantees objectivity of evaluation by sidestepping learning altogether: the
policy network parameters are generated using Random Weight Guessing (RWG),
making our method agnostic to (i) the classic RL setup, (ii) any learning
algorithm, and (iii) hyperparameter tuning. We show that this approach isolates
the environment complexity, highlights specific types of challenges, and
provides a proper foundation for the statistical analysis of the task's
difficulty. We test our approach on a variety of classic control benchmarks
from the OpenAI Gym, where we show that small untrained networks can provide a
robust baseline for a variety of tasks. The networks generated often show good
performance even without gradual learning, incidentally highlighting the
triviality of a few popular benchmarks.
