Policy Information Capacity: Information-Theoretic Measure for Task
  Complexity in Deep Reinforcement Learning
  Progress in deep reinforcement learning (RL) research is largely enabled by
benchmark task environments. However, analyzing the nature of those
environments is often overlooked. In particular, we still do not have agreeable
ways to measure the difficulty or solvability of a task, given that each has
fundamentally different actions, observations, dynamics, rewards, and can be
tackled with diverse RL algorithms. In this work, we propose policy information
capacity (PIC) -- the mutual information between policy parameters and episodic
return -- and policy-optimal information capacity (POIC) -- between policy
parameters and episodic optimality -- as two environment-agnostic,
algorithm-agnostic quantitative metrics for task difficulty. Evaluating our
metrics across toy environments as well as continuous control benchmark tasks
from OpenAI Gym and DeepMind Control Suite, we empirically demonstrate that
these information-theoretic metrics have higher correlations with normalized
task solvability scores than a variety of alternatives. Lastly, we show that
these metrics can also be used for fast and compute-efficient optimizations of
key design parameters such as reward shaping, policy architectures, and MDP
properties for better solvability by RL algorithms without ever running full RL
experiments.
