Characterizing the Action-Generalization Gap in Deep Q-Learning
  We study the action generalization ability of deep Q-learning in discrete
action spaces. Generalization is crucial for efficient reinforcement learning
(RL) because it allows agents to use knowledge learned from past experiences on
new tasks. But while function approximation provides deep RL agents with a
natural way to generalize over state inputs, the same generalization mechanism
does not apply to discrete action outputs. And yet, surprisingly, our
experiments indicate that Deep Q-Networks (DQN), which use exactly this type of
function approximator, are still able to achieve modest action generalization.
Our main contribution is twofold: first, we propose a method of evaluating
action generalization using expert knowledge of action similarity, and
empirically confirm that action generalization leads to faster learning;
second, we characterize the action-generalization gap (the difference in
learning performance between DQN and the expert) in different domains. We find
that DQN can indeed generalize over actions in several simple domains, but that
its ability to do so decreases as the action space grows larger.
