Identifying Critical States by the Action-Based Variance of Expected
  Return
  The balance of exploration and exploitation plays a crucial role in
accelerating reinforcement learning (RL). To deploy an RL agent in human
society, its explainability is also essential. However, basic RL approaches
have difficulties in deciding when to choose exploitation as well as in
extracting useful points for a brief explanation of its operation. One reason
for the difficulties is that these approaches treat all states the same way.
Here, we show that identifying critical states and treating them specially is
commonly beneficial to both problems. These critical states are the states at
which the action selection changes the potential of success and failure
substantially. We propose to identify the critical states using the variance in
the Q-function for the actions and to perform exploitation with high
probability on the identified states. These simple methods accelerate RL in a
grid world with cliffs and two baseline tasks of deep RL. Our results also
demonstrate that the identified critical states are intuitively interpretable
regarding the crucial nature of the action selection. Furthermore, our analysis
of the relationship between the timing of the identification of especially
critical states and the rapid progress of learning suggests there are a few
especially critical states that have important information for accelerating RL
rapidly.
