Methodical Advice Collection and Reuse in Deep Reinforcement Learning
  Reinforcement learning (RL) has shown great success in solving many
challenging tasks via use of deep neural networks. Although using deep learning
for RL brings immense representational power, it also causes a well-known
sample-inefficiency problem. This means that the algorithms are data-hungry and
require millions of training samples to converge to an adequate policy. One way
to combat this issue is to use action advising in a teacher-student framework,
where a knowledgeable teacher provides action advice to help the student. This
work considers how to better leverage uncertainties about when a student should
ask for advice and if the student can model the teacher to ask for less advice.
The student could decide to ask for advice when it is uncertain or when both it
and its model of the teacher are uncertain. In addition to this investigation,
this paper introduces a new method to compute uncertainty for a deep RL agent
using a secondary neural network. Our empirical results show that using dual
uncertainties to drive advice collection and reuse may improve learning
performance across several Atari games.
