Student-Initiated Action Advising via Advice Novelty
  Action advising is a budget-constrained knowledge exchange mechanism between
teacher-student peers that can help tackle exploration and sample inefficiency
problems in deep reinforcement learning (RL). Most recently, student-initiated
techniques that utilise state novelty and uncertainty estimations have obtained
promising results. However, the approaches built on these estimations have some
potential weaknesses. First, they assume that the convergence of the student's
RL model implies less need for advice. This can be misleading in scenarios with
teacher absence early on where the student is likely to learn suboptimally by
itself; yet also ignore the teacher's assistance later. Secondly, the delays
between encountering states and having them to take effect in the RL model
updates in presence of the experience replay dynamics cause a feedback lag in
what the student actually needs advice for. We propose a student-initiated
algorithm that alleviates these by employing Random Network Distillation (RND)
to measure the novelty of a piece of advice. Furthermore, we perform RND
updates only for the advised states to ensure that the student's own learning
does not impair its ability to leverage the teacher. Experiments in GridWorld
and MinAtar show that our approach performs on par with the state-of-the-art
and demonstrates significant advantages in the scenarios where the existing
methods are prone to fail.
