Privacy-Preserving Kickstarting Deep Reinforcement Learning with
  Privacy-Aware Learners
  Kickstarting deep reinforcement learning algorithms facilitate a
teacher-student relationship among the agents and allow for a well-performing
teacher to share demonstrations with a student to expedite the student's
training. However, despite the known benefits, the demonstrations may contain
sensitive information about the teacher's training data and existing
kickstarting methods do not take any measures to protect it. Therefore, we use
the framework of differential privacy to develop a mechanism that securely
shares the teacher's demonstrations with the student. The mechanism allows for
the teacher to decide upon the accuracy of its demonstrations with respect to
the privacy budget that it consumes, thereby granting the teacher full control
over its data privacy. We then develop a kickstarted deep reinforcement
learning algorithm for the student that is privacy-aware because we calibrate
its objective with the parameters of the teacher's privacy mechanism. The
privacy-aware design of the algorithm makes it possible to kickstart the
student's learning despite the perturbations induced by the privacy mechanism.
From numerical experiments, we highlight three empirical results: (i) the
algorithm succeeds in expediting the student's learning, (ii) the student
converges to a performance level that was not possible without the
demonstrations, and (iii) the student maintains its enhanced performance even
after the teacher stops sharing useful demonstrations due to its privacy budget
constraints.
