Neural-to-Tree Policy Distillation with Policy Improvement Criterion
  While deep reinforcement learning has achieved promising results in
challenging decision-making tasks, the main bones of its success --- deep
neural networks are mostly black-boxes. A feasible way to gain insight into a
black-box model is to distill it into an interpretable model such as a decision
tree, which consists of if-then rules and is easy to grasp and be verified.
However, the traditional model distillation is usually a supervised learning
task under a stationary data distribution assumption, which is violated in
reinforcement learning. Therefore, a typical policy distillation that clones
model behaviors with even a small error could bring a data distribution shift,
resulting in an unsatisfied distilled policy model with low fidelity or low
performance. In this paper, we propose to address this issue by changing the
distillation objective from behavior cloning to maximizing an advantage
evaluation. The novel distillation objective maximizes an approximated
cumulative reward and focuses more on disastrous behaviors in critical states,
which controls the data shift effect. We evaluate our method on several Gym
tasks, a commercial fight game, and a self-driving car simulator. The empirical
results show that the proposed method can preserve a higher cumulative reward
than behavior cloning and learn a more consistent policy to the original one.
Moreover, by examining the extracted rules from the distilled decision trees,
we demonstrate that the proposed method delivers reasonable and robust
decisions.
