Re-understanding Finite-State Representations of Recurrent Policy
  Networks
  We introduce an approach for understanding control policies represented as
recurrent neural networks. Recent work has approached this problem by
transforming such recurrent policy networks into finite-state machines (FSM)
and then analyzing the equivalent minimized FSM. While this led to interesting
insights, the minimization process can obscure a deeper understanding of a
machine's operation by merging states that are semantically distinct. To
address this issue, we introduce an analysis approach that starts with an
unminimized FSM and applies more-interpretable reductions that preserve the key
decision points of the policy. We also contribute an attention tool to attain a
deeper understanding of the role of observations in the decisions. Our case
studies on 7 Atari games and 3 control benchmarks demonstrate that the approach
can reveal insights that have not been previously noticed.
