Maximum Causal Tsallis Entropy Imitation Learning
  In this paper, we propose a novel maximum causal Tsallis entropy (MCTE)
framework for imitation learning which can efficiently learn a sparse
multi-modal policy distribution from demonstrations. We provide the full
mathematical analysis of the proposed framework. First, the optimal solution of
an MCTE problem is shown to be a sparsemax distribution, whose supporting set
can be adjusted. The proposed method has advantages over a softmax distribution
in that it can exclude unnecessary actions by assigning zero probability.
Second, we prove that an MCTE problem is equivalent to robust Bayes estimation
in the sense of the Brier score. Third, we propose a maximum causal Tsallis
entropy imitation learning (MCTEIL) algorithm with a sparse mixture density
network (sparse MDN) by modeling mixture weights using a sparsemax
distribution. In particular, we show that the causal Tsallis entropy of an MDN
encourages exploration and efficient mixture utilization while Boltzmann Gibbs
entropy is less effective. We validate the proposed method in two simulation
studies and MCTEIL outperforms existing imitation learning methods in terms of
average returns and learning multi-modal policies.
