Path Consistency Learning in Tsallis Entropy Regularized MDPs
  We study the sparse entropy-regularized reinforcement learning (ERL) problem
in which the entropy term is a special form of the Tsallis entropy. The optimal
policy of this formulation is sparse, i.e.,~at each state, it has non-zero
probability for only a small number of actions. This addresses the main
drawback of the standard Shannon entropy-regularized RL (soft ERL) formulation,
in which the optimal policy is softmax, and thus, may assign a non-negligible
probability mass to non-optimal actions. This problem is aggravated as the
number of actions is increased. In this paper, we follow the work of Nachum et
al. (2017) in the soft ERL setting, and propose a class of novel path
consistency learning (PCL) algorithms, called {\em sparse PCL}, for the sparse
ERL problem that can work with both on-policy and off-policy data. We first
derive a {\em sparse consistency} equation that specifies a relationship
between the optimal value function and policy of the sparse ERL along any
system trajectory. Crucially, a weak form of the converse is also true, and we
quantify the sub-optimality of a policy which satisfies sparse consistency, and
show that as we increase the number of actions, this sub-optimality is better
than that of the soft ERL optimal policy. We then use this result to derive the
sparse PCL algorithms. We empirically compare sparse PCL with its soft
counterpart, and show its advantage, especially in problems with a large number
of actions.
