Maximum Entropy Reinforcement Learning with Mixture Policies
  Mixture models are an expressive hypothesis class that can approximate a rich
set of policies. However, using mixture policies in the Maximum Entropy
(MaxEnt) framework is not straightforward. The entropy of a mixture model is
not equal to the sum of its components, nor does it have a closed-form
expression in most cases. Using such policies in MaxEnt algorithms, therefore,
requires constructing a tractable approximation of the mixture entropy. In this
paper, we derive a simple, low-variance mixture-entropy estimator. We show that
it is closely related to the sum of marginal entropies. Equipped with our
entropy estimator, we derive an algorithmic variant of Soft Actor-Critic (SAC)
to the mixture policy case and evaluate it on a series of continuous control
tasks.
