Variance Reduced Advantage Estimation with $\delta$ Hindsight Credit
  Assignment
  Hindsight Credit Assignment (HCA) refers to a recently proposed family of
methods for producing more efficient credit assignment in reinforcement
learning. These methods work by explicitly estimating the probability that
certain actions were taken in the past given present information. Prior work
has studied the properties of such methods and demonstrated their behaviour
empirically. We extend this work by introducing a particular HCA algorithm
which has provably lower variance than the conventional Monte-Carlo estimator
when the necessary functions can be estimated exactly. This result provides a
strong theoretical basis for how HCA could be broadly useful.
