GradientDICE: Rethinking Generalized Offline Estimation of Stationary
  Values
  We present GradientDICE for estimating the density ratio between the state
distribution of the target policy and the sampling distribution in off-policy
reinforcement learning. GradientDICE fixes several problems of GenDICE (Zhang
et al., 2020), the state-of-the-art for estimating such density ratios. Namely,
the optimization problem in GenDICE is not a convex-concave saddle-point
problem once nonlinearity in optimization variable parameterization is
introduced to ensure positivity, so any primal-dual algorithm is not guaranteed
to converge or find the desired solution. However, such nonlinearity is
essential to ensure the consistency of GenDICE even with a tabular
representation. This is a fundamental contradiction, resulting from GenDICE's
original formulation of the optimization problem. In GradientDICE, we optimize
a different objective from GenDICE by using the Perron-Frobenius theorem and
eliminating GenDICE's use of divergence. Consequently, nonlinearity in
parameterization is not necessary for GradientDICE, which is provably
convergent under linear function approximation.
