Conservative Distributional Reinforcement Learning with Safety
  Constraints
  Safety exploration can be regarded as a constrained Markov decision problem
where the expected long-term cost is constrained. Previous off-policy
algorithms convert the constrained optimization problem into the corresponding
unconstrained dual problem by introducing the Lagrangian relaxation technique.
However, the cost function of the above algorithms provides inaccurate
estimations and causes the instability of the Lagrange multiplier learning. In
this paper, we present a novel off-policy reinforcement learning algorithm
called Conservative Distributional Maximum a Posteriori Policy Optimization
(CDMPO). At first, to accurately judge whether the current situation satisfies
the constraints, CDMPO adapts distributional reinforcement learning method to
estimate the Q-function and C-function. Then, CDMPO uses a conservative value
function loss to reduce the number of violations of constraints during the
exploration process. In addition, we utilize Weighted Average Proportional
Integral Derivative (WAPID) to update the Lagrange multiplier stably. Empirical
results show that the proposed method has fewer violations of constraints in
the early exploration process. The final test results also illustrate that our
method has better risk control.
