Escaping from Zero Gradient: Revisiting Action-Constrained Reinforcement
  Learning via Frank-Wolfe Policy Optimization
  Action-constrained reinforcement learning (RL) is a widely-used approach in
various real-world applications, such as scheduling in networked systems with
resource constraints and control of a robot with kinematic constraints. While
the existing projection-based approaches ensure zero constraint violation, they
could suffer from the zero-gradient problem due to the tight coupling of the
policy gradient and the projection, which results in sample-inefficient
training and slow convergence. To tackle this issue, we propose a learning
algorithm that decouples the action constraints from the policy parameter
update by leveraging state-wise Frank-Wolfe and a regression-based policy
update scheme. Moreover, we show that the proposed algorithm enjoys convergence
and policy improvement properties in the tabular case as well as generalizes
the popular DDPG algorithm for action-constrained RL in the general case.
Through experiments, we demonstrate that the proposed algorithm significantly
outperforms the benchmark methods on a variety of control tasks.
