A Dynamic Penalty Function Approach for Constraints-Handling in
  Reinforcement Learning
  Reinforcement learning (RL) is attracting attention as an effective way to
solve sequential optimization problems that involve high dimensional
state/action space and stochastic uncertainties. Many such problems involve
constraints expressed by inequality constraints. This study focuses on using RL
to solve constrained optimal control problems. Most RL application studies have
dealt with inequality constraints by adding soft penalty terms for violating
the constraints to the reward function. However, while training neural networks
to learn the value (or Q) function, one can run into computational issues
caused by the sharp change in the function value at the constraint boundary due
to the large penalty imposed. This difficulty during training can lead to
convergence problems and ultimately lead to poor closed-loop performance. To
address this issue, this study proposes a dynamic penalty (DP) approach where
the penalty factor is gradually and systematically increased during training as
the iteration episodes proceed. We first examine the ability of a neural
network to represent a value function when uniform, linear, or DP functions are
added to prevent constraint violation. The agent trained by a Deep Q Network
(DQN) algorithm with the DP function approach was compared with agents with
other constant penalty functions in a simple vehicle control problem. Results
show that the proposed approach can improve the neural network approximation
accuracy and provide faster convergence when close to a solution.
