Constrained Reinforcement Learning for Dynamic Optimization under
  Uncertainty
  Dynamic real-time optimization (DRTO) is a challenging task due to the fact
that optimal operating conditions must be computed in real time. The main
bottleneck in the industrial application of DRTO is the presence of
uncertainty. Many stochastic systems present the following obstacles: 1)
plant-model mismatch, 2) process disturbances, 3) risks in violation of process
constraints. To accommodate these difficulties, we present a constrained
reinforcement learning (RL) based approach. RL naturally handles the process
uncertainty by computing an optimal feedback policy. However, no state
constraints can be introduced intuitively. To address this problem, we present
a chance-constrained RL methodology. We use chance constraints to guarantee the
probabilistic satisfaction of process constraints, which is accomplished by
introducing backoffs, such that the optimal policy and backoffs are computed
simultaneously. Backoffs are adjusted using the empirical cumulative
distribution function to guarantee the satisfaction of a joint chance
constraint. The advantage and performance of this strategy are illustrated
through a stochastic dynamic bioprocess optimization problem, to produce
sustainable high-value bioproducts.
