Chance Constrained Policy Optimization for Process Control and
  Optimization
  Chemical process optimization and control are affected by 1) plant-model
mismatch, 2) process disturbances, and 3) constraints for safe operation.
Reinforcement learning by policy optimization would be a natural way to solve
this due to its ability to address stochasticity, plant-model mismatch, and
directly account for the effect of future uncertainty and its feedback in a
proper closed-loop manner; all without the need of an inner optimization loop.
One of the main reasons why reinforcement learning has not been considered for
industrial processes (or almost any engineering application) is that it lacks a
framework to deal with safety critical constraints. Present algorithms for
policy optimization use difficult-to-tune penalty parameters, fail to reliably
satisfy state constraints or present guarantees only in expectation. We propose
a chance constrained policy optimization (CCPO) algorithm which guarantees the
satisfaction of joint chance constraints with a high probability - which is
crucial for safety critical tasks. This is achieved by the introduction of
constraint tightening (backoffs), which are computed simultaneously with the
feedback policy. Backoffs are adjusted with Bayesian optimization using the
empirical cumulative distribution function of the probabilistic constraints,
and are therefore self-tuned. This results in a general methodology that can be
imbued into present policy optimization algorithms to enable them to satisfy
joint chance constraints with high probability. We present case studies that
analyze the performance of the proposed approach.
