Meta Reinforcement Learning for Adaptive Control: An Offline Approach
  Meta-learning is a branch of machine learning which trains neural network
models to synthesize a wide variety of data in order to rapidly solve new
problems. In process control, many systems have similar and well-understood
dynamics, which suggests it is feasible to create a generalizable controller
through meta-learning. In this work, we formulate a meta reinforcement learning
(meta-RL) control strategy that takes advantage of known, offline information
for training, such as the system gain or time constant, yet efficiently
controls novel systems in a completely model-free fashion. Our meta-RL agent
has a recurrent structure that accumulates "context" for its current dynamics
through a hidden state variable. This end-to-end architecture enables the agent
to automatically adapt to changes in the process dynamics. Moreover, the same
agent can be deployed on systems with previously unseen nonlinearities and
timescales. In tests reported here, the meta-RL agent was trained entirely
offline, yet produced excellent results in novel settings. A key design element
is the ability to leverage model-based information offline during training,
while maintaining a model-free policy structure for interacting with novel
environments. To illustrate the approach, we take the actions proposed by the
meta-RL agent to be changes to gains of a proportional-integral controller,
resulting in a generalized, adaptive, closed-loop tuning strategy.
Meta-learning is a promising approach for constructing sample-efficient
intelligent controllers.
