Assured Learning-enabled Autonomy: A Metacognitive Reinforcement
  Learning Framework
  Reinforcement learning (RL) agents with pre-specified reward functions cannot
provide guaranteed safety across variety of circumstances that an uncertain
system might encounter. To guarantee performance while assuring satisfaction of
safety constraints across variety of circumstances, an assured autonomous
control framework is presented in this paper by empowering RL algorithms with
metacognitive learning capabilities. More specifically, adapting the reward
function parameters of the RL agent is performed in a metacognitive
decision-making layer to assure the feasibility of RL agent. That is, to assure
that the learned policy by the RL agent satisfies safety constraints specified
by signal temporal logic while achieving as much performance as possible. The
metacognitive layer monitors any possible future safety violation under the
actions of the RL agent and employs a higher-layer Bayesian RL algorithm to
proactively adapt the reward function for the lower-layer RL agent. To minimize
the higher-layer Bayesian RL intervention, a fitness function is leveraged by
the metacognitive layer as a metric to evaluate success of the lower-layer RL
agent in satisfaction of safety and liveness specifications, and the
higher-layer Bayesian RL intervenes only if there is a risk of lower-layer RL
failure. Finally, a simulation example is provided to validate the
effectiveness of the proposed approach.
