An Algorithmic Theory of Metacognition in Minds and Machines
  Humans sometimes choose actions that they themselves can identify as
sub-optimal, or wrong, even in the absence of additional information. How is
this possible? We present an algorithmic theory of metacognition based on a
well-understood trade-off in reinforcement learning (RL) between value-based RL
and policy-based RL. To the cognitive (neuro)science community, our theory
answers the outstanding question of why information can be used for error
detection but not for action selection. To the machine learning community, our
proposed theory creates a novel interaction between the Actor and Critic in
Actor-Critic agents and notes a novel connection between RL and Bayesian
Optimization. We call our proposed agent the Metacognitive Actor Critic (MAC).
We conclude with showing how to create metacognition in machines by
implementing a deep MAC and showing that it can detect (some of) its own
suboptimal actions without external information or delay.
