Pessimism About Unknown Unknowns Inspires Conservatism
  If we could define the set of all bad outcomes, we could hard-code an agent
which avoids them; however, in sufficiently complex environments, this is
infeasible. We do not know of any general-purpose approaches in the literature
to avoiding novel failure modes. Motivated by this, we define an idealized
Bayesian reinforcement learner which follows a policy that maximizes the
worst-case expected reward over a set of world-models. We call this agent
pessimistic, since it optimizes assuming the worst case. A scalar parameter
tunes the agent's pessimism by changing the size of the set of world-models
taken into account. Our first main contribution is: given an assumption about
the agent's model class, a sufficiently pessimistic agent does not cause
"unprecedented events" with probability $1-\delta$, whether or not designers
know how to precisely specify those precedents they are concerned with. Since
pessimism discourages exploration, at each timestep, the agent may defer to a
mentor, who may be a human or some known-safe policy we would like to improve.
Our other main contribution is that the agent's policy's value approaches at
least that of the mentor, while the probability of deferring to the mentor goes
to 0. In high-stakes environments, we might like advanced artificial agents to
pursue goals cautiously, which is a non-trivial problem even if the agent were
allowed arbitrary computing power; we present a formal solution.
