Bayesian Robust Optimization for Imitation Learning
  One of the main challenges in imitation learning is determining what action
an agent should take when outside the state distribution of the demonstrations.
Inverse reinforcement learning (IRL) can enable generalization to new states by
learning a parameterized reward function, but these approaches still face
uncertainty over the true reward function and corresponding optimal policy.
Existing safe imitation learning approaches based on IRL deal with this
uncertainty using a maxmin framework that optimizes a policy under the
assumption of an adversarial reward function, whereas risk-neutral IRL
approaches either optimize a policy for the mean or MAP reward function. While
completely ignoring risk can lead to overly aggressive and unsafe policies,
optimizing in a fully adversarial sense is also problematic as it can lead to
overly conservative policies that perform poorly in practice. To provide a
bridge between these two extremes, we propose Bayesian Robust Optimization for
Imitation Learning (BROIL). BROIL leverages Bayesian reward function inference
and a user specific risk tolerance to efficiently optimize a robust policy that
balances expected return and conditional value at risk. Our empirical results
show that BROIL provides a natural way to interpolate between return-maximizing
and risk-minimizing behaviors and outperforms existing risk-sensitive and
risk-neutral inverse reinforcement learning algorithms. Code is available at
https://github.com/dsbrown1331/broil.
