Tight Bayesian Ambiguity Sets for Robust MDPs
  Robustness is important for sequential decision making in a stochastic
dynamic environment with uncertain probabilistic parameters. We address the
problem of using robust MDPs (RMDPs) to compute policies with provable
worst-case guarantees in reinforcement learning. The quality and robustness of
an RMDP solution is determined by its ambiguity set. Existing methods construct
ambiguity sets that lead to impractically conservative solutions. In this
paper, we propose RSVF, which achieves less conservative solutions with the
same worst-case guarantees by 1) leveraging a Bayesian prior, 2) optimizing the
size and location of the ambiguity set, and, most importantly, 3) relaxing the
requirement that the set is a confidence interval. Our theoretical analysis
shows the safety of RSVF, and the empirical results demonstrate its practical
promise.
