Sample Complexity of Robust Reinforcement Learning with a Generative
  Model
  The Robust Markov Decision Process (RMDP) framework focuses on designing
control policies that are robust against the parameter uncertainties due to the
mismatches between the simulator model and real-world settings. An RMDP problem
is typically formulated as a max-min problem, where the objective is to find
the policy that maximizes the value function for the worst possible model that
lies in an uncertainty set around a nominal model. The standard robust dynamic
programming approach requires the knowledge of the nominal model for computing
the optimal robust policy. In this work, we propose a model-based reinforcement
learning (RL) algorithm for learning an $\epsilon$-optimal robust policy when
the nominal model is unknown. We consider three different forms of uncertainty
sets, characterized by the total variation distance, chi-square divergence, and
KL divergence. For each of these uncertainty sets, we give a precise
characterization of the sample complexity of our proposed algorithm. In
addition to the sample complexity results, we also present a formal analytical
argument on the benefit of using robust policies. Finally, we demonstrate the
performance of our algorithm on two benchmark problems.
