Robust Entropy-regularized Markov Decision Processes
  Stochastic and soft optimal policies resulting from entropy-regularized
Markov decision processes (ER-MDP) are desirable for exploration and imitation
learning applications. Motivated by the fact that such policies are sensitive
with respect to the state transition probabilities, and the estimation of these
probabilities may be inaccurate, we study a robust version of the ER-MDP model,
where the stochastic optimal policies are required to be robust with respect to
the ambiguity in the underlying transition probabilities. Our work is at the
crossroads of two important schemes in reinforcement learning (RL), namely,
robust MDP and entropy regularized MDP. We show that essential properties that
hold for the non-robust ER-MDP and robust unregularized MDP models also hold in
our settings, making the robust ER-MDP problem tractable. We show how our
framework and results can be integrated into different algorithmic schemes
including value or (modified) policy iteration, which would lead to new robust
RL and inverse RL algorithms to handle uncertainties. Analyses on computational
complexity and error propagation under conventional uncertainty settings are
also provided.
