A Relation Analysis of Markov Decision Process Frameworks
  We study the relation between different Markov Decision Process (MDP)
frameworks in the machine learning and econometrics literatures, including the
standard MDP, the entropy and general regularized MDP, and stochastic MDP,
where the latter is based on the assumption that the reward function is
stochastic and follows a given distribution. We show that the
entropy-regularized MDP is equivalent to a stochastic MDP model, and is
strictly subsumed by the general regularized MDP. Moreover, we propose a
distributional stochastic MDP framework by assuming that the distribution of
the reward function is ambiguous. We further show that the distributional
stochastic MDP is equivalent to the regularized MDP, in the sense that they
always yield the same optimal policies. We also provide a connection between
stochastic/regularized MDP and constrained MDP. Our work gives a unified view
on several important MDP frameworks, which would lead new ways to interpret the
(entropy/general) regularized MDP frameworks through the lens of stochastic
rewards and vice-versa. Given the recent popularity of regularized MDP in
(deep) reinforcement learning, our work brings new understandings of how such
algorithmic schemes work and suggest ideas to develop new ones.
