Steady State Analysis of Episodic Reinforcement Learning
  This paper proves that the episodic learning environment of every
finite-horizon decision task has a unique steady state under any behavior
policy, and that the marginal distribution of the agent's input indeed
converges to the steady-state distribution in essentially all episodic learning
processes. This observation supports an interestingly reversed mindset against
conventional wisdom: While the existence of unique steady states was often
presumed in continual learning but considered less relevant in episodic
learning, it turns out their existence is guaranteed for the latter. Based on
this insight, the paper unifies episodic and continual RL around several
important concepts that have been separately treated in these two RL
formalisms. Practically, the existence of unique and approachable steady state
enables a general way to collect data in episodic RL tasks, which the paper
applies to policy gradient algorithms as a demonstration, based on a new
steady-state policy gradient theorem. Finally, the paper also proposes and
experimentally validates a perturbation method that facilitates rapid
steady-state convergence in real-world RL tasks.
