Optimizing for the Future in Non-Stationary MDPs
  Most reinforcement learning methods are based upon the key assumption that
the transition dynamics and reward functions are fixed, that is, the underlying
Markov decision process is stationary. However, in many real-world
applications, this assumption is violated, and using existing algorithms may
result in a performance lag. To proactively search for a good future policy, we
present a policy gradient algorithm that maximizes a forecast of future
performance. This forecast is obtained by fitting a curve to the
counter-factual estimates of policy performance over time, without explicitly
modeling the underlying non-stationarity. The resulting algorithm amounts to a
non-uniform reweighting of past data, and we observe that minimizing
performance over some of the data from past episodes can be beneficial when
searching for a policy that maximizes future performance. We show that our
algorithm, called Prognosticator, is more robust to non-stationarity than two
online adaptation techniques, on three simulated problems motivated by
real-world applications.
