Policy Gradients Incorporating the Future
  Reasoning about the future -- understanding how decisions in the present time
affect outcomes in the future -- is one of the central challenges for
reinforcement learning (RL), especially in highly-stochastic or partially
observable environments. While predicting the future directly is hard, in this
work we introduce a method that allows an agent to "look into the future"
without explicitly predicting it. Namely, we propose to allow an agent, during
its training on past experience, to observe what \emph{actually} happened in
the future at that time, while enforcing an information bottleneck to avoid the
agent overly relying on this privileged information. This gives our agent the
opportunity to utilize rich and useful information about the future trajectory
dynamics in addition to the present. Our method, Policy Gradients Incorporating
the Future (PGIF), is easy to implement and versatile, being applicable to
virtually any policy gradient algorithm. We apply our proposed method to a
number of off-the-shelf RL algorithms and show that PGIF is able to achieve
higher reward faster in a variety of online and offline RL domains, as well as
sparse-reward and partially observable environments.
