Effective Reinforcement Learning through Evolutionary Surrogate-Assisted
  Prescription
  There is now significant historical data available on decision making in
organizations, consisting of the decision problem, what decisions were made,
and how desirable the outcomes were. Using this data, it is possible to learn a
surrogate model, and with that model, evolve a decision strategy that optimizes
the outcomes. This paper introduces a general such approach, called
Evolutionary Surrogate-Assisted Prescription, or ESP. The surrogate is, for
example, a random forest or a neural network trained with gradient descent, and
the strategy is a neural network that is evolved to maximize the predictions of
the surrogate model. ESP is further extended in this paper to sequential
decision-making tasks, which makes it possible to evaluate the framework in
reinforcement learning (RL) benchmarks. Because the majority of evaluations are
done on the surrogate, ESP is more sample efficient, has lower variance, and
lower regret than standard RL approaches. Surprisingly, its solutions are also
better because both the surrogate and the strategy network regularize the
decision-making behavior. ESP thus forms a promising foundation to decision
optimization in real-world problems.
