Learning MDPs from Features: Predict-Then-Optimize for Sequential
  Decision Problems by Reinforcement Learning
  In the predict-then-optimize framework, the objective is to train a
predictive model, mapping from environment features to parameters of an
optimization problem, which maximizes decision quality when the optimization is
subsequently solved. Recent work on decision-focused learning shows that
embedding the optimization problem in the training pipeline can improve
decision quality and help generalize better to unseen tasks compared to relying
on an intermediate loss function for evaluating prediction quality. We study
the predict-then-optimize framework in the context of sequential decision
problems (formulated as MDPs) that are solved via reinforcement learning. In
particular, we are given environment features and a set of trajectories from
training MDPs, which we use to train a predictive model that generalizes to
unseen test MDPs without trajectories. Two significant computational challenges
arise in applying decision-focused learning to MDPs: (i) large state and action
spaces make it infeasible for existing techniques to differentiate through MDP
problems, and (ii) the high-dimensional policy space, as parameterized by a
neural network, makes differentiating through a policy expensive. We resolve
the first challenge by sampling provably unbiased derivatives to approximate
and differentiate through optimality conditions, and the second challenge by
using a low-rank approximation to the high-dimensional sample-based
derivatives. We implement both Bellman--based and policy gradient--based
decision-focused learning on three different MDP problems with missing
parameters, and show that decision-focused learning performs better in
generalization to unseen tasks.
