Off-Policy Evaluation in Partially Observed Markov Decision Processes
  We consider off-policy evaluation of dynamic treatment rules under the
assumption that the underlying system can be modeled as a partially observed
Markov decision process (POMDP). We propose an estimator, partial history
importance weighting, and show that it can consistently estimate the stationary
mean rewards of a target policy given long enough draws from the behavior
policy. Furthermore, we establish an upper bound on its error that decays
polynomially in the number of observations (i.e., the number of trajectories
times their length), with an exponent that depends on the overlap of the target
and behavior policies, and on the mixing time of the underlying system. We also
establish a polynomial minimax lower bound for off-policy evaluation under the
POMDP assumption, and show that its exponent has the same qualitative
dependence on overlap and mixing time as obtained in our upper bound. Together,
our upper and lower bounds imply that off-policy evaluation in POMDPs is
strictly harder than off-policy evaluation in (fully observed) Markov decision
processes, but strictly easier than model-free off-policy evaluation.
