State Relevance for Off-Policy Evaluation
  Importance sampling-based estimators for off-policy evaluation (OPE) are
valued for their simplicity, unbiasedness, and reliance on relatively few
assumptions. However, the variance of these estimators is often high,
especially when trajectories are of different lengths. In this work, we
introduce Omitting-States-Irrelevant-to-Return Importance Sampling (OSIRIS), an
estimator which reduces variance by strategically omitting likelihood ratios
associated with certain states. We formalize the conditions under which OSIRIS
is unbiased and has lower variance than ordinary importance sampling, and we
demonstrate these properties empirically.
