Statistical Inference After Adaptive Sampling in Non-Markovian
  Environments
  There is a great desire to use adaptive sampling methods, such as
reinforcement learning (RL) and bandit algorithms, for the real-time
personalization of interventions in digital applications like mobile health and
education. A major obstacle preventing more widespread use of such algorithms
in practice is the lack of assurance that the resulting adaptively collected
data can be used to reliably answer inferential questions, including questions
about time-varying causal effects. Current methods for statistical inference on
such data are insufficient because they (a) make strong assumptions regarding
the environment dynamics, e.g., assume a contextual bandit or Markovian
environment, or (b) require data to be collected with one adaptive sampling
algorithm per user, which excludes data collected by algorithms that learn to
select actions by pooling the data of multiple users. In this work, we make
initial progress by introducing the adaptive sandwich estimator to quantify
uncertainty; this estimator (a) is valid even when user rewards and contexts
are non-stationary and highly dependent over time, and (b) accommodates
settings in which an online adaptive sampling algorithm learns using the data
of all users. Furthermore, our inference method is robust to misspecification
of the reward models used by the adaptive sampling algorithm. This work is
motivated by our work designing experiments in which RL algorithms are used to
select actions, yet reliable statistical inference is essential for conducting
primary analyses after the trial is over.
