On the Optimality of Batch Policy Optimization Algorithms
  Batch policy optimization considers leveraging existing data for policy
construction before interacting with an environment. Although interest in this
problem has grown significantly in recent years, its theoretical foundations
remain under-developed. To advance the understanding of this problem, we
provide three results that characterize the limits and possibilities of batch
policy optimization in the finite-armed stochastic bandit setting. First, we
introduce a class of confidence-adjusted index algorithms that unifies
optimistic and pessimistic principles in a common framework, which enables a
general analysis. For this family, we show that any confidence-adjusted index
algorithm is minimax optimal, whether it be optimistic, pessimistic or neutral.
Our analysis reveals that instance-dependent optimality, commonly used to
establish optimality of on-line stochastic bandit algorithms, cannot be
achieved by any algorithm in the batch setting. In particular, for any
algorithm that performs optimally in some environment, there exists another
environment where the same algorithm suffers arbitrarily larger regret.
Therefore, to establish a framework for distinguishing algorithms, we introduce
a new weighted-minimax criterion that considers the inherent difficulty of
optimal value prediction. We demonstrate how this criterion can be used to
justify commonly used pessimistic principles for batch policy optimization.
