Settling the Sample Complexity of Model-Based Offline Reinforcement
  Learning
  This paper is concerned with offline reinforcement learning (RL), which
learns using pre-collected data without further exploration. Effective offline
RL would be able to accommodate distribution shift and limited data coverage.
However, prior algorithms or analyses either suffer from suboptimal sample
complexities or incur high burn-in cost to reach sample optimality, thus posing
an impediment to efficient offline RL in sample-starved applications.
  We demonstrate that the model-based (or "plug-in") approach achieves
minimax-optimal sample complexity without burn-in cost for tabular Markov
decision processes (MDPs). Concretely, consider a finite-horizon (resp.
$\gamma$-discounted infinite-horizon) MDP with $S$ states and horizon $H$
(resp. effective horizon $\frac{1}{1-\gamma}$), and suppose the distribution
shift of data is reflected by some single-policy clipped concentrability
coefficient $C^{\star}_{\text{clipped}}$. We prove that model-based offline RL
yields $\varepsilon$-accuracy with a sample complexity of \[ \begin{cases}
\frac{H^{4}SC_{\text{clipped}}^{\star}}{\varepsilon^{2}} &
(\text{finite-horizon MDPs})
\frac{SC_{\text{clipped}}^{\star}}{(1-\gamma)^{3}\varepsilon^{2}} &
(\text{infinite-horizon MDPs}) \end{cases} \] up to log factor, which is
minimax optimal for the entire $\varepsilon$-range. Our algorithms are
"pessimistic" variants of value iteration with Bernstein-style penalties, and
do not require sophisticated variance reduction.
