Learning to Stop with Surprisingly Few Samples
  We consider a discounted infinite horizon optimal stopping problem. If the
underlying distribution is known a priori, the solution of this problem is
obtained via dynamic programming (DP) and is given by a well known threshold
rule. When information on this distribution is lacking, a natural (though
naive) approach is "explore-then-exploit," whereby the unknown distribution or
its parameters are estimated over an initial exploration phase, and this
estimate is then used in the DP to determine actions over the residual
exploitation phase. We show: (i) with proper tuning, this approach leads to
performance comparable to the full information DP solution; and (ii) despite
common wisdom on the sensitivity of such "plug in" approaches in DP due to
propagation of estimation errors, a surprisingly "short" (logarithmic in the
horizon) exploration horizon suffices to obtain said performance. In cases
where the underlying distribution is heavy-tailed, these observations are even
more pronounced: a ${\it single \, sample}$ exploration phase suffices.
