Sequential Nonparametric Testing with the Law of the Iterated Logarithm
  We propose a new algorithmic framework for sequential hypothesis testing with
i.i.d. data, which includes A/B testing, nonparametric two-sample testing, and
independence testing as special cases. It is novel in several ways: (a) it
takes linear time and constant space to compute on the fly, (b) it has the same
power guarantee as a non-sequential version of the test with the same
computational constraints up to a small factor, and (c) it accesses only as
many samples as are required - its stopping time adapts to the unknown
difficulty of the problem. All our test statistics are constructed to be
zero-mean martingales under the null hypothesis, and the rejection threshold is
governed by a uniform non-asymptotic law of the iterated logarithm (LIL). For
the case of nonparametric two-sample mean testing, we also provide a finite
sample power analysis, and the first non-asymptotic stopping time calculations
for this class of problems. We verify our predictions for type I and II errors
and stopping times using simulations.
