Comparing Sequential Forecasters
  Consider two or more forecasters, each making a sequence of predictions for
different events over time. We ask a relatively basic question: how might we
compare these forecasters, either online or post-hoc, while avoiding
unverifiable assumptions on how the forecasts or outcomes were generated? This
work presents a novel and rigorous answer to this question. We design a
sequential inference procedure for estimating the time-varying difference in
forecast quality as measured by any scoring rule. The resulting confidence
intervals are nonasymptotically valid and can be continuously monitored to
yield statistically valid comparisons at arbitrary data-dependent stopping
times ("anytime-valid"); this is enabled by adapting variance-adaptive
supermartingales, confidence sequences, and e-processes to our setting.
Motivated by Shafer and Vovk's game-theoretic probability, our coverage
guarantees are also distribution-free, in the sense that they make no
distributional assumptions on the forecasts or outcomes. In contrast to a
recent work by Henzi and Ziegel, our tools can sequentially test a weak null
hypothesis about whether one forecaster outperforms another on average over
time. We demonstrate their effectiveness by comparing probability forecasts on
Major League Baseball (MLB) games and statistical postprocessing methods for
ensemble weather forecasts.
