Breaking hypothesis testing for failure rates
  We describe the utility of point processes and failure rates and the most
common point process for modeling failure rates, the Poisson point process.
Next, we describe the uniformly most powerful test for comparing the rates of
two Poisson point processes for a one-sided test (henceforth referred to as the
"rate test"). A common argument against using this test is that real world data
rarely follows the Poisson point process. We thus investigate what happens when
the distributional assumptions of tests like these are violated and the test
still applied. We find a non-pathological example (using the rate test on a
Compound Poisson distribution with Binomial compounding) where violating the
distributional assumptions of the rate test make it perform better (lower error
rates). We also find that if we replace the distribution of the test statistic
under the null hypothesis with any other arbitrary distribution, the
performance of the test (described in terms of the false negative rate to false
positive rate trade-off) remains exactly the same. Next, we compare the
performance of the rate test to a version of the Wald test customized to the
Negative Binomial point process and find it to perform very similarly while
being much more general and versatile. Finally, we discuss the applications to
Microsoft Azure. The code for all experiments performed is open source and
linked in the introduction.
