Likelihood-Free Frequentist Inference: Confidence Sets with Correct
  Conditional Coverage
  Many areas of science make extensive use of computer simulators that
implicitly encode likelihood functions of complex systems. Classical
statistical methods are poorly suited for these so-called likelihood-free
inference (LFI) settings, particularly outside asymptotic and low-dimensional
regimes. Although new machine learning methods, such as normalizing flows, have
revolutionized the sample efficiency and capacity of LFI methods, it remains an
open question whether they produce confidence sets with correct conditional
coverage for small sample sizes. This paper unifies classical statistics with
modern machine learning to present (i) a practical procedure for the Neyman
construction of confidence sets with finite-sample guarantees of nominal
coverage, and (ii) diagnostics that estimate conditional coverage over the
entire parameter space. We refer to our framework as likelihood-free
frequentist inference (LF2I). Any method that defines a test statistic, like
the likelihood ratio, can leverage the LF2I machinery to create valid
confidence sets and diagnostics without costly Monte Carlo samples at fixed
parameter settings. We study the power of two test statistics (ACORE and BFF),
which, respectively, maximize versus integrate an odds function over the
parameter space. Our paper discusses the benefits and challenges of LF2I, with
a breakdown of the sources of errors in LF2I confidence sets.
