Minimax Estimation of Conditional Moment Models
  We develop an approach for estimating models described via conditional moment
restrictions, with a prototypical application being non-parametric instrumental
variable regression. We introduce a min-max criterion function, under which the
estimation problem can be thought of as solving a zero-sum game between a
modeler who is optimizing over the hypothesis space of the target model and an
adversary who identifies violating moments over a test function space. We
analyze the statistical estimation rate of the resulting estimator for
arbitrary hypothesis spaces, with respect to an appropriate analogue of the
mean squared error metric, for ill-posed inverse problems. We show that when
the minimax criterion is regularized with a second moment penalty on the test
function and the test function space is sufficiently rich, then the estimation
rate scales with the critical radius of the hypothesis and test function
spaces, a quantity which typically gives tight fast rates. Our main result
follows from a novel localized Rademacher analysis of statistical learning
problems defined via minimax objectives. We provide applications of our main
results for several hypothesis spaces used in practice such as: reproducing
kernel Hilbert spaces, high dimensional sparse linear functions, spaces defined
via shape constraints, ensemble estimators such as random forests, and neural
networks. For each of these applications we provide computationally efficient
optimization methods for solving the corresponding minimax problem (e.g.
stochastic first-order heuristics for neural networks). In several
applications, we show how our modified mean squared error rate, combined with
conditions that bound the ill-posedness of the inverse problem, lead to mean
squared error rates. We conclude with an extensive experimental analysis of the
proposed methods.
