A Free Lunch with Influence Functions? Improving Neural Network
  Estimates with Concepts from Semiparametric Statistics
  Parameter estimation in empirical fields is usually undertaken using
parametric models, and such models readily facilitate statistical inference.
Unfortunately, they are unlikely to be sufficiently flexible to be able to
adequately model real-world phenomena, and may yield biased estimates.
Conversely, non-parametric approaches are flexible but do not readily
facilitate statistical inference and may still exhibit residual bias. We
explore the potential for Influence Functions (IFs) to (a) improve initial
estimators without needing more data (b) increase model robustness and (c)
facilitate statistical inference. We begin with a broad introduction to IFs,
and propose a neural network method 'MultiNet', which seeks the diversity of an
ensemble using a single architecture. We also introduce variants on the IF
update step which we call 'MultiStep', and provide a comprehensive evaluation
of different approaches. The improvements are found to be dataset dependent,
indicating an interaction between the methods used and nature of the data
generating process. Our experiments highlight the need for practitioners to
check the consistency of their findings, potentially by undertaking multiple
analyses with different combinations of estimators. We also show that it is
possible to improve existing neural networks for `free', without needing more
data, and without needing to retrain them.
