Ensemble Kalman Inversion: A Derivative-Free Technique For Machine
  Learning Tasks
  The standard probabilistic perspective on machine learning gives rise to
empirical risk-minimization tasks that are frequently solved by stochastic
gradient descent (SGD) and variants thereof. We present a formulation of these
tasks as classical inverse or filtering problems and, furthermore, we propose
an efficient, gradient-free algorithm for finding a solution to these problems
using ensemble Kalman inversion (EKI). Applications of our approach include
offline and online supervised learning with deep neural networks, as well as
graph-based semi-supervised learning. The essence of the EKI procedure is an
ensemble based approximate gradient descent in which derivatives are replaced
by differences from within the ensemble. We suggest several modifications to
the basic method, derived from empirically successful heuristics developed in
the context of SGD. Numerical results demonstrate wide applicability and
robustness of the proposed algorithm.
