Ensemble Neural Networks (ENN): A gradient-free stochastic method
  In this study, an efficient stochastic gradient-free method, the ensemble
neural networks (ENN), is developed. In the ENN, the optimization process
relies on covariance matrices rather than derivatives. The covariance matrices
are calculated by the ensemble randomized maximum likelihood algorithm (EnRML),
which is an inverse modeling method. The ENN is able to simultaneously provide
estimations and perform uncertainty quantification since it is built under the
Bayesian framework. The ENN is also robust to small training data size because
the ensemble of stochastic realizations essentially enlarges the training
dataset. This constitutes a desirable characteristic, especially for real-world
engineering applications. In addition, the ENN does not require the calculation
of gradients, which enables the use of complicated neuron models and loss
functions in neural networks. We experimentally demonstrate benefits of the
proposed model, in particular showing that the ENN performs much better than
the traditional Bayesian neural networks (BNN). The EnRML in ENN is a
substitution of gradient-based optimization algorithms, which means that it can
be directly combined with the feed-forward process in other existing (deep)
neural networks, such as convolutional neural networks (CNN) and recurrent
neural networks (RNN), broadening future applications of the ENN.
