Sequential Training of Neural Networks with Gradient Boosting
  This paper presents a novel technique based on gradient boosting to train a
shallow neural network (NN). Gradient boosting is an additive expansion
algorithm in which a series of models are trained sequentially to approximate a
given function. A neural network can also be seen as an additive model where
the scalar product of the responses of the last hidden layer and its weights
provide the final output of the network. Instead of training the network as a
whole, the proposed algorithm trains the network sequentially in $T$ steps.
First, the bias term of the network is initialized with a constant
approximation that minimizes the average loss of the data. Then, at each step,
a portion of the network, composed of $J$ neurons, is trained to approximate
the pseudo-residuals on the training data computed from the previous
iterations. Finally, the $T$ partial models and bias are integrated as a single
NN with $T \times J$ neurons in the hidden layer. Extensive experiments in
classification and regression tasks are carried out showing a competitive
generalization performance with respect to neural networks trained with
different standard solvers, such as Adam, L-BFGS and SGD. Furthermore, we show
that the proposed method design permits to switch off a number of hidden units
during test (the units that were last trained) without a significant reduction
of its generalization ability. This permits the adaptation of the model to
different classification speed requirements on the fly.
