Exploiting Spline Models for the Training of Fully Connected Layers in
  Neural Network
  The fully connected (FC) layer, one of the most fundamental modules in
artificial neural networks (ANN), is often considered difficult and inefficient
to train due to issues including the risk of overfitting caused by its large
amount of parameters. Based on previous work studying ANN from linear spline
perspectives, we propose a spline-based approach that eases the difficulty of
training FC layers. Given some dataset, we first obtain a continuous piece-wise
linear (CPWL) fit through spline methods such as multivariate adaptive
regression spline (MARS). Next, we construct an ANN model from the linear
spline model and continue to train the ANN model on the dataset using gradient
descent optimization algorithms. Our experimental results and theoretical
analysis show that our approach reduces the computational cost, accelerates the
convergence of FC layers, and significantly increases the interpretability of
the resulting model (FC layers) compared with standard ANN training with random
parameter initialization followed by gradient descent optimizations.
