Semi-Implicit Back Propagation
  Neural network has attracted great attention for a long time and many
researchers are devoted to improve the effectiveness of neural network training
algorithms. Though stochastic gradient descent (SGD) and other explicit
gradient-based methods are widely adopted, there are still many challenges such
as gradient vanishing and small step sizes, which leads to slow convergence and
instability of SGD algorithms. Motivated by error back propagation (BP) and
proximal methods, we propose a semi-implicit back propagation method for neural
network training. Similar to BP, the difference on the neurons are propagated
in a backward fashion and the parameters are updated with proximal mapping. The
implicit update for both hidden neurons and parameters allows to choose large
step size in the training algorithm. Finally, we also show that any fixed point
of convergent sequences produced by this algorithm is a stationary point of the
objective loss function. The experiments on both MNIST and CIFAR-10 demonstrate
that the proposed semi-implicit BP algorithm leads to better performance in
terms of both loss decreasing and training/validation accuracy, compared to SGD
and a similar algorithm ProxBP.
