Empirical study of PROXTONE and PROXTONE$^+$ for Fast Learning of Large
  Scale Sparse Models
  PROXTONE is a novel and fast method for optimization of large scale
non-smooth convex problem \cite{shi2015large}. In this work, we try to use
PROXTONE method in solving large scale \emph{non-smooth non-convex} problems,
for example training of sparse deep neural network (sparse DNN) or sparse
convolutional neural network (sparse CNN) for embedded or mobile device.
PROXTONE converges much faster than first order methods, while first order
method is easy in deriving and controlling the sparseness of the solutions.
Thus in some applications, in order to train sparse models fast, we propose to
combine the merits of both methods, that is we use PROXTONE in the first
several epochs to reach the neighborhood of an optimal solution, and then use
the first order method to explore the possibility of sparsity in the following
training. We call such method PROXTONE plus (PROXTONE$^+$). Both PROXTONE and
PROXTONE$^+$ are tested in our experiments, and which demonstrate both methods
improved convergence speed twice as fast at least on diverse sparse model
learning problems, and at the same time reduce the size to 0.5\% for DNN
models. The source of all the algorithms is available upon request.
