Improving Neural Network Training in Low Dimensional Random Bases
  Stochastic Gradient Descent (SGD) has proven to be remarkably effective in
optimizing deep neural networks that employ ever-larger numbers of parameters.
Yet, improving the efficiency of large-scale optimization remains a vital and
highly active area of research. Recent work has shown that deep neural networks
can be optimized in randomly-projected subspaces of much smaller dimensionality
than their native parameter space. While such training is promising for more
efficient and scalable optimization schemes, its practical application is
limited by inferior optimization performance. Here, we improve on recent random
subspace approaches as follows: Firstly, we show that keeping the random
projection fixed throughout training is detrimental to optimization. We propose
re-drawing the random subspace at each step, which yields significantly better
performance. We realize further improvements by applying independent
projections to different parts of the network, making the approximation more
efficient as network dimensionality grows. To implement these experiments, we
leverage hardware-accelerated pseudo-random number generation to construct the
random projections on-demand at every optimization step, allowing us to
distribute the computation of independent random directions across multiple
workers with shared random seeds. This yields significant reductions in memory
and is up to 10 times faster for the workloads in question.
