SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for
  Large-Scale Deep Learning Systems
  Deep Learning (DL) algorithms are the central focus of modern machine
learning systems. As data volumes keep growing, it has become customary to
train large neural networks with hundreds of millions of parameters to maintain
enough capacity to memorize these volumes and obtain state-of-the-art accuracy.
To get around the costly computations associated with large models and data,
the community is increasingly investing in specialized hardware for model
training. However, specialized hardware is expensive and hard to generalize to
a multitude of tasks. The progress on the algorithmic front has failed to
demonstrate a direct advantage over powerful hardware such as NVIDIA-V100 GPUs.
This paper provides an exception. We propose SLIDE (Sub-LInear Deep learning
Engine) that uniquely blends smart randomized algorithms, with multi-core
parallelism and workload optimization. Using just a CPU, SLIDE drastically
reduces the computations during both training and inference outperforming an
optimized implementation of Tensorflow (TF) on the best available GPU. Our
evaluations on industry-scale recommendation datasets, with large fully
connected architectures, show that training with SLIDE on a 44 core CPU is more
than 3.5 times (1 hour vs. 3.5 hours) faster than the same network trained
using TF on Tesla V100 at any given accuracy level. On the same CPU hardware,
SLIDE is over 10x faster than TF. We provide codes and scripts for
reproducibility.
