Accelerating SLIDE Deep Learning on Modern CPUs: Vectorization,
  Quantizations, Memory Optimizations, and More
  Deep learning implementations on CPUs (Central Processing Units) are gaining
more traction. Enhanced AI capabilities on commodity x86 architectures are
commercially appealing due to the reuse of existing hardware and virtualization
ease. A notable work in this direction is the SLIDE system. SLIDE is a C++
implementation of a sparse hash table based back-propagation, which was shown
to be significantly faster than GPUs in training hundreds of million parameter
neural models. In this paper, we argue that SLIDE's current implementation is
sub-optimal and does not exploit several opportunities available in modern
CPUs. In particular, we show how SLIDE's computations allow for a unique
possibility of vectorization via AVX (Advanced Vector Extensions)-512.
Furthermore, we highlight opportunities for different kinds of memory
optimization and quantizations. Combining all of them, we obtain up to 7x
speedup in the computations on the same hardware. Our experiments are focused
on large (hundreds of millions of parameters) recommendation and NLP models.
Our work highlights several novel perspectives and opportunities for
implementing randomized algorithms for deep learning on modern CPUs. We provide
the code and benchmark scripts at https://github.com/RUSH-LAB/SLIDE
