Training Recommender Systems at Scale: Communication-Efficient Model and
  Data Parallelism
  In this paper, we consider hybrid parallelism -- a paradigm that employs both
Data Parallelism (DP) and Model Parallelism (MP) -- to scale distributed
training of large recommendation models. We propose a compression framework
called Dynamic Communication Thresholding (DCT) for communication-efficient
hybrid training. DCT filters the entities to be communicated across the network
through a simple hard-thresholding function, allowing only the most relevant
information to pass through. For communication efficient DP, DCT compresses the
parameter gradients sent to the parameter server during model synchronization.
The threshold is updated only once every few thousand iterations to reduce the
computational overhead of compression. For communication efficient MP, DCT
incorporates a novel technique to compress the activations and gradients sent
across the network during the forward and backward propagation, respectively.
This is done by identifying and updating only the most relevant neurons of the
neural network for each training sample in the data. We evaluate DCT on
publicly available natural language processing and recommender models and
datasets, as well as recommendation systems used in production at Facebook. DCT
reduces communication by at least $100\times$ and $20\times$ during DP and MP,
respectively. The algorithm has been deployed in production, and it improves
end-to-end training time for a state-of-the-art industrial recommender model by
37\%, without any loss in performance.
