SqueezeBERT: What can computer vision teach NLP about efficient neural
  networks?
  Humans read and write hundreds of billions of messages every day. Further,
due to the availability of large datasets, large computing systems, and better
neural network models, natural language processing (NLP) technology has made
significant strides in understanding, proofreading, and organizing these
messages. Thus, there is a significant opportunity to deploy NLP in myriad
applications to help web users, social networks, and businesses. In particular,
we consider smartphones and other mobile devices as crucial platforms for
deploying NLP models at scale. However, today's highly-accurate NLP neural
network models such as BERT and RoBERTa are extremely computationally
expensive, with BERT-base taking 1.7 seconds to classify a text snippet on a
Pixel 3 smartphone. In this work, we observe that methods such as grouped
convolutions have yielded significant speedups for computer vision networks,
but many of these techniques have not been adopted by NLP neural network
designers. We demonstrate how to replace several operations in self-attention
layers with grouped convolutions, and we use this technique in a novel network
architecture called SqueezeBERT, which runs 4.3x faster than BERT-base on the
Pixel 3 while achieving competitive accuracy on the GLUE test set. The
SqueezeBERT code will be released.
