BiT: Robustly Binarized Multi-distilled Transformer
  Modern pre-trained transformers have rapidly advanced the state-of-the-art in
machine learning, but have also grown in parameters and computational
complexity, making them increasingly difficult to deploy in
resource-constrained environments. Binarization of the weights and activations
of the network can significantly alleviate these issues, however is technically
challenging from an optimization perspective. In this work, we identify a
series of improvements which enables binary transformers at a much higher
accuracy than what was possible previously. These include a two-set
binarization scheme, a novel elastic binary activation function with learned
parameters, and a method to quantize a network to its limit by successively
distilling higher precision models into lower precision students. These
approaches allow for the first time, fully binarized transformer models that
are at a practical level of accuracy, approaching a full-precision BERT
baseline on the GLUE language understanding benchmark within as little as 5.9%.
