DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
  Large-scale pre-trained language models such as BERT have brought significant
improvements to NLP applications. However, they are also notorious for being
slow in inference, which makes them difficult to deploy in real-time
applications. We propose a simple but effective method, DeeBERT, to accelerate
BERT inference. Our approach allows samples to exit earlier without passing
through the entire model. Experiments show that DeeBERT is able to save up to
~40% inference time with minimal degradation in model quality. Further analyses
show different behaviors in the BERT transformer layers and also reveal their
redundancy. Our work provides new ideas to efficiently apply deep
transformer-based models to downstream tasks. Code is available at
https://github.com/castorini/DeeBERT.
