LV-BERT: Exploiting Layer Variety for BERT
  Modern pre-trained language models are mostly built upon backbones stacking
self-attention and feed-forward layers in an interleaved order. In this paper,
beyond this stereotyped layer pattern, we aim to improve pre-trained models by
exploiting layer variety from two aspects: the layer type set and the layer
order. Specifically, besides the original self-attention and feed-forward
layers, we introduce convolution into the layer type set, which is
experimentally found beneficial to pre-trained models. Furthermore, beyond the
original interleaved order, we explore more layer orders to discover more
powerful architectures. However, the introduced layer variety leads to a large
architecture space of more than billions of candidates, while training a single
candidate model from scratch already requires huge computation cost, making it
not affordable to search such a space by directly training large amounts of
candidate models. To solve this problem, we first pre-train a supernet from
which the weights of all candidate models can be inherited, and then adopt an
evolutionary algorithm guided by pre-training accuracy to find the optimal
architecture. Extensive experiments show that LV-BERT model obtained by our
method outperforms BERT and its variants on various downstream tasks. For
example, LV-BERT-small achieves 79.8 on the GLUE testing set, 1.8 higher than
the strong baseline ELECTRA-small.
