GroupBERT: Enhanced Transformer Architecture with Efficient Grouped
  Structures
  Attention based language models have become a critical component in
state-of-the-art natural language processing systems. However, these models
have significant computational requirements, due to long training times, dense
operations and large parameter count. In this work we demonstrate a set of
modifications to the structure of a Transformer layer, producing a more
efficient architecture. First, we add a convolutional module to complement the
self-attention module, decoupling the learning of local and global
interactions. Secondly, we rely on grouped transformations to reduce the
computational cost of dense feed-forward layers and convolutions, while
preserving the expressivity of the model. We apply the resulting architecture
to language representation learning and demonstrate its superior performance
compared to BERT models of different scales. We further highlight its improved
efficiency, both in terms of floating-point operations (FLOPs) and
time-to-train.
