BERTology Meets Biology: Interpreting Attention in Protein Language
  Models
  Transformer architectures have proven to learn useful representations for
protein classification and generation tasks. However, these representations
present challenges in interpretability. In this work, we demonstrate a set of
methods for analyzing protein Transformer models through the lens of attention.
We show that attention: (1) captures the folding structure of proteins,
connecting amino acids that are far apart in the underlying sequence, but
spatially close in the three-dimensional structure, (2) targets binding sites,
a key functional component of proteins, and (3) focuses on progressively more
complex biophysical properties with increasing layer depth. We find this
behavior to be consistent across three Transformer architectures (BERT, ALBERT,
XLNet) and two distinct protein datasets. We also present a three-dimensional
visualization of the interaction between attention and protein structure. Code
for visualization and analysis is available at
https://github.com/salesforce/provis.
