Transformers and the representation of biomedical background knowledge
  BioBERT and BioMegatron are Transformers models adapted for the biomedical
domain based on publicly available biomedical corpora. As such, they have the
potential to encode large-scale biological knowledge. We investigate the
encoding and representation of biological knowledge in these models, and its
potential utility to support inference in cancer precision medicine - namely,
the interpretation of the clinical significance of genomic alterations. We
compare the performance of different transformer baselines; we use probing to
determine the consistency of encodings for distinct entities; and we use
clustering methods to compare and contrast the internal properties of the
embeddings for genes, variants, drugs and diseases. We show that these models
do indeed encode biological knowledge, although some of this is lost in
fine-tuning for specific tasks. Finally, we analyse how the models behave with
regard to biases and imbalances in the dataset.
