The Utility of General Domain Transfer Learning for Medical Language
  Tasks
  The purpose of this study is to analyze the efficacy of transfer learning
techniques and transformer-based models as applied to medical natural language
processing (NLP) tasks, specifically radiological text classification. We used
1,977 labeled head CT reports, from a corpus of 96,303 total reports, to
evaluate the efficacy of pretraining using general domain corpora and a
combined general and medical domain corpus with a bidirectional representations
from transformers (BERT) model for the purpose of radiological text
classification. Model performance was benchmarked to a logistic regression
using bag-of-words vectorization and a long short-term memory (LSTM)
multi-label multi-class classification model, and compared to the published
literature in medical text classification. The BERT models using either set of
pretrained checkpoints outperformed the logistic regression model, achieving
sample-weighted average F1-scores of 0.87 and 0.87 for the general domain model
and the combined general and biomedical-domain model. General text transfer
learning may be a viable technique to generate state-of-the-art results within
medical NLP tasks on radiological corpora, outperforming other deep models such
as LSTMs. The efficacy of pretraining and transformer-based models could serve
to facilitate the creation of groundbreaking NLP models in the uniquely
challenging data environment of medical text.
