Pre-training A Neural Language Model Improves The Sample Efficiency of
  an Emergency Room Classification Model
  To build a French national electronic injury surveillance system based on
emergency room visits, we aim to develop a coding system to classify their
causes from clinical notes in free-text. Supervised learning techniques have
shown good results in this area but require a large amount of expert annotated
dataset which is time consuming and costly to obtain. We hypothesize that the
Natural Language Processing Transformer model incorporating a generative
self-supervised pre-training step can significantly reduce the required number
of annotated samples for supervised fine-tuning. In this preliminary study, we
test our hypothesis in the simplified problem of predicting whether a visit is
the consequence of a traumatic event or not from free-text clinical notes.
Using fully re-trained GPT-2 models (without OpenAI pre-trained weights), we
assess the gain of applying a self-supervised pre-training phase with unlabeled
notes prior to the supervised learning task. Results show that the number of
data required to achieve a ginve level of performance (AUC>0.95) was reduced by
a factor of 10 when applying pre-training. Namely, for 16 times more data, the
fully-supervised model achieved an improvement <1% in AUC. To conclude, it is
possible to adapt a multi-purpose neural language model such as the GPT-2 to
create a powerful tool for classification of free-text notes with only a small
number of labeled samples.
