Robustly Pre-trained Neural Model for Direct Temporal Relation
  Extraction
  Background: Identifying relationships between clinical events and temporal
expressions is a key challenge in meaningfully analyzing clinical text for use
in advanced AI applications. While previous studies exist, the state-of-the-art
performance has significant room for improvement.
  Methods: We studied several variants of BERT (Bidirectional Encoder
Representations using Transformers) some involving clinical domain
customization and the others involving improved architecture and/or training
strategies. We evaluated these methods using a direct temporal relations
dataset which is a semantically focused subset of the 2012 i2b2 temporal
relations challenge dataset.
  Results: Our results show that RoBERTa, which employs better pre-training
strategies including using 10x larger corpus, has improved overall F measure by
0.0864 absolute score (on the 1.00 scale) and thus reducing the error rate by
24% relative to the previous state-of-the-art performance achieved with an SVM
(support vector machine) model.
  Conclusion: Modern contextual language modeling neural networks, pre-trained
on a large corpus, achieve impressive performance even on highly-nuanced
clinical temporal relation tasks.
