CEHR-BERT: Incorporating temporal information from structured EHR data
  to improve prediction tasks
  Embedding algorithms are increasingly used to represent clinical concepts in
healthcare for improving machine learning tasks such as clinical phenotyping
and disease prediction. Recent studies have adapted state-of-the-art
bidirectional encoder representations from transformers (BERT) architecture to
structured electronic health records (EHR) data for the generation of
contextualized concept embeddings, yet do not fully incorporate temporal data
across multiple clinical domains. Therefore we developed a new BERT adaptation,
CEHR-BERT, to incorporate temporal information using a hybrid approach by
augmenting the input to BERT using artificial time tokens, incorporating time,
age, and concept embeddings, and introducing a new second learning objective
for visit type. CEHR-BERT was trained on a subset of Columbia University Irving
Medical Center-York Presbyterian Hospital's clinical data, which includes 2.4M
patients, spanning over three decades, and tested using 4-fold cross-validation
on the following prediction tasks: hospitalization, death, new heart failure
(HF) diagnosis, and HF readmission. Our experiments show that CEHR-BERT
outperformed existing state-of-the-art clinical BERT adaptations and baseline
models across all 4 prediction tasks in both ROC-AUC and PR-AUC. CEHR-BERT also
demonstrated strong transfer learning capability, as our model trained on only
5% of data outperformed comparison models trained on the entire data set.
Ablation studies to better understand the contribution of each time component
showed incremental gains with every element, suggesting that CEHR-BERT's
incorporation of artificial time tokens, time and age embeddings with concept
embeddings, and the addition of the second learning objective represents a
promising approach for future BERT-based clinical embeddings.
