Pre-training of Graph Augmented Transformers for Medication
  Recommendation
  Medication recommendation is an important healthcare application. It is
commonly formulated as a temporal prediction task. Hence, most existing works
only utilize longitudinal electronic health records (EHRs) from a small number
of patients with multiple visits ignoring a large number of patients with a
single visit (selection bias). Moreover, important hierarchical knowledge such
as diagnosis hierarchy is not leveraged in the representation learning process.
To address these challenges, we propose G-BERT, a new model to combine the
power of Graph Neural Networks (GNNs) and BERT (Bidirectional Encoder
Representations from Transformers) for medical code representation and
medication recommendation. We use GNNs to represent the internal hierarchical
structures of medical codes. Then we integrate the GNN representation into a
transformer-based visit encoder and pre-train it on EHR data from patients only
with a single visit. The pre-trained visit encoder and representation are then
fine-tuned for downstream predictive tasks on longitudinal EHRs from patients
with multiple visits. G-BERT is the first to bring the language model
pre-training schema into the healthcare domain and it achieved state-of-the-art
performance on the medication recommendation task.
