Transformer Query-Target Knowledge Discovery (TEND): Drug Discovery from
  CORD-19
  Previous work established skip-gram word2vec models could be used to mine
knowledge in the materials science literature for the discovery of
thermoelectrics. Recent transformer architectures have shown great progress in
language modeling and associated fine-tuned tasks, but they have yet to be
adapted for drug discovery. We present a RoBERTa transformer-based method that
extends the masked language token prediction using query-target conditioning to
treat the specificity challenge. The transformer discovery method entails
several benefits over the word2vec method including domain-specific (antiviral)
analogy performance, negation handling, and flexible query analysis (specific)
and is demonstrated on influenza drug discovery. To stimulate COVID-19
research, we release an influenza clinical trials and antiviral analogies
dataset used in conjunction with the COVID-19 Open Research Dataset Challenge
(CORD-19) literature dataset in the study. We examine k-shot fine-tuning to
improve the downstream analogies performance as well as to mine analogies for
model explainability. Further, the query-target analysis is verified in a
forward chaining analysis against the influenza drug clinical trials dataset,
before adapted for COVID-19 drugs (combinations and side-effects) and on-going
clinical trials. In consideration of the present topic, we release the model,
dataset, and code.
