Mining Adverse Drug Reactions from Unstructured Mediums at Scale
  Adverse drug reactions / events (ADR/ADE) have a major impact on patient
health and health care costs. Detecting ADR's as early as possible and sharing
them with regulators, pharma companies, and healthcare providers can prevent
morbidity and save many lives. While most ADR's are not reported via formal
channels, they are often documented in a variety of unstructured conversations
such as social media posts by patients, customer support call transcripts, or
CRM notes of meetings between healthcare providers and pharma sales reps. In
this paper, we propose a natural language processing (NLP) solution that
detects ADR's in such unstructured free-text conversations, which improves on
previous work in three ways. First, a new Named Entity Recognition (NER) model
obtains new state-of-the-art accuracy for ADR and Drug entity extraction on the
ADE, CADEC, and SMM4H benchmark datasets (91.75%, 78.76%, and 83.41% F1 scores
respectively). Second, two new Relation Extraction (RE) models are introduced -
one based on BioBERT while the other utilizing crafted features over a Fully
Connected Neural Network (FCNN) - are shown to perform on par with existing
state-of-the-art models, and outperform them when trained with a supplementary
clinician-annotated RE dataset. Third, a new text classification model, for
deciding if a conversation includes an ADR, obtains new state-of-the-art
accuracy on the CADEC dataset (86.69% F1 score). The complete solution is
implemented as a unified NLP pipeline in a production-grade library built on
top of Apache Spark, making it natively scalable and able to process millions
of batch or streaming records on commodity clusters.
