PubMedQA: A Dataset for Biomedical Research Question Answering
  We introduce PubMedQA, a novel biomedical question answering (QA) dataset
collected from PubMed abstracts. The task of PubMedQA is to answer research
questions with yes/no/maybe (e.g.: Do preoperative statins reduce atrial
fibrillation after coronary artery bypass grafting?) using the corresponding
abstracts. PubMedQA has 1k expert-annotated, 61.2k unlabeled and 211.3k
artificially generated QA instances. Each PubMedQA instance is composed of (1)
a question which is either an existing research article title or derived from
one, (2) a context which is the corresponding abstract without its conclusion,
(3) a long answer, which is the conclusion of the abstract and, presumably,
answers the research question, and (4) a yes/no/maybe answer which summarizes
the conclusion. PubMedQA is the first QA dataset where reasoning over
biomedical research texts, especially their quantitative contents, is required
to answer the questions. Our best performing model, multi-phase fine-tuning of
BioBERT with long answer bag-of-word statistics as additional supervision,
achieves 68.1% accuracy, compared to single human performance of 78.0% accuracy
and majority-baseline of 55.2% accuracy, leaving much room for improvement.
PubMedQA is publicly available at https://pubmedqa.github.io.
