How to Pre-Train Your Model? Comparison of Different Pre-Training Models
  for Biomedical Question Answering
  Using deep learning models on small scale datasets would result in
overfitting. To overcome this problem, the process of pre-training a model and
fine-tuning it to the small scale dataset has been used extensively in domains
such as image processing. Similarly for question answering, pre-training and
fine-tuning can be done in several ways. Commonly reading comprehension models
are used for pre-training, but we show that other types of pre-training can
work better. We compare two pre-training models based on reading comprehension
and open domain question answering models and determine the performance when
fine-tuned and tested over BIOASQ question answering dataset. We find open
domain question answering model to be a better fit for this task rather than
reading comprehension model.
