Adding more data does not always help: A study in medical conversation
  summarization with PEGASUS
  Medical conversation summarization is integral in capturing information
gathered during interactions between patients and physicians. Summarized
conversations are used to facilitate patient hand-offs between physicians, and
as part of providing care in the future. Summaries, however, can be
time-consuming to produce and require domain expertise. Modern pre-trained NLP
models such as PEGASUS have emerged as capable alternatives to human
summarization, reaching state-of-the-art performance on many summarization
benchmarks. However, many downstream tasks still require at least moderately
sized datasets to achieve satisfactory performance. In this work we (1) explore
the effect of dataset size on transfer learning medical conversation
summarization using PEGASUS and (2) evaluate various iterative labeling
strategies in the low-data regime, following their success in the
classification setting. We find that model performance saturates with increase
in dataset size and that the various active-learning strategies evaluated all
show equivalent performance consistent with simple dataset size increase. We
also find that naive iterative pseudo-labeling is on-par or slightly worse than
no pseudo-labeling. Our work sheds light on the successes and challenges of
translating low-data regime techniques in classification to medical
conversation summarization and helps guides future work in this space. Relevant
code available at
\url{https://github.com/curai/curai-research/tree/main/medical-summarization-ML4H-2021}.
