{'fraction_non_alphanumeric': 0.03137020258141721, 'fraction_numerical': 0.013106533321936347, 'mean_word_length': 4.987376322074377, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards Clinical Encounter Summarization: Learning to Compose Discharge Summaries from Prior Notes

The records of a clinical encounter can be extensive and complex, thus placing a premium on tools that can extract and summarize relevant information. This paper introduces the task of generating discharge summaries for a clinical encounter. Summaries in this setting need to be faithful, traceable, and scale to multiple long documents, motivating the use of extract-then-abstract summarization cascades. We introduce two new measures, faithfulness and hallucination rate for evaluation in this task, which complement existing measures for fluency and informativeness. Results across seven medical sections and five models show that a summarization architecture that supports traceability yields promising results, and that a sentence-rewriting approach performs consistently on the measure used for faithfulness (faithfulness-adjusted $F_3$) over a diverse range of generated sections.


Introduction
Clinical notes in the electronic health record (EHR) are used to document the patient's progress and interaction with clinical professionals. These notes contain rich and diverse information, including but not limited to admission notes, nursing notes, radiology notes, and physician notes. The information the clinicians need, however, is often buried in the sheer amount of text, as the number of clinical notes in an encounter can be in the hundreds. Finding the information can be time-consuming; time that is already in short supply for the clinicians to attend to the patients (Weiner and Biondich, 2006;Sinsky et al., 2016), and can even contribute to the worsening physician burnout crisis (Tawfik et al., 2018;. Summarization has the potential to help clinicians make sense of these clinical notes. In this paper, we aim to make progress toward summarizing one of the most common information sources clinicians interact with -the patient's clinical encounter. A clinical encounter ( Figure 1) documents an interaction between a patient and a healthcare provider (e.g., a visit to the hospital), including structured and unstructured data. Our work focuses on the unstructured clinical notes.
A natural target for summarization is the discharge summary: a specialized clinical note meant to be a summary of the clinical encounter, typically written at the time of patient discharge. Each section (e.g., past medical history, brief hospital course, medications on admission) in the discharge summary represents a different aspect of the encounter. By building a system to extract and compose these medical sections from prior clinical notes in the same encounter, we can summarize the information in a format clinicians are already trained to read and understand.
There are significant challenges ahead, however. In this work, we identify three main challenges of summarizing a clinical encounter: (1) an evidencebased fallback that allows traceable inspection, (2) the faithfulness of the summary, and (3) the long text in a clinical encounter. We believe that all three challenges need to be properly addressed before a discussion about deployment can happen. Thus, this work focuses on measuring and understanding how existing state-of-the-art summarization systems perform on these challenges. Additionally, we propose an extractive-abstractive summarization pipeline that directly addresses the evidence-based fallback challenge and the long text challenge. For the third challenge, faithfulness, we introduce an evaluation measure that uses a medical NER system, inspired by recent work on faithfulness in summarization (Maynez et al., 2020;Zhang et al., 2020).

Contributions
• We identify three challenges for summarizing clinical encounters: (1) faithfulness, (2) evidence, and (3) long text. • We introduce the task of discharge summary composition from prior clinical notes. • We evaluate our proposed extractive-abstractive pipeline for multi-document summarization with medical NER-based scores and ROUGE across seven discharge summary sections. • We create a collection derived from a public database (MIMIC III); a potential benchmark for clinical multi-document summarization.

Evidence, Faithfulness, and Long Text
In this section, we identify the three main challenges in discharge summary composition.
Evidence. A summary should be displayed with means for the clinician to inspect and understand where the information come from. In this respect, extractive summarization has a clear advantage over abstractive summarization, as the source of extracted content can be easily traced and displayed in context. However, abstractive summarization does benefit from a more fluent generation and the potential to function as a writing aid to alleviate the clinicians' documentation burden. The challenge lies in how to design the system such that evidence can be traced.
Faithfulness. Like any model supporting clinical decision making, measuring and understanding the faithfulness of the model output is important. As abstractive summarization systems are evaluated by their ability to generate fluent output, faithfulness can be a challenge to these models. Addressing this problem is an active area of research (Maynez et al., 2020;Zhang et al., 2020).

Extract and then Abstract
These challenges are common in summarization.
In particular, one of the main challenges in multidocument abstractive summarization is to summarize a large number of documents. While significant progress has been made to scale the abstractive models (Beltagy et al., 2020;Zaheer et al., 2020), recent work still involves using an extractive model (e.g., tf-idf based cosine similarity (Liu et al., 2018), logistic regression (Liu and Lapata, 2019a)) to limit the number of paragraphs before abstraction.
Here we proposed a similar extractiveabstractive pipeline. However, what is different in a clinical context is that we wish to place more weight on the extractor than rely on the abstractor to summarize a large amount of text. This decision is motivated by the fact that extractive models are inherently better at being faithful to the source, as they do not introduce novel information. This characteristic makes them ideal candidates for clinical summarization.
Our proposed extractor-abstractor pipeline involves two stages ( Figure 2). The first stage functions as a recall-oriented extractive summarization system to extract relevant sentences from prior documents. The extracted sentences are then passed through post-processing steps that remove duplicated sentences and arrange them to form an extractive summary. The second stage is an abstractive summarization system that aims to take the extractive summary from the previous step and smooths out irrelevant or duplicated information. We describe the details of implementations and how to scale this pipeline to very long text in Section 7.  Another advantage of this pipeline is that it provides a clear path of evidence-based fallbacks (Figure 2). Notably, both the extractor and the abstractor in the extractor-abstractor pipeline are capable of producing full summaries. Clinicians can reference the extractive summary if they find the abstractive summary problematic or if the abstractor model has low confidence. The extractive summary also has another level of fallback. The extracted sentences came from the source documents, so we can also display the extracted sentences in context or even use the extractor as a highlighter.

Measuring Faithfulness
Following prior work, we report ROUGE-n (n = {1, 2}) to measure n-gram overlap as a proxy for informativeness, and ROUGE -L (longest common subsequence, with possible gaps) as a proxy for fluency (Lin and Hovy, 2003;Maynez et al., 2020). However, as Schluter (2017) and Cohan and Goharian (2016) have argued, ROUGE alone is insufficient and possibly misleading for measuring informativeness, specifically when it comes to faithfulness and factualness.
In a summarization setting, a faithful summary refers to a summary that does not contain information from outside of the source. On the other hand, a factual summary allows information not presented in the source, as long as the information is factually correct. In the setting of clinical sum-marization, we argue that faithfulness is far more important. Novel information appearing in a summary that has no support from the source, whether factual or not, can affect the transparency of the model.
A downside of this definition of faithfulness, however, is that it does not take reference summaries into account. Any extracted sentences (e.g., the first three sentences) from the source are always faithful by definition. Such extraction, however, might not be a summary relevant to this task. Figure 3 helps us illustrate the relationship between source documents, reference summary, and systemgenerated summary using a Venn diagram. 1 A desirable summary, especially in a clinical setting, is faithful to the source and relevant as measured by the reference summary. In Figure 3, this region corresponds to B + C, the ideal set of information a clinical summarization system should target. Based on this observation, we define Faithfulness-adjusted Precision as C System and Faithfulness-adjusted Recall as C B+C . Intuitively, faithfulness-adjusted precision measures how much information in the system-generated summary is both relevant and faithful. Similarly, faithfulnessadjusted recall measures the amount of faithful and relevant information that has been included by the system. In a clinical setting, recall is often more important than precision; it is better to overextract and have clinicians ignore or remove the irrelevant content than have missing content. While our extractive-abstractive pipeline provides a series of fallbacks that allows clinicians to inspect what could be missing by looking at the context of the extracted sentences, under-extraction can still hap-pen. We therefore report a recall-oriented measure to combine the two above measures: Faithfulnessadjusted F β , where we set β = 3. In this setting, faithfulness-adjusted recall is three times more important than faithfulness-adjusted precision (Van Rijsbergen, 1979). 2 Hallucination is perhaps the leading concern of applying abstractive summarization system in a clinical setting. If one defines hallucination as a system generating content that is not faithful to the source, we can identify hallucination as the region F + G. G, the information that is not present in neither the source nor the reference, is particularly problematic. We therefore measure Incorrect Hallucination Rate as G System . However, an important underlying assumption of these measures is that the regions in Figure 3 are quantifiable. While there are many ways to approximate these regions, as a starting point, we use the medical named entity recognition (NER) system in SciSpacy (Neumann et al., 2019). The SciSpacy NER matches any spans in the text which might be an entity in UMLS, a large biomedical database, and transforms the text into a set of medical entities. The cardinalities of the sets and their overlaps can then be used to calculate the above measures.

Related Work
Clinical Summarization. Most literature on clinical summarization focuses on extractive summarization, due to the risk involved in a clinical application (Demner-Fushman and Lin, 2006;Feblowitz et al., 2011;Liang et al., 2019;Moen et al., 2016). For abstractive summarization, summarization of radiology reports has been a topic of interest in NLP research recently. Zhang et al. (2018) show promising results generating assessment section of a chest x-ray radiology report from the findings and background section. MacAvaney et al. (2019) improved this model through the incorporation of domain-specific ontologies. However, such generated reports may not be clinically sound, and the models generate sentences inconsistent with the patient's background. Therefore, in subsequent work (Zhang et al., 2020) added a reinforcement learning based fact-checking mechanism to generate a clinically consistent assessment. Lee (2018) explores the generation of the Chief Complaint of emergency department cases from age Dataset Input Ouput # Data Gigaword 10 1 10 1 10 6 CNN/DailyMail 10 2 -10 3 10 1 10 5 WikiSum 10 2 -10 6 10 1 -10 3 10 6 Our Dataset 10 4 -10 5 *10 0 -10 3 10 3 Table 1: Size comparison of summarization datasets. *For stats of the output sections, see Table 2. group, gender, and discharge diagnosis code. Ive et al. (2020) follow a closely related approach of extracting keyphrases from mental health records to generate synthetic notes. They further evaluate the quality of generated synthetic data for downstream tasks. Work from Lee (2018) generates clinical notes by conditioning transformer-based models on a limited window of past patient data. In our work, instead of focusing on purely extractive or abstractive clinical summarization, we proposed an extractive-abstractive pipeline as a framework for clinical multi-document summarization.

Faithfulness in Summarization.
Recognizing the limitation of the existing measures and the danger of hallucination in summarization systems, faithfulness in summarization has gained attention recently (Kryscinski et al., 2020;Cao et al., 2017). Recent work on faithfulness evaluation in summarization involves using textual entailment (Maynez et al., 2020) or question answer generation (Arumae and Liu, 2019;Wang et al., 2020). For radiology summarization, Zhang et al. (2020) proposed using a radiology information extraction system to extract a pre-defined set of 14 pieces of factual information tailored to radiology reports.
In this paper, we approximate information overlap using the overlap of medical named entities. We argue that the domain of clinical encounter summarization is very different from the domains of most textual entailment tasks or question answer generation tasks. It is often much more specific, allowing us to apply the medical NER model. However, it is not as specific as the radiology summarization task, where a set of pre-defined information can more easily be identified.

Dataset
We derive our dataset from the MIMIC III database v1.4 (Johnson et al., 2016): a freely accessible, English-language, critical care database consisting of a set of de-identified, comprehensive clinical data of patients admitted to the Beth Israel Dea-coness Medical Center's Intensive Care Unit (ICU). The database includes structured data such as medications and laboratory results and unstructured data such as clinical notes written by medical professionals. For this work, we will focus on the unstructured data.
The challenge for adapting the MIMIC III database for our purpose, however, is that MIMIC III is incomplete. Due to the way that MIMIC III was collected, not all clinical notes are available; only notes from ICU, radiology, echo, ECG, and discharge summary (Johnson and Shivade, 2020) are guaranteed to be available. It is important to note that the incompleteness is not a property of the problem we are trying to address; it is a property of that database. We limit the incompleteness issue by focusing on the subset of encounters that contain at least one admission note (a clinical note written at the time of admission) as a proxy for completeness. This leaves us about 10% of the total encounters, or around 6,000 encounters.
We identify seven medical sections in the discharge summary as our targets for summarization: (1) chief complaint, (2) family history, (3) social history, (4) medications on admission, (5) past medical history, (6) history of present illness, and (7) brief hospital course. These medical sections were chosen based on their high prevalence in discharge summaries and their length diversity (see Table 2).
Target Section Extraction. To extract the target medical sections from the discharge summary, we use a regular expression based approach to identify the medical section headers' variants from the training set. We then collect the content from the target medical section header and stop right before the next section header in the discharge summary. Around one hundred randomly selected extracted medical sections are manually examined to ensure no missing content or over-extraction. For each of these target medical sections, we then collect all the prior clinical notes (according to the chart date timestamp in MIMIC III) as their source documents. On average, the source documents consist of 64 documents and 36,3567 words. Table 1 shows a comparison with other dataset.
After the rule-based target extraction, we split the 6,000 encounters based on the subject id to prevent data leakage. Each section is split into training, validation, and test set (80/10/10) using the same set of subject ids. If the rule-based target extraction returns nothing, the encounter is excluded. See Table 2 for the statistic of sample size.

Models and Experiments
As explained in Section 3, our proposed pipeline involves an extractive summarization component and an abstractive summarization component. This section identifies a set of existing extractors and abstractors across a diverse range of different approaches to understand what models are suitable for encounter-level clinical summarization. To understand the robustness of these approaches, we train and test these models across seven medical sections with a diverse range of length.
Extractors. Since our goal is to summarize an encounter conditioned on a targeted medical section, we focus our attention on supervised extractors. Supervised extractive summarization is often framed as a sentence extraction problem. Each sentence is encoded into a representation used to determine whether the sentence should be included in the extracted summary. RNN or transformer-based attention are often used to encode the surrounding sentences as context.
RNN+RL ext : Chen and Bansal (2018) proposed a method to use reinforcement learning to fine-tune a pretrained RNN sentence extractor to a pointer network operating over sentences. By modeling the next sentence to extract (including the extra "end-of-extraction" sentence) as the action space, the current extracted sentences as the state space, and by using ROUGE between reference summary sentence and rewritten extracted sentence (rewritten by a separate pretrained abstractor) as the reward, the authors repurposed the sentence extractor to extract sentences from the source documents and reorder them as they might appear in the summary.
PRESUMM ext : Liu and Lapata (2019b) proposed Presumm, a family of summarization models. Here we are especially interested in the extractive summarization variant that uses a modified pretrained BERT model to encode sentences to determine whether the sentence should be included in the extracted summary. While the model has been shown to achieve competitive results, applying a BERT encoder to very long text can be challenging in terms of memory limitations. Thus, we apply a split-map-reduce framework, where the long text is split into smaller units during training and inference. After inference, each smaller unit's extracted sentences are then concatenated back together in the same order as appeared in the original source. Since the model only assigns scores to sentences, we sweep the score cutoff threshold on the validation set using ROUGE-L score, and apply that cutoff on the test set.
Abstractors. In our extractive-abstractive pipeline, abstractors play a role in mapping the extracted sentences to the reference summary. Here we include two abstractor variants: 3 RNN+RL abs : Similar to RNN+RL ext , however, after each sentence is extracted, it is immediately rewritten by passing through a pretrained sentence-level abstractor. The goal is to rewrite each extracted sentence to the format of what might appear in the reference summary. This sentencerewriting approach has the disadvantage of only having a local view when rewriting (thus no merging of information). However, the advantage is that the memory limitation of sentence-level rewriting does not grow with the number of sentences, so it can be applied to longer summaries.
BART: Lewis et al. (2019) propose BART as a transformer variant that uses a bidirectional encoder similar to BERT and an autoregressive (left to right) decoder similar to GPT. The model has competitive performance for summarization, and thus is our choice for transformer-based abstractor. In contrast to the sentence-rewriting approach of RNN+RL abs , we train BART to rewrite all the extracted sentences directly to the summary.
Baselines. Since clinical encounter summarization is a new task, there are no baselines from prior work. Following prior work on summarization, we include two special baselines: (1) ORACLE ext : Extraction by using the reference summary; for each sentence in the reference summary, greedily select the source sentence in the source document that yields the maximum ROUGE-L score. (2) RULE-BASED ext : apply the same rule-based target section extraction method in in Section 6 that was used to construct the dataset. Instead of applying to the discharge summary, we apply the same extraction method to the prior clinical documents.
Evaluating the extractor-abstractor pipeline. For the two extractive models, RNN+RL ext and PRESUMM ext , as well as the two extraction baselines, we report ROUGE scores as well as our proposed factualness-adjusted {precision/recall/F 3 } scores across the seven medical sections.
For the abstractive models, we measure the combinations of abstractive models with extractive models in our proposed pipeline. This implies measuring the performance of three models (two pointer-generator models shown in Appendix): RNN+RL abs (uses RNN+RL ext as the extractor), RNN+RL ext + BART, and PRESUMM ext + BART. For the abstractors, we additionally measure incorrect hallucination rate defined in Section 4.

Results and Discussion
Extractive summarization and abstractive summarization are often applied in different settings and should thus be compared separately. For full results table, see Appendix A. Here we highlight the main findings. In Figure 4a, we highlight the ROUGE-L scores (ROUGE-1 and ROUGE-2 have a similar pattern) of the two extractive summarization systems compared to the oracle and rule-based extractive summary. An interesting observation is the effect of length: average word count of the reference summary medical section. RNN+RL ext outperforms PRESUMM ext on shorter sections, and vice-versa for the longer sections. This difference can be partially attributed to the way cutoff is being done at the extractors. For RNN+RL ext , an RL agent is trained to decide when to stop extracting sentences. For the shorter sections, the RL learns to stop at just a few sentences (e.g., a typical chief complaint has two sentences, family history has on average 2.6 sentences). On longer sections, however, we find that the RL agent has difficulty stopping, causing over-extraction. In contrast, for PRESUMM ext , a score cutoff threshold is tuned on the development set using the ROUGE-L score. This approach has a more balanced performance but suffers at short sections. Another factor contributing to the lead of PRESUMM ext in the longer sections is our split-map-reduce framework, which enables the extractive model to conduct inference over all the clinical documents.
Interestingly, the baseline RULE-BASED ext performs surprisingly well on Rouge for the two shortest sections. Upon inspection, most of the extraction is just the medical section's title, without any content. This observation is backed up by the lower faithfulness-adjusted recall of this baseline.
For abstractive summarization, we highlight the ROUGE-L of the three abstractors in Figure 4b. Interestingly, after being abstracted by     BART, both RNN+RL ext and PRESUMM ext converged to roughly the same ROUGE-L scores. This suggests that in our extractor-abstractor pipeline, BART is effective in taking different extracted summaries and smoothing them into the format and content expected for the medical sections. On the other hand, RNN+RL abs outperforms BART at the shorter sections, and even ORACLE ext at the family history section. Note that ORACLE ext is not necessarily an upper-bound for the abstractive summarization models; abstractors allow rewriting content in prior notes into the format of discharge summary. Sentence seqmentation (the basic unit of extraction) can also be noisy in clinical notes. On the other hand, the curve for RNN+RL abs is almost identical to RNN+RL ext in Figure 4a, with a constant increase. This is largely attributed to the sentence-rewriting of the sentence-level abstractor that allows RNN+RL abs to keep the benefit of its extractor counterpart, while rewriting the content to reduce over-extracted sentences. Table 2 shows our faithfulness adjusted measures. For the extractors, RNN+RL ext outperforms all other extractors on faithfulnessadjusted F 3 and even outperforms ORACLE ext in the brief hospital course section. This is possible because ORACLE ext is selected using ROUGE-L, not faithfulness-adjusted F 3 . For the abstractors, a similarly good performance is found for RNN+RL abs , where its precision consistently increases compared to RNN+RL ext . The good performance of RNN+RL ext,abs can largely be attributed to the high recall that has hurt their ROUGE-L performance in Figure 4. Interestingly, the two BART models again perform roughly the same, with recall of RNN+RL ext + BART higher than PRESUMM ext + BART. For the longest section, generation for  BART proves to be difficult, as indicated by the large drop of recall, whereas the sentence-wise rewriting strategy of RNN+RL abs has scaled better to longer sections. The overall incorrect hallucination rate shown in Figure 5 is relatively low, with the notable exception of the family history section. Inspection of the generated summaries shows that the most common hallucination of both BART systems is the phrase "no family history". Interestingly, the ground truths corresponding to these hallucinations are mostly variations of the term "non-contributory"; inspection of the source also shows that the family history section was often left blank. That being said, there are still cases of hallucinations where "no family history" is followed by a condition (e.g., arrhythmia, cardiomyopathies) that is not mentioned in the source. Table 3 shows a qualitative analysis of a randomly chosen summary of the past medical section. In this case, RNN+RL ext over-extracted content from the previous sections. However, after passing through BART, BART successfully smooths out the noise and generates the same output as Presumm ext + BART. In this case, RNN+RL abs happens to be hallucinating (mapping cr 1.3 to cr 1.5). All summarization systems missed "# stable angina on long acting nitrate"; mention of "angina" is actually not present in the prior clinical notes.

Conclusion and Future Work
We present a novel clinical summarization taskdischarge summary composition by summarizing prior clinical documents, derived from a public database (MIMIC III). By summarizing the vast number of clinical notes in a format clinicians are already trained to read and understand, our work has the potential to reduce the time clinicians spend on making sense of the data, allowing them to allocate more time to the patients.
We view this work as a promising first step to measure how existing models perform on the task and share the task with the community. One limitation of this work is that if there is novel information available only when writing the discharge summary, there will be no way of summarizing it. It is also important to note that since we are using MIMIC III for training and evaluation, the results shown are biased to the dataset, as MIMIC III is an English-language collection from the ICU of a single hospital, and may not necessary be applicable to other clinical setting.
We identify three main challenges: (1) faithfulness, (2) evidence, and (3) long text. An extractorabstractor pipeline is proposed to provide a natural way of fallback with an increasing amount of evidence at each fallback and also enable scaling to very long documents. To investigate the risk of hallucination and faithfulness in the summaries, we evaluate with a NER-based measure on top of ROUGE. Adapting state-of-the-art summarization models, our experiments over seven medical sections demonstrate the potential for the extractorabstractor pipeline and represent a framework towards a set of enabling technologies that can assist clinicians to better make sense of the vast amount of unstructured data in the EHR.

Ethical Considerations
Deidentification. Our dataset is derived from the public database MIMIC III v1.4 (Johnson et al., 2016). Johnson et al. (2016) deidentified the database in accordance with the Health Insurance Portability and Accountability Act (HIPAA) standards. This standard requires removing all eighteen identifying data elements, including patient name, telephone number, address, and dates. These fields are replaced with placeholders. A constant (but different per patient) offset is applied to shift the dates. Patients over 89 years old were mapped to over 300, in compliance with HIPAA.
Although under U.S. federal guidelines, secondary use of fully deidentified, publicly available data is exempt from institutional review board (IRB) review (45 CFR § 46.104, "Exempt research"), we still consider the dataset sensitive. We are careful to treat it as such. During training and error analysis, we of course do not attempt to identify individuals, and when the qualitative analysis is shown, we double-check to avoid showing potentially identifiable information.
Population. In MIMIC III, out of the 38,161 patients, 71.34% are White, 7.69% Black, 2.38% other, 2.37% Asian, and the rest unknown. Most of the patients in MIMIC III were older adults, with the most common age group being 71-80, followed by the 61-70 age group. (Dai et al., 2020).
Broader Impact. Clinical application has the genuine potential to affect people's lives. As we have emphasized in Section 1 and Section 9, this work is not about a discussion for deployment, but rather a first step in understanding how the current existing summarization models perform. Importantly, we need to understand the failure modes of these systems and how to address these failures. Our emphasis on faithfulness and traceability of summarization reflects those beliefs. Hopefully, the three challenges we identify are the first of many future steps that will make progress toward alleviating the documentation burden of clinicians and ultimately result in a better quality of care for patients. Yuhao Zhang, Derek Merck, Emily Tsai, Christopher D. Manning, and Curtis Langlotz. 2020. Optimizing the factual correctness of a summary: A study of summarizing radiology reports. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5108-5120, Online. Association for Computational Linguistics.

A Appendix: Full Results
See Table 4 and Table 5 for the full scores for all models on all seven sections.
B Appendix: Qualitative Analysis Table 6 shows a random example on social history section.

C Appendix: Reproducibility
Here we describe the training details of the models for reproducibility.
RNN+NL ext and RNN+NL abs . Both models are trained following the original recipe (Chen and Bansal, 2018). The training setup involves the following steps: (1) use gensim to train a word2vec embedding from scratch from the training set of the source documents, (2) construct pseudo pairs of sentences (source sentence, summary sentence): for each summary sentence, greedily finds the onebest source sentence using ROUGE-L recall, (3) use the pseudo pairs to train an RNN extractor, (4) use the pseudo pairs to train an pointer-generator that rewrites the sentences, and (5) train an RL agent that fine-tunes the RNN extractor with the sentence-rewriting pointer-generator. Model is trained on one V100 GPU, with an Adam optimizer of learning rate 1e-3. Here we use the same set of hyperparameters as Chen and Bansal (2018). For more details, please refer to the original paper. For each of the seven medical sections, we follow the training recipe, and repeat it five times. The reported models are chosen based on the validation set. We found that the RL fine-tuning step can potentially be very unstable. For the longer sections (e.g., brief hospital course and history of present illness), the RL fine-tuning can even fail to converge.
PRESUMM ext . We use the original implementation released with PRESUMM (Liu and Lapata, 2019b). Learning rate is set to 2e-3 and extractor dropout rate is set to 0.1, following the original paper. bert-base-uncased is used as the pretrained BERT model. We made three important changes: (1) increase the maximum tokens the encoder can consume to 1024 tokens, (2) in the data preprocessing step, we construct pseudo pairs of sentences that will be later used to train the extractor: for each summary sentence, greedily finds the one-best source sentence using ROUGE-L recall, and (3) before the training begin, we split the source documents and their labels into segments smaller than 1024 tokens. After inference finishes, we concatenate the segments (together with a extraction score for each sentence) back together in the original order.
For each of the seven medical sections, we train the model on 4 V100 GPUs, with 150,000 training steps and model checkpointing every 2,000 steps. We report the model with the lowest model loss on the validation set. Since the model only assigns scores to sentences, we sweep the threshold of score cutoff on the validation set using ROUGE-L score, and apply that cutoff on the test set.

POINTGEN.
We use an open implementation of pointer-generator (See et al., 2017), implemented with PyTorch and AllenNlp. 4 Our model follows the original paper and has 256-dimensional hidden states and 128-dimensional word embeddings. The vocabulary size is set to 50k words for both source and target. The model is optimized using Adagrad with learning rate 0.15 and an initial accumulator value of 0.1, and trained on one v100 GPU for 50 epochs with early stopping on the validation set.
BART. We use the Fairseq (Ott et al., 2019) implementation of BART-large (Lewis et al., 2019) as it is shown to achieve the state-of-the-art ROUGE scores for abstractive summarization. We fine-tune the BART-large model with the standard learning rate of 3 × 10 −5 . We utilize a machine with 8 GPUs and batch size of 2048 input tokens per GPU. We train for a maximum of 10 epochs with early stopping to select the checkpoint with the smallest loss on the validation set. During decoding, we use beam search with beam size of 6. We restrict the generation length to be between 10 to 300 tokens.