{'fraction_non_alphanumeric': 0.027972591696896414, 'fraction_numerical': 0.01007658202337767, 'mean_word_length': 4.862948960302457, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards Automated Anamnesis Summarization: BERT-based Models for Symptom Extraction

Professionals in modern healthcare systems are increasingly burdened by documentation workloads. Documentation of the initial patient anamnesis is particularly relevant, forming the basis of successful further diagnostic measures. However, manually prepared notes are inherently unstructured and often incomplete. In this paper, we investigate the potential of modern NLP techniques to support doctors in this matter. We present a dataset of German patient monologues, and formulate a well-defined information extraction task under the constraints of real-world utility and practicality. In addition, we propose BERT-based models in order to solve said task. We can demonstrate promising performance of the models in both symptom identification and symptom attribute extraction, significantly outperforming simpler baselines.


Introduction & Related Work
Thorough documentation of a patient's clinical encounters remains a highly sought goal in modern healthcare systems. Especially of importance are the patient's initial complaints, which often guide diagnostic measures. These are usually delineated in a pa- * Equal contribution tient monologue during an initial anamnesis interview between physician and patient. During such conversations, documentation is often recorded in handwritten or typed form. This is very time effective, but usually results in brief, non-standardized, and potentially incomplete notes which are difficult to query or investigate. The professional is often forced to trade off documentation completeness, patient attention and time expenditure. A system that automatically extracts relevant information from such monologues could greatly improve documentation completeness and time efficiency.
Recent methods achieve promising results for neural abstractive summarization (Shi et al. 2018). However, such sequence to sequence approaches usually require large amounts of training data, which is particularly hard to obtain in the medical domain (Quiroz et al. 2019).
Other work investigates symptom extraction from clinical text. Jackson et al. (2017) extract symptoms from discharge notes, and others classify chief complaints from Electronic Health Records (Lee et al. 2019, Chapman et al. 2005 (Si et al. 2019, Chang et al. 2020. In this paper, we focus on a dataset of German patient monologues. Unfortunately, there are much fewer resources available for the German language; e.g. no domain specific pretrained BERT models exist, and publicly available medical data is scarce. This significantly increases the difficulty of creating effective models when dealing with German text. Our goal in this work is to extract all symptoms out of German patient monologues. Our contributions are threefold: i) we present a dataset that consists of written descriptions of patient conditions, ii) we propose a pragmatically grounded task in which each description is annotated with a structured summary of reported symptoms and their attributes, and iii) we present BERTbased models to solve the task.

Dataset
We collect publicly accessible text-based patient written condition descriptions (henceforth denoted as "post") from German medical forums. We select posts that describe gastrointestinal conditions.
After conducting a survey of active medical professionals in Germany, we find that extracting concise symptom information from the often non-technical and verbose patient descriptions is most relevant in the anamnesis. Although the history of treatment, condition, and medication are also useful, we focus on symptom extraction as a first step.
For each forum post, we label the described symptoms according to a hierarchical collection of common symptoms. Similar to Liu et al. (2019), we further annotate attributes that convey additional information, categorized as one of several categories: location, description, time, frequency and action. The survey participants confirm that this serves as a useful summary. We formulate rigorous rules to reduce room for interpretation. To ensure consistency, we label a large portion of the posts twice and merge the results in a final review.
In total, the dataset consists of 125 unique posts, totaling 592 symptoms and 1276 attributes, of which 729 were unique post segments. Further details and dataset statistics can be found in Appendix A.

Methods
To infer the labelings, we divide the task into two stages: Symptom Classification and Attribute Extraction. In both stages, we make use of a pretrained German

Symptom Classification
Baseline Models. We present two baseline models. The first is an MLP with a sigmoid output layer, trained on TF-IDF (Ramos 2003) features. The second is a pretrained BERT model followed by a sigmoid classification layer. For both models, the dimension of the output layer corresponds to the number of output symptoms in the training set.
BERT Symptom Query Model. The baseline classifiers can naturally only predict symptoms that occur in the training dataset. We propose a symptom query model which is able to generalize to unseen symptoms. We feed the symptom description -a short, layman description of the symptom -along with the patient written text into the model, and attempt to predict whether the symptom is present in the text or not. Due to the large amount of possible negative samples, we randomly select one negative sample for each positive sample in a post.
Curriculum Learning. To gain further performance, we employ a curriculum learning scheme (Bengio et al. 2009). The hierarchical nature of the symptom collection allows us to form negative samples which are gradually more complex to classify. For instance, symptoms which are close in the hierarchy are more difficult to differentiate.
Augmented Descriptions. To improve generalization ability to new symptoms, we extract all text segments in the corpus that were annotated as symptoms and introduce them as alternative layman descriptions of the respective symptom during training. This approach yields an order of magnitude increase in the amount of positive samples.

Attribute Extraction.
To extract the relevant attributes for a given symptom we adopt a question answering (QA) style approach. The model receives the post and the symptom description as input and predicts which text sections constitute attributes. Unlike in many QA tasks, (e.g. Rajpurkar et al. 2016, Liu et al. 2019), we often require multiple answers to a single question, as a symptom can have multiple attributes of the same type. We introduce two methods to solve this task. For each of the methods, we train six models; a separate model for each of the five attribute types, and a "general" model that is trained to predict all attributes at once (see Appendix B.2).

Start-End
Prediction. Conceptually similar to the QA model described by Devlin et al. (2019), we combine BERT with two final shared linear layers applied to all tokens. For each token, we predict if it is the start or the end of an attribute.
To account for the multiple possible answers, we use the sigmoid activation function and consider multiple start-end pairs. We often observe very long predictions due to mismatched start and end tokens. This occurs when two attributes are predicted correctly, but the start of one attribute is matched with the end of the other, resulting in a very long token sequence. To combat this, we employ a heuristic post-processing method to form the final predictions (see Appendix B.2.1).
Contiguous Prediction. To circumvent mismatching start and end tokens, we propose to consider each token separately. We append a linear layer after the BERT model, with which we independently classify whether each token is part of a given attribute. We then construct the predictions as all longest sequences of tokens that all have predicted probabilities greater than a threshold of 0.7.

Experiments
The dataset is split into train, test and validation sets. The test set is formed using 20% of all labelings which were identified as correct after double labeling; we observe that these labelings are of higher quality. The validation set is formed using 10% of all remaining labelings. We evaluate the model that achieved the highest validation F1 score (sum of F1 scores over all attributes for the general attribute extraction models) and report it's scores on the held out test set. All F1 scores are micro averaged across all classes. All models are implemented using PyTorch (Paszke et al. 2019).

Results
Symptom Classification. We observe that the symptom query model architecture outperforms both the TF-IDF + MLP and the sigmoid classifier baselines. Both curriculum learning and data augmentation appear to be beneficial and together yield a performance increase of 0.065 F1 compared to the base BERT Symptom Query model (see Table 1). We further notice that the symptom query model with augmented descriptions generalizes better to unseen symptoms, as illustrated in Appendix D.
Attribute Extraction. We report results by attribute type in Table 2. Both methods appear to produce useful predictions (see Appendix F). We observe that the "general" models that are trained on all attributes at once outperform the respective separately trained counterparts on most attributes.
The location attribute is predicted particularly well by almost all models, with F1 scores up to 0.7. We hypothesize that attributing a location to the correct symptom is simpler than correctly matching the other, more generic attributes: "head" is likely to be an attribute to "headache", while e.g. "rarely" could be a frequency attribute to almost any symptom.

Conclusion
In this work, we propose an approach to extract symptom information under the constraints of real-world utility and practicality. To the best of our knowledge, we are the first to attempt such a task in German. We demonstrate promising performance even on a relatively small dataset using a data augmentation technique and a curriculum learning scheme.
Although we observe limitations in model performance, particularly in the attribute extraction task, we are optimistic that this can be improved with a larger dataset. Further work could investigate the extraction of other relevant information such as preexisting conditions and medication. With such improvements, we see potential for practical, automated anamnesis summarization in the near future.

B.1. Symptom Classification
Our architecture (see Figure 1) consists of the pretrained BERT model, followed by a linear layer which takes the concatentation of the mean of all token hidden states, and the pooler output. 1

B.2. Attribute Extraction
We use the pretrained BERT model, followed by a linear layer that is applied to each token's BERT embedding (see Figure 2). The linear layer predicts two probabilities, p s and p e for each token (and for each attribute in case of the general models) when using the Start-End method, and a single probability p inside per token in case of the Contiguous method (see Table 5).

B.2.1. Start-End Post-Processing
To generate the predictions for the Start-End method, we consider each combination of tokens t i , t j with i ≤ j and compute p range We construct the predicted attributes as all token sequences t i , . . . , t j where p range i,j > τ startend = 0.7. To avoid predicting intersecting attributes, we also require 2 3 · p range i,j > p s k and 2 3 · p range i,j > p e k for all k ∈ {i + 1, . . . , j − 1}.
Additionally, we limit the maximum number of tokens to avoid long predictions due to mismatched start and end tokens.

B.3. Training
Following HuggingFace's question-answering examples, 2 we use the Adam optimizer (Kingma and Ba 2015) with learning rate 3e-5 and batch size 32. 3 We train for 40 epochs, and run evaluation every 50 steps, with early stopping using the validation set. We use the default BERT dropout configurations for all symptom classification models and increase them to 0.2 for the attribute extraction models.

Appendix D. Generalization Ability
In Table 6 we observe that the symptom query model seems to perform better on symptoms that appear less frequently in the train set. The difference decreases as symptom frequency increases. We hypothesize that the difference can be attributed to better generalization ability of the BERT SQ architecture.