{'fraction_non_alphanumeric': 0.03709739263803681, 'fraction_numerical': 0.023149923312883437, 'mean_word_length': 4.975085910652921, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Secondary Use of Clinical Problem List Entries for Neural Network-Based Disease Code Assignment

Clinical information systems have become large repositories for semi-structured annotated healthcare data, which have reached a critical mass that makes them interesting for supervised data-driven neural network approaches. We explored automated coding of 50 character long clinical problem list entries using the International Classification of Diseases (ICD-10) and evaluated three different types of network architectures on the top 100 ICD-10 three-digit codes. A fastText baseline reached a macro-averaged F1-measure of 0.83, followed by a character-level LSTM with a macro-averaged F1-measure of 0.84. Top performing was a downstreamed RoBERTa model using a custom language model with a macro-averaged F1-measure of 0.88. A neural network activation analysis together with an investigation of the false positives and false negatives unveiled inconsistent manual coding as a main limiting factor.


Introduction and Motivation
The clinical information systems (CIS) of large healthcare providers host electronic health records (EHRs) in database systems which accumulated real-world data of millions of patients over many years, in varying degrees of standardization and structure. Tables that associate free text entries with codes from controlled vocabularies can be considered human-annotated data sets, which lend themselves to data-intense deep learning methods.
The CIS of a large, multicentre public hospital provider in Austria, includes such a table with millions of short clinical problem descriptions in German language (maximum 50 characters), most of which manually annotated with a code from the International Classification of Diseases . This table fulfils three purposes: (i) collection of content that automatically fills the "Diagnoses" section when narrative discharge summaries are created, (ii) display of a problem-list like scrollable textbox in the CIS frontend, and (iii) provision of disease codes for administrative purposes.
Due to the technical limitation of 50 characters, the often-lengthy ICD-10 descriptions are usually overwritten by the users, which results in an overly compact language, characterized by ellipsis, context-dependent abbreviations and acronyms, non-standardised numeric values, spelling variants and errors. These text snippets exemplify most idiosyncrasies of clinical language (Meystre et al., 2008;Névéol et al., 2018;Wang et al., 2018).
Our work is centred on this resource. We wanted to investigate to what extent clinical realworld data is suited as training material for different types of multi-class classification approaches for automatic assignment of codes from the classification system ICD-10, and where it reaches its limits. We applied three different types of neural network (NN) architectures: a shallow NN (fast-Text), a long short-term memory (LSTM) and a transformer-based architecture (RoBERTa) for our experimental secondary use-case scenario of clinical routine documentation.

Background and Related Work
This section gives an overview of related approaches that focus on assignment of disease codes using NNs trained with clinical text, bypassing numerous equally interesting results of applying these networks to other clinical domains (Topol, 2019;Lipton et al., 2015) and tasks, e.g., medication extraction (Tao et al., 2017;Jagannatha and Yu, 2016;Chalapathy et al., 2016;Zeng et al., 2017) or document deidentification (Lee et al., 2016;Dernoncourt et al., 2017).
Emphasis to process this kind of data had also been given within several task-specific academic challenges 1 (Huang and Lu, 2015;Dörendahl et al., 2019;Kelly et al., 2019). A recent systematic literature review regarding ICD coding systems was published by Kaur et al. (2021). Gehrmann et al. (2017) applied convolutional neural networks (CNNs) to the recognition of ten disease phenotypes using 1,610 manually annotated discharge summaries from the MIMIC-III corpus. CNNs outperformed n-gram models and concept-based prediction models using the clinical information extractor cTakes in combination with random forest and linear regression models, yielding a macro-average F 1 -measure of 0.76. In a salience analysis, they showed that the NN was able to identify phrases that were meaningful for their corresponding phenotype class. Karmakar (2018) compared CNNs, LSTMs, both with and without attention, and an LSTM with a hierarchical attention approach. Using the MIMIC-III data set with 5,000 documents and the seventeen most prominent "level 1" ICD-9 codes, the attention-based CNN outperformed the other models with an F 1 -measure of 0.73. Using the twenty most common "level 5" ICD-10 codes with 52,600 records, the CNN performed best, with an F 1measure of 0.73. For the top seventeen "level 1" ICD-9 codes on the same data set, an F 1 -measure of 0.80 was reached using the CNN with attention. Gao et al. (2017) applied hierarchical attention networks (HAN, Yang et al. (2016)) for information extraction from cancer pathology reports using 942 de-identified texts. Twelve ICD-O topography codes and four histological grading stages were recognized by the network. Their HAN outperformed conventional machine learning (ML) and deep learning (DL) methods, reaching F 1 -measures of 0.85 and 0.71, respectively.
How to process the textual content of large EHRs using NNs exploiting billions of data points and to yield results comparable to human experts was demonstrated by Rajkomar et al. (2018) and Liang et al. (2019). The latter achieved a macro-averaged F 1 -measure of 0.89 by classifying thirteen disease phenotype classes, and Rajkomar et al. (2018) reported the following weighted AUROCs: admission diagnosis -0.87, diagnosis 24 hours after admission -0.89, and diagnosis at discharge -0.90. 1 https://n2c2.dbmi.hms.harvard.edu The main difference to the aforementioned studies is that our experiment uses very short, ICD-10 annotated problem list entries in German that were created as part of routine documentation by physicians.

Data Set
We used ∼1.9 million unique de-identified problem list entries and ignored all entries without ICD codes. A 90/10 split was carried out for training and test set preparation. We trained one model using the top 100 occurring three-digit ICD-10 codes, which covers about 50% of all codes in the data set. Therefore, we scaled up (Wang et al., 2016;Buda et al., 2018) all training samples of the highly imbalanced data set to the most frequent code, resulting in ∼6 million training samples (100 times ∼60k observations). The test set based distribution of the 100 codes under investigation remained unchanged (∼93k observations).  Zolotov and Kung (2017). We used a simple rulebased tokenizer 2 and normalized all resulting tokens to lower case. The training parameters are listed in Appendix A.1. Figure 2 shows the LSTM (Hochreiter and Schmidhuber, 1997) architecture and the input representation of a problem list entry. Character inputs are modelled as 122 dimensional onehot encoded vectors out of the training set plus an out-of-character-dictionary feature dimension in accordance to Zhang and LeCun (2015). A softmax layer outputs the final class probability distribution for the multi-class setting. In this case, we refrained from any pre-processing, taking into account every possible character for the sequence modelling problem. Deeplearning4j 3 was used as implementation library. The training parameters are listed in Appendix A.2.

RoBERTa
For transformer-based architectures (Vaswani et al., 2017), we decided to apply RoBERTa (Liu et al., 2019), an optimized version of bidirectional encoder representations from transformers (BERT) (Devlin et al., 2019). We refrained from the reuse of pretrained German language models due to the characteristics of the short text fragments under investigation, with a high frequency of domain-specific abbreviations and Latin loanwords. Instead, we build our own language model, leveraging the available list entries in combination with byte pair encoding for tokenization in order to support the downstream task with a first understanding of the language under scrutiny. To this end we used ktrain (Maiya, 2020), a lightweight wrapper library e.g. for hugging face (

Neural Network Interpretation
One important aspect to consider when choosing machine learning (ML) algorithms is their ability to explain and give feedback why a certain decision has been made.
Particularly in health care use cases, it is often expected that decision support systems are able to explain the path that led to a certain suggestion. This is reinforced by regulatory frameworks, in which an explanation component is mandatory wherever ML algorithms are applied in realworld clinical decision support settings (Goodman and Flaxman, 2017). The complexity of this task varies from system to system. It is straightforward with decision trees, in which the training process returns a tree that allows full tracing of core decision strategies. In the case of NNs, however, this process poses challenges that are more complex. In this work we were interested in a characterwise feedback for the overall classification result via the inspection of certain class probabilities at certain positions of the LSTM sequence model (Karpathy, 2015), to identify activation levels at this most fine granular input representation scheme.

Evaluation Metrics
We evaluated the performance of the trained model on the test data set using precision, recall, and F 1 -measure per ICD-10 code, as well as macro-evaluation statistics for the overall multiclass classification problem to emphasize on equal importance of each code assignment (Sokolova and Lapalme, 2009). The following definitions are used: true positives (TPs) -the number of correctly assigned codes; true negatives (TNs) -the number of correctly unassigned codes; false positives (FPs) -the number of incorrectly assigned codes; False negatives (FNs) -the number incorrectly unassigned codes. Exploiting this definition, precision P = TP / (TP + FP), recall R = TP / (TP + FN) and F 1 -measure = 2 · P · R / (P + R). Macro performance statistics were averaged over individual evaluation metrics per class.

Results and Discussion
Evaluation of the test set with 92,832 problem list entries yielded an overall macro-averaged F 1measure of 0.83 for the fastText baseline, 0.84 for the LSTM, modelling the input as sequence of characters and 0.88 for the RoBERTa approach. Tables 2, 1 and 3 show the evaluation on an ICD-10 three-digit level. We got the highest F 1measure value for the ICD-10 code P07 (disorders related to short gestation and low birth weight, not elsewhere classified) and the lowest one for Z03 (medical observation and evaluation for suspected diseases and conditions) for all three different NN architectures. Interestingly, the LSTM relying solely on character-level inputs without any preprocessing performed not significantly different from the fastText baseline, taking into account a certain degree of input normalization and different levels of character n-grams for the generation of the overall embedding representation of the problem list entry. This provides strong evidence that different sequence combinations of characters are interpreted by the LSTM network in context for the best possible ICD-10 code assignment. In this work, we therefore decided to investigate a network activation analysis of the LSTM. Figure 4 shows a character-level based heat map of different problem list entries, which can be interpreted to give a visual feedback on which ICD-10 code is the preferred one at a certain character po-sition taking into account its preceding characters. The intensity level corresponds to the class level probability for a given code at a given position and varies between zero and one. A low-level intensity therefore corresponds to a low probability for the class depicted at the right of the figure. In contrast, a high intensity level provides high evidence at a given character position for a certain code. The correct code is highlighted in green, together with the ICD code under investigation. What can be seen at the top of Figure 4 is the network stimulation for the correct class I25 (chronic ischaemic heart disease) for the German input "Kononare Herzkrankh. 1 Gefäß -1 x DES in LAD" (coronary heart disease. 1 vessel -1 x DES in LAD). This snippet exhibits a typing mistake "Kononare" instead of "Koronare" (coronary), an ad-hoc abbreviation "Herzkrankh." for "Herzkrankheit" (heart disease) and two acronyms. "DES" stands for "drugeluting stent" and "LAD" for "left anterior descending (artery)". Especially the last part of this input string strongly supports the code I25. In contrast, the same string has a very low feedback activation for the ICD-10 class E11 (type 2 diabetes mellitus), as expected.

Heat Map
The centre of Figure 4 displays the result of the experiment in reverse. We see the network activation for the input "Diab. mellitust Typ 2, HbA1c: 43 mmol/mol" (diabetes mellitus type 2, HbA1c: 43 mmol/mol) and its corresponding correct code E11. Again, there are ad-hoc abbreviations "Diab." for "Diabetes" and the typing mistake "mellitust" (correct "mellitus"). With the occurrence of the digit "2" in the input string, the network responds with a very high activation at this position for type 2 diabetes mellitus. "HbA1c" (glycated hemoglobin) is also an important diabetes biomarker. Conversely, there is basically no feedback activity for the class I25, resulting in low probability values for the whole input sequence.
At the bottom of Figure 4 the input activation for the same string is contrasted with E10 (type 1 diabetes mellitus) and E14 (unspecified diabetes mellitus). As seen before with the appearance of "2" there is strong evidence for E11. Nevertheless, for E10 and E14, there are clearly observable activation levels. This inspection supports our assumption that, for certain ICD-10 code sections, the network will intermix classes if there is no clear consistent exclusive coding. Nevertheless, the network tries to interpret the input as well as possible with respect to the training data. We will provide a more detailed error analysis in the next section.

Error Analysis
By conducting a confusion matrix analysis on the codes E14 (unspecified diabetes mellitus) and Z03 (medical observation and evaluation for suspected diseases and conditions), where we obtained low performance measurements, we aim to identify problems using training data at the quality level of clinical routine documentation.
E14 When investigating FPs, in most cases E11 was automatically coded as E14. On the other hand, E14 was very often coded as E11 regarding the fraction of the FNs. Having a more detailed inspection on the false negative E14 codes, very often the NN correctly assigned E11 for type 2 diabetes mellitus even though it was human-coded as E14, e.g. "Diabetes mellitus Typ II (HbA1c 9,1%)". Here, obviously, the network corrected a wrong code. We could inspect an intermixture within this section of codes as coded by humans, which obviously influences the trained system.
Z03 What we could inspect from the TPs related to the code Z03 via an activation analysis is that important information for that specific class are contained in the token "Ausschluss" (exclusion) and the abbreviation "V.a." (suspicion of). Very often FP entries that contained these information entities were classified by the NN as Z03, even though physicians had given these entries a different code, e.g., "Ausschluss einer relevanten KHK" (exclusion of a relevant KHK) -KHK being an acronym for "koronare Herzkrankheit" (coronary heart disease) -was coded as I25. The responsible FNs were a mixture of (i) occurring patterns that were nevertheless neglected, e.g., "V.a." for correct class interpretation, and (ii) problem list entries getting the code Z03, with the distinct pattern of the class Z03 be missing from the coders' point of view (ellipses problem). Therefore, the NN assigned its disease code, e.g., "Melanom Stadium IIC (T4bN0M0, AJCC2009:T4b)" (melanoma stage IIC (T4bN0M0, AJCC2009:T4b) was assigned C43 (malignant melanoma of skin) by the NN instead of Z03 as originally coded by the physician.

System Limitations
One main limitation of the study, despite the good macro-averaged multi-class classification results of 0.88 using RoBERTa, is its computational cost for language model generation and downstreaming to the problem domain. The good evaluation results despite training the necessary RoBERTa models just for one epoch might be explained via the fact that through the upsampling of the highly imbalanced data set, multi-epoch training is simulated. Also, for language model generation, perplexity should be used as a quality measurement (Liu et al., 2019). For this, much higher computational resources would have to be leveraged via GPU or TPU clusters for processing clinical narratives at large scale on-premises due to data protection considerations.
In this study, we have used 100 ICD-10 threedigit level codes for the initial investigation if the described resource can be used in a secondary use case scenario for NN model training. Reflecting the fact of a holistic coding system covering thousands of possible codes e.g. for ICD-10-GM, fast similarity search in a contextualized embedding space (Johnson et al., 2017) seems to be the technically more feasible for making the best use of our data.

Conclusion and Outlook
We presented an approach for the assignment of short clinical problem list entries to ICD-10 threedigit codes using three different NN architectures: a shallow NN (fastText), a recurrent NN (LSTM) and a transformer-based architecture (RoBERTa).
We identified a semi-structured data source (ICD-10 code, 50-character long problem list entry) and hypothesized that a clinical language interpretation model for ICD-10 coding can be learned at the level of clinical routine documentation with all its specific language idiosyncrasies. We took the top 100 occurring three-digit codes explaining ∼50% of all coded problem list oc-currences with duplicates removed, up-sampled them to accommodate for the imbalanced data set characteristics and trained the three networks using ∼6 million samples. ∼93,000 problem list entries were used as a test set, where RoBERTa performed best, with an overall macro-averaged F 1 -measure of 0.88. The fastText baseline evaluated with a macro-averaged F 1 -measure of 0.83, the LSTM approach modelling the problem purely on character-level reached a macro-averaged F 1measure of 0.84. In a character-level based heat map, we visualized LSTM network activations to a given input. Thus, we could trace classification decisions back. TP, FPs and FN analyses revealed that the trained network suffered from coding inconsistencies (e.g. E14 versus E11). Nevertheless, it tended to correct wrongly assigned codes, because the majority of the ICD-10 codes had been correctly assigned by the physicians. Future work should investigate in more detail quality aspects of the identified data resource (gold standard creation, interrater reliability), as they have a major influence on the predictive performance. Further, experiments will also measure to what level of distractions the model is robust in doing correct ICD-10 classifications, varying for example input tokens to a certain Levenshtein distance k or applying token-based transposition / rephrasing. Clinical language domain (Alsentzer et al., 2019) BERT models in German are not openly available at the moment but would be of high interest for the clinical NLP community in the future.
The preliminary experiments presented in this paper emphasize the potential of secondary use scenarios of annotated semi-structured clinical real-world data for building NN-based applications such as the assignment of disease codes. Taking advantage of such resources is needed for data-driven applications. There is a yet unexploited potential of NNs to bridge the gap between standardized code entries and the language used in clinical documentation tasks. Identifying and leveraging annotated language resources as presented in this paper could be one step in this direction.

A.1 Training Parameters fastText
The skip-gram embeddings were trained using default parameters (vector dimensionality: 100; subword: between 3 and 6 characters; learning rate: 0.05; epochs: 5). The NN was trained for 25 epochs for all 100 classes using the corpus specific pre-trained skip-gram embeddings from the step before, applying default parameters (learning rate: 0.05).

A.3 Training Parameters RoBERTa
Model optimization was performed on an NVIDIA Titan V with the following parameters: