{'fraction_non_alphanumeric': 0.030899327995884376, 'fraction_numerical': 0.012411176489501945, 'mean_word_length': 5.108012568735271, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Visual Acuity Prediction on Real-Life Patient Data Using a Machine Learning Based Multistage System

In ophthalmology, intravitreal operative medication therapy (IVOM) is widespread treatment for diseases such as the age-related macular degeneration (AMD), the diabetic macular edema (DME), as well as the retinal vein occlusion (RVO). However, in real-world settings, patients often suffer from loss of vision on time scales of years despite therapy, whereas the prediction of the visual acuity (VA) and the earliest possible detection of deterioration under real-life conditions is challenging due to heterogeneous and incomplete data. In this contribution, we present a workflow for the development of a research-compatible data corpus fusing different IT systems of the department of ophthalmology of a German maximum care hospital. The extensive data corpus allows predictive statements of the expected progression of a patient and his or her VA in each of the three diseases. Within our proposed multistage system, we classify the VA progression into the three groups of therapy"winners","stabilizers", and"losers"(WSL scheme). Our OCT biomarker classification using an ensemble of deep neural networks results in a classification accuracy (F1-score) of over 98 %, enabling us to complete incomplete OCT documentations while allowing us to exploit them for a more precise VA modelling process. Our VA prediction requires at least four VA examinations and optionally OCT biomarkers from the same time period to predict the VA progression within a forecasted time frame. While achieving a prediction accuracy of up to 69 % (macro average F1-score) when considering all three WSL-based progression groups, this corresponds to an improvement by 11 % in comparison to our ophthalmic expertise (58 %).


Introduction and motivation
High-resolution imaging of the central retina utilizing optical coherence tomography (OCT) plays a key role in the diagnosis and monitoring of the most common macular diseases such as age-related macular degeneration (AMD), diabetic macular edema (DME), and retinal vein occlusion (RVO) 1,2 . Furthermore, detailed analysis of different biomarkers on OCT scans is now the basis for treatment decisions as several biological markers provide not only information on diagnosis of these particular eye conditions, but also play an important role in predicting the treatment response. With the increasing amount of available data for therapeutic strategies, the identification of biomarkers with predictive values, as well as different medication options such as intravitreal operative medication therapies (IVOM), it is challenging for ophthalmologists to individualize the therapy for each patient. Artificial intelligence (AI) based algorithms should in the future help to find optimal individual therapeutic strategies for each patient. Real-world studies have already demonstrated how important the early treatment begin and the compliance with the therapy are [3][4][5] . The study group of Gerding et al. was one of the first ones to classify patients with AMD into three treatment groups based on visual acuity (VA) and central retinal thickness: therapy "winners", "stabilizers", and "losers" (WSL scheme) 6 . This interdisciplinary cooperation between IT specialists and ophthalmologists aims at analyzing the patients' data according to the WSL scheme while identifying predictive values for several OCT biomarkers. The results should help ophthalmologists to better define their therapy strategy for each patient in everyday practice. Moreover, Schmidt-Erfurth et al. recently reported the potential of AI-based approaches for targeted optimization of diagnosis and therapy for eye diseases 7 . In their contributions, they furthermore describe the impact of deep learning (DL) for the prediction of patient progressions in the earlier stages of AMD utilizing OCT biomarkers 8,9 .

Related work
In the following sections, we discuss ophthalmic research on data mining from clinical IT systems (section 1.1.1), text and OCT image processing (section 1.1.2), and the use of the processed data for patient progression modelling (section 1.1.3). Besides the shown patient-wise information, ophthalmic doctors are furthermore enabled to on-site annotate their predictions regarding the expected VA progression.

Availability of ophthalmic data in research
In recent years, more and more ophthalmic data sets are being released 10,11 . A recent review article identified 94 open access data sets containing 507 724 images and 125 videos from 122 364 patients with diabetic retinopathy, glaucoma, and AMD being disproportionately over-represented in comparison to other eye diseases. However, the documentation of demographic characteristics such as age, sex, and ethnicity was reported to be of poor data quality even at the aggregate level 12 . In 2017, we proposed a prototypical workflow to aggregate ophthalmic text and image data of all patients from the Department of Ophthalmology of the maximum care hospital Klinikum Chemnitz gGmbH in Chemnitz, Germany. We combined data mining and basic natural language processing utilizing the interface of the clinic's practice management software Turbomed (CompuGroup Medical) and extracted a set of preliminary diagnostic patient data in order to determine the ratio of patients with VA improvement, stabilization, and deterioration 13 .

Ophthalmic text and OCT image processing
While widely being used for general text processing, natural language processing systems have recently been demonstrated to robustly extract medication information from clinical notes to study VA, intraocular pressure, and medication outcomes of cataract and glaucoma surgeries 14,15 to develop predictive models for low-vision prognosis 16 as well as to predict glaucoma progressions 17 . Following De Fauw et al., especially machine learning (ML) and deep learning based approaches enable a more precise progression modelling as recent advances prove their applicability and capabilities within the domain 18 . Moreover, recognition of OCT biomarkers 19 allows further VA and treatment based medical forecasts, including ensemble-based solutions to OCT segmentation 20 that enable the completion of incomplete OCT documentations. However, automated thresholding algorithms can yield an improved reproducibility of OCT parameters while allowing a more sensitive diagnosis of pathologies when, e.g., discriminating between healthy and impaired macular perfusion 21 .

Patient progressing modelling
Schmidt-Erfurth et al. investigate the influence of hyperreflective foci as OCT biomarkers during the progression of geographic atrophy within the context of AMD 8   Proposed medical text and image data mining workflow for the following descriptive statistics and VA modelling based on our three base systems of the electronic health record system (EHR or Turbomed system), the clinical information system (CIS system), and the OCT system. Our patient progression visualization and modelling dashboard is shown in Fig. 1. Currently, the classification and prediction step of treatment adjustment is still in development. However, given all data, a treatment recommendation can be realized.
Moreover, time-dependent sequence modelling using recurrent neural networks (RNN) 23 constitutes a promising approach to treatment prediction 24 . At this point it is also noted that approaches to patient progression modelling exist that explore conventional models. These include mathematical models, e.g., to determine the effect of the anti-vascular endothelial growth factor on VA via medication concentration and tolerance 25 , as well as regression-based approaches, e.g., to predict VA in diabetic retinopathy via the foveal avascular zone area 26 .

Contribution of this work
In this contribution, we present an IT system architecture that aggregates patient information for more than 49 000 patients from different categories of various multimedia data in the form of text and images within multiple heterogeneous ophthalmic data resources. The resulting data corpus enables predictive statements to be made about the expected progression of a patient's VA after at least four VA examinations in each of the three diseases AMD, DME, and RVO. A more fine-grained analysis is conducted to reveal the influence of medical co-existing factors such as other diseases in this real-world setting. Within our proposed multistage system, an ensemble of deep neural networks allows the completion of incomplete or missing OCT documentations. In order to conduct a patient progression modelling, we define a fundamental use case for the prediction of the VA by aggregating different patient information as input of the subsequent VA forecast after a given time period. We present an evaluation formalism and discuss the resulting predictions in comparison to those of a human annotator as our ophthalmic expertise (an experienced ophthalmologist). In order to enable ophthalmic doctors to annotate their predictions regarding the patient-wise expected VA progression, our proposed patient progression visualization and modelling dashboard gives an overview over our aggregated data with patient-wise information of, e.g., general patient information and VA, OCT biomarkers, diagnosis, and medication information (Fig. 1).

Methods and implementation
This section provides the methods and their implementation related to the patient progression modelling. Firstly, the proposed system's architecture for data acquisition, preprocessing, analysis, and prediction is introduced (section 2.1). To allow a unified data processing, the role of the related terminologies for medical application (ophthalmic ontology) is addressed (section 2.2),
whereas the data fusion and cleaning is introduced (section 2.3). Following, descriptive statistics are derived (section 2.4). Finally, the principles of the patient progression modelling within the context of our ML-and DL-based approaches are introduced (section 2.5). The related implemented models and approaches to patient progression modelling as well as OCT biomarker classification are provided via open science from the within this contribution leveraged machine learning framework 1 .

System architecture
The from data as well as text mining obtained ophthalmic core data set is based on the categories of general patient information (G), VA-based patient information (V), OCT scans and biomarkers (O), diseases (D), as well as treatments and medications (T). These categories are obtained from different base systems, the electronic health record system (EHR or Turbomed system), the clinical information system (CIS system), and an OCT system (Heidelberg system) (see also Fig. 2). While all three systems contain basic patient information, the OCT system provides first and foremost OCT scans (categories G and O). The EHR system consists of one large database which contains all relevant patient information within the categories of G, V, D, and T. As the treatment and medication information within the EHR system is not always complete, the clinic's CIS system is additionally utilized in order to retrieve missing medication information (category CIS). Following the retrieved data of all three base systems in the form of BDT (EHR system), CSV (CIS system), as well as E2E files (OCT system), all relevant information for a patient-based data merging is processed. Finally, all patient information is chronologically synchronized, whereas sensitive patient information is pseudo-anonymized. These results are then presented via our patient progression visualization and modelling dashboard ( Fig. 1).

Text processing and ontology
The challenge of the ophthalmic text processing is given by the heterogeneity and incompleteness of the underlying data itself, which are in turn documented by many different medical doctors. For text processing and ontology creation, the programming language Python with the Natural Language Toolkit (NLTK) 27 was utilized. Our implementation encompasses a set of category-specific rules handling abbreviations, negations, and synonyms as well as orthography and grammar variants and mistakes. For this purpose, recognized text strings are mapped into a unique ontology via a dictionary-based stemming 28 . The results have been qualitatively inspected for reliability and plausibility. A thorough quantitative evaluation of our text processing algorithms is still subject to further investigations and will be part of our future research. Therefore, it is beyond the scope of this contribution to further explain the underlying (German) text processing in detail. In the following, we demonstrate some arbitrary strings from the EHR system's diagnosis with customized (German) abbreviations and how they are mapped to diseases, whereby " * " denotes a placeholder for any string that is not handled otherwise. The mapping is case-insensitive and was specifically adjusted for German doctors (Table 1).

Data fusion and cleaning
Our practice management software supports database exports via the BDT file format. Here, we extracted electronic health records from over 10 years ranging from 2010 to 2020, including over 49 000 patients and more than 130 000 examinations. Our six main categories (G, V, O, D, T, and OP) currently span a total of more than 30 subcategories. More than 18 000 IVOMs are available from 2013 to 2020 after a fusion of the EHR and CIS systems' data. All data were linked to the over 12 000 OCT volumes exported from the OCT system via the patient ID (see also Fig. 2, patient-based data merging). Subsequently, the obtained merged data table is further processed and cleaned. Firstly, it is cleaned by mapping the arbitrary text strings from the medical letters to diseases. The medical letters contain several terms and abbreviations for specific diseases as different letters stem from several different doctors. Secondly, unspecific and invalid terms are revised. For example, the entry "eye side" of treatments is only allowed to have the entries "left" or "right", while invalid entries such as "-" are removed.

Category-centered data organization and descriptive statistics
After the data cleaning and fusion, the data are focused towards description of i) anamnesis, ii) intravitreal operative medications (IVOMs), iii) diseases, as well as iv) visual acuity. The main table is changed towards a patient-centered description (Fig. 2). The aim is to record the start and (if available) the end dates of anamnesis entries, diseases, and IVOM therapy cycles for each patient. We designed our tables via two lines per patient as the diseases are to a large degree eye independent, hence we include both eyes as separate lines. For the medical doctors, the category-centered tables allow an easier filtering, sorting, and a more compact view of the data for particular medical information. Finally, all tables are visualized in the statistical-description module. In addition to illustrating a single data set, the tables can also be combined to show cross-referenced data correlations. From all available visualization, including up to 30 combinations of visual acuity and disease statistics, we illustrate in this work due to space constraints the statistics of the aforementioned diseases as well as a disease statics under comorbidities.

Patient progression modelling
For the OCT biomarker classification and patient progression modelling via VA prediction, a set of prominent as well as conventional approaches from machine learning and deep learning are adapted by utilizing the machine learning library TensorFlow with Keras as its front end. Furthermore, scikit-learn is deployed for all machine learning based models and evaluation procedures. Since extracted medical data from patients' EHRs originate from ophthalmologists' documentations, they are defined as our ground truth and serve as training as well as test data.

OCT biomarker classification
In order to complete incomplete OCT biomarker documentations, our multistage system consists of an ensemble of different models for the local (slice-wise) OCT classification, which in turn enables a global (scan-wise) OCT biomarker classification. Within our proposed approach, OCT biomarkers are firstly classified using the provided OCT images of the OCT scans' slices (OCT B-scans). For the following scan-wise OCT biomarker classification, the slice-wise classifications are combined by utilizing the respectively obtained classification confidences. Based on the obtained OCT biomarker classifications, the subsequent VA modelling is realized as a time series prediction using, among others, different ML-and DL-based models such as multilayer perceptrons (MLP) 29 and recurrent neural networks 30, 31 .

Visual acuity prediction
To allow a WSL-based grouping of VA values, the logMAR score of each VA value is derived via VA logMAR = − log 10 VA dec 32 . We define a decimal VA range of 0.04 to 2.0, corresponding to a logMAR range of 1.  Table 5 gives an overview over our ophthalmic data set with its data organization for VA prediction. To predict the VA at a given date, a window with previous patient information of minimum window size 4 and maximum window size 10 is utilized (a). For this purpose, the shown data organization from lines 1 to 10 is deployed, whereas the data within the fields "treatment",    Table 2. OCT biomarker classification results for the slice-wise as well as the scan-wise OCT biomarker classification. We conducted our test runs with a randomized data set split ratio of 80/10/10 for training, validation, and test set averaged over five runs. For our OCT biomarker data set overview, see Table 4.
"OCT biomarker", as well as "additional data" is further detailed in (b). Here, "l/r" denotes the availability of data for both eyes. When translating the information from (a) and (b), factorized data vectors are obtained (c) which are in turn applied to the model learning and evaluation procedures. In (c), "−1" denotes a numeric placeholder when no information is present within the respective data fields. For the sake of simplicity, the processing step of data factorization was omitted. For example, the data entry of the patient's sex can result in three different values: 0 for male, 1 for female, and 2 for patients of diverse sex. Shown are the values for the first window of size 4 in Fig. 1.

Test results, evaluation, and discussion
The following sections provide our evaluation regarding the predictive statistics of therapy winners, stabilizers, and losers (section 3.1). To allow the inclusion of OCT biomarkers into the patient progression and VA modelling process, incomplete or missing OCT documentations are completed (section 3.2). Finally, given each patient's medical data, the following patient progression and VA prediction utilizes, i.a., the resulting OCT biomarker completions (section 3.3).

Predictive statistics
For the following predictive statistics, a statistical analysis was conducted for the aforementioned diseases, whereby the progression of the VA was classified into therapy winners, stabilizers, and losers via the first and the last VA measurement of a patient (section 2.4). The size of our data corpus and its harmonisation as described in sections 2.1 to 2.3 allows different kind of statistical surveys, e.g., separated according to disease, time periods, and comorbidities. The data originate from a maximum care medical hospital and thus indicate effects of real-world scenarios for the aforementioned ascertainments. Our statistical investigations consequently allow us to make statistical predictions under real-life conditions for questions such as "If a patient has exudative AMD, what are the future prognoses for this patient?". For the three aforementioned diseases, the outcomes are shown in Fig. 3. We observed that in average especially for AMD a deterioration of the visual acuity might be expected. , and vice versa. Statistical significance tests are currently not the focus of this work. However, we believe that the data allow us to interpret the increase in the population of the therapy loser group as real effect. This means that more and more patients are experiencing deterioration of vision over longer periods of time. The representation of comorbidities is shown in Fig. 3d as a proof of concept. Even here, if DME and epiretinal membrane (ERM) occur simultaneously, it becomes apparent that only about 12-25 patients are included in each substatistic and the direct comparison with the DME-only group would probably not yet stand up to statistical tests. More data will be needed in the future, e.g., by merging several ophthalmic hospitals into a common research data infrastructure.  (Table 6b) as well as our annotations (ophthalmic expertise) averaged over five runs. "/" denotes model predictions without additional medical data and VA values only. For our VA modelling data set overview with data explanations and vectors, see Table 5. Figures 5 and 6 as well as Table 6 show our evaluation results in further detail.

OCT biomarker classification
Our 6 OCT biomarkers of interest are separated into two states, physiological and pathological, defining two distinct classes. These take the values interrupted (pathological) vs. preserved (physiological) for external limiting membrane (ELM) and ellipsoid zone (ellipsoid) as well as present (pathological) vs. not present (physiological) for foveal depression, scars, pigment epithelial detachment (PED), and subretinal fibrosis, respectively. An OCT biomarker data set overview with the available OCT B-scans per OCT biomarker and related classes is shown in Table 4. For our OCT biomarker OCT slice extraction, we determined an intermediate subset of slices with a slice range of 8 to 18 from 25 in total. Since different OCT scans may possess different original image resolutions, their image resolution was scaled to an initial size of 256 × 256 pixels. Table 2 shows our classification results for the slice-wise and the scan-wise OCT classification using different prominent approaches from machine learning and deep learning averaged over five runs. In comparison, the selected models DenseNet-201 34 and ResNet-152V2 35 show the best classification accuracies in F1-score with mean classification accuracies of 81.8 and 82.3 % for our 6 OCT biomarkers. When comparing the results for the different OCT biomarkers, ELM and ellipsoid show the best scan-wise accuracies, whereas subretinal fibrosis and PED represent the more challenging OCT biomarkers. For the following scan-wise OCT classification, a random forest classifier 33 is already sufficient in order to achieve high classification accuracies of up to 99.9 %. Finally, we obtain the resulting mean classification accuracies in F1-score of 82.3 (slice-wise) and 98.2 % (scan-wise).

Visual acuity prediction
The principles of VA prediction are illustrated in Fig. 4 for an exemplary patient whose first diagnosis was cataract in both eyes and DME in the right eye in 02/2014. Shown is the VA progression over a timespan of 3 years, in which the patient had 6 IVOMs with Eylea and Lucentis. Given a minimum window size of 4, the VA predictions for the 5th to the last examination are based on the previous examinations with their underlying medical patient data. For evaluation, the horizontal lines indicate our thresholds for therapy winners, stabilizers, and losers (see also section 3.1).

Evaluation principles
Out of 49 000 patients, 7 878 patients with VA series of length ≥ 5 exist within our data, resulting in over 100 000 separate VA series of length 5. With the three diseases AMD, DME, and RVO, 1 496 patients with VA series of length ≥ 5 exist, resulting in 14 026 separate VA series of length 5. For the evaluation, all changes within all consecutive VA-based predictions are considered. Hence, all predictions as shown in Fig. 4 are utilized in order to formulate a prediction capability assessment. Ophthalmic expertise MLP MLP-LDA Figure 5. VA modelling results' precision (P) and recall (R) plot for our annotations (ophthalmic expertise) as well as our approaches to VA prediction with MLP and MLP-LDA based on our WSL scheme (see also Table 6).
The VA-based prediction accuracy is calculated via all VA predictions and related local ∆VA i . For the modelling process, the completed OCT documentations (section 3.2) and the related additional data is retrieved. For this purpose, Table 5 gives an extensive overview over the leveraged data set as well as the related data organization.

Training and test setup
Our training setup for VA prediction includes: the Glorot initializer 36 for weight initialization, the Adam optimizer 37 with a standard learning rate of 0.001 and exponential decay rates of 0.9 and 0.999, as well as a batch size of 32. We conducted our test runs with a randomized data set split ratio of 80/10/10 for training, validation, and test set. Table 3 shows our prediction results using different statistical approaches as well as prominent approaches from machine learning and deep learning averaged over five runs. For this purpose, our statistical estimator predicts VA progressions utilizing the statistical distribution of our WSL scheme within our train set. The moving average (MA) estimator averages the given window of VA values, whereas the weighted MA estimator weights recent VA values more strongly in order to formulate a baseline prediction. Whereas our baseline approaches result in prediction accuracies of up to 40.8 % in macro average F1-score (bagging regressor 33 ), a more realistic setting includes the completed OCT documentations, but also the in Table 5 shown additional data. Our MLP-based predictor results in the second highest prediction accuracy, whereas the addition of OCT biomarkers allows their inclusion in the VA modelling process, resulting in a prediction accuracy of 44.6 % with an improvement by 4.4 %. Therefore, the inclusion of OCT biomarkers allows their modelling as crucial information and influential visual factors when no OCT classifications are provided. Following our MLP model, a meta model was realized which classifies the predicted VA values via our WSL scheme. While the MLP predicts the VA values, our so-called MLP-LDA model utilizes these predictions by further classifying them via linear discriminant analysis, whereas the obtained results are cross-validated. We obtain a final prediction accuracy of 69 % (+24.4 %), while the predictions of our ophthalmic expertise reach 58 %. Figure 5 gives an overview over our obtained main results in precision and recall for therapy winners, stabilizers, and losers. Considering the class-wise scores, MLP-LDA strikes a balance between all three progression groups, whereas a trade-off between therapy winners and losers with therapy stabilizers is observable. Finally, Table 6 depicts an extensive VA modelling results overview. In comparison to MLP and MLP-LDA, our annotations show increased scores for therapy stabilizers while achieving the highest general accuracy in true positives. However, MLP-LDA achieves a by 11 % increased macro average F1-score while showing only a by 5.1 % reduced accuracy in true positives. As therapy stabilizers are predominantly present within our data corpus, we conclude that in general a high class-wise prediction accuracy, especially for the critical progression group of therapy losers, has to be enforced in order to realize a (semi-)automated recommender system.

Conclusion and outlook
In this contribution, we developed an IT system architecture that aggregates patient-wise information for more than 49 000 patients from different categories of various multimedia data in the form of text and images within multiple heterogeneous ophthalmic data resources stemming from a German hospital of maximum care. As the prediction of a patient's progression is challenging within this real-world setting, our realized workflow allows a first processing of medical patient data to enable a visual acuity prediction, OCT biomarker classification, as well as a general statistical evaluation and visualization. The resulting data corpus allows predictive statements of the expected progression of a patient and his or her visual acuity in each of the three diseases of AMD, DME, and RVO. Our data revealed that especially exudative AMD results in a notable high amount of therapy "losers" (56 % regarding a time span of 3 to 6 years). A more fine-grained analysis is able to reveal the influence of medical co-existing factors such as other diseases. As proof of concept, we showed exemplary DME with an epiretinal membrane, yet, in some cases statistical surveys of comorbidities in combination with different observation time windows, the data situation is still too weak to derive reliable correlations. For the following visual acuity based treatment progression modelling, incomplete OCT documentations are completed by classifying the OCT scans' slices (OCT B-scans), which in turn allows the classification of OCT scans when only single OCT slices are provided. Based on the obtained OCT slice classifications, a scan-wise OCT classification of the OCT biomarkers ELM, ellipsoid zone, foveal depression, scars, PED, and subretinal fibrosis resulted in an overall classification accuracy (F1-score) of over 98 %. Finally, the completed OCT documentations are combined with additional medical data, defining our ophthalmic feature vectors for visual acuity prediction. In comparison to different approaches from machine learning and deep learning, we achieve an overall prediction accuracy of up to 69 % (macro average F1-score), whereas our ophthalmic expertise reaches an overall prediction accuracy of 58 % (+11 %). However, as the influence of the OCT biomarkers is not yet fully understood, further investigations have to be conducted. Future contributions can build on these initial results in order to determine an optimal time for a change in medication or therapy. In addition, patient-related data of the different categories available and their relevance for the modelling process have to be further investigated in order to allow the realization of a (semi-)automated recommender system.

Classification definitions and terminology
The macro average F1-score is calculated via the class-wise F1-scores f , where F denotes the set of all class-wise F1-scores following our WSL scheme: macro average F1-score = 1 /|F|·∑ f ∈F f , whereas the class-wise F1-scores are in turn calculated via F1-score = 2 · (precision · recall) /(precision+recall) with precision = TP /TP+FP and recall = TP /TP+FN (TP = classification true positives, FP = false positives, and FN = false negatives).  Macro average F1-score 77.5 Table 6. Extensive VA modelling results overview for our baseline approaches without additional data (VA values only), ML-and DL-based approaches with additional data, as well as annotations (ophthalmic expertise) of one single run.