{'fraction_non_alphanumeric': 0.026948989412897015, 'fraction_numerical': 0.021632522113754066, 'mean_word_length': 4.926127104834329, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards automated patient data cleaning using deep learning: A feasibility study on the standardization of organ labeling

Data cleaning consumes about 80% of the time spent on data analysis for clinical research projects. This is a much bigger problem in the era of big data and machine learning in the field of medicine where large volumes of data are being generated. We report an initial effort towards automated patient data cleaning using deep learning: the standardization of organ labeling in radiation therapy. Organs are often labeled inconsistently at different institutions (sometimes even within the same institution) and at different time periods, which poses a problem for clinical research, especially for multi-institutional collaborative clinical research where the acquired patient data is not being used effectively. We developed a convolutional neural network (CNN) to automatically identify each organ in the CT image and then label it with the standardized nomenclature presented at AAPM Task Group 263. We tested this model on the CT images of 54 patients with prostate and 100 patients with head and neck cancer who previously received radiation therapy. The model achieved 100% accuracy in detecting organs and assigning standardized labels for the patients tested. This work shows the feasibility of using deep learning in patient data cleaning that enables standardized datasets to be generated for effective intra- and interinstitutional collaborative clinical research.


INTRODUCTION
Data cleaning consumes about 80% of the time spent on data analysis for clinical research projects (Dasu andJohnson 2003, Wickham 2014). Data cleaning improves the quality of data by detecting and removing errors and irregularities caused by inconsistencies or misspellings during data entry, missing information, and the integration of heterogeneous data sources (Rahm and Do 2000). This is not a one-time task, but one that must be performed iteratively as new issues are detected and new data are collected. Several research groups have focused on schema translation and schema integration, and some efforts have focused on identifying and eliminating duplicates (Tang, Nan 2014, Galhardas et al 2000b, Galhardas et al 2000a, Hernández and Stolfo 1998, Lee et al 1999, Monge 2000, but data cleaning has received very little attention (Tang 2014). This is in part due to the breadth of activities that encompass data cleaning (Wickham 2014, Raman and Hellerstein 2001, Lakshmanan et al 1996. Furthermore, the increasing popularity of artificial intelligence in medicine and the collection of big data (Lee and Yoon 2017, Roski et al 2014, Scruggs et al 2015 make it necessary to build automated data cleaning tools (Hu et al 2014, Chen and Zhang 2014. Organ labelling in radiation therapy is often inconsistent, even within an institution, due to different physician preferences, treatment planning systems, and variations that occur over time. These inconsistencies are further widened inter-institutionally. This poses a problem for clinical research, especially for multi-institutional collaborative clinical research, which relies on sharing and using standardized data across multiple sites (Cochrane et al 2007, Denton et al 2016, Gagliardi et al 2012, Mayo et al 2015, Nyholm et al 2016. Naming inconsistencies and the lack of naming standards in radiation therapy have led to misinterpretation of critical details, resulting in treatment errors and radiation therapy misadministration (Santanam et al 2013, Santanam et al 2012. For example, inaccurate contours or modifying or renaming structures caused miscommunication during treatment planning that resulted in 84 of the 500 (17%) misadministration events (Authority 2009).To illustrate this problem, Table 1 shows the inconsistencies in organ labeling for three patients with head and neck cancer at our institution. In patient 1, "R" and "RT" are alternatively used to represent "right" and are prefixed to the organ label, but in patient 2, "R" is placed after the organ label. Also, patient 1 appears to have two right parotids. In patients 2 and 3, numerous abbreviations appear, such as C5, T2, r1 (167), and r2 (163), among others, that were created for optimization or plan evaluation purposes. These structures do not adhere to AAPM Task Group 263 standards that require a character "z" to be prefixed to such structures so that they appear towards the end of the structure list. Also, in patients 1, 2, and 3, the right parotid gland is given three different labels: "RT PAROTID," "Parotid R," and "ParotidGland_R," respectively. This illustrates the difficulty in standardizing workflows and generating large cohorts of data for clinical research.  (157) This paper reports our work to standardize radiation therapy naming conventions of organs for specific tumor sites (prostate and head and neck) in accordance with nomenclature recommended by Mayo et al (2015). We used a convolutional neural network (CNN) to automatically identify each organ from the CT image before assigning labels. This is an initial effort towards automated patient data cleaning using deep learning. In the next section, we present details on the problem of organ labeling and elaborate on the CNN structure used. Then, we briefly discuss the selection of hyperparameters and the fine tuning of the model.
In section 3, we present the results from prostate and head and neck cases. Finally, we give our conclusions and discuss the potential next steps for using deep learning for data cleaning in radiation oncology.

Organ Labeling Problem
Organ labeling in radiation therapy is the process where the physician manually contours critical organs near the treatment site and assigns labels to them. In many institutions, there are currently no automated processes to assist physicians in this task. Our novel approach addresses the issue of organ labeling. We present a two-step process of assigning standardized labels to the organs identified: Step 1. Automatically identify the organs contoured for a specific treatment site, and Step 2. Assign standardized labels in accordance with Mayo et al's standardized organ nomenclature (Mayo et al 2015).
In the first step, we acquire organ contour points along with their labels from the RT Structure DICOM file (Mildenberger et al 2002). The contour points are used as the raw input data, while the organ labels are used as the ground truth. Next, we generate 2D binary masks for each organ on a slice by slice basis and then fuse these masks onto a single image to generate a composite image that corresponds to each slice in the dataset. The extracted organ labels are used to annotate the dataset with output class labels. The set of composite images and the class labels are used as inputs to train the organ identification model. Finally, the model is tested on previously unseen composite images to determine the accuracy in identifying different organs. Formally, the organ identification model takes as input 2D composite images constructed from physician-delineated contour points where = [ 1,1 , … , , ], such that , corresponds to the th organ of patient and outputs a sequence of predicted labels = [ ,1 , … , , ], such that , can take at most one class per patient corresponding to the th organ. The cross-entropy objective function (De Boer et al 2005, Deng 2006) is optimized for every training example to assess the model's accuracy in identifying organs in unseen test samples, as follows: where (•) corresponds to the probability assigned to the input , having value , .
The organ identification problem can be modeled as a traditional multi-class classification problem that can be solved using a supervised machine learning model, such as support vector machines (SVMs) (Steinwart andChristmann 2008, Hearst et al 1998). Given the nature of the organ identification problem, where the input was an image and the output its corresponding organ label, a CNN was the best choice (Krizhevsky et al 2012). The CNN automatically extracts key localized high-dimensional features from the 2D composite images through learned weights and biases and pooling, while alleviating the effects of the curse of dimensionality (Indyk and Motwani 1998).

Patient Data
We trained and tested the organ labeling model on data from patients with head and neck or prostate cancer. For the prostate study, we collected data from 100 patients and generated a total of ~ 40000 composite images. We considered five critical organs for this study: prostate, bladder, rectum, left femur head, and right femur head. For the head and neck study, we collected data from 54 patients and generated a total of ~ 45000 composite images. We considered nine critical organs for this study: spinal cord, brainstem, larynx, mandible, esophagus, left parotid, right parotid, left cochlea, and right cochlea.
The 2D composite images used as input was a novel representation of the raw data acquired in the form of organ contour points. Each composite image consisted of organ masks that had contour points for the specific slice in consideration. Finally, we changed the pixel intensities (pixel manipulation) for each composite image, such that, for a specific organ class, that organ mask was assigned the highest pixel value, while the remaining organ masks present in that composite image were assigned a pixel value of half that

Activation Function Softmax
Optimizer Adam Default parameters Table 3. Summary of hyperparameters used.

Model Architecture
The CNN model used in the organ identification step consists of three convolutional and subsampling layers and two fully connected layers (Fig. 2). Neurons constitute the convolutional/fully-connected layers that have learnable weights and biases. The input to the CNN was an × × composite image, where n was the height, m was the width, and was the number channels, which was set to 1 (grayscale images).
The convolutional layer had k kernels of size × × , where j was set to 3, and > . The value of was doubled at each of the three convolutional layers, such that = 64, 128, 256 at layers 1, 2, and 3, respectively. The filter size enabled the extraction of locally connected structures that were convolved with the image to produce feature maps. These feature maps were then subsampled using 2 × 2 contiguous max pooling to retain the most prominent localized features in the image. The resulting feature maps decreased in size by half after the completion of each of the three max pooling layers (128 × 64 to 64 × 32; 64 × 32 to 32 × 16; 32 × 16 to 16 × 8). Prior to max pooling, an additive bias and nonlinearity was applied to each feature map. We used the rectified linear activation function (ReLU) in the hidden layers, which is piece-wise linear and saturates at zero when the input is less than 0. Finally, we used two fully connected layers along with the softmax activation function in the output layer to provide the probabilities of the computed output classes (Dunne and Campbell 1997).
The composite images generated enabled the CNN to effectively learn high-dimensional localized features and leveraged its ability to extract key image-contrast features because of the pixel intensity manipulation performed on the composite images. Creating composite images for each organ class increased the size of the dataset by a factor of 9 (the number of organs considered) for head and neck cases and by a factor of 5 for prostate cases. For this study, the data samples were split into training and testing sets that were randomly selected with no overlapping and no bias among the organ classes. A data split ratio of 80/20 was used for training and testing, respectively. For both prostate and head and neck cases, the physicianannotated contours from the CT simulations were used as the ground truth. The model reported multi-class 9 and 5 vectors of probabilities that corresponded to the organs considered for head and neck and prostate cases, respectively.

Model Parameter Optimization
We performed network optimization strategies, such as batch normalization, after each convolutional layer and before applying the ReLU activation function (Ioffe and Szegedy 2015). We used a sparse implementation of the Adam algorithm as the optimizer with default parameters (Kingma and Ba 2014) and a random search approach to identify the best hyperparameter values (Bergstra and Bengio 2012). This not only improved the accuracy of organ identification, but it also reduced the number of batch iterations required to reach this accuracy. See Table 3 for details on the hyperparameter values.
Once the CNN model has identified the organ, the next step is to assign a standardized label to it. For this, we created site-specific templates of standardized names for the critical organs, as recommended by Mayo et al (2015). Next, we mapped the output organ classes to the corresponding names in the template to assign a standardized label for each of the organs detected. See Figure 2 for an overview of the steps involved in identifying the organs and labeling them.

RESULTS
For this study, we considered patients with head and neck and prostate cancer. For the prostate cases, the organ labeling model learned high-dimensional features that identified the 5 organ classes with 100% accuracy and assigned standardized labels as recommended by Mayo et al (2015). Despite inconsistencies n organ contouring and labeling, the model extracted key features that allowed for successful organ identification. The CNN was trained on batches of 150 data samples after normalization. The data was randomly split into training data (80%) and testing data (20%) with no overlap and no data bias. The CNN model reported an accuracy of 100% within 5 epochs.
For the head and neck cases, the organ labeling model was able to learn high-dimensional features that identified the 9 organ classes with 100% accuracy and assigned standardized labels as recommended by Mayo et al (2015). Despite inconsistencies in organ contouring and labeling, the model extracted key features that allowed for successful organ identification. The CNN was trained on batches of 150 data samples after normalization. The data was randomly split into training data (80%) and testing data (20%) with no overlap and no data bias. The CNN model reported an accuracy of 100% within 10 epochs.
The number of slices considered for each patient was determined independently. First, we computed the minimum and maximum range of CT slice numbers for each organ class. Then, we determined the biggest range of CT slice numbers that contained contour points from all the organ classes. The number of slices per patient was equal to the number of composite images generated. Based on this methodology, the slices having contour points from multiple organs resulted in composite images with multiple organ masks. In some cases, the composite image contained a single organ mask, which was considered the most difficult case. Yet, the CNN model identified the organ accurately with the help of high-dimensional intensity features generated by the pixel intensity manipulation performed on the composite images. Hence, a combination of structural, spatial, and intensity features enabled the model to learn inter-image relationships, despite the input being 2D composite images.
The model's accuracy and training speed changed drastically when using different learning rates. Figure   4(a) illustrates the accuracy in identifying the different organs, and 4(b) shows the loss achieved for different learning rates. For a learning rate of 10 −1 , the accuracy was flat at 10%, which represented a failure to identify the organs. For a learning rate of 10 −2 , the model reported an accuracy of 89% then plateaued at the same level. The model took 184 batch iterations to converge. For a learning rate of 10 −4 , the CNN reported an accuracy of100%, but it took 305 batch iterations to converge. For a learning rate of 10 −3 , the CNN performed very efficiently, taking 90 batch iterations to converge with a 100% organ identification accuracy. Thus, 10 −3 was the most effective learning rate for training the model.
We saw similar trends in the cost minimization for the different learning rates. With a learning rate of 10 −1 , the cost remained flat at 3.52 after running the model for 184 batch iterations. This learning rate did not minimize the distance substantially between the identified organs and corresponding ground truth class labels. The cost function reduced the distance to 0.234 in 184 batch iterations with a learning rate 12 of 10 −2 , while a cost reduction of 0.129 was reported by the model at a learning rate of 10 −4 . The best cost reduction of 0.092 was seen with a learning rate of 10 −3 , which also converged the quickest, in 90 batch iterations.
To understand how the CNN learned high-dimensional features, we visually illustrated the weights of the model at different stages of the training process for two learning rates: 10 −3 and 10 −1 (Fig. 5). We displayed the initial randomized weights and the learned weights after the convolutional step and the fullyconnected network optimization in 3D space after computing the top 3 principal component analysis (PCA) modes (Abdi andWilliams 2010, Wold et al 1987) using Tensorflow's Tensorboard (Abadi et al 2016).
This helped us understand how the model was performing and how to better fine tune it. The top row shows how the weights were optimized while training the CNN model for the learning rate 10 −3 . We observed two weight clusters (WC) after the convolutional step and a single WC after the fully-connected step. This indicates that the model extracted key localized features and accurately identified organ classes. During the computation of the top 3 PCA modes for learning rate 10 −3 , we observed 4% total variance for the initial random weights and 98% total variance after the convolutional step, while a total variance of 99.9% was observed after the completion of the fully-connected network optimization. This indicated that the model had learned high-dimensional features. In Figure 5, the bottom row illustrates how the weights were optimized while training the model for a learning rate of 10 −1 . Here, we noticed many WCs after the convolutional step, indicating that the model was learning conflicting features. Our observation of WCs after the fully-connected step emphasized this further. This coincided with the results observed from Figure   4, where the model failed to converge and identified organs with an accuracy of roughly 10%.
In Figure 6, we visualize previously unseen test data samples that have been identified by the CNN model. The organ classes are represented in 3D space with color coding assigned to each organ class after computing the top three PCA modes. The total variance of the top 3 PCA modes was 93.8%, while the individual variances were 74.6%, 17.5%, and 1.7%, respectively. The nine organ classes were wellclustered and separated from each other. This representation of the test data provides a way of visualizing not only the variations that occur between class clusters, but also the subtle differences that occur within a class cluster.

DISCUSSION AND CONCLUSIONS
Traditionally, patient data cleaning has accounted for approximately 80% of the time spent on data analysis in clinical research projects. In this work, we reported an initial effort towards automated patient data cleaning, specifically, the task of standardizing organ labels. Assigning standardized labels to the organs is crucial to avoiding the misinterpretation of critical details, which can result in treatment errors and radiation therapy misadministration. We have shown that deep learning can achieve accurate organ identification and standardized labeling without manual intervention, thus eliminating errors and inconsistencies in labeling patient data. We tested the model on patients with head and neck or prostate cancer. In total, 154 patients' data (100 prostate cases and 54 head and neck cases) were utilized in this study. Our model identified organ contours with an accuracy of 100% and assigned standardized labels. In our future work on patient data cleaning, we focus on two areas: increasing the number of organs labeled for head and neck cases and contouring errors. We intend to incorporate a comprehensive list of head and neck organs (this was not possible in this study due to the small patient data set that we had to work with) as prescribed by our institution. Our goal is to develop a 3D model that accepts volumetric anatomical data to identify organs automatically. This will allow the model to learn high-dimensional volumetric features. We intend to add this model to our current clinical system to automate the procedure of standardizing organ labels and retrospectively applying the model to existing patient data to promote clinical research, especially multi-institutional collaborative clinical research.
We are also working to identify and correct contour errors using deep learning. Errors such as incorrect contouring, contour-label mismatch, and volume fill (not closing the manually contoured volume), among others, could result in treatment errors and significantly lengthen the treatment planning process. Currently, at our institution, it takes 7-10 days to plan treatment for head and neck cases after the CT simulation. If contouring errors are detected, the required rework would increase treatment planning time between 28% to 57%. In addition, this also adds logistic complexity to the whole process, because it involves many people, including the physician, dosimetrist, physicists, and the Quality Assurance (QA) team, hence the need for automated tools.