{'fraction_non_alphanumeric': 0.038758416817211365, 'fraction_numerical': 0.0162588273936607, 'mean_word_length': 5.0072344623479115, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 3, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Chest X-Ray Analysis of Tuberculosis by Deep Learning with Segmentation and Augmentation

The results of chest X-ray (CXR) analysis of 2D images to get the statistically reliable predictions (availability of tuberculosis) by computer-aided diagnosis (CADx) on the basis of deep learning are presented. They demonstrate the efficiency of lung segmentation, lossless and lossy data augmentation for CADx of tuberculosis by deep convolutional neural network (CNN) applied to the small and not well-balanced dataset even. CNN demonstrates ability to train (despite overfitting) on the pre-processed dataset obtained after lung segmentation in contrast to the original not-segmented dataset. Lossless data augmentation of the segmented dataset leads to the lowest validation loss (without overfitting) and nearly the same accuracy (within the limits of standard deviation) in comparison to the original and other pre-processed datasets after lossy data augmentation. The additional limited lossy data augmentation results in the lower validation loss, but with a decrease of the validation accuracy. In conclusion, besides the more complex deep CNNs and bigger datasets, the better progress of CADx for the small and not well-balanced datasets even could be obtained by better segmentation, data augmentation, dataset stratification, and exclusion of non-evident outliers.


I. INTRODUCTION
Due to relatively cheap price and easy accessibility chest X-ray (CXR) imaging is used widely for health monitoring and diagnosis of many lung diseases (pneumonia, tuberculosis, cancer, etc.). Manual analysis and detection by CXR of marks of these diseases is carried out by expert radiologists, which is a long and complicated process. Nevertheless, the modern evolution of general-purpose graphic processing cards (GPU) hardware [1] and software for medical image analysis [2], especially deep learning techniques [3], allows scientists to detect automatically many lung diseases from CXR images at a level exceeding certified radiologists [4]. Despite these successes the strong belief exists among experts that deep learning techniques become efficient for the very big datasets (>10 4 images), because for the smaller datasets (<10 3 images) they produce bad predictions (if any at all) with the very low accuracies. This paper is dedicated to the description of the lung segmentation technique in combination with lossless and lossy data augmentation which allow us to get the statistically reliable predictions of lung diseases (availability of tuberculosis) for such a small dataset (<10 3 images) even.

II. PROBLEM AND RELATED WORK
Recently, promising results were obtained in the field of computer-aided medical image analysis for an assessment of lung diseases by deep learning from CXR image analysis [2,[4][5][6]. Availability of open datasets with labeled CXR images allowed data scientists to apply their deep learning algorithms for anatomical structure detection, segmentation, computeraided detection (CADe) of suspicious regions, and computeraided diagnosis (CADx) [2]. Various datasets with CXR images were released for public domain recently: JSRT dataset with 247 images of cancer [7]; LIDC dataset with ~10 3 images [8]; Montgomery County (MC) dataset with 138 images, Shenzhen Hospital (SH) dataset with 662 images [9,10] (Fig. 1), and ChestX-ray14 [11], which is the largest publicly available database of CXR images to the moment (112,120 images of 30,805 unique patients). Several hardships with CADx of lung diseases on the basis of the small datasets (<10 3 images) are related with presence of high variability of image sizes, image quality, and influence of other regions outside of lungs (heart, shoulders, ribs, clavicles, etc.). This variability is originated by presence of patients of different gender, nationality, age, substance abuse, professional activity, general health state, and other parameters of patients under investigations. Such variability is strengthened by some additional artifacts like labels created by radiologists and/or doctors (like white rectangles with text and symbols shown in top part on Fig.1a). Recently, efficiency of some dimensionality reduction techniques, like lung segmentation, bone shadow exclusion, and t-distributed stochastic neighbor embedding (t-SNE) for exclusion of outliers, was demonstrated for analysis of CXR 2D images to identify marks of lung cancer [5][6]. Despite these attempts, CADx remains a challenge in medical image applications, especially, in the view of high variability of lungs and other regions outside of lungs due to different personal parameters of patients under investigations. The aim of this work is to demonstrate the efficiency of the proposed lung segmentation technique in combination with lossless and lossy data augmentation methods to get the statistically reliable predictions of lung diseases (availability of tuberculosis) even for the relatively small datasets like SH (<10 3 images) [9,10].

III. EQUIPMENT, METHODS, AND DATA
The deep CNN [5] was trained on various general-purpose graphic processing cards (GPU) by NVIDIA: Tesla K40 (Kepler microarchitecture), GTX 1080 Ti (Pascal), and the latest and most powerful GPU-card, Tesla V100 (Volta). The official data on performance, bandwidth, number of cores, and the actual training times with the correspondent speedups (for the same dataset and runtime configuration) are shown in Table 1. It worth to note that the CNN model was not optimized here for calculations by tensor processing unit (TPU) in V100 card. That is why its speedup is not so impressive in this work, but the results on the optimized version will be reported elsewhere [12]. Shenzhen Hospital (SH) dataset of CXR images was acquired from Shenzhen No. 3 People's Hospital in Shenzhen, China. It contains normal and abnormal CXR images with marks of tuberculosis. The exploratory data analysis shown that this small dataset is a not well-balanced dataset with regard to availability/absence of disease, age, gender (Fig. 2). The image widths and heights are not equal and these differences lead to the wide and asymmetric distribution of image areas (Fig. 3). The current and previous attempts to perform training for such small CXR datasets without any pre-processing were performed and failed [5,6]. Fortunately, "external segmentation" of the left and right lung fields in CXR images (exclusion of outside regions which are not pertinent to lungs) was demonstrated to be effective to provide successful training and, moreover, to increase the accuracy of predictions [5,6]. To perform lung segmentation, i.e. to cut the left and right lung fields from the lung parts in standard CXRs, the manually prepared masks were used ( Fig. 1, central) that allowed to get the additional dataset from SH, namely, the segmented SH dataset ( Fig. 1, right). Despite the high variability of lung segmentation masks (due to the different lung shapes and borders) the distribution of lung mask areas was much closer to the normal one in comparison to the distribution of image areas, which was asymmetric and contained the long tail for smaller sizes (Fig. 3). Some examples of the most similar, dissimilar, and average lung masks are shown in Fig. 4. This variability is explained by presence of outliers, for example, some portion of children with the smaller lungs (see the local peaks for age<20 in Fig. 2).

IV. RESULTS
The deep CNN [5] was trained in GPU mode by means of TensorFlow machine learning framework [13] to predict presence (336 images) or absence (326 images) of tuberculosis in them. The dataset was split into 8:1:1 parts for training, validation and test parts respectively. Images were rescaled 1./255 and resized to 2048×2048 and distributed among training, validation and test parts. Several training and validation runs up to the 100 epochs for the CNN with dropout on CXR images from the segmented SH dataset were performed for: a) the segmented SH dataset; b) the segmented SH dataset with lossless and c) lossy data augmentation.

A. Segmented Dataset
The examples of accuracy and loss values after some runs are shown in Fig. 5 (accuracy) and Fig. 6 (loss): for training and validation, after averaging their values over the runs (red line in Fig. 5 and Fig. 6), and after smoothing the averaged time series (blue line in Fig. 5 and Fig. 6) by the locally weighted polynomial regression method [14]. The pronounced overtraining (overfitting) can be observed after comparison of these training and validation results (Fig. 5 and Fig. 6). The averaged and smoothed value of training accuracy (Fig. 5a) is going with steps up to the theoretical maximum of 1 and the training loss (Fig. 6a) is going to 0. At the same time the averaged and smoothed value of the validation accuracy (Fig. 5b) is no more than 0.76 and the validation loss (Fig. 6b) is very high and growing. Such overfitting can be explained by the small size of the dataset, which can be decreased by the well-known data augmentation methods. For this purpose, the lossless and lossy types of data augmentation were applied and their role investigated.

B. Effect of Lossless Data Augmentation
The lossless data augmentation for 2D images included the following transformations: mirror-like reflections (left-right and up-down) and rotations by 90n degrees, where n = 1,2,3. This allowed to increase efficiently (by 8 times) the whole dataset and obtain the more realistic results on accuracy and loss during training and validation. The training accuracy did not exceed 0.9 (Fig. 7a), the validation accuracy did not exceed 0.7 (Fig. 7b), and the training loss (Fig. 8a) was going to 0 with much lower rate. But the most important aspect that validation loss (Fig. 8a) now has the global minimum (approximately at 3000 steps) that corresponds to the most realistic estimation of the validation accuracy, which is equal to 0.7±0.1.

C. Effect of Lossy Data Augmentation
The additional lossy data augmentation for these 2D images included rotations by 5 degrees. In this case the training accuracy was lower than 0.85 (Fig. 9a) with the lower standard deviation (of the averaged and smoothed value), the validation accuracy was lower than 0.65 (Fig. 9b), and the training loss ( Fig. 10a) was not lower than 0.35. But again the most important aspect that validation loss (Fig. 10a) has the global minimum (approximately at 2500 steps) that corresponds to the most realistic estimation of the validation accuracy, which is equal to 0.64±0.1.
Despite the lower number of training/validation runs for datasets with lossless data augmentation (3) and lossy data augmentation (2) in comparison to 7 runs for the segmented dataset, the standard deviation of the averaged and smoothed value (denoted by blue whisker lines in Fig. 5-10) of the validation accuracy was always lower for the augmented datasets ( Fig. 7b and Fig. 9b). And the more interesting observation related with the much lower validation loss for the augmented datasets ( Fig. 8b and Fig. 10b) in comparison to the segmented dataset (Fig. 6b), but just up to the moment when the validation loss reaches the local minimum ( Fig. 8b and Fig. 10b), which corresponds to non-overfitted training.

V. DISCUSSION AND CONCLUSIONS
The careful analysis of training and validation values of accuracy and loss (Fig. 11) for images from the segmented dataset and pre-processed datasets with lossless and lossy data augmentation allows us to derive the following observations and conclusions. The training rate is highest for the dataset obtained after lung segmentation (Fig. 11a, red lines), but it can be explained by the evident overtraining (overfitting). It is confirmed by the growth of the validation loss (Fig. 11b, red dash line) and the large difference between training (Fig. 11a, red solid line) and validation (Fig. 11a, red dash line) accuracies, and also by the large difference between training (Fig. 11b, red solid line) and validation (Fig. 11b, red dash line) losses. Despite the fact that the training rates for datasets with both lossless (Fig. 11a, blue solid line) and lossy (Fig. 11a, green solid line) data augmentations are much lower than the training rate for segmented dataset (Fig. 11a, red solid line), the data augmentation allowed us to avoid overfitting. Moreover, the loss values for both datasets with data augmentation are very close (Fig. 11b, green and blue lines) up to the moment when the validation loss reaches the local minimum ( Fig. 8b and Fig. 10b), which corresponds to non-overfitted training. For the dataset with lossless data augmentation the training and validation accuracy values (Fig. 11a, blue lines) become equal at some stage, and validation accuracy reaches the maximum just near the local minimum of the validation loss. But training and validation accuracy values for the dataset with lossy data augmentation never become close (Fig. 11a, green lines) and difference between them always grows during training and validation. This can be considered as a consequence of evident information loss during lossy data augmentation (actually rotation up to 5 degrees). Actually, it allows us to decrease the validation loss, but with decrease of validation accuracy. The different lossy data augmentation techniques can lead to the various positive and negative effects on CADx predictions [15], but the role of the information loss should be investigated more thoroughly. Moreover, influence of the outliers and long tails in lung area distribution can be excluded by naïve dimensionality reduction (due to "internal" bone segmentation) and t-SNE dimensionality reduction method with the significant increase of accuracy even for the very low portion of the filtered our outliers (5%) [6]. In this sense, the more careful exploratory data analysis and preprocessing with filtering out lung groups for people of various ages, genders, and geographical origin will be very promising.
The results obtained here demonstrate that the segmentation and data augmentation can improve the accuracy of prediction of lung diseases (availability of tuberculosis) even for the relatively small datasets like SH (662 images) [9,10]. Recently, the influences of inhomogeneity and outliers in CXR images was investigated for JSRT dataset with CXR images of cancer [5,6]. The pre-processed JSRT dataset obtained after lung segmentation, bone shadow exclusion, and filtering out the outliers by t-SNE demonstrated the highest training rate and best accuracy in comparison to the other pre-processed datasets. The current work on SH dataset with CXR images of tuberculosis shown that the similar lung segmentation technique allow to train (unfortunately with overfitting) deep CNN on such small and not well-balanced dataset like SH (that was impossible for the original SH dataset without segmentation). Moreover, the lossless data augmentation allows us to get the reliable training without overfitting and with steady validation loss decrease and global minimum. The effect of the additional lossy data augmentation (rotation by 5 degrees here) has the same tendency. But influence of lossy data augmentation in the wider sense (with more rotation, scaling, distortion, etc.) on CADx predictions for CXR images with marks of tuberculosis is open question yet and should be investigated along with other lung datasets [15].
The first evident way for improvement is related with increase of the datasets from the current number of 662 images (SH) up to > 100 000 images (ChestXRay dataset). In this sense the progress can be reached by sharing the similar datasets around the world in the spirit of open science, volunteer data collection, processing and computing [16][17][18]. And the second way is based on usage of more complex deep CNNs like CheXNet with >100 layers [4] and their fine tuning that can be very crucial for efficiency of the models used [19][20][21]. But in the context of this paper the better results in CADxrelated research by deep learning could also be reached by: • the more detailed exploratory data analysis with cutting the dataset in the better balanced subsets, for example, to separate lung images of children and adults (which have the very different lung area), to separate them by gender, nationality, age, substance abuse, professional activity, general health state, and other parameters of patients under investigations; • the better "external" lung and "internal" bone segmentation [5]; • exclusion of non-evident outliers, e.g, by dimensionality reduction methods (like t-SNE or other) [6]; • lossless data augmentation by other means (shifts, translations, etc.); • lossy data augmentation which under work now [15].
In conclusion, the results obtained here demonstrate the efficiency of lung segmentation, lossless and lossy data augmentation for CADx of tuberculosis by the deep CNN applied to the small and not well-balanced dataset even. The deep CNN demonstrates ability to train (despite overfitting) on the pre-processed dataset obtained after lung segmentation in contrast to the original not-segmented dataset. Addition of lossless data augmentation to the segmented dataset leads to the lowest validation loss (without overfitting) and nearly the same accuracy (within the limits of standard deviation) in comparison to the original and other pre-processed datasets after lossy data augmentation. The pre-processed dataset obtained after lung segmentation and the limited lossy data augmentation shows the lower validation loss with the lower validation accuracy. But the effect of lossy augmentation should be investigated more thoroughly with regard to the concrete datasets and their heterogeneity [15].

ACKNOWLEDGMENTS
The work was partially supported by Huizhou Science and Technology Bureau and Huizhou University (Huizhou, P.R.China) in the framework of Platform Construction for China-Ukraine Hi-Tech Park Project # 2014C050012001. The manually segmented lung masks were prepared by students and teachers of National Technical University of Ukraine "Igor Sikorsky Kyiv Polytechnic Institute" and can be used as an open science dataset under CC BY-NC-SA 4.0 license (https://www.kaggle.com/yoctoman/shcxr-lung-mask).