{'fraction_non_alphanumeric': 0.02899117235089091, 'fraction_numerical': 0.015309944949346885, 'mean_word_length': 4.8744737849215465, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Going Deeper through the Gleason Scoring Scale: An Automatic end-to-end System for Histology Prostate Grading and Cribriform Pattern Detection

The Gleason scoring system is the primary diagnostic and prognostic tool for prostate cancer. In recent years, with the development of digitisation devices, the use of computer vision techniques for the analysis of biopsies has increased. However, to the best of the authors' knowledge, the development of algorithms to automatically detect individual cribriform patterns belonging to Gleason grade 4 has not yet been studied in the literature. The objective of the work presented in this paper is to develop a deep-learning-based system able to support pathologists in the daily analysis of prostate biopsies. The methodological core of this work is a patch-wise predictive model based on convolutional neural networks able to determine the presence of cancerous patterns. In particular, we train from scratch a simple self-design architecture. The cribriform pattern is detected by retraining the set of filters of the last convolutional layer in the network. From the reconstructed prediction map, we compute the percentage of each Gleason grade in the tissue to feed a multi-layer perceptron which provides a biopsy-level score.mIn our SICAPv2 database, composed of 182 annotated whole slide images, we obtained a Cohen's quadratic kappa of 0.77 in the test set for the patch-level Gleason grading with the proposed architecture trained from scratch. Our results outperform previous ones reported in the literature. Furthermore, this model reaches the level of fine-tuned state-of-the-art architectures in a patient-based four groups cross validation. In the cribriform pattern detection task, we obtained an area under ROC curve of 0.82. Regarding the biopsy Gleason scoring, we achieved a quadratic Cohen's Kappa of 0.81 in the test subset. Shallow CNN architectures trained from scratch outperform current state-of-the-art methods for Gleason grades classification.


Introduction
Worldwide, prostate cancer (PCa) is the second most common cancer in men, with 1.3 million new patients in 2018 [1]. According to the World Health Organisation, the yearly number of new cases will increase by more than 40% in this decade [2]. The main tool to diagnose PCa, once clinical explorations or blood test suggest its presence, is the prostate biopsy. Small portions of the tissue are extracted with a needle, laminated, stained with Hematoxylin and Eosin (H&E) and finally stored in crystal. Then, the sample is analysed under the microscope by the pathologist, determining the presence and grade of cancerous patterns depending on the morphology and organisation of the glands, nuclei and lumen using the Gleason grading system [3]. In this system, different cancer patterns in the tissue are grouped in different grades according to the prognosis of the cancer. In particular, for two-dimensional tissue slides, the Gleason grades (GG) range from 3 to 5, correlating inversely with the degree of gland differentiation of the tissue. The Gleason grade 3 (GG3) includes atrophic well differentiated and dense glandular regions. The GG4 contains cribriform, ill-formed, large-fused and papillary glandular patterns. Finally, GG5 includes isolated cells or file of cells, nests of cells without lumina formation and pseudo-roseting patterns.
Examples of patterns belonging to different grades are presented in Figure 1.
Pathologists classify by visual inspection the tissue regions, detecting the presence of one or more Gleason patterns and, finally, diagnose the combined Gleason score according to the most prominent grades (e.g. the combined grade 5 + 4 = 9 would be assigned to a sample in which the main cancerous Gleason grade is 5 followed by the grade 4). Therefore, the combined Gleason score ranges from 6 to 10, and it is assigned to the whole biopsy. This score is currently the best marker of prostate cancer prognosis and it defines the treatment to apply [4]. However, the Gleason scoring of histological prostate biopsies is a high time-consuming and repetitive task, which has intra and inter pathologist variability. Moreover, after the last International Society of Urological Pathology (ISUP) Consensus Conference in 2014 [5], new guidelines have been included that increase the pathologists' workload. In particular, it is recommended to also report the percentage of Gleason grade 4 in the sample, mainly for regions scored as 3 + 4 = 7, where a higher percentage of Gleason grade 4 indicates the 4 convenience of an earlier treatment [6], and the presence of cribriform glandular patterns, which indicate worse prognosis than the presence of other Gleason grade 4 patterns [7,8]. Computer-Aided Diagnosis systems (CAD) support the work of pathologists and increase the objectivity in the this process. These are based on the digitisation of the histological crystals, obtaining whole slide images (WSIs) and developing computer vision algorithms to detect the cancerous regions inside the biopsy (or WSI).
Computer vision algorithms have been widely used to analyse histological PCa images. This section summarises the works previously presented in the CADs literature for prostate cancer detection, classifying them according to three factors: the kind of images included in the analysed database, the objectives addressed by CAD systems, and the techniques proposed to achieve them.
Regarding the images, mainly three types of histological images have been used: WSIs, prostactetomies and Tissue Micro Arrays (TMAs). TMAs are clusters of representative tumor areas extracted manually by pathologists [9].
TMAs are used for testing new techniques in a large number of different tumour samples. One of the main limitations of TMAs lies in the small amount of tissue that can be included in each samples, which may not be representative of the whole tumor region in epithelial tumors with heterogeneous patterns [10]. This is the case of prostate cancer, which has different patterns for each Gleason grade, as previously mentioned. Non-cancerous patterns that could confuse CAD systems, as the inflamed tissue or benign multi-nucleation, could be lost using TMAs. Thus, the strategy based on TMA analysis is not used in clinical practice [11] and it is more convenient to develop CAD systems based on raw WSI analysis. A model trained using large databases of WSIs could be used for both WSIs and prostactetomies. The works in [12,13,14,15,16,17,18] follow the strategy of WSI analysis, while in [19,20,21] the authors use TMAs to develop the CAD models.

5
With regard to the objectives to be addressed, some works focus just on the detection of prostate cancer against non-cancerous tissue [13,17] or on the first-stage prostate cancer detection [22]. A full analysis of Gleason grades from 3 to 5 is usually limited by the size of the collected database, and the low prevalence of Gleason  Finally, concerning the techniques used to deal with the different mentioned objectives, the most common approach to analysed both is to perform a patchbased strategy (see Figure 2). The motivation for using this strategy is the large size of both TMAs and, especially WSIs, together with hardware limitations.  In this case, a precision of 0.10 is reported in this work. In addition, the primary and secondary grades are not just related to the proportion of the different grades in the tissue, but also to the severity of each grade (e.g. GG5 could be diagnosed as secondary grade even having less proportion than GG4 or GG3 in 7 the tissue).
The objective of this work is to develop an automatic Computer-Aided Diagnosis system working on WSIs and able to support pathologists in the analysis of the biopsy during the diagnosis process. The tasks of this analysis, to be included in the pathologists' report, are: • Detection of the cancerous regions in the tissue according to the Gleason grading system.
• Calculation of the percentage of each Gleason grade in the biopsy.
• Gleason scoring of the whole biopsy, taking into account not only the grade proportion but also its severity.
This work is developed using our collected database SICAPv2, the largest public database of prostate biopsies with pixel-level annotations of Gleason grades, specifying the presence of cribriform patterns. In the following lines, we summarise the main contributions of this paper. The different blocks of our system are presented in Figure 3. First, we develop a patch-level predictor of

Materials: SICAP database
The database presented in this paper, SICAPv2, is, to the best of the authors's knowledge, the largest public collection of prostate H&E biopsies with local-level annotations of Gleason grades. SICAPv2 is an extension the database introduced in [17] and will be publicy available after the publication of this paper.
After analysing the literature, four main prostate cancer tissue image databases were found. The largest database with prostate biopsies was released by The Cancer Genome Atlas project 1 [25] with up to 720 prostate biopsy slides. Nevertheless, the lack of annotations at both the local and biopsy levels of the Gleason grades restricts the use of these data. The database shared by Arvaniti et al. [19] includes pixel-level annotations of Gleason patterns from 886 small regions of slides (cores of TMAs). Unfortunately, as discussed earlier, those cores do not represent the heterogeneous patterns of local structures of prostate cancer and benign lesions, so they lack clinical relevance for the slide-level Gleason score diagnosis. Similar limitations are found in the recent database from the challenge Gleason19 in the MICCAI 2019 conference 2 , with 331 cores annotated by different pathologists, and the dataset used in [16], composed by 625 isolated patches. Although those databases contribute to the validation of different algorithms, the lack of large databases with clinical reference of heterogeneous patterns has been a limiting factor for the scientific community to develop deep-learning-based methods which demand a large amount of data.  The local cancerous patterns were annotated using an in-house software based on the OpenSeadragon libraries [26], following the Gleason scale and indicating the presence of cribriform glandular structures. In order to process the large WSIs, these were down-sampled to 10x resolution and divided into patches of size 512 2 and overlap of 50% between them. Those values were previously optimised for the detection of cancerous patterns in [17]. A mask of the presence of tissue in the patches was obtained by applying the Otsu threshold method.
To develop the model able to predict the main Gleason grade, patches with less than 20% of tissue were excluded. In addition, patches without cancerous patterns annotated by the pathologists belonging to cancerous biopsies where also discarded. After this procedure, the database contains 4417 non-cancerous patches, 1635 labelled as GG3, 3622 as GG4, and 665 as GG5. Note that if one patched contained more than one annotated grade, the majority grade was assigned as label. 763 GG4 patches also contain annotated cribriform glandular regions. A summary of the database description is presented in Table 1. The data collected by Arvaniti et al. [19] was also utilised to validate the models produced in our study. The cores were resized to match the resolution used in our models and patched to the dimensions used in our database. By this approach, each one of these cores is approximately equivalent to one of our patches. Thus, 115 non-cancerous images, 274 patches labelled as GG3, 210 GG4, and 104 GG5 were used to validate our work in an external database.
Also, the patches shared by Gerytch et al. [13] were used in our work for the validation of our proposed model. After normalisation of the images to match our methodology, 32 non-cancerous images, 95 patches labelled as GG3, 216 GG4, and 70 GG5 were obtained.

Patch-Level Gleason Grading
The patch-level classification in the different Gleason grades is carried out by means of convolutional neural networks. We propose a self-designed base-model architecture (from now on called F SConv) which consists of a simple convo-lutional architecture with three convolutional layers and dimensional reduction operation employing max-pooling layers ( Table 2).
After the automatic feature extraction blocks (base model), we introduce as top model a global-max-pooling layer. To show the superior performance of this architecture, different configurations already applied in the literature to the same problem, have been also tested as top models and are described next.
One of the main approaches is the flattening of the activation volume resulting from the final convolutional block and the class prediction through consecutive fully-connected layers. In this case, overfitting is addressed by means of a random dropout of a percentage of the neurons in each training iteration.
Nevertheless, these top-model architectures include a large number of parameters to optimise, increasing the complexity of the model, and they are sensitive to the location of the structures in the image. This problem is usually dealt with data augmentation techniques, applying, for example, random rotations and translations to the images. Other approaches propose the convenience of using global-average pooling on the last feature maps as regulariser to make the model translation-invariant and decrease its complexity [27]. This technique is used in [19] for the prediction of prostate cancer Gleason degree with fine-tuned models. Due to the use of a patch-based strategy with sliding window, the location and amount of the cancerous structures in the image is not controlled.
Thus, as shown in Figure   For comparison, together with the proposed architecture trained from scratch, we fine-tuned several well-known architectures: VGG19 [28], ResNet-50 [29], In-ceptionV3 [30] and MobileNetV2 [31]. All of them were pre-trained in the Imagenet data set [32]. For the feature extraction stage, the base model from those pre-trained models is extracted and partially retrained. This strategy is usually used to transfer the knowledge obtained in extracting features from a large database to specific domains where the amount of data is limited. Nevertheless, the patterns of the images used during the training are very different from the histology ones. To keep just the low-level features (contours, combination of basic colours, general shapes, etc.) from the pre-trained models, the weights of just the first convolutional blocks are frozen, while the rest are re-trained to adapt the model to the specific application. The layer from which the freezing strategy is applied is empirically optimised for each model, and is specified in the experimental part of the paper, in Section 4.2.
The output layer for all the different configurations is composed of one neuron per class with soft-max activation function to obtain the final probability per class. In the training process, we use categorical cross-entropy as loss function, modified to deal with the class imbalance in the training set as follows: where y and y contain the one-hot-encoded reference labels and predicted prob-

Cribriform Pattern Detection
The detection of cribriform structures in GG4 patches is also carried out using convolutional neural networks. Due to the complexity of the task and the

Experiments
In this section we present the results of the different experiments carried out to show the performance of the proposed approach for the different classification tasks: patch-level classification, cribriform pattern detection and WSI scoring.
In all cases, when possible, we also present a comparison with current state of the art methods and discuss the obtained results.

Database Partitioning and Metrics
In order to train the models and optimise the hyperparameters involved in this process, the database was divided following a cross-validation strategy. In particular, each patient was exclusively assigned to one fold with the aim of avoiding overestimation of the performance of the system [21] and ensuring its ability of generalisation. Thus, the database was divided into 5 groups containing approximately 20% of the patches each one. Notice that this process was carried out trying to guarantee the class balance character between sets. A summary of the resulting partition is presented in Table 3. Notice that four of the five sets were used to tune the hyper-parameters involved in the developed algorithms while the remaining partition was employed to test the final predictive system. For the evaluation of the patch-level Gleason grade prediction, a cross-validation strategy was used with the four validation cohorts, while for the WSI-level prediction of Gleason scores those sets were joined to apply a leave-one-out strategy per patient in training.
In order to objectively evaluate the performance of the trained models the following metrics were used: accuracy, F1-score, and Cohen's quadratic kappa Cohewhere c indicates the predicted classes.
However, an automatic method should be less penalised when classifying a GG5 tissue as GG4 than as NC, even more so when taking into account the inter and intra-observer variability. In the literature, this fact is addressed using the Cohen's quadratic kappa (κ) metric [33]. The metric κ ranges from −1 to 1, being directly proportional to the level of agreement between observers (-1 no agreement, 1 total agreement). Although there is not objective interpretation of which are the reasonable values for κ in medical applications, recent proposals [34] define a moderate agreement if κ is higher than 0.6, while a strong agreement is stated when κ is higher than 0.8.
The patch-level Gleason grading models are evaluated using all the aforementioned figures of merit.
In order to evaluate the system for the detection of cribriform patterns, the area under the Receiver Operating Characteristic (ROC) curve (AU C) was used. In medical applications, a system is considered reliable if the AU C value exceeds 0.80 [35]. The predicted labels are obtained by thresholding the scores (cribriform if the probability is above 0.5), and then evaluated by means of ACC, sensitivity and specificity.
Regarding the evaluation of the WSI-level Gleason scoring, the Cohen's quadratic kappa was used.

Patch-Level Gleason Grading
In the case of the patch-level Gleason grading model, in this section besides the obtained results using SICAPv2 database, we also discuss its performance in an external database.   Regarding the results obtained in the fine-tuned models, the use of architectures with residual blocks provided slightly worse results than the sequential approach, similarly as the previous results reported in the literature where sequential models used to outperform residual ones [17,19,20]. In relation to the use of different top models, no differences were found in the accuracy of the fine-tuned architectures, observing similar results for all of them. In relation to F SConv architecture, interesting results were obtained while testing the use of different top models. The best performing architecture to validate the system is the one with global-max pooling, F SConv+GMP. The outperforming of the global-max pooling compared to the fully-connected configuration could be explained by the reduction in the number of weights to be optimised (see Table 5 Table 4). This is the first time in the literature that self-defined architectures trained from scratch outperform fine-tuned architectures from the state-of-the-art pre-trained in Imagenet for Gleason grading. Moreover, the reduced amount of parameters (2 × 10 7 in the VGG19+GMP+FC model against 6 × 10 5 in the F SConv+GMP model, see Table 5), makes more convenient the F SConv architecture for deployment. Thus, the model F SConv+GMP was trained using all the images in the crossvalidation sets in order to evaluate its performance in the external test cohort.

FSConv Architecture Benchmarking
The results of the proposed model for the test set and a comparison of them with previous state-of-the-art works are reported in Table 6. κ value increases up to 0.77 in the test subset for F SConv+GMP. In comparison with previous studies, our results outperform the state of the art, obtaining almost a strong agreement between our model and the pathologist, while just moderate agreement (κ = 0.55 [19]) was obtained previously in the test set. Figure 7 shows the performance evaluation of F SConv. In particular, the confusion matrix for validation and test subsets are presented. From this figure, it can be observed that most of the errors occur between adjacent classes.

Model Interpretation
One of the main drawbacks of deep learning models in medical practice is the lack of interpretability. This fact creates distrust in the clinicians, the final users of CAD systems. To deal with this problem, in this research we study the interpretability of the trained models by means of the Class Activation Maps technique (CAMs). Both VGG19+GMP+FC (the best fine-tuned model) and F SConv+GMP models are compared in this section using CAMs.
This technique was proposed in [36] as a procedure to obtain a heatmap indicating the regions of the input image to which the model is paying attention to predict certain class. CAMs for both models are obtained for images correctly classified (see Figure 8) and for images miss-classified by the VGG19 model (see Figure 9). These illustrations are organised as follows: the first row corresponds to the original patch, and the second and third rows show the CAMs for VGG19 and F SConv models, respectively. In Figure 8 each column shows an example per class: NC, GG3, GG4 and GG5 accordingly. The main difference in the results obtained by VGG19 and F SConv is the best differentiation between GG3 and GG4 by the second model (see Table 4), the most difficult task in the pathologists' work. In Figure 9 three of those cases are presented in each column: two cases predicted by the VGG19 as GG3 and one as GG5, respectively. Those cases were correctly classified as GG4 by F SConv model. we can extract interesting insights (see Figure 8). In the case of GG4, the map is activated in a small nest belonging to a fused-glands structure with irregular cribriform shape. Regarding the GG3 image, the dot indicates thick cytoplasm in different medium-sized tubular glands. In the image marked as GG5, the CAM highlights single isolated cells with hyperchromasia. Less interpretable is the CAM obtained in the NC image, where any gland is detected. We speculate that the model carries out this classification by dismissing the presence of cancer- ous patterns. Regarding the cases where VGG19 miss-classifies GG4 in Figure   9, a correct detection of the regions of interest is observed. However, these glandular regions are not correctly classified as GG4, while F SConv model does it just paying attention to closed lumens in small ill-formed glands. At this stage of understanding, we believe that this fact is the cause of the different performance by both models. VGG19 focuses the prostate cancer detection on detecting epithelial and glandular regions, and these structures present a larger heterogeneity than its basic components (colour and size of individual glands, diameter and opening degree of lumens in the glandular region, etc.). This could be the reason why the VGG19 generalises slightly worse than F SConv.

Validation on External Databases
With the purpose of testing the generalization capability of the trianed model, F SConv net was validated on two external databases. The databases used were shared by Arvaniti et al. [19] and Gerytch et al. [13]. The first database is composed of 886 cores from Tissue-Micro Arrays digitised at 40× magnification, and the second has 625 patches of prostate histology images at 20× magnification. Each core was resized to 10× resolution and a central patch with dimensions 512 2 was extracted. For both databases, the ground truth was generated following the procedure in [19]. Non-cancerous patches were ex-tracted from images with only benign structures annotated, labels GG3, GG4, and GG5 were assigned to patches with only the corresponding grade annotated.
Examples of the obtained images from the Arvaniti et al. and Gerytch et al.
databases are presented in the first and second rows of Figure 10, respectively.
Note that the H&E stain color images are different from those appearing in the SICAPv2 database (see Figure 1 for examples of the images used to train the developed models). To normalise the colour distribution of the images in external databases, the method presented in [37] was used after applying a channel-wise histogram matching of the external images to a SICAPv2 database reference image. This image was selected by the expert pathologists involved in this work based on its structural and colour properties. Then, our best performing model, i.e. F SConv, was used to predict and evaluate our performance on the external databases. Table 7 and Figure 11 show the obtained figures of merit and confusion matrices, respectively.   The obtained results in Arvaniti et al. database were slightly worse than the ones reached in our test cohort. The macro-averaged F1 score was 0.57, while 0.65 was obtained in the test cohort (see Table 6). To the best of the authors's knowledge, this is the first time in the literature that a model trained for patchlevel Gleason grading in tested on an external database. This is a challenging task, taking into account the known inter-pathologist variability of the Gleason grading task and the differences in the histology sample preparation. Thus, the difference in the results could be explained by those factors. In comparison to the results obtained in [19] on this database, the reported κ in the test subset was 0.55 (see Table 6), while the κ obtained by our model was 0.64. Our Concretely, at filters conv 1 , conv 2 and conv 3 (see Table 2 for F SConv architecture details). The output probability of each model was used to compute the Receiver Operative Curve (ROC) and evaluate the Area Under Curve (AUC).
Then, probabilities were thresholded to output a positive classification when they are above 50%. The results obtained for the cross-validation set are presented in Table 8, and the Receiver-Operative-Curve in Figure 12 (a). The best results were obtained for the validation set by the network whose weights were frozen up to the layer conv 2 . Thus, just the last layer, conv 3 and the output neuron were trained. The accuracy obtained through this configuration was 0.8225, with a sensitivity and specificity of 0.8993 and 0.5223, respectively. The reached AUC was 0.8225. Slightly better results were obtained by this model in the test subset. The ROC computed in the test subset is presented in Figure 12 (b), and it encloses an AUC of 0.8240. This value is at the permissible confidence level of systems for medical applications, above 0.80 [35]. Although the accuracy value decreases to 0.7239, the sensitivity and specificity are more balanced, with values 0.7168 and 0.7586, respectively. To the best of the authors's knowledge, this is the first time that the detection of cribriform patterns in histology prostate images is addressed and evaluated, so that it is not possible to establish comparison with previous works. Nevertheless, the studies comparing the inter-observer variability of the Gleason patterns classification show the challenging character of this task. In [38] the reproducibility in this problem was studied with 23 genitourinary pathologists.
The consensus was achieved for cribriform glands in only 23% of the cases, and a consensus was not reached in how to classify the complex fused glands with cribriform shapes. We observed that the misclassified instances in our approach were mainly due to this kind of pattern. In Figure 13 few representative examples are presented, being (d), (e), and (f) images with complex fused glands that the model misclassified as cribriform pattern. Therefore, the results obtained by the model are auspicious, and its main limitation is the misclassification of patterns with large inter-pathologist variability.

WSI-Level Gleason Scoring
Once the patch-level prediction is performed with model F SConv, the probability maps for each Gleason grade are obtained, as specified in the subsec- higher resolution window to facilitate visualisation. The example in Figure 14 is a biopsy with Gleason score 3 + 4 = 7, the biopsy in Figure 15 corresponds to a 3 + 3 = 6 sample and the case in Figure 16, 5 + 5 = 10. Finally, a non-cancerous case is presented in Figure 17. In the case presented with Gleason score 3 + 4 = 7 (see Figure 14), the GG3 and GG4 regions are correctly classified. In a subsequent review of this case, pathologists detected that some glands in the right region without pathologist's annotations in the ground truth and classified as GG3 by the model were  This proposed approach is compared with the method proposed by Arvaniti [19] using T = 10% as minimum number of pixels with a certain label to be consider the corresponding grade in the WSI grading. The confusion matrix at biopsy level obtained for both methods is presented in Figure 18, and Cohen's quadratic kappa (κ) was calculated as a figure of merit.

Conclusions and future work
In this work, we have proposed and validated end-to-end approaches to automatically support the pathologists analysis of prostate whole slide images.
This support includes the pixel-level prediction of Gleason grades, cribriform patterns detection, calculation of the percentage of each grade in the tissue and finally the scoring of the entire biopsy.
We have compared fine-tuned state-of-the-art architectures and self-designed convolutional neural network architectures trained from scratch for the patch- It is important to note that this work brings an important contribution to the scientific community: the SICAPv2 database, the largest public database containing pixel-level annotations of prostate biopsies.
Further research will focus on developing convolutional-neural-network architectures that combine low and high-level features in the classification stage, as well as the inclusion in those models the prediction of all the individual cancerous patterns (i.e. ill-fused, papillary or large-fused) as the cribriform one, in an end-to-end training. Furthermore, the SICAPv2 database will be enlarged with additional annotated whole slide images.