{'fraction_non_alphanumeric': 0.030005733579664903, 'fraction_numerical': 0.011690131872332293, 'mean_word_length': 5.145038167938932, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learn like a Pathologist: Curriculum Learning by Annotator Agreement for Histopathology Image Classification

Applying curriculum learning requires both a range of difficulty in data and a method for determining the difficulty of examples. In many tasks, however, satisfying these requirements can be a formidable challenge. In this paper, we contend that histopathology image classification is a compelling use case for curriculum learning. Based on the nature of histopathology images, a range of difficulty inherently exists among examples, and, since medical datasets are often labeled by multiple annotators, annotator agreement can be used as a natural proxy for the difficulty of a given example. Hence, we propose a simple curriculum learning method that trains on progressively-harder images as determined by annotator agreement. We evaluate our hypothesis on the challenging and clinically-important task of colorectal polyp classification. Whereas vanilla training achieves an AUC of 83.7% for this task, a model trained with our proposed curriculum learning approach achieves an AUC of 88.2%, an improvement of 4.5%. Our work aims to inspire researchers to think more creatively and rigorously when choosing contexts for applying curriculum learning.


Introduction
Curriculum learning [1] is an elegant idea inspired by human learning that proposes that neural networks should be trained on examples in a specified order based on difficulty (typically easy to hard), as opposed to the random ordering that is currently common in practice. As such, curriculum learning requires both that there exists some range of difficulty among training examples and that we define a method for ranking examples. In most cases, however, it is unclear whether a range of difficulty exists among the examples, and even when a range of difficulty exists, an ideal ranking function is rarely available. In this paper, we try to answer the question: are there tasks with domain-specific properties that are naturally appropriate for addressing these challenges of curriculum learning? Interest in using deep learning to analyze histopathology images (stained tissues and cells that are typically manually examined under a microscope) has increased in recent years, with neural networks achieving pathologist-level performance on a variety of tasks [2][3][4][5][6][7][8]. In this paper, we propose that histopathology image analysis is a suitable scenario for curriculum learning for two reasons. First, due to the nature of histopathology images, we know that a range of difficulty in examples exists for many tasks. Second, medical image datasets typically have annotations from multiple clinicians-these annotations can be leveraged as a natural proxy for ranking example difficulty. Specifically, our paper makes the following contributions: 1. We contend that histopathology image classification is a natural scenario for applying curriculum learning, and we propose a curriculum learning approach that leverages annotator agreement as a proxy for difficulty. 2. We evaluate our proposed approach on a colorectal polyp classification dataset, for which a baseline model achieved an AUC of 83.7% and our best single-stage baseline achieved an AUC of 84.6%. When trained with curriculum learning, our model's AUC improves to 88.2%, outperforming the average pathologist annotator on our test set in terms of Cohen's κ [9].
The rest of our paper is outlined as follows. §2 analyzes the challenges of curriculum learning and presents our intuitions on why histopathology image classification is a suitable context for curriculum learning. §3 describes our task and dataset. §4 presents the main results of our proposed curriculum learning approach. §5 compares the value of annotator agreement with two previously-proposed methods of scoring difficulty. §6 shows how the increase in AUC from curriculum learning translates to improvements in performance relative to pathologist performance. §7 discusses the implications of our work. §8 puts our study in the context of prior work on curriculum learning for medical imaging and concludes our paper.

Curriculum Learning Intuitions
Curriculum learning. One of the earliest works demonstrating the benefit of curriculum learning [1] posits that learning occurs better when examples are not randomly presented but instead organized in a meaningful order that gradually shows more concepts and complexity. Although the intuition behind this approach seems obvious in the context of human and animal learning, it is often unclear how to best apply this strategy for training neural networks. As such, a diverse set of approaches has been explored in this area of research. These approaches generally first score examples by difficulty and then train models using a schedule based on example difficulty, where easier examples are typically seen first and harder examples are seen later. For instance, Bengio et al.'s original work [1] explored a noising-based curriculum for shape detection and a vocabulary-size based task for language modeling. As popular recent examples, Weinshall et al. [10] use the confidence of a pre-trained classifier as an estimator for difficulty; Korbar et al. [11] use a schedule with self-defined easy and hard examples for learning of audio-visual temporal synchronization; Ganesh and Corso [12] propose to incrementally learn labels instead of learning difficult examples; and various teacher-student frameworks have been proposed in the context of curriculum learning [13,14].
Challenges of curriculum learning. Despite the appeal of teaching machines to learn like humans, curriculum learn-ing has been seen by some [10] as mostly remaining in the fringes of machine learning research. Based on the strategies of prior work, we broadly see two central challenges that arise when applying curriculum learning.
First, curriculum learning assumes that a range of easy and hard examples exists. Although it could be argued that this is a true statement for any given dataset for at least some definition of easy and hard, the distribution of example difficulties likely varies based on the nature of the task and the dataset. Since the added value of curriculum learning comes from utilizing the varying degrees of difficulty in a task, tasks with a smaller range of example difficulty are less conducive to effective curriculum learning. Empirically, Weinshall et al., present some evidence related to this claim, showing that curriculum learning had a larger improvement compared with regular training when applied to tasks that were more difficult and likely included challenging examples (e.g., distinguishing small mammals in CIFAR-100), than when applied to tasks that were easier and did not have examples that were difficult to classify (e.g., discriminating between 5 well-separated classes in CIFAR-100) [10].
Second, a curriculum learning approach must somehow categorize examples as easy or hard. Prior work has tried to address this challenge in many ways, including trying to discover inherent patterns in the data [10], using handpicked heuristics [15], and creating custom training progressions [11]. Though sometimes effective, these methods can be difficult to implement, and it is often unclear whether an approach that works on one dataset will also work on another. Indeed, scoring images by difficulty is often the core problem addressed in many curriculum learning papers.
Our intuitions. We contend that histopathology image classification is an important task that naturally addresses the challenges above-and could benefit from curriculum learning-based on the following two observations: • A range of example difficulty exists in many histopathology image classification tasks. We believe this to be true for several domain-specific reasons. (1) Because pathological disease develops over time, there is a progression from normal tissue to diseased tissue. Since many diseases are classified into discrete classes, there must be some points in this progression that lie on the margins of two classes and are therefore hard to diagnose.
(2) Pathology residents learn to read images by first studying classic examples of diseases and then learning to diagnose more-challenging cases over time, implying that human instructors acknowledge some notion of easy and hard examples [16]. (3) Inter-annotator agreement is moderate or low on many disease classification tasks [7,17], suggesting that some images are hard to classify. Knowing that a curriculum exists is the first step to applying curriculum learning.
• Image-level annotator agreement can be leveraged as a proxy for example difficulty. As medical image datasets are commonly annotated by several trained clinicians, we can leverage the extent of agreement for each image as a proxy for the difficulty of that image. By definition, images with high agreement are easy to classify, as everyone agrees on them, and images with lower agreement are harder to classify. If these human notions of difficulty translate to a helpful curriculum for training neural networks, then many tasks with annotator agreement data already contain a curriculum that can be used to improve model performance. [17][18][19][20][21][22][23][24][25][26][27][28][29].

Histopathology Dataset 1
In this paper, we focus on the task of colorectal polyp classification, a challenging and clinically-important task in pathology. As shown in Table 1, our dataset contains 3,152 images in total, each annotated with a binary label of either hyperplastic polyp (HP) or sessile serrated adenoma (SSA).
Colorectal polyp classification task. Colonoscopy is a common screening program in the United States [30], and so classification of colorectal polyps (growths inside the colon lining that can lead to colonic cancer if left untreated) is one of the highest-volume tasks in pathology. Our task focuses on the clinically-important binary distinction between hyperplastic polyps (HPs) and sessile serrated adenomas (SSAs), a challenging problem [31][32][33][34][35]. Pathologically, SSAs are characterized by broad-based crypts, often with complex structure and heavy serration [36].
Data collection and annotation. For our data collection, we scanned 328 Formalin Fixed Paraffin-Embedded (FFPE) whole-slide images of colorectal polyps, which were originally diagnosed as either hyperplastic polyps (HPs) or sessile serrated adenomas (SSAs), from patients at our tertiary medical institution. From these 328 whole-slide images, we then extracted 3,152 patches (image portions of size 224 × 224 pixels) representing diagnostically-relevant regions of interest for HPs or SSAs. The seven practicing boardcertified gastrointestinal pathologists at our tertiary institution then independently labeled each of the 3,152 images in our dataset as either HP or SSA.
Train-test split and gold-standard labels. Images were split randomly by whole-slide such that images from the same whole-slide either all went into the training set or all went into the testing set. As shown in Table 1, we used a training set of 2,175 images (∼70% of images) and a testing set of 977 images (∼30% of images). In our testing set, we use the majority vote of labels as the gold-standard label, a common choice in the literature [18, 23-29, 37, 38]. 1 We plan to make our dataset and annotations publicly available to facilitate further research.  - Table 2: Pairwise annotator agreement (%) for our seven annotators (indexed as A1, A2, A3, A4, A5, A6, and A7).    Figure 4 shows examples for each level of agreement. On average, each pair of pathologists had an average agreement of 72.9%, and each pathologist agreed with the majority vote 83.2% of the time, indicating that our task has non-neglible disagreement, even among pathologist annotators who all specialize in gastroenterology.

Curriculum learning: annotator agreement
We propose a curriculum learning framework that leverages annotator agreement to rank images by difficulty. Specifically, we define images with high annotator agreement to be easy and images with low annotator agreement to be hard. For our dataset, which was labeled by seven annotators, we partition our images into four discrete levels of difficulty: very easy (7/7 agreement among annotators), easy (6/7 agreement among annotators), hard (5/7 agreement among annotators), and very hard (4/7 agreement among annotators). An overview schematic of this setup can be seen in Figure 1. For our training schedule, we train our network on progressively harder images in four stages: • Stage 1: Very easy images only • Stage 2: Very easy + easy images • Stage 3: Very easy + easy + hard images • Stage 4: Very easy + easy + hard + very hard images At each stage, we make sure to include images from the previous stages to prevent catastrophic forgetting [39]. Though our current model uses four levels of difficulty and four stages of training, our general framework could be used in any scenario where annotator agreement data is available.
Experimental Setup. For our model, we use ResNet-18, a common choice for classifying histopathology images. Specifically, we follow the DeepSlide repository [7] for histopathology image classification, training our model for 50 epochs (well past convergence) using stochastic data augmentation with the Adam optimizer [40], initial learning rate of 1 × 10 −3 , and learning rate decay factor of 0.91.
For more-robust evaluation, for each model we consider the five highest AUCs on the test set, which are evaluated at every epoch. We report the mean and standard deviation of these values calculated over 20 different random seeds.
Baselines. Our primary baseline is the vanilla-training model where all training images are given a label determined by the majority vote of annotator labels, a common gold-standard in the literature [18, 23-29, 37, 38]. We also explore variations of single-stage training in which only certain images, selected based on annotator agreement, are used for training. For all experiments, our test set is fixed and contains images from all levels of difficulty, although stratified analyses are also presented.
As shown in the first block of Table 3, our vanilla baseline that uses all images achieves an AUC of 83.7%. Interestingly, the network trained on only very easy and easy images achieved an overall AUC of 84.6%, an almost 1% improvement over the vanilla baseline. As shown by the performances on the test set when stratified by difficulty, this network trained on very easy and easy images does better on very easy and easy images in the test set while doing worse on hard and very hard images, leading to a higher overall AUC because there are more very easy and easy images than hard and very hard image in the testing set.
Annotator agreement-based curriculum learning. The second block of Table 3 Table 3: Histopathology image classification model trained using a curriculum learning framework outperforms single-stage training baselines by an AUC of 3.6-4.5%. Image difficulty is determined by annotator agreement in four discrete categories: very easy (7/7 annotator agreement), easy (6/7 agreement), hard (5/7 agreement), and very hard (4/7 agreement). ∆ + indicates the level of statistical significance in improvement over the vanilla baseline of training with all images: * indicates p ≤ 0.05; *** indicates p ≤ 0.001. Means and standard deviations shown are for 20 random seeds.
curriculum learning scheme at each stage of training. In this curriculum learning scheme, the model already outperforms all single-stage models at the second stage, achieving an AUC of 85.5%, and at the third stage, achieves an AUC of 88.2%, the highest of any model we train. Moreover, this model also achieved the highest performance when stratified by very easy, easy, and hard images in the test set, outperforming earlier stages that trained only on very easy and easy examples. These results suggest that training on harder images in a curriculum framework not only improves performance on hard images, but also improves performance on easy images, a finding consistent with Korbar et al. [11].
Perhaps strikingly, model performance actually decreases in the fourth stage of training that includes very hard examples, as performance on very hard images in the test set increases but performance on other images in the test set decreases. One explanation for this slight dip in performance is that very hard images, which have only 4/7 pathol-ogist agreement, could be too challenging to analyze accurately (even for expert humans), so their features might not be beneficial for training machine learning models either.
In terms of statistical significance, our curriculum learning model at the second, third, and fourth stages outperforms the vanilla-training model with p ≤ 0.001, based on a two-sample t-test for means.
Anti-curriculum learning. To further validate that the improvement in model performance is indeed a result of the intentionally selected images at each stage, we train a model using an anti-curriculum scheme, which reverses the learning schedule (i.e., the model first trains on hard images and then trains on progressively easier images). As shown in the third block in Table 3, no models trained using the anticurriculum framework outperform the vanilla baseline.
Visualization. For a qualitative examination of how the model changes throughout training, we compute GradCAM heatmaps [41] to visualize the model's predictions at each of the four stages of curriculum learning. In Figure 5, we show examples of SSA images where curriculum learning was both successful and unsuccessful, as subjectively examined by our pathologists. In the successful examples, the model seemed to focus on broad-based crypts (a defining characteristic of SSAs) more heavily in Stage 3 of curriculum training, our best-performing model. In the unsuccessful examples, on the other hand, the model seemed to focus on broad-based crypts more heavily in earlier stages.
Other curriculum baselines. As a further baseline, we also asked a pathologist to directly score the difficulty of each image on a scale from 1-4, with 1 as very easy to classify and 4 as very hard to classify. We also ran a four-stage experiment using this difficulty measure, as shown in the fourth block of Table 3, but find that curriculum learning here does not improve performance, possibly because the manual difficulty scores from a single pathologist are too subjective. Moreover, we tested a control curriculum with the same training scheme as the annotator agreement experiment, except images in each stage were selected randomly (fifth block of Table 3). As expected, this control curriculum performed about the same as our vanilla baseline.

Annotator agreement vs model confidence
This section compares various proxies for difficulty in terms of their usefulness for curriculum learning. Whereas our model so far has used annotator agreement as a proxy for example difficulty, prior work has proposed that the output confidence of a machine learning classifier can also be a proxy for example difficulty [10,42]. First, we perform a sanity check to see whether classifier output confidence correlates with annotator agreement. As we might expect, they do-predicted confidence distributions of a model pretrained on ImageNet and fine-tuned on our dataset appeared substantially different for different levels of annotator agreement ( Figure 8 in the Supplementary Materials).
We conduct a simple ablation study to compare annotator agreement and model output confidence as proxies for image difficulty in curriculum learning. For simplicity, we use a two-step curriculum learning scheme-where training is done in one stage containing only easy images and a following stage containing a mixture of easy and hard images-and use a single-step pacing schedule [42].
We evaluate the following three proxies for difficulty: 1. Self-taught scoring, where the classifier with randomly initialized weights is pre-trained on our dataset, and output confidences are used to sort examples by difficulty. 2. Transfer learning, where a classifier pre-trained on Im-ageNet is fine-tuned on our dataset, and output confidences are used to sort examples by difficulty. 3. Coarse annotator agreement, a simplified version of our curriculum learning scheme above, where images are divided into two categories of either easy (6/7 or 7/7 agreement) or hard (4/7 or 5/7 agreement), instead of the four categories used in §4.
The self-taught scoring and transfer learning proxies assign each image with a confidence score: images with confidence score greater than a threshold τ are classified as easy and images with confidence score less than τ are classified as hard. We choose τ such that the proportion of easy and hard images was the same as the natural distribution of easy and hard images from coarse annotator agreement. Then, a new classifier is trained in two stages: (1) easy images only, and (2) easy images and hard images combined at various ratios. Our coarse annotator agreement method in this section follows this same training scheme of two stages. Figure 6 shows the results for the various ratios of hard images we used in the second stage of training. Without making any general claims, we see that on our dataset, annotator agreement appears to be a more useful proxy for difficulty than model output confidence. One possible explanation for this result is that although the transfer learning and self-taught scoring approaches work (marginally, in our case), much of the information provided by the pre-trained classifier is shared with the resulting classifier (i.e., much of Single-Stage Baseline Annotator Agreement (Ours) Self-Taught Scoring Transfer Learning Figure 6: Performance of curriculum learning models that use proxies of transfer learning [42], self-taught scoring [42], and annotator agreement (ours). For all tested ratios of hard images in the second stage of training, curriculum learning by annotator agreement outperforms transfer learning and self-taught scoring.
the added value of the pre-trained classifier can already be discovered by the resulting classifier), whereas using annotator agreement as a proxy provides new information about difficulty that the model would not have had access to otherwise. Moreover, we find that including a greater proportion of easy images than hard images in the second stage of training is important for preventing catastrophic forgetting, a finding consistent with prior work [10,11,42]. Using only hard images in the second stage was especially problematic for the self-taught scoring and transfer learning models, possibly because the images most challenging for these proxy models to learn will also be challenging for the learner model to optimize.

Comparison with Pathologist Performance
For more context on the significance of the improvement that curriculum learning brings to model performance, in this section, we compare the performance of our models with that of pathologists. Specifically, we frame the predictions of each model as the annotations of an additional pathologist, and we compare these predictions with the annotations of actual pathologists in terms of Cohen's κ [9], a common measure of inter-annotator agreement.
For our models, we select the best-performing curriculum learning model (Stage 3) and compare it with the vanilla-baseline model that was trained on all images in a single stage. As our models output a continuous distribution of probabilities for HP and SSA, we evaluate each model at several different confidence thresholds (a lower threshold results in higher recall, whereas a higher threshold results in higher precision). For average pathologist performance, we compute the Cohen's κ between all pairs of pathologists, and for each of the seven pathologists we show the mean Cohen's κ of all six pairs involving that pathologist. Figure 7 shows these results comparing our models with both individual pathologists and the average of all pathologists. We see that there is a wide range of Cohen's κ for individual pathologists-the mean of our pathologists' Cohen's κ scores was 0.450, which is in the moderate range of 0.41-0.60 [43] (a similar study found a Cohen's κ of 0.380 found among four pathologists [35]). Our curriculum learning model (AUC = 88.2%) outperforms the pathologist average for multiple thresholds and the baseline model (AUC = 83.7%) for all thresholds. In particular, adding a curriculum schedule increases performance from the baseline's maximum κ = 0.384 to the curriculum learning model's maximum κ = 0.473, the difference needed to outperform the pathologists' average (κ = 0.450).

Discussion
Human and machine notions of difficulty. Our study has presented a transparent analysis of the requirements of curriculum learning, proposing that histopathology image analysis tasks present a range of difficulty among examples and that readily-available annotator agreement can be used as a natural proxy for ranking images. Experimentally, we found that using this natural proxy as a curriculum to train classifiers can yield significant performance improvements.
Some prior work has demonstrated that what makes an image difficult for neural networks to classify might not always match what makes it difficult for human annotators, an observation that recent work on adversarial examples takes advantage of [44]. Our study explores a converse idea for histopathology image classification, finding that machine notions of difficulty do correlate with human annotator agreement and contending that annotator agreement can be a useful proxy for facilitating curriculum learning.
Implications on medical image analysis. Our work also has implications for how labels are used in the histopathology image analysis and medical image analysis domains. Much prior work in these domains has used majority voting or senior pathologist resolution to resolve labels, only retaining a single label for each image without distinguishing between images with high annotator and low annotator agreement. Our method could instead leverage the annotation agreement for curriculum learning. In Table 4 in the supplementary materials section, we list past work for which multiple annotator agreement levels appears to be available and therefore our method could be applicable (although most are private datasets).
Certainly, not every scenario in medical imaging is ideal for applying curriculum learning. For instance, both pathologists and deep learning models have achieved high performance in distinguishing high-grade lung cancers and normal tissue, which is considered a relatively easy task that is less likely to exhibit a range of difficulty among examples. Another negative example could be detecting bone fractures: since bone fractures are typically caused by instantaneous traumatic events, there is no progression of disease development, so we are less certain that a range of example difficulty exists. Potential scenarios conducive to curriculum learning by annotator agreement should ideally involve a progression of disease development and be challenging problems where even specialized pathologists might disagree. In particular, we believe that cancer datasets can benefit from curriculum learning because of the inherent natural progression of cancers (i.e., cancer develops over time and is not sudden). This could include tasks such as distinguishing among subtypes of lung adenocarcinoma or assessing tumor proliferation in breast cancer tissue samples.
Limitations. Though we intentionally addressed a common, clinically-important, and diagnostically-challenging problem of colorectal polyp classification for empirical evaluation-the best dataset that we were able to collect and annotate with multiple annotators at this time-our study nonetheless only uses a single dataset. As such, although our approach seems effective in its current form, we consider our results as an invitation for further exploration in this direction rather than a validation of curriculum learning for all histopathology classification tasks.
Furthermore, our dataset contained annotations from seven pathologist annotators, allowing us to categorize images into four discrete levels of difficulty. For medical image datasets that have fewer annotators and therefore fewer categorized levels of annotation difficulty, we suspect that the benefits of our approach could be slightly diminished. For example, our coarse annotator agreement method in §5, which only used two levels of annotator agreement, achieved a smaller performance improvement (1.5%) than our four-level annotator agreement method (4.5%). Moreover, our dataset size is modest in the medical imaging domain, and so whether these curriculum methods work for other dataset sizes is still relatively unexplored. We believe, however, that our curriculum learning methodology might still be worth exploring for these datasets with fewer annotators, as even small improvements in performance are important given the high cost of data annotation and the importance of accuracy toward patient outcomes.

Further Related Work and Conclusions
Although we argue that histopathology imaging is a promising context for curriculum learning, we are not the first to explore curriculum learning in the medical imaging domains. Oksuz et al. [45] used curriculum learning for artifact detection by first training on heavily-corrupted images and then introducing less-corrupted images, thereby improving performance on borderline cases. Maicas et al. [46] used a meta-training approach called teacher-student curriculum learning to improve breast screening classification on a weakly-labeled dataset. Moreover, Jesson et al. [47] used adaptive curriculum sampling to better detect lung nodules in extreme class-imbalance scenarios, and Oksuz et al. [45] applied a curriculum based on disease severity levels in radiology reports (e.g., mild, moderate, severe).
These prior studies have demonstrated empirical evaluation of curriculum learning and helped inspire our work, but their methods tend to be complicated and use inductive biases specific to certain datasets. For instance, the approach of Oksuz et al. [45] only applies to artifact detection, the approach of Jesson et al. [47] is specific to segmentation tasks, and disease-severity information from radiology reports used in Oksuz et al. [45] might not exist for many datasets and can be challenging to parse. Our method, on the other hand, is easy to implement, presents no modifications to network architecture, and only requires annotator agreement data, which is often readily available.
Based on a thoughtful analysis of the assumptions in curriculum learning, we have presented a simple yet effective curriculum learning framework which leverages easilyobtained annotator agreement data. In histopathology image analysis, where data collection and annotation can be especially costly, it is important to combine the natural properties of classification tasks with the most-appropriate inductive biases. We aim to have provided a well-motivated argument for more intentional application of curriculum learning to readers from both computer vision and medical imaging analysis backgrounds.