{'fraction_non_alphanumeric': 0.038664270739742435, 'fraction_numerical': 0.015573525007487272, 'mean_word_length': 4.47752624671916, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Memory Classifiers: Two-stage Classification for Robustness in Machine Learning

The performance of machine learning models can significantly degrade under distribution shifts of the data. We propose a new method for classification which can improve robustness to distribution shifts, by combining expert knowledge about the ``high-level"structure of the data with standard classifiers. Specifically, we introduce two-stage classifiers called memory classifiers. First, these identify prototypical data points -- memories -- to cluster the training data. This step is based on features designed with expert guidance; for instance, for image data they can be extracted using digital image processing algorithms. Then, within each cluster, we learn local classifiers based on finer discriminating features, via standard models like deep neural networks. We establish generalization bounds for memory classifiers. We illustrate in experiments that they can improve generalization and robustness to distribution shifts on image datasets. We show improvements which push beyond standard data augmentation techniques.


Introduction
Modern machine learning (ML) systems have achieved extraordinary performance in several areas, such as computer vision and natural language processing. However, recent work has illustrated that seemingly benign data corruptions, that humans can readily withstand, are potentially detrimental to the efficacy of ML systems [Recht et al., 2018, Azulay andWeiss, 2018]. For vision, this has been observed for corruptions like pixelation, compression, blur, and other transforms [Hendrycks and Dietterich, 2019]. This lack of robustness has limited the application of ML techniques in safety-critical applications. Efforts to improve the robustness against such corruptions have been attempted in the form of data augmentation , Lim et al., 2021. However, many systems remain vulnerable even to small degrees of corruption. An alternative classical approach is to infuse ML models with appropriate inductive biases [Goyal and Bengio, 2020, Battaglia et al., 2018, Gülçehre and Bengio, 2016. Such priors are available in at least two distinct ways: as model architectures such as deep neural nets (DNNs) [Fukushima, 1980, LeCun et al., 1989 and as synthetic examples [Kambhampati, 2021].
Figure 1: The figure shows sample images with the corresponding labels assigned by different classifiers. The image in the middle equals the one on the left plus Gaussian noise. We additionally show the classifier accuracies under Gaussian noise with severity level 5 [Hendrycks and Dietterich, 2019] Hence, for an 8-bit RGB color scheme, α = (255, 0, 0) generates images with a red patch of size w×w, inside an L×L image whose pixels are sampled from the distribution P . Now consider a toy color dataset D with three distinct classes corresponding to the color channels generated by the above process. The dataset contains 3 × 10 3 train and 3 × 10 2 test datapoints. The task is to predict the color of the patch contained in the image. We train a SOTA CNN architecture ResNet50, and-as expected given the simplicity of the dataset-it achieves 100% test accuracy. However, perhaps surprisingly, these classifiers are not robust to corruptions like Gaussian noise [Hendrycks and Dietterich, 2019], even on this simple dataset; see Figure 1.
A standard way to improve accuracy against such corruptions is through data-augmentation methods such as AugMix . We train a ResNet50 architecture on the color dataset using AugMix. The prediction accuracy on the perturbed test dataset improves, but remains well below satisfactory levels. For instance, under Gaussian noise, the accuracy of the network trained with AugMix drops to 33%, which is close to random guessing, as this is a balanced three-class dataset. The details of this experiment are in the Appendix A.5.
This task does not pose problems to humans, as the color dataset has a clear feature-color of the patch-that one can use. However, deep nets fail spectacularly, and thus seemingly do not learn this obvious feature. Our experiments show that this is the case in more realistic image datasets for high stakes medical applications. Motivated by these examples, in this paper we aim to address the following research question: How do we introduce expert information into classifiers, to improve robustness without compromising performance?
To this end we introduce memory classifiers. In this framework, we first allow the designer to write a distance function which captures a notion of similarity among images. This step is particularly useful in applications such as medical diagnosis, where physicians often have useful intuitions to share, and we illustrate this in one of our case studies. Next, using the distance metric we cluster the dataset around training datapoints and treat them as memories. Afterwards, we assign a cluster to each test datapoint according to the most "similar" memory, and apply classifier trained for the cluster. The distance function in the case for image applications, we leverage the rich body of work on digital image processing, e.g., Gonzalez et al. [2009]. This offers us tools to capture the high-level properties of an image, which can be potentially more robust.
We provide both theoretical and empirical evidence to support the benefits of this architecture. We develop generalization bounds, and empirically show that it has robustness to small distribution shifts. Intuitively, the memory-formation step identifies representative datapoints that are well-separated in the space of datapoints, which reduces sample complexity and increases robustness.
Contributions: 1. We propose memory classifiers, a new two-stage method for classification. The first step is to select representative data points, or memories, from the training dataset, representing different clusters of the data. We learn a classifier for each such cluster. To select memories, we propose a method to encode domain knowledge via a similarity metric. 2. We give rigorous generalization bounds for memory classifiers. 3. We show improvements in classification robustness across different types of ML architectures, by evaluating robustness against natural distribution shifts on two image datasets. Further, any technique that improves the robustness of a neural network architecture can be incorporated into the within-cluster classifiers. Step 1: The test datapoint gets mapped to a memory datapoint using a matching algorithm.
Step 2: Once matched, the final decision is issued by a within-cluster classifier.

Overview
Figure 2 provides an overview for the computational workflow of memory classifiers. A memory classifier is composed of a collection of memories, each equipped with a separate classifier. These individual classifiers could be of distinct types (DNNs, random forests, etc). Our two-stage memory classifier has the following steps: Step 1: Memory Selection. The memory selection algorithm maps a datapoint to a memory m using a similarity score sim. A higher value of sim(x 1 , x 2 ) ∈ [0, 1] means that the datapoints x 1 and x 2 are more similar. Formally, given a set of memories M + = {(m 1 , b 1 ), (m 2 , b 2 ), . . . , (m q , b q )}, where b k ∈ [0, 1] are similarity score thresholds, and a test datapoint x t , we find the closest memory which satisfies the threshold condition by computing: (1) If sim(m k , x t ) ≥ b k , we then select the classifier associated with memory m k , otherwise we select an "out-ofboundary" classifier. 1 Step 2: Classification. Use x t as an input to the classifier associated with memory m k , to predict a label. If the "out-of-boundary" classifier is selected, label the input x t as an unknown class.

Memory Classifiers: Formal Analysis
Consider a selector function s : X × [q + 1] := {1, 2, . . . , q + 1} → {0, 1}, which chooses the k-th memory for input x when s(x, k) = 1. Further, s chooses the out-of-distribution classifier when s(x, q + 1) = 1. A memory classifier is then defined by the function F : X → R, which first uses s to select a memory k and then applies a classifier h k , so that: Here h k is the within-cluster classifier associated with the chosen memory k, for k ≤ q, and h q+1 is the out-ofboundary classifier. We classify x according to the sign of F (x), breaking ties arbitrarily. Now, we provide a data-dependent generalization bound for memory classifiers. We let the training and testing samples be independently and identically distributed (iid), sampled from a fixed unknown distribution D over the space X × {−1, +1}. For a binary label y ∈ {−1, 1}, we denote the classification error of a classifier F as R(F ) = P (x,y)∼D [yF (x) ≤ 0]. The empirical error over a training set T = {(x 1 , y 1 ), . . . , (x n , y n )} is where P T denotes the empirical distribution defined by T . We assume that for all k ∈ [q+1], h k are chosen from a hypothesis space H k , while s is chosen from a hypothesis space S. Let the number of points mapped to memory k and correctly classified be n + k . Let the empirical Rademacher complexities [Koltchinskii and Panchenko, 2002, Bartlett and Mendelson, 2002, Shalev-Shwartz and Ben-David, 2014] of the hypothesis spaces H k , S over the training set T beR T (H k ),R T (S), respectively. Consider 0 < δ ≤ 1/q, and let C(n, q, δ) = 1 δ log q n 2 + log ρ 2 n log q + log(4/δ) n . ( Assume that the memories are chosen from the training datapoints, so M ⊂ {x 1 , . . . , x n }. This always holds in our experiments. Then, for any memory classifier, with probability at least 1 − δ, we have for some constant κ > 0, The details of the proof are in Appendix A, and build on the results of [DeSalvo et al., 2015]. The proof also requires a delicate bounding of the number of possible selector functions induced by memory classifiers; which requires a careful combinatorial analysis, and also uses the Massart Lemma from learning theory [Shalev-Shwartz and Ben-David, 2014]. In (4), we have a nontrivial generalization bounds if the number q of memories is of smaller order than n 1/4 / log(n) 1/2 , and if the Rademacher complexity of each H k isR T (H k ) 1/q. Consistent with this, the memory selection algorithm 1 attempts to minimize the number of memories to achieve good generalization.

Learning Memory Classifiers
Learning a memory classifier has two stages. In the first stage, we identify datapoints in the training data as memories, such that all data points are within a certain similarity threshold of the memories, while aiming to minimize the number of memories. In the second stage, we train a classifier for data cluster belonging to each memory. We explain the two steps in detail next. Clustering with Memories: We aim to cluster the data around the memories, which should be datapoints in the training set. Given a dataset, the Partitioning Around Medoids (PAM) method [Park andJun, 2009, Xu andTian, 2015] tries to select a cluster M = {m 1 , m 2 , . . . , m q } of q memories such that the following clustering objective is maximized: for the similarity score sim : X × X → [0, 1]. The naive implementation of PAM has a runtime complexity of O(n 2 q 2 ) [Schubert and Rousseeuw, 2018]. Even the faster variants are usually too slow for large image datasets. Thus, we use a simple variant of the Clustering Large Applications based upon Randomized Search (CLARANS) [Ng and Han, 2002] algorithm. This integrates randomized global search with local cluster improvement.

Selecting memories
Memory Initialization: Here we provide a simple intuitive explanation of the memory set initialization workflow. The goal is to place memories in the input space densely enough, with the constraint that every training datapoint occurs within a similarity score boundary b t of some memory. The exact details are in Algorithm 2 in the Appendix 3.1. Essentially, we iterate the following process -a random datapoint is picked as a memory, and compared with all datapoints in the set in a single linear pass. The datapoints which end up being marked as similar are admitted onto the set of datapoints accepted by the current memory. This continues until the full set of training datapoints is covered by a set of memories M . This provides a warm start for any subsequent algorithm optimizing the choice of memories.
Learning Memories: Starting with a dataset D of size n, the aim is to build a set M = {m 1 , m 2 , . . . , m q } of size q which minimizes the clustering objective. Choosing the memories can be simplified by viewing it as a search through a graph G [Ng and Han, 2002] with subsets D q ⊂ D as its nodes. A subset of size q defines a choice for the memory set M . Definition 3.1 (Memory Search Graph G). The undirected memory search graph G is represented by an ordered pair (V, E), where the set of nodes V is the collection of subsets D q of size q of the original dataset D. An edge e ∈ E exists between two nodes D 1 q and D 2 Associated with each node of the graph is a clustering objective cost. This allows for a greedy algorithm: starting from a node one can visit a series of neighboring nodes with increasing scores in the search process. This is followed by a combination of Global resets and Local minimization to approximate the optimal choice.

Algorithm 1 Generating Memories
if NewScore > CurrentScore then output M Algorithm 1 creates the memories which are used to implement the memory classifier. Analogous to the standard CLARANS algorithm [Ng and Han, 2002], a node in G is surrounded by q(n − q) neighbors, which can be large set We warm start the process with a reasonable choice of an initial node in G, and greedily search for local improvements for a fixed number of iterations. The number of memories is controlled by the similarity score thresholds b t . The iterations are limited by Z g and Z l .

Training a Classifier
For each memory, the training datapoints in the cluster associated to it form the training set for the classifier assigned to that memory. We can then use standard learning procedures to train these classifiers, as we explain in detail for each particular example.

Experiments
We apply our framework to two classification tasks: acne lesion classification for dermatology and leaf severity assessment for botany. When applying machine learning tools to specific problems, there are often additional visual cues that can inform the decision of a human observer and should ideally be useful to a machine learning algorithm. These features, if carefully extracted, can help improve the accuracy of classification; and possibly help in improving robustness as well-in line with the long history of feature engineering techniques [Nixon and Aguado, 2012]. The challenge is to incorporate them into a modern machine learning pipeline.

Case Study: Acne Lesion Classification
Acne vulgaris, or simply acne, is a common disease that affects approximately 85% of people over their lifespan [Abas et al., 2016]. The effects of acne and the resulting scars can be psychosocial and possibly psychological, in some cases over the entire lifespan. Distinguishing and counting primary acne lesions, commonly divided into noninflammatory (comedones, either white or blackheads) and inflammatory ones (pustules, papules, and nodules), is necessary to assess acne severity [Tan et al., 2012]. Counting lesions, however, is a very time-consuming task which is usually only performed in clinical trials and required training raters to achieve higher reliability [Tan et al., 2006]. The classification task is to assign each primary lesion to one of the three classes commonly used in acne severity scales [Bernardis et al., 2020]: comedones (ClassA), pustules and papules (ClassB), and nodules (ClassC). Figure 3 shows   sample lesion crops for the 3 classes. All images were taken from a retrospective deidentified dataset of 63 pediatric patients diagnosed with acne, ages 2-21, at the Children's Hospital of Philadelphia and selected to adequately span the range of acne types encountered in the clinic. An expert labeled each image outlining all primary lesions. Individual lesion images were then obtained by cropping out regions containing each lesion, using a binary mask, and resizing them into images of size 300 × 300.
A clinician uses features such as the inflammation (redness), lesion size, amount of bulge, among others, when classifying a lesion. These features are usually context-dependent. For instance, the degree of inflammation relies on cues about the skin tone in its immediate neighborhood. Thus, the redness of a lesion belonging to the same class can differ across patients. Another important cue is the size of the lesion. This can depend on the relative size of a lesion compared to other lesions on the same face image.
Features: We compute the following three features: 1) Lesion size Sz ∈ R, i.e., number of pixels that a lesion occupies in an image, normalized by the median size of lesions over the original face image; 2) Skin redness Rd ∈ R, i.e., an indicator of inflammation. We compare the red channel histogram of a lesion with a neighboring skin patch of similar size, as a measure of the degree of redness of the skin. The histogram comparison was implemented using standard computer vision tools [his]. Finally, 3) Saturation Sat ∈ {0, 1} due to over-exposure. Although not a clinical feature, Sat is an important visual marker of a lesion. Some skin lesions can suffer from over-exposure due to larger than average height as compared to the surrounding face area. This causes light to reflect off of the lesion, making it less detectable.
Experiments: The dataset consists of a total of 1683 lesions extracted from the 63 face images. Out of these, 1153 lesions from 36 face images are used for training, and 530 lesions from 26 face images for testing. To construct the similarity metric, we train a depth-three decision tree T , which uses features Sz, Rd, and Sat to classify among the three classes -(ClassA, ClassB, ClassC). This gives a binary valued similarity metric, such that sim(X 1 , X 2 ) = 1 if they are assigned the same class by T . This results in a total of 3 memories. The numbers on each image are the assigned severity labels.
The individual leaf classifiers are trained by transfer learning from a pre-trained deep neural network. We trained the networks with an SGD optimizer for 40 epochs, with a batch size of 24, learning rate of 0.01, and a momentum of 0.9. When comparing with a data-augmentation scheme, we trained the networks for individual memories after initializing the network weights with a classifier trained on the whole data augmented training set.  Figure 5, and the detailed table is in the Appendix. We observe that memory classifiers can improve upon these SOTA data-augmentation techniques. We observe an improvement of around 9% across varying severity levels averaged across different perturbation types.
Comparison with feature augmentation methods: An alternative way to include features in the training process is to add them the features extracted by the CNN [Lee et al., 2016, Li andSrikumar, 2019]. Hence, we concatenate the features extracted by a CNN with additional ones we implemented, and train a classifier. We perform our experiments on five candidate architectures for lesion classifiers: ResNet18, ResNet34, ResNet50, VGG11, and VGG16 He et al.
[2015], Simonyan and Zisserman [2014]. The clean accuracies on the test set are presented in Table 1. The baselines are trained using the same network architecture as the leaf classifiers for a fair comparison, since our node classifiers can be arbitrary. In Table 1, we observe that memory classifiers are at least as good, or better than, the baseline classifiers.

Case Study: Leaf Disease Severity
Machine learning has the ability to transform the agricultural industry through crop yield prediction, intelligent spraying, and disease diagnosis. The task of identifying plant diseases is a promising problem for machine learning. In  this case study, we focus on a dataset of coffee leaves affected by multiple biotic stress factors, such as leaf miners, rust, brown leaf spots, and cercospora leaf spots. Our dataset contains 1747 images of arabica coffee leaves, including healthy and diseased leaves. We use 80% of the dataset for training, and the remainder for test. We refer the reader to Esgario et al.
[2019] for more details. In Figure 4, we show a sample of leaves with varying severity levels, depending on the degree and type of the affected area. This is available in the dataset as a severity score SV ∈ {0, . . . , 4}, where a higher score denotes more severity. The classification task is to predict the severity level from the input image.
Features: As a pre-processing step, we partition a given image into different segments via Felzenszwalb and Huttenlocher [2004] and perform a color value thresholding in HSV color space to obtain the green, brown, and discolored regions. We over-approximate the leaf region by constructing a convex hull of the green pixels. This ensures that any holes created by discoloration are included as an area of the leaf. We then define functions F d : X → [0, 1] and F b : X → [0, 1] as the number of pixels discovered as discolored or brown as a fraction of the green pixels. Figure 4 in Appendix, shows sample masks overlaid on the original image. Such features can inform health of a leaf: a plant with mostly green leaves without discolored or brown areas is likely to be healthy. A decision tree T : [0, 1] 2 → {0, . . . , 4} based on these features classifies the image into one of the five severity levels, so that T (F d (x), F b (x)) ∈ SV . Given the label computed by T , two leaf images with the same predicted class are assigned a sim score of 1, and 0 otherwise.

Experiments:
We train leaf classifiers and baseline classifiers as discussed in the previous case study. It takes ∼ 30 minutes for training our memory classifier on a single NVIDIA Quadro RTX 6000 GPU. In Esgario et al. [2019], the authors perform transfer learning on a pre-trained ResNet50 architecture to predict the severity score. Here we train the networks on the original image, instead of cropping out the leaf area as a preprocessing step as in Esgario et al. [2019]. This makes the problem more challenging, and causes a 4% drop in test accuracy. We study the robustness of different neural network architectures and data augmentation methods similarly to the previous case study. We subject the leaf images to 18 different corruptions across 5 different severity levels.
Comparison with data augmentation methods: As mentioned in the previous case study, we use data augmentation for improving the accuracy, using NoisyMix, AugMix, CutMix, ManifoldMix and Adversarial Training. The results are presented in Table 2. We observe that across different data augmentation schemes, memory classifiers achieve consistently better accuracy under different degrees of severity in the image noise. We plot the results in Figure 6.
Comparison with feature augmentation methods: We augment the features extracted by a CNN with the features discussed above, and train a baseline classifier using them. We repeat this experiment on 5 different CNN architectures for both the leaf classifiers and the standalone CNN as the baseline (Table 6). The clean accuracies of the memory classifiers and the baseline network are comparable. However, memory classifiers are consistently more robust than the simple feature augmented counterparts.

Related Work
An early attempt at case-based reasoning-which is distantly related to our approach of clustering to a small set of memories-can be found in Kolodner [1992]. There has been a growing interest in using these ideas for image Figure 6: Robust accuracy on corrupted leaf dataset across five severity levels.
classification. The motivations range from interpretability [Chen et al., 2019] to solving few shot learning [Snell et al., 2017], and even one shot learning [Vinyals et al., 2016]. Our work is related, but substantially different due to our reliance on memories; this leads to further differences in implementation and learning guarantees. Another closely related idea is of coresets e.g., Mirzasoleiman et al.
[2019], etc. , which uses a subset of representative data samples for training. This can improve training time, while usually achieving similar performance as using the full data. In contrast, our clustering step via the similarity function approach aims to improve performance and robustness. Our workflow is structurally similar to the idea of Mixtures-of-Experts [Shazeer et al., 2017] (MoE) models but is conceptually different. A challenge in MoE models is to define their regions of activity. The novelty in our work is that these regions are defined via the memories in conjunction with the distance metric. Other prior work includes efforts to de-bias DNNs by designing training sets [Li et al., 2020]; understanding how shape-bias disagreement occurs in DNNs [Islam et al., 2021]; and using Gabor filters to simulate the visual cortex [Dapello et al., 2020]

Discussion
We show that using robust features to compute a similarity metric offers a viable path towards robustness. Clustering the dataset using the robust features allows the CNN to improve on inductive biases present in each cluster. Consider the example of the color dataset discussed in Section 1. The features extract the color of the patch in the image, which decides the similarity. For the cluster belonging to a memory of the green patch, the trained CNN is more likely to label an image as a green patch even after corruption, simply because of the distribution of the training data. This explains some of the robustness we observe. Our experiments suggest that this robustness also extends to more complex datasets. Construction of such additional robust features will be explored in the future.

A Appendix
A.1 Generalization properties Definitions and notations. We will use a few definitions and notations: for a positive integer q, we denote [q] := {1, 2, . . . , q}. We denote by I(A) the indicator of the event A, which equals to unity (i.e., "1") if event A is true, and equals to zero if event A is false.
Now we provide the proof of the data-dependent generalization bound (4) for memory classifiers. We let the training and testing samples be independently and identically distributed (iid), sampled from a fixed unknown distribution D over the space X ×{−1, +1}. Let S be the hypothesis class that corresponds to using s with the thresholds b k , k ∈ [q], as described in the text. Let S k = {s(·, k) : s ∈ S}, and define S k similarly. Consider a set of iid random variables ε i , i ∈ [n], distributed uniformly over {−1, 1}. For a hypothesis class H, let the empirical Rademacher complexity of the hypothesis space H over the training set T beR T (H) = n −1 E ε1,...,εn sup h∈H n i=1 ε i h(x i ) [Koltchinskii and Panchenko, 2002, Bartlett and Mendelson, 2002, Shalev-Shwartz and Ben-David, 2014.
A key insight is that memory classifiers can be thought of as cascade classifiers [DeSalvo et al., 2015]. From their Theorem 1 (with p = 1), we have Thus, it remains to bound the Rademacher complexity of the selector function class S k .
Recall that M = {m 1 , . . . , m q } is the set of memories. Then, for an input x ∈ X , and a memory m k with k ∈ [q], the selector function s chooses the memories most similar to x from the set M by computing: . If b k is the larger above, then some hypothesis in H q+1 is chosen. Thus, we have S k ⊂ S k ∪ H q+1 . Since S k consists of 0-1 valued classifiers, and H q+1 consists of a constant classifier (thus its empirical Rademacher complexity is of order O(1/n 1/2 ), we find, for some constant κ > 0, The number of hypotheses in S k that can be formed based on n datapoints is at most N n,q = n · n−1 q−1 . This is because we can choose the first memory m k as any of the n datapoints (or the virtual memory), and then we can choose the remaining q memories as any subset of the remaining n − 1 datapoints. Each such choice determines at most one distinct function s(·, k), which gives the desired bound.
Let A = {a s : s(·, k) ∈ S k } be the set of binary vectors indexed by s, for each s containing the indices of the datapoints such that s(x i , k) = 1. Thus a s ∈ {0, 1} n , and a s,i = 1 iff s(x i , k) = 1. We have a s 2 ≤ n 1/2 , because a s ∈ {0, 1} n .
This finishes the proof. , for details. One can observe that adding noise does not completely impair the feature extraction, and thus helps in improving the robustness of our classification process. Figure 7 demonstrates the different masks overlaid on the original image. The number of pixels discovered as discolored/brown as a fraction of the green pixels are returned by the functions F d and F b , respectively. We also demonstrate how these features are robust to natural perturbations.

A.3 Ablation on Different CNN Architectures
We present the results for different CNN architectures in Table 3, and Table 4 for the leaf severity and lesion classification datasets. The clean test accuracies obtained by the memory classifiers are comparable to those of standard neural nets, across the datasets and network architectures. The performances are in most cases comparable or even better than for the original network.

A.4 Augmented Features Results
An alternate way to add features to the ones extracted by a DNN is by feature augmentation, including them in the penultimate layer. The intuition is that the final layer might be able learn how to incorporate them with features extracted from the images. The results for this approach are presented in Tables 5 and 6, for the skin and leaf datasets, respectively.

A.5 Color Dataset Details
We generate the color dataset as described in Section 1, with L = 500 and w = L/10. The three classes are generated using the parameter p set to [(255, 0, 0), (0, 255, 0), (0, 0, 255)]. This generates images belonging to three color classes. The dataset has 3000 training and 300 test images.
We train the following baselines: Table 4: Leaf Disease Severity Results (Network Ablation). For a given network architecture, we compare the evaluation on a standalone DNN classifier and the same network used in a memory classifier setting. The memories were formed using features corresponding to different proportions of discoloration in the leaf. We report the average accuracy (in percentage) across 5 severity levels for different types of image corruption. For a given architecture and corruption, we highlight the ones with better accuracy.
1. ResNet50: The network is trained using transfer learning from a pre-trained model. We use an SGD optimizer to train for 50 epochs, with a learning rate of 0.01, weight decay 0.0005, batch size of 48 and momentum of 0.9. The trained network has 100% accuracy on the clean test images.
2. AugMix: The training uses a similar ResNet50 backbone network, with similar hyper-parameters and training algorithm, as discussed in . The trained network in this case has a clean test accuracy of 99%.
Memory Classifier: The feature extractor in this case study predicts the color of the image. For pre-processing, we partition the image into segments via the method from Felzenszwalb and Huttenlocher [2004]. Out of the T largest segments, we find the one with the highest intensity over the different color channels. In our experiments we set T = 20. Among the segments, the color channel with the highest intensity assigns the color to an image. Clustering the dataset with this feature forms three memories. The node classifiers for each memory cluster use the Augmix data-augmentation technique to train a ResNet50 architecture. This again achieves a test accuracy of 100%.  Table 7: Color data augmentation results. We present the accuracies of classifiers for predicting the color across different corruptions and severities. The Baseline and memory classifier uses a ResNet50 backbone with AugMix augmentation. We highlight the rows pertaining to different kinds of image noise.

A.6 Intializing Memories
Intializing Memories: Certain "high-level" properties of images from the same class typically cluster well. For instance, the class of images containing a red traffic light usually contain a set of bright red pixels Thus, we begin by efficiently identifying such broad-ranging categories. While we know that having a small number of memories can help with generalization, the specific number of memories is often not known a priori. We begin by placing memories in the input space densely enough, with the constraint that every training data point occurs within a similarity score threshold b t of some memory.
We demonstrate this in Algorithm 2. A randomly picked data point is compared with all data points in the set Reject-edSet. This happens in a single linear pass. Data points marked as similar enough are included into the set of data points for the current memory. Further analysis of these data points is not required for the current algorithm. This continues until there are any data points left. Algorithm 2 always terminates, as the size of RejectedSet decreases in each iteration. The output provides a warm start for any subsequent algorithm optimizing the choice of memories.