{'fraction_non_alphanumeric': 0.04874202027788209, 'fraction_numerical': 0.019176367505319813, 'mean_word_length': 4.638108680310515, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The shape and simplicity biases of adversarially robust ImageNet-trained CNNs

Increasingly more similarities between human vision and convolutional neural networks (CNNs) have been revealed in the past few years. Yet, vanilla CNNs often fall short in generalizing to adversarial or out-of-distribution (OOD) examples which humans demonstrate superior performance. Adversarial training is a leading learning algorithm for improving the robustness of CNNs on adversarial and OOD data; however, little is known about the properties, specifically the shape bias and internal features learned inside adversarially-robust CNNs. In this paper, we perform a thorough, systematic study to understand the shape bias and some internal mechanisms that enable the generalizability of AlexNet, GoogLeNet, and ResNet-50 models trained via adversarial training. We find that while standard ImageNet classifiers have a strong texture bias, their R counterparts rely heavily on shapes. Remarkably, adversarial training induces three simplicity biases into hidden neurons in the process of"robustifying"CNNs. That is, each convolutional neuron in R networks often changes to detecting (1) pixel-wise smoother patterns, i.e., a mechanism that blocks high-frequency noise from passing through the network; (2) more lower-level features i.e. textures and colors (instead of objects);and (3) fewer types of inputs. Our findings reveal the interesting mechanisms that made networks more adversarially robust and also explain some recent findings e.g., why R networks benefit from a much larger capacity (Xie et al. 2020) and can act as a strong image prior in image synthesis (Santurkar et al. 2019).


Introduction
Convolutional neural networks (CNNs), specifically, AlexNet (Krizhevsky et al., 2012), have been increasingly found to exhibit interesting correspondences with human object recognition (Rajalingham et al., 2018;Serre, 2019;Cichy et al., 2016). However, a remarkable difference between ImageNettrained CNNs and humans is that while the CNNs leverage mostly texture cues to categorize images, humans recognize an object by relying on its outline or silhouette (Geirhos et al., 2019). Interestingly, this shape bias of human vision was hypothesized to enable the superior performance of humans on out-of-distribution (OOD) data (Geirhos et al., 2018), which CNNs often misclassify.
For example, simply adding Gaussian noise to the input image can dramatically reduces AlexNet's top-1 accuracy from ∼57% to ∼11% (Hendrycks and Dietterich, 2019). More severely, adding imperceptible, pixel-wise changes to an input image of a school bus yields a visually-identical image that would cause state-of-the-art CNNs to mislabel "ostrich" (Szegedy et al., 2014). An interesting class of CNNs that are state-of-the-art on such OOD noisy or adversarial data are adversarially-robust CNNs (hereafter, R networks), i.e. the networks that are trained to correctly label adversarial examples instead of real examples (Madry et al., 2018). That is, adversarial training has substantially improved model robustness to some types of out-of-distribution and adversarial data (Madry et al., 2018;. Therefore, we aim to understand the relationship between the OOD performance and shape bias of R networks by answering three main questions. First, it remains unknown whether a shape bias exists in a R network trained on the large-scale ImageNet (Russakovsky et al., 2015), which often induces a large texture bias into vanilla CNNs (Geirhos et al., 2019), e.g. to separate ∼150 four-legged species in ImageNet. Thus, an important question is: Q1: On ImageNet, do adversarially-robust networks (a.k.a., R networks) prefer shapes over textures? This question is interestingly also because Santurkar et al. (2019) found that ImageNet-trained R networks act as a strong image texture prior, i.e. they can be successfully used for many pixel-wise image optimization tasks .
Second, Geirhos et al. (2019) found that CNNs trained to be more shape-biased (but not via adversarial training) can generalize better to many unseen ImageNet-C (Hendrycks and Dietterich, 2019) image corruptions than S networks, which have a much weaker shape bias . Therefore, we ask: Q2: If an R network has a stronger preference for shapes than standard ImageNet-trained CNNs (hereafter, S networks), will it perform better on OOD images? It is worth noting that R networks, which were trained solely on adversarial examples, often underperform S networks on real test sets . Beside being more robust to adversarial attacks, some CNNs trained via AdvProp ) (a variant of Madry et al. (2018)'s adversarial training framework that trains CNNs on both real and adversarial data) obtain a similarly high accuracy on real data as S networks; however, it is unknown whether AdvProp CNNs are shape-or texture-biased (which we answer in Table 2).
Third, while most prior work studied the behaviors of R CNNs as a black-box classifier, little is known about the internal network characteristics of R networks and, what en-able their shape bias and OOD performance. Therefore, we attempt to shed light into the last question: Q3: How did adversarial training change the hidden neural representations to make CNNs more shape-biased and adversarially-robust?
In this paper, we harness two common datasets in ML interpretability and neuroscience-cue-conflict (Geirhos et al., 2019) and NetDissect (Bau et al., 2017)-to answer the three questions above via a systematic study across three different convolutional architectures-AlexNet (Krizhevsky et al., 2012), GoogLeNet (Szegedy et al., 2015), and ResNet-50 (He et al., 2016)-trained to perform image classification on the large-scale ImageNet dataset (Russakovsky et al., 2015). We find that: 1 1. R classifiers trained on ImageNet prefer shapes over textures ∼67% of the time-a stark contrast to S classifiers, which prefer shapes at ∼25% (Sec. 3.1). 2. Consistent with the strong shape bias, R classifiers interestingly outperform S counterparts on texture-less, distorted images (stylized and silhouetted images) (Sec. 3.2.2). 3. Adversarial training makes CNNs more robust by (a) blocking pixel-wise input noise via smooth filters (Sec. 3.3.1); (b) reducing the set of high-activating inputs to simple patterns (Sec. 3.3.2). 4. Units that detect texture patterns (according to Net-Dissect) are not only useful to texture-based image classification but can be also highly useful to shapebased classification (Sec. 3.4). 5. By aligning NetDissect and cue-conflict frameworks, we found that hidden neurons in R networks are surprisingly neither strongly shape-biased nor texture-biased, but instead generalists that detect low-level features (Sec. 3.4).
The exciting differences between human vision and R CNNs revealed by our work can inform future research into what enables the strong generalization capability of human vision and how to improve state-of-the-art CNNs.

Networks and Datasets
Networks To understand the effects of adversarial training across a wide range of architectures, we compare each pair of S and R models while keeping their network architectures constant. That is, we conduct all experiments on two groups of classifiers: (a) standard AlexNet, GoogLeNet, & ResNet-50 (hereafter, ResNet) models pre-trained on the 1000-class 2012 ImageNet dataset; and (b) three adversarially-robust counterparts i.e. AlexNet-R, GoogLeNet-R, & ResNet-R which were trained via adversarial training (see below). Training A standard classifier with parameters was trained to minimize the cross-entropy loss over pairs of (training example , ground-truth label ) drawn from the ImageNet training set : 1 All code and data will be available on github upon publication.
On the other hand, we trained each R classifier via Madry et al. (2018) adversarial training framework where each real example is changed by a perturbation Δ: and  is the perturbation range allowed within an 2 norm. Hyperparameters The S models were downloaded from PyTorch model zoo (PyTorch, 2019). We adversarially trained all R models using the library , using the same hyperparameters in Santurkar et al., 2019;Bansal et al., 2020) because these previous works have shown that adversarial training significantly changed the inner-workings of ImageNet CNNsi.e. becoming a strong image prior with perceptually-aligned deep features. That is, adversarial examples were generated using Projected Gradient Descent (PGD) (Madry et al., 2018) with an 2 norm constraint of 3, a step size of 0.5, and 7 PGD-attack steps. R models were trained using an SGD optimizer for 90 epochs with a momentum of 0.9, an initial learning rate of 0.1 (which is reduced 10 times every 30 epochs), a weight decay of 10 −4 , and a batch size of 256 on 4 Tesla-V100 GPU's.
Compared to the standard counterparts, R models have substantially higher adversarial accuracy but lower ImageNet validation-set accuracy (Table 1). To compute adversarial accuracy, we perturbed validation-set images with the same PGD attack settings as used in training. Correctly-labeled image subsets: ImageNet-CL Following Bansal et al. (2020), to compare the behaviors of two networks of identical architectures on the same inputs, we tested them on the largest ImageNet validation subset (hereafter, ImageNet-CL) where both models have 100% accuracy. The sizes of the three subsets for three architectures-AlexNet, GoogLeNet,693,20,238,and 27,343. On modified ImageNet images (e.g., adversarial in Fig. 1b), we only tested each CNN pair on the modified images whose original versions exist in ImageNet-CL. That is, we wish to gain deeper insights into how CNNs behave on correctly-classified images, and then how their behaviors change when some input feature (e.g. textures or shapes) is modified.

Do ImageNet adversarially-robust networks prefer shapes or textures?
It is important to know which input feature a classifier uses when making decisions. While standard ImageNet networks often carry a strong texture bias (Geirhos et al., 2019), it is unknown whether their adversarially-robust counterparts would be heavily texture-or shape-biased. Here, we test this hypothesis by comparing S and R models on the well-known cue-conflict dataset (Geirhos et al., 2019). That is, we feed "stylized" images provided by Geirhos et al. (2019) that contain contradicting texture and shape cues (e.g. elephant skin on a cat silhouette) and count the times a model uses textures Table 1 Top-1 accuracy (%) on 50K-image ImageNet validation-set and PGD adversarial images (number of iterations = 7; = 3; step size = 0.5). or shapes (i.e. outputting or ) when it makes a correct prediction. Experiment Our procedure follows Geirhos et al. (2019). First, we excluded 80 images that do not have conflicting cues (e.g. cat textures on cat shapes) from their 1,280-image dataset. Each texture or shape cue belongs to one of 16 MS COCO (Caesar et al., 2018) coarse labels (e.g. or ). Second, we ran the networks on these images and converted their 1000-class probability vector outputs into 16-class probability vectors by taking the average over the probabilities of the fine-grained classes that are under the same COCO label. Third, we took only the images that each network correctly labels (i.e. into the texture or shape class), which ranges from 669 to 877 images (out of 1,200) for 6 networks and computed the texture and shape accuracies over 16 classes. Results On average, over three architectures, R classifiers rely on shapes ≥ 67.08% of the time, i.e. ∼2.7× higher than 24.56% of the S models (Table 2). In other words, by replacing real with adversarial examples, adversarial training causes the heavy texture bias of ImageNet classifiers (Geirhos et al., 2019;Brendel and Bethge, 2019) to drop substantially (∼2.7×). Additionally, we also test two adversarially-robust models that are trained on a mix of real and adversarial data via AdvProp  with a PGD 2 of 1 and 5, respectively (see full hyperparameters in the training code). These AdvProp models obtain a similarly high accuracy on both validation as well as adversarial images (Table 1; PGD1  & PGD5). Interestingly, AdvProp models are also heavily texture-biased (Table 2; 68.24% vs. 31.75%) but their preferences are in between that of the vanilla and R models. Moreover, as CNNs are trained with increasingly more adversarial perturbations and less real data (from Standard → PGD1 → PGD5 → Robust), the texture bias also decreases (from 77.79%→ 68.24% → 63.11% → 29.63%; Table 2). This result presents a strong evidence that real ImageNet images strongly induce a texture bias and adversarial images induce a shape bias into CNNs.

Do robust networks generalize to unseen types
of distorted images? Geirhos et al. (2019) found that some training regimes that encourage CNNs to focus more on shape can improve their performance on unseen image distortions. Sec. 3.1 shows that training with more adversarial examples makes a CNN more shape-biased. Therefore, it is interesting here to understand how adversarial training improves CNN generalization performance.
That is, we test R models on two types of controlled images where either shape or texture cues are removed from the original, correctly-labeled ImageNet images ( Fig. 1c-f). Note that when both shape and texture cues are present, e.g. in cue-conflict images, R classifiers consistently prefer shape over texture, i.e. a shape bias (Table 2). However, this bias may not necessarily carry over to the images where only either texture or shape cues are present.

Performance on shape-less images
We create shape-less images by dividing each ImageNet-CL image into a grid of × even patches where ∈ {2, 4, 8} and re-combining them randomly into a new "scrambled" version ( Fig. 1c). On average, over three grid types, we observed a larger accuracy drop in R models compared to S models, ranging from 1.6× to 2.04× lower accuracy (Table 3d). That is, R models become substantially less accurate than S models when shape cues are removed by patch-shuffling-another evidence for their exclusive reliance on shapes (instead of textures). See Fig. 2 for the predictions of ResNet and ResNet-R on scrambled patches. Table 3 Adversarially-robust (R) models outperform vanilla (S) models when both are under PGDadversarial attacks (b). R models consistently outperform S model in texture-less (e,f) distortions. Here, we report top-1 accuracy scores (%) on the transformed images whose original real versions were correctly-labeled (a) by both S and R models. "Scrambled" column (c) shows the mean accuracy scores over three patch-scrambling types (details in Fig. 3).

Network
(a) Real ImageNet

Performance on texture-less images
Following Geirhos et al. (2019), we test R models on three types of texture-less images where the texture is increasingly removed: (1) stylized ImageNet images where textures are randomly modified; (2) binary, black-and-white, i.e. B&W, images (Fig. 1e); and (3) silhouette images where the texture information is completely removed (Fig. 1f). Stylized ImageNet To construct a set of stylized ImageNet images (see Fig. 1d), we take all ImageNet-CL images (Sec. 2) and change their textures via a stylization procedure in Geirhos et al. (2019), which harnesses the style-transfer technique (Gatys et al., 2016) to apply a random style to each ImageNet "content" image. B&W images For all ImageNet-CL images, we use the same process described in Geirhos et al. (2019) to generate silhouettes, but we do not manually select or modify the images. We use ImageMagick (2020) to binarize ImageNet images into B&W images via the following steps: Silhouette For all ImageNet-CL images, we obtain their segmentation maps via a PyTorch DeepLab-v2 model (Chen et al., 2017) pre-trained on MS COCO-Stuff. We used the ImageNet-CL images that belong to a set of 16 COCO superclasses in Geirhos et al. (2019) (e.g. , , ). When evaluating classifiers, an image is considered correctly labeled if its ImageNet predicted label is a subclass of the correct COCO superclass ( Fig. 1f; → ). Results Consistently, on all three texture-less image sets, R models outperformed their S counterparts (Table 3df)-a remarkable generalization capability, especially on B&W and silhouette images where little to no texture information is available.

What internal mechanisms make adversarially trained CNNs more robust than standard CNNs?
We have shown that after adversarial training, R models are more robust than S models on new adversarial examples generated for these pre-trained models via PGD attacks (Table 3b). Furthermore, on non-adversarial, highfrequency images, R models may also outperform S models (Table A1a; AlexNet-R) (Yin et al., 2019;Gilmer et al., 2019).
We aim to understand the internal mechanisms that make R CNNs more robust to high-frequency noise by analyzing the networks at the weight (Sec. 3.3.1) and neuron (Sec. 3.3.2) levels.

Weight level: Smooth filters to block pixel-wise noise
Smoother filters To explain this phenomenon, we visualized the weights of all 64 filters (11×11×3), in both AlexNet and AlexNet-R, as RGB images. We compare each AlexNet filter with its nearest filter (via Spear- ResNet-R   Figure 3: Standard CNNs substantially outperform R models on scrambled versions of the ImageNet-CL images due to their capability of recognizing images using textures. Here, we report top-1 accuracy (%) on the scrambled images whose original versions (ImageNet-CL) were correctly-labeled by both standard and R classifiers (hence, the 100% for 1×1 blue bars).
man rank correlation) in AlexNet-R. Remarkably, R filters appear qualitatively much smoother than their counterparts (Fig. 4a). The R filter bank is also less diverse, e.g. R edge detectors are often black-and-white in contrast to the color-ful AlexNet edges (Fig. 4b). A similar contrast was also seen for the GoogL-eNet and ResNet models (Fig. A1). We also quantify the smoothness, in total variation (TV) (Rudin et al., 1992), of the filters of all 6 models ( Table 4) and found that, on average, the filters in R networks are much smoother.
For example, the mean TV of GoogLeNet-R is about 1.5 times smaller than that of GoogLeNet.
In almost all layers, R filters are smoother than S filters (Fig. A9).

Blocking pixel-wise noise
We hypothesize that the smoothness of filters makes R classifiers more robust against noisy images.
To test this hypothesis, we computed the total variation of the channels across 5 layers when feeding ImageNet-CL images and their noisy versions ( Fig. 1c; ImageNet-C Level 1 additive noise ∼ (0, 0.08)) to S and R models.
At , the smoothness of R activation maps remains almost unchanged before and after noise addition ( Fig. 5a; yellow circles are on the diagonal line). In contrast, the filters in standard AlexNet allow Gaussian noise to pass through, yielding larger-TV channels ( Fig. 5a; blue cir- AlexNet-R ( ). R channels have similar TV before and after adding noise, suggesting that kernels filter out the added noise. In higher layers ( and ), R channels are consistently more invariant to the input noise than S channels ( dots are clustered around the diagonal line while dots have higher variance). See Fig. A4 for the same scatter plot (a) for all five layers. cles are mostly above the diagonal). That is, the smooth filters in R models indeed can filter out pixel-wise Gaussian noise despite that R models were not explicitly trained on this image type! Furthermore, AlexNet-R and ResNet-R consistently perform better in different noise distortions (Table A1 (a)) compared to their S models.
In higher layers, it is intuitive that the pixel-wise noise added to the input image might not necessarily cause activation maps, in both S and R networks, to be noisy because higher-layered units detect more abstract concepts. However, interestingly, we still found that R channels to have consistently less mean TV (Fig. 5b-c).
Our result suggests that most of the de-noising effects take place at lower layers (which contain more generic features) instead of higher layers (which contain more task-specific features).

Neuron level: Robust neurons prefer lower-level and fewer inputs
Here, via NetDissect framework, we wish to characterize how adversarial training changed the hidden neurons in R networks to make R classifiers more adversarially robust. Network Dissection (hereafter, NetDissect) is a common framework for quantifying the functions of a neuron by computing the Intersection over Union (IoU) between each activation map (i.e. channels) and the human-annotated segmentation maps for the same input images.
That is, each channel is given an IoU score per humandefined concept (e.g. or ) indicating its accuracy in detecting images of that concept. A channel is tested for its accuracy on all ∼1,400 concepts, which span across six coarse categories: , , , , , and (Bau et al., 2017) (c.f. Fig. A11 for example NetDissect images in and concepts).
Following Bau et al. (2017), we assign each channel a main functional label i.e. the concept that has the highest IoU with. In both S and R models, we ran NetDissect on all 1152, 5808, and 3904 channels from, respectively, 5, 12, and 5 main convolutional layers (post-ReLU) of the AlexNet, GoogLeNet, and ResNet-50 architectures (c.f. Sec. A for more details of layers used). AlexNet GoogLeNet ResNet Figure 6: Total channel increases (%) in R models. For all three architectures, the numbers of NetDissect and detectors in R models increase, e.g. by 117% and 7%, respectively, for AlexNet, while the number of units decreases by 28%. See Fig. A3 for layer-wise plots for AlexNet in other object and color category.

Shift to detecting more low-level features i.e. colors and textures
We found a consistent trend-adversarial training resulted in substantially more filters that detect colors and textures (i.e. in R models) in exchange for fewer object and part detectors.
For example, throughout the same GoogLeNet architecture, we observed a 102% and a 34% increase of color and texture detectors, respectively, in the R model, but a 20% and a 26% fewer object and part detectors, compared to the S model (c.f. Fig. 6). After adversarial training, ∼11%, 15%, and 10% of all hidden neurons (in the tested layers) in AlexNet, GoogLeNet, and ResNet, respectively, shift their roles to detecting lower-level features (i.e. textures and colors) instead of higher-level features (see feature visualizations in Fig. 10).
Across three architectures, the increases in and channels are often larger in higher layers. The largest functional shifts appearing in higher layers can be because the higher-layered units are more task-specific (Nguyen et al., 2016a;Yosinski et al., 2014).

Consistent findings with ResNet CNNs trained on Stylized ImageNet
We also compare the shape-biased ResNet-R with ResNet-SIN, i.e. a ResNet-50 trained exclusively on stylized ImageNet images where textures are removed via stylization (Geirhos et al., 2019). ResNet-SIN also has a strong shape bias of 81.37%. 2 Interestingly, similar to ResNet-R, ResNet-SIN also has more low-level feature detectors (colors and textures) and fewer high-level feature detectors (objects and parts) than the vanilla ResNet (Fig. 7). In contrast, finetuning this ResNet-SIN on ImageNet remarkably changes the model to be texture-biased (at a 79.7% texture bias) and to contain fewer and more and units ( Fig. 7; ResNet-SIN+IN vs. ResNet-SIN).
That is, training or finetuning on ImageNet tend to cause CNNs to be more texture-biased and contain more high-level features (i.e. detecting objects and parts). In contrast, train-2 _ in https://github.com/rgeirhos/texture-vs-shape/. See Table 4   (2) ResNet-R was trained via PGD-adversarial training; (3) ResNet-SIN was trained on texture-removed ImageNet from (Geirhos et al., 2019); and (4) ResNet-SIN+IN was a ResNet-SIN that was then finetuned on ImageNet. Training on textureremoved ImageNet or adversarial examples consistently produces CNNs that are (a) heavily shape-biased; (b) contain more low-level, features; (c) fewer high-level, detectors compared to training on ImageNet which produces texture-biased CNNs (ResNet and ResNet-SIN+IN).
ing on adversarial examples or texture-distorted images cause CNNs to focus more on shapes and learn more generic, lowlevel features. Shift to detecting simpler objects Analyzing the concepts in the category where we observed largest changes in channel count, we found evidence that neurons change from detecting complex to simpler objects. That is, for each Net-Dissect concept, we computed the difference in the numbers of channels between the S and R model. In the same category, AlexNet-R model has substantially fewer channels detecting complex concepts e.g. −30 , −13 , and −11 detectors ( Fig. A10b; rightmost columns), compared to the standard network. In contrast, the R model has more channels detecting simpler concepts, e.g. +40 and +12 channels ( Fig. A10b; leftmost columns). The top-49 images that highest-activated R units across five layers also show their strong preference for simpler backgrounds and objects (Fig. 10). Shift to detecting fewer unique concepts The previous sections have revealed that neurons in R models often prefer images that are pixel-wise smoother (Sec. 3.3.1) and of lower-level features (Sec. 3.3.2), compared to S neurons. Another important property of the complexity of the function computed at each neuron is the diversity of types of inputs detected by the neuron (Nguyen et al., 2016b(Nguyen et al., , 2019. Here, we compare the diversity score of NetDissect concepts detected by units in S and R networks. For each channel , we calculated a diversity score, i.e. the number of unique concepts that detects with an IoU score ≥ 0.01.
Interestingly, on average, an R unit fires for 1.16 times Texture ( ) scores over all channels in the entire network ("All") or in a NetDissect category ("Object", "Color", and "Texture"). While AlexNet-R has more and channels ( above ), these R channels are not heavily shape-or texture-biased. In contrast, the corresponding channels in AlexNet are heavily texture-biased ( is almost 2× of ). , which has a NetDissect label of under category. The unit prefers circular patterns including car wheels and clock. Right: Example cue-conflict images originally labeled by AlexNet as (top) or (bottom) but that were given a different label after the unit is ablated. For example, "clock2-bicycle1" is a cue-conflict image that has the shape of a clock and the texture of a bicycle. Qualitatively, the unit helps AlexNet detect clocks and cars using shapes (top) and reddish pink cars, birds, chairs, and bicycles using textures (bottom). The unit has Shape and Texture scores of 18 and 22, respectively. See Fig. A6 & Fig. A7 for more examples. fewer unique concepts than an S unit (22.43 vs. 26.07; c.f. Fig. A12a). Similar trends were observed in ResNet (Fig. A12b). Qualitatively comparing the highest-activation training-set images by the highest-IoU channels in both networks, for the same most-frequent concepts (e.g. ), often confirms a striking difference: R units prefer a less diverse set of inputs (Fig. 10). As R hidden units fire for fewer concepts, i.e. significantly fewer inputs, the space for adversarial inputs to cause R models to misbehave is strictly smaller.

Which neurons are important for shape-based or texture-based image classification?
To understand how the found changes in R neurons (Sec. 3.3) relate to the shape bias of R CNNs (Sec. 3.1), here, we zero out every channel, one at a time, in S and R networks and measure the performance drop in recognizing shapes and textures from cue-conflict images. Shape & Texture scores For each channel, we computed a "Shape score", i.e. the number of images originally correctly labeled into the class by the network but that, after the ablation, are labeled differently (examples in Fig 9a-b). Similarly, we computed a Texture score per channel. The Shape and Texture scores quantify the importance of a channel in image classification using shapes and textures, respectively. First, we found that the channels labeled by Net-Dissect are not only important to texture-based but also shape-based classification. That is, on average, zero-ing out these channels caused non-zero Texture and Shape scores ( Fig. 8; Texture and are above 0). See Fig. 9 for an example of channels with high Shape and Texture scores.
This result is aligned with the fact that R networks consistently have more units (Fig. 6) but are shape-biased (Sec. 3.1).
Second, the units are, as expected, highly texturebiased in AlexNet (Fig. 8 Texture; is almost 2× of ). However, surprisingly, those units in AlexNet-R are neither strongly shape-biased nor texture-biased ( Fig. 8; Texture ≈ ). That is, across all three groups of the , , and , R neurons appear mostly to be generalist, low-level feature detectors. This generalist property might be a reason for why R networks are more effective in transfer learning than S networks (Salman et al., 2020).
Finally, the contrast above between the texture bias of S and R channels (Fig. 8) reminds researchers that the single semantic label assigned by NetDissect to each neuron is not describing a full picture of what the neuron does and how it helps in downstream tasks. To the best of our knowledge, this is the first work to align the NetDissect and cue-conflict frameworks to study how individual neurons contribute to the generalizability and shape bias of the entire network.

Related Work
Shape bias On smaller-scaled datasets e.g. CIFAR-10, Zhang and Zhu (2019) also found that R networks rely heavily on shapes (instead of textures) to classify images. However, such shape bias may not necessarily generalize to highdimensional, large-scale datasets such as ImageNet. Different from Zhang and Zhu (2019), here, we study networks trained on ImageNet and its variants (e.g. Stylized ImageNet). To the best of our knowledge, we are the first to reveal (1) the shape bias of ImageNet-trained R networks; (2) the roles of neurons in R networks; and (3) their internal robustification mechanisms. We also found that our adversarially-trained CNNs and the CNNs trained on texture-removed images by Geirhos et al. (2019) both similar have a strong shape bias and contain simpler and more generic features than standard ImageNet-trained CNNs. Simplicity bias Deep neural networks tend to prioritize learning simple patterns that are common across the training set (Arpit et al., 2017). Furthermore, deep ReLU networks often prefer learning simple functions (Valle-Perez et al., 2019;De Palma et al., 2019), specifically low-frequency functions (Rahaman et al., 2019), which are more robust to random parameter perturbations. Along this direction, here, we have shown that R networks (1) have smoother weights (Sec. 3.3.1), (2) prefer even simpler and fewer inputs (Sec. 3.3.2) than standard deep networks-i.e. R networks represent even simpler functions. Such simplicity biases are consistent with the fact that gradient images of R networks are much smoother  and that R classifiers act as a strong image prior for image synthesis . Robustness Each R neuron computing a more restricted function than an S neuron (Sec. 3.3.2) implies that R models would require more neurons to mimic a complex S network. This is consistent with recent findings that adversarial training requires a larger model capacity . In Sec. 3.3.1, we found that adversarial training produces network that are robust to images with additive Gaussian noise. Interestingly, Ford et al. (2019) found the reverse is also true: Training CNNs with additive Gaussian noise can improve the performance on adversarial examples.
While AdvProp did not yet show benefits on ResNet , it might be interesting future work to find out whether EfficientNets trained via AdvProp also have shape and simplicity biases. Furthermore, simplicity biases may be incorporated as regularizers into future training algorithms to improve model robustness. For example, encouraging filters to be smoother might improve robustness to high-frequency noise.
Also aligned with our findings, Rozsa and Boult (2019) found that explicitly narrowing down the non-zero input regions of ReLUs can improve adversarial robustness.

Discussion and Conclusion
We found that R networks heavily rely on shape cues in contrast to S networks. One may fuse an S network and a R network (two channels, one uses texture and one uses shape) into a single, more robust, interpretable ML model. That is, such model may (1) have better generalization on OOD data than S or R network alone and (2) enable an explanation to users on what features a network uses to label a given image.
Our study on how individual hidden neurons contribute to the R network shape preference (Sec. 3.4) revealed that texture-detector units are equally important to the texturebased and shape-based recognition. This is in contrast to a common hypothesis that texture detectors should be exclusively only useful to texture-biased recognition. Our surprising finding suggests that the categories of stimuli in the well-known Network Dissection (Bau et al., 2017) need to be re-labeled and also extended with low-frequency patterns e.g. single lines or silhouettes in order to more accurately quantify hidden representations.
It might be interesting future work to improve model performance by (1) training them jointly on adversarial examples and texture-less images; and (2) adding smoothness prior to network gradients and filters. Also, adding a smoothness prior to the gradient images or filters of convolutional networks may improve their generalization capability.
In sum, our work has revealed several intriguing properties of adversarially-trained networks, providing insights for future designs of classifiers robust to out-of-distribution examples.

B. Kernel smoothness visualization
We visualize the weights of the 6 models and plot them side by side to compare their kernel smoothness (Fig. A1). The kernels in R models a consistently smoother than its counter part. For a more straight forward visualization, we use Spearman rank correlation score to pair the similar kernels in AlexNet and AlexNet-R (Fig. A2).

C. Object and color detectors of AlexNet
We compared the layer-wise difference of and detectors of AlexNet and AlexNet-R (Fig. A3). We can see in Fig. A3a that AlexNet has more object detectors and fewer color detectors compared to AlexNet-R.

D. Total variance (TV) of AlexNet & AlexNet-R on clean/noisy images
The total variances of AlexNet & AlexNet-R plot (Fig. A4) shows that the early layer i.e.
in AlexNet-R can filtered out Gaussian noise. The total variances of clean and noisy input results in smilar value in in AlexNet-R, this means the noise has been filtered by .

E. ImageNet-C evaluation
In Table A1, we evaluate the validation accuracy of 6 models on 15 common types of image corruptions in ImageNet-C. It turns out that shape biased model does not necessary mean better generalization on these image corruptions. But the R models of AlexNet and ResNet do shows better performance in noise and blur distortions.
ResNet-R Figure A1: All 64 filters of in each standard network (left) and its counterpart (right). The filters of R models (right) are smoother and less diverse compared to those in standard models (left). Especially, the edge filters of standard networks are noisier and often contain multiple colors in them.

F. Examples of shape-less and texture-less images
We randomly choose one image in 7 COCO coarser classes (out of 16) and plot the corresponding shape-less and textureless version (Fig. A5).

G. Visualizing channel preference via cue-conflict and NetDissect
In Fig. A6-A8, we show three samples of the channel preferences experiment. In each of the sample, Top is the top-49 images of the channels (Similar to Fig. 10). On the Middle & Bottom, we zero-out the corresponding channel and re-run the conflict test to find out the images that were mis-classified. i.e. Fig.A6 the clock images in Middle Figure A2: filters of AlexNet-R are smoother than the filters in standard AlexNet. In each column, we show an AlexNet filter filter and their nearest filter (bottom) from the AlexNet-R. Above each pair of filters are their Spearman rank correlation score (e.g. : 0.36) and their total variation (TV) difference (i.e. smoothness differences). Standard AlexNet filters are mostly noisier than their nearest R filter (i.e. positive TV differences).  , AlexNet-R have fewer object detectors but more color detector units compared to standard AlexNet. The differences between the two networks increase as we go from lower to higher layers. Because both networks share an identical architecture, the plots here demonstrate a substantial shift in the functionality of the neurons as the result of adversarial training-detecting more colors and textures and fewer objects. Similar trends were also observed between standard and R models of GoogLeNet and ResNet-50 architectures.
were classified into shape category by cue-conflict test. After zeroing-out the channel, the network lose the ability to classify the image into shape category.  Figure A4: Each point shows the Total Variation (TV) of the activation maps on clean and noisy images for an AlexNet or AlexNet-R channel. We observe a striking difference in : The smoothness of R channels remains unchanged before and after noise addition, explaining their superior performance in classifying noisy images. While the channel smoothness differences (between two networks) are gradually smaller in higher layers, we still observe R channels are consistently smoother. : Applying different transformation that remove shape/texture on real images. We randomly show an example of 7 out of 16 COCO coarser classes. See Table 3 for classification accuracy scores on different images distortion dataset in 1000 classes(Except for Silhouette). *Note: Silhouette are validate in 16 COCO coarse classes. Figure A6: AlexNet with Shape and Texture scores of 18 and 22, respectively. It has a NetDissect label of (IoU: 0.0568) under category. Although this neuron is in NetDissect category, the misclassified images suggest that this neuron helps in both shape-and texture-based recognition. Top: Top-49 images that highestactivated this channel. Middle: Mis-classified images in category (18 images). Bottom: Mis-classified images in category (22 images). Figure A7: AlexNet-R with Shape and Texture scores of 17 and 19, respectively. It has a NetDissect label of (IoU: 0.0409) under category. This neuron has almost equal Shape and Texture scores and is useful in detecting both the shape and textures of knives and bottles at the same time. Top: Top-49 images that highest-activated this channel. Middle: Mis-classified images in category. Bottom: Mis-classified images in category. Figure A8: AlexNet with Shape and Texture scores of 3 and 17, respectively. It has a NetDissect label of (IoU: 0.0542) under category. This is a heavily texture-biased neuron that helps networks detect animals by their fur textures. Mean TV

ResNet50
ResNet50-R (c) Mean layer-wise kernel TV of ResNet and ResNet-R Figure A9: For all main layers, AlexNet-R filters are smoother (i.e. lower TV mean) than their counterparts in AlexNet (a). The same observation was found in GoogLeNet-R vs. GoogLeNet comparison (b). In ResNet-R, its early filters at are also smoother than those in ResNet. Mean diversity score ResNet ResNet-R (b) ResNet layer-wise mean diversity Figure A12: In each plot, we show the mean diversity scores across all channels in each layer. Both AlexNet-R and ResNet-R consistently have channels with lower diversity scores (i.e. detecting fewer unique concepts) than the standard counterparts.