{'fraction_non_alphanumeric': 0.02934737281067556, 'fraction_numerical': 0.016941201000834027, 'mean_word_length': 5.027332704995287, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Understanding the effect of sparsity on neural networks robustness

This paper examines the impact of static sparsity on the robustness of a trained network to weight perturbations, data corruption, and adversarial examples. We show that, up to a certain sparsity achieved by increasing network width and depth while keeping the network capacity fixed, sparsified networks consistently match and often outperform their initially dense versions. Robustness and accuracy decline simultaneously for very high sparsity due to loose connectivity between network layers. Our findings show that a rapid robustness drop caused by network compression observed in the literature is due to a reduced network capacity rather than sparsity.


Introduction
Deep learning methods are increasingly used for solving complex tasks, yet little is known about the choice of the best architecture, the required model size, capacity, and the trade-offs involved. A common strategy is to train overparameterized models and compress them into smaller representations (Hoefler et al., 2021). This works remarkably well with an almost negligible drop in accuracy (Gale et al., 2019;Blalock et al., 2020), and is crucial to make use of these models in resource-constrained environments. Recent works, however, shows that test accuracy does not capture how model compression impacts the generalization properties of these models (Hooker et al., 2020;Entezari & Saukh, 2019).
Related literature refers to robustness as the network generalization ability to small shifts in the distribution that humans are usually robust to. There is a growing body of work studying methods for building robust models. Recent studies (Shankar et al., 2020;Recht et al., 2019) found that image classification models show a consistent accuracy drop when evaluated on ImageNet (Deng et al., 2009) and Im-ageNetV2 (Recht et al., 2019), while humans achieve the same accuracy. Another line of research aims at minimizing the worst case expected error over a set of probability distributions by applying distributionally robust optimization (Shafieezadeh-Abadeh et al., 2015;Duchi et al., 2020;Sagawa et al., 2020). A similar line of work focuses on finding models that have low performance drop on adversarial examples (Biggio & Roli, 2018;Madry et al., 2019).
A recent study by Hooker et al. (2020) shows that model compression, and to a smaller extent quantization, result in tremendous robustness degradation. At the same time, Golubeva et al. (2021) found that wider networks of the same capacity (same number of parameters) yield better performance. Model compression leads simultaneously to sparser and lower capacity networks, yet the contribution of both effects is mixed. Understanding the impact of these effects on model robustness in isolation is crucial when optimizing machine learning models for resource-constrained devices. This work evaluates the effect of model sparsification while keeping the network capacity, defined by the total number of parameters, fixed.
Contributions. We hypothesise that sparsity alone does not hurt model robustness when the network capacity is fixed and provide empirical evidence to support this hypothesis in a number of settings. We run our study on a range of network architectures (MLPs, VGG and ResNets), datasets (MNIST, CIFAR-10, CIFAR-100), robustness tests (weight perturbations, data corruptions, adversarial examples) and evaluate the overall and per class network performance. We observe that for randomly initialized models with a static sparsity pattern applied before or after training, network sparsification does not hurt or even improves robustness to a certain sparsity compared to a dense network of the same capacity. Robustness and accuracy decline simultaneously for very high sparsity due to loose connectivity between network layers. We show that our hypothesis holds when introducing sparsity by increasing network width and depth in separate experiments, applied before and after training. These findings show that a rapid robustness drop caused by network compression observed in the literature is due to a reduced network capacity rather than sparsity. Figure 1. Robustness to weight perturbations, sparsification by increasing width. We add multiplicative Gaussian noise zi ∼ N (µ, w 2 i σ 2 i ) to each weight and evaluate model performance. We observe that as we move towards higher sparsity levels, the performance first increases then decreases in extreme sparsity levels. We note that such increase is happening earlier for simpler tasks like MNIST. This performance improvement indicates a flatter loss landscape around the minima suggesting better generalization.  . Robustness to data corruption, sparsification by increasing width. We evaluate the performance of the models on corrupted datasets MNIST-C, CIFAR10-C and CIFAR100-C. We observe that as we move towards higher sparsity levels, the performance first increases then decreases in extreme sparsity levels. We note that such increase is happening earlier for simpler tasks like MNIST.

Experimental Framework
We hypothesise that sparsity, while keeping the number of parameters fixed, does not hurt network robustness. We support our hypothesis by exhaustive tests covering multiple datasets, network architectures, model and data corruptions, sparsity levels, sparsification methods and schedules. The details are given below.
Datasets and architectures. The datasets used in the experiments include MINST (LeCun & Cortes, 2010), CIFAR-10 ( Krizhevsky et al., 2009), and CIFAR-100) (Krizhevsky et al., 2009). We fix the number of weights in each network architecture (one layer MLP, VGG16 (Simonyan & Zisserman, 2015), ResNet18 (He et al., 2015)) throughout all experiments, by increasing the width or depth and introducing the proper corresponding sparsity. See sparsification methods for more details. We use one layer MLP with 2 7 hidden units, VGG with 11 layers, and ResNet18 as base architectures. We refer to these vanilla architectures as to 100 %-networks before sparsification. Note that for both ResNet and VGG our vanilla implementation uses the layer width of 16 as the base architecture, which is lower than 64 used in the original architecture. We use width to set the number of output channels for the first layer and use the same width ratios as the respective vanilla architectures for the following layers. All networks were trained using SGD with momentum 0.9. Details for each model family are provided in Section A.
Sparsification methods. Existing literature covers multiple ways to make use of sparsity during and after model training including static and dynamic sparsity (e.g., β-Lasso (Neyshabur, 2020)), iterative hard thresholding (e.g., Lottery Ticket Hypothesis with various pruning strategies (Frankle & Carbin, 2018;Renda et al., 2020)) and others. (Hoefler et al., 2021) provides a comprehensive survey on pruning strategies. Sparsification without changing the number of parameters was investigated in (Golubeva et al., 2021). In their study static sparsity showed the most prominent impact on network performance and is thus adopted in this work.
We sparsify a network while preserving its capacity by changing the network's width or depth. When sparsifying by increasing width, we leverage the approach introduced . Robustness to weight perturbations. Sparsification after training by increasing width. We add multiplicative Gaussian noise zi ∼ N (µ, w 2 i σ 2 i ) to each weight and evaluate performance on test data. As we move towards higher sparsity levels, the performance decreases in extreme sparsity levels. . Robustness to data corruption. Sparsification after training by increasing width. We evaluate on the corrupted datasets MNIST-C, CIFAR10-C and CIFAR100-C. sompare to static sparsity at the prior to training, robustness degrades sooner.
in (Golubeva et al., 2021): every layer of the network is sparsified by removing weights at random in proportion to the layer size, using a static mask generated at initialization. This approach is referred to as static sparsity. We build on its publicly available implementation (Golubeva et al., 2021). Sparsifying by increasing network depth involves duplicating layers and then applying a static random mask to sparsify the weight tensors. When sparsifying by increasing depth, we consider MLP with 2 9 hidden units in each layer, and add layers of the same size. For VGG and ResNet we build architecture families VGG11, VGG13, VGG16 and ResNet18, ResNet34, ResNet50 all enjoying the default width of 64.
Sparsification schedules. In addition to static sparsity applied prior to network training, we also investigate network pruning after training by removing a certain amount of weights with the lowest magnitude to match the required sparsity level. Note that no fine-tuning is applied.
Model perturbation. Similarly to (von Oswald et al., 2021), we perturb model weights by applying Gaussian noise z i ∼ N (µ, w 2 i σ 2 i ) in proportion to the magnitude of each weight w i , i ∈ L, and then measure the difference in the loss δL = E z [L(w i + z) − L(w i )]. Accuracy drop due to model perturbation is related to the flatness of the loss landscape around the obtained optimum. Robustness to weight perturbation could also represents a proxy for quantization error. This error is introduced in neural network compression by weight quantization in the literature (Novac et al., 2021).
Corrupted data. We apply numerous algorithmically generated corruptions, similar to the ones evaluated in (Hooker et al., 2020) (e.g., blur, contrast, pixelation) to all datasets used in this paper. This allows us investigating how sensitive the sparsified models are to data corruptions of different severity which humans are oblivious to. Our corrupted datasets are MNIST-C (Mu & Gilmer, 2019), CIFAR10-C and CIFAR100-C (Hendrycks & Dietterich, 2019).
When applying sparsity, we evaluate both the overall model performance and its performance on the most sensitive class. We follow the methodology introduced in (Hooker et al., 2020) and evaluate the change to class level recall compared to the overall model accuracy. The obtained results are presented below.

Results
Perturbed model weights. We first investigate the networks that were sparsified while growing the width to keep their capacity fixed. Figure 1 shows that as we move towards higher sparsity levels, the test performance first increases then decreases in extreme sparsity levels. We note that such increase is happening earlier for simpler tasks like MNIST. We observe that sparse configurations are indeed in flatter regions of weight space as δL increases more slowly with δz i . This suggests better robustness and generalization around the minima (Pittorino et al., 2020;Jiang et al., 2019). Each point in this plot shows the mean over five networks trained from different initializations. When sparsification is applied while increasing network depth, the maximum accuracy and robustness are achieved for smaller depth values in all experiments. Note that keeping a network connected while increasing its depth, in contrast to width, becomes difficult with higher sparsity. The results are summarized in Section B.1. The outcome across all experiments consistently suggests that sparsification alone does not undermine network robustness to weight perturbations as long as sufficient network connectivity is maintained.
Corrupted data. Figure 2 evaluates the performance of the models on corrupted datasets MNIST-C, CIFAR10-C and CIFAR100-C. We observe that as we move towards higher sparsity levels, the test performance first increases then decreases in extreme sparsity levels. We note that such increase is happening earlier for simpler tasks like MNIST. Each point in Figure 2is mean performance over three trained networks. For each network we randomly sample 1000 examples from a dataset and add five noise samples in each run. On CIFAR10-C and CIFAR100-C our evaluation considers corruption severity of two and four as classified by (Hendrycks & Dietterich, 2019). Detailed results for specific corruption types can be found in Section B.2. The results for the achieved performance of networks sparsified by increasing depth are also shown in Section B.1. We note that VGG networks experience convergence issues as the network sparsity approaches 10% due to lacking connectivity between layers. This is not the case for MLP and ResNet which also converge for lower percentage of remaining weights. We attribute these differences to the power of skip connections in ResNet and low overall tested network depths (1,2,4 and 8) for MLP.
Sensitive classes. Similarly to (Hooker et al., 2020), sen-sitive classes are considered those with the lowest recall. For each sparsity level, we train five models, evaluate them on the test data and report the minimum recall among all classes. (Hooker et al., 2020) shows that there are some particular examples in each class that a pruned network forgets easily. However, we observe that as the networks get wider (or deeper) and sparser the minimum recall does not decrease. Sparsification does not disproportionally affect sensitive classes, which may not be noticeable by just looking at the overall accuracy. This is due to the fact that the capacity of the networks is fixed. The results are shown in Figure 8 in Section B.1.
Adversarial attacks. Figure 3 shows the robustness of sparsified networks when applying adversarial attacks to perturb test data. We observe a consistent trend for robustness to all adversarial attacks (BIM (Kurakin et al., 2016), APGD (Croce & Hein, 2020), PGD (Madry et al., 2019), FGSM (Goodfellow et al., 2014)). Similar to perturbed model weights and corrupted data, as we have less remaining weights, test performance for adversarial examples is first improved and then decreases for extreme sparsity levels where the overall (clean) network accuracy drops. Dense VGG networks trained on MNIST show the highest accuracy decline in the presence of all attacks, while sparsification helps to improve adversarial robustness.
Post-training sparsification. Figure 4 depicts the results for post-training sparsification for MLP and VGG architectures challenged with perturbed model weights. The results indicate a similar trend to the experiments with static sparsity applied at initialization. For VGG we observe a slight improvement followed by an accuracy drop. However, the performance does deteriorate sooner than with static sparsity. For MLP the results show stagnating accuracy and a slight drop in performance on CIFAR-10. We attribute this to the simplicity of our sparsification method and a relatively low number of weights in the one layer MLP. Similar results are obtained on corrupted datasets visualized in Figure 5.

Conclusion
In this work we hypothesise that sparsity, while keeping the number of parameters fixed, does not hurt network robustness. We provide experimental evidence to support this claim based on several standard architectures, datasets, sparsification methods and measures of robustness. Our observation is that network sparsification often helps to improve robustness compared to a dense model, yet the benefits decline together with the overall model accuracy for high sparsity levels. This is due to the increasingly loose connectivity between layers which complicates optimization. Since network capacity rather than sparsity causes accuracy and robustness drop of compressed models, designing pruning methods that treat network capacity and sparsity separately can lead to better compressed models. In addition, our work emphasizes the need for training procedures that better support sparse operations, which would allow for a faster and more memory efficient training of sparse networks.

A. Implementation Details
We used Caliban (Ritchie et al., 2020) to manage all experiments in a reproducible environment in Google Cloud's AI Platform. Each point in plots show the mean value taken over at least five different runs.
A.1. Training for MLP • Width experiments: we use VGG11 and ResNet18 changing the width of the first layer and adapting the following layers according to the ratios in the vanilla version of the networks.
• Depth experiments: we change the architecture: VGG11, VGG13, VGG16 and ResNet18, ResNet34, ResNet50 while keeping the default width of the vanilla networks.

B.1. Sparsification by increasing network depth
In addition to increasing network width before applying random static sparsification, we also test sparsification by increasing network depth. In this case, we duplicate layers while keeping their width unchanged. In contrast to wider networks, high sparsity levels severely impairs connectivity between layers of deeper networks which make these more difficult to train and converge. VGG experience convergence issues when sparsifying by increasing depth when sparsity approaches 10 %. ResNet appears less susceptible to this issue due to the presence of skip connections, whereas MLPs are more robust due to a considerably lower depth in our settings (1,2,4 and 8). Our experiments with deeper MLP networks and high sparsity also reveal convergence issues similarly to VGG. We run experiments for increasing depth for all architectures and datasets up to a point where the overall network accuracy starts to drop and as long as network training converges. Figure 6 and Figure 7 show the results for weight perturbations and corrupted data of varying severity. The results are similar and show that robustness improves as long as network accuracy without intervention remains steady. Figure 8 shows the results for the lowest recall among all classes. Network accuracy on the most sensitive classes does not decline with sparsity. The test is conducted without weight perturbation or data corruption.

B.2. Detailed results for corrupted data
The plots presented in Figure 9, Figure 10 and Figure 11 provide details on the impact of individual corruption methods on network performance. We show details for MLP, VGG and ResNet architectures trained on MNIST, CIFAR-10 and CIFAR-100. Sparsification is achieved by increasing network width. Although the effect of data corruptions on model performance varies widely, it can be observed that in all cases a sparser network matches the accuracy of the vanilla 100 % network. This observation holds up to high sparsity levels where the overall model performance declines. . Robustness to weight perturbations. Sparsification by increasing network depth We add multiplicative Gaussian noisezi ∼ N (µ, w 2 i σ 2 i ) to each weight and evaluate model performance. There is a sweetspot corresponding to optimal sparsity. With higher depth and sparsity network connectivity declines leading to simultaneous accuracy and robustness drop. . Robustness to corrupted data. Sparsification by increasing network depth Corrupted datasets MNIST-C, CIFAR10-C and CIFAR100-C. There is a sweetspot corresponding to optimal sparsity. With higher depth and sparsity network connectivity declines leading to simultaneous accuracy and robustness drop.