{'fraction_non_alphanumeric': 0.03338079941716064, 'fraction_numerical': 0.014670331489883101, 'mean_word_length': 4.8286045165026055, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Holistic Deep Learning

There is much interest in deep learning to solve challenges in applying neural network models in real-world environments. In particular, three areas have received considerable attention: adversarial robustness, parameter sparsity, and output stability. Despite numerous attempts to solve these problems independently, little work simultaneously addresses the challenges. In this paper, we address the problem of constructing holistic deep learning models by proposing a novel formulation that solves these issues in combination. Real-world experiments on both tabular and MNIST datasets show that our formulation can simultaneously improve the accuracy, robustness, stability, and sparsity over traditional deep learning models among many others.

However, some significant drawbacks can prevent Deep Learning from being viable for critical tasks, including healthcare, policy-making, or self-driving decisions. First, it has been shown that many state-of-the-art DNNs are highly vulnerable to adversarial inputs, and several highly effective techniques have been devised for finding such inputs (Goodfellow et al., 2014;Carlini and Wagner, 2016). This is a dangerous phenomenon in real-world scenarios. For self-driving cars, for example, it would be unacceptable to use a machine learning algorithm that may confuse a street sign because of an adversarial sticker. Robustness is therefore a desirable property.
Second, due to the stochastic nature of neural network training, Deep Learning models can suffer from instability and high dependence on the specific train/test split. Such variability can significantly hurt the resulting model's interpretability and brings into question the utility and applicability of the model's predictions, see Bertsimas and Paskov (2020) for further discussion on this. Indeed, the question of how to construct models that are robust to the specific choice of training set has traditionally been overlooked.
Third, the high number of parameters in the neural networks prevents efficient computation and storage. Most neural networks have millions of non-zero parameters that must be stored and accessed for evaluation. This severely limits the applicability of these models in memory-limited and low-power environments (e.g., phones and smart watches). If a neural network could be sparsified so that it has only a small number of non-zero weights, this could greatly compress the network size and further improve the explainability of the underlying model.
Numerous methods have been devised to address robustness or sparsity but there is no unifying groundwork to produce a holistic model that would maintain high accuracy while being robust to adversarial attacks, stable with relation to the choice of training split, sparse, and not too expensive to train. This paper aims to propose such a combination through a novel comprehensive methodology named Holistic Deep Learning (HDL).
Our contributions are the following: 1. We design a novel Holistic Deep Learning framework that successfully trains robust, stable, and sparse feed-forward neural networks at scale.
2. We show that it is possible to train models that do not exhibit any trade-off and do not sacrifice natural accuracy performance for better robustness.
3. We report an extensive benchmark of all methodologies on various tasks including a large selection of UCI datasets and the MNIST imaging benchmark, highlighting the superior performance of Holistic Deep Learning over standalone approaches.
The paper is organized as follows: Section 2 outlines the current state of robust, sparse, and stable methods in the discipline; Section 3 elucidates the key challenges of Holistic Deep Learning and explains detailed methodologies to resolve these challenges, as well as the operational principles underlying our Machine Learning models; Section 4 outlines the data and the design of the experiments we report in Section 5. Section 6 deals with conclusions from the results and validates the effectiveness of our HDL framework.

Related Work
This section provides an overview of the deep learning methodologies to build robust or sparse neural networks and defines the notion of stability.

Robust Neural Networks
Adversarial robustness measures a neural network's resilience against adversarial inputs (Szegedy et al., 2014b) that are original inputs perturbed slightly in a way that causes them to be misclassified by a neural network. It has been shown that many state-of-the-art DNNs are highly vulnerable to adversarial inputs, and several highly effective techniques have been devised for finding such inputs (Goodfellow et al., 2014;Carlini and Wagner, 2016). This is a dangerous phenomenon of practical concern for real-world scenarios (e.g., self-driving cars, face recognition).
Since this weakness was exposed, numerous heuristics for robustifying neural networks were proposed. However, most of them do not substantially increase robustness over undefended neural networks against a specific attack (Athalye et al., 2018). In terms of effectiveness, the adversarial training proposed by Madry et al. (2017) demonstrates the best performance but at a substantial computational cost. It is based on data augmentation with adversarial examples found by iterative projected gradient descent (PGD) with random starting points.
To the best of our knowledge, a defense mechanism that can balance effectiveness and efficiency is yet to be found.
On the other hand, robust optimization tools have been scarcely utilized in the context of neural networks. Robust optimization is a flexible framework that provides theoretical guarantees of the solutions found by numerical solvers (Ben-Tal et al., 2009;Bertsimas and den Hertog, 2021;Bertsimas et al., 2019) proposed robust formulations of Logistic Regression, Support Vector Machine, and Decision Tree classifiers models and were able to improve their performance.

Defense against adversarial attacks
Most defense strategies against adversarial attacks can be subdivided into five categories: • Adversarial training: the training data is augmented with adversarial examples to make the models more robust against them (Szegedy et al., 2014b;Madry et al., 2017;Tramèr et al., 2017;Yu et al., 2019).
• Manifold projections: an input sample is projected in a lower dimensional space in which the neural network has been trained to be particularly robust (Ilyas et al., 2017;Lamb et al., 2018).
• Regularization in the loss function: an additional penalty term is added to the optimized objective function to upper bound or to approximate the adversarial loss (Hein and Andriushchenko, 2017;Yan et al., 2018).

Sparsity in Deep Learning
Sparsity is a desirable property in machine learning because it can save computational resources, facilitate interpretations, and prevent overfitting, leading to improved performance. Sparsity corresponds to selecting a small subset of the original number of features to make predictions.
With the explosion of the number of neural network parameters in deep learning, there is also interest in using sparser networks, in terms of the number of neurons and connections, to decrease computational costs and save memory space.
There are two main sparsity paradigms in deep learning: (i) train-then-sparsify, by pruning neurons or connections after training completion, and (ii) sparsify-during-training, by learning a sparse structure from scratch while training.
Pruning consists of reducing the size of the neural network by removing unnecessary neurons or connections after training the network. Janowsky (1989) and Lecun et al. (1989) initially proposed to re-train ("fine-tune") the models after the sparsification to obtain higher accuracy. This general pruning method has been widely investigated, particularly the scheme to choose which connections to prune as explained by Hoefler et al. (2021). Han et al. (2015) proposed, for example, to prune the less important connections using their magnitude as a proxy, i.e., all weights under a specific threshold are set to zeroes. Many other variations exist, using, for instance, convex optimization (Aghasi et al., 2018), computing the sums of absolute weights of the filters of each layer and pruning the ones with the smallest sums (Li et al., 2016), or computing the average percentage of zeroes (Hu et al., 2016), or determining what channels contribute to a network's discriminative power (Zhuang et al., 2018).
On the other hand, multiple methodologies exist to train sparse networks from scratch by changing the connectivity patterns directly during training (e.g., (Bellec et al., 2017;Mocanu et al., 2017;Mostafa and Wang, 2019)).
Overall, the literature provided numerous techniques for inducing sparsity, and the set of models and datasets used as benchmarks has grown very large. We refer the reader to Gale et al. (2019) and Hoefler et al. (2021) for more comprehensive surveys.
We decided to employ the continuous sparsification framework from Savarese et al. (2020) to implement sparsity in our holistic framework. The core idea (named the "Lottery Ticket Hypothesis") is to utilize successive rounds of retraining to arrive at a sparse initialization for the neural network that could be trained to high accuracy. They propose a practical algorithm that uses a deterministic approximation to sparsify the weights with 0 regularization on dense layers and convolutional neural networks. The methodology approximates an objective function with 0 penalty using a series of sigmoid functions that converge to the true 0 penalty as the "hardness parameter" β tends to infinity. The training process gradually increases the low β value over time to suppress the magnitudes of certain trained weights. These weights are then utilized as the new initialization point for multiple rounds of retraining until the termination condition is reached (see Section 3.3 for more details).

Stability
The stability of a machine learning model, with respect to the initialization of its parameters and the choice of a training split, is an important property to understand. The sensitivity to initial parameter choices has been studied extensively (see Michel et al. (1988) and Novak et al. (2018) for more information), but the sensitivity to the choice of training split has not been treated adequately. Indeed, the closest work to this comes from either cross-validation or distributionally robust optimization. Cross-validation can be used to measure the variability from the choice of training split, albeit at a significant increase in computational cost, see Hastie et al. (2001) or Krogh and Vedelsby (1994) for more information. Additionally, distributionally robust optimization has been used to quantify the worst-case generalization error in the presence of shifts in distribution or regime, see Staib and Jegelka (2019) and Goldwasser et al. (2020), but this work remains largely theoretical. In both cases, while a greater understanding of the sensitivity to the choice of training split is achieved, neither succeeds in providing a prescription for explicitly constructing models that are robust to the choice of training split. This latter setting has been studied by Bertsimas and Paskov (2020) and Bertsimas et al. (2021b) and we adopt their notion of stability for our present work, which we summarize below.
The variability of a method's output over different training sets represents an important measure of its stability. Indeed, this can be quantified via the following methodology: 1. Assuming the data has already been split into training/testing sets, build a model on the training data and record the prediction assigned to each point.
2. Repeat this procedure s times, keeping track of the prediction assigned to each point.
3. This will yield, for each point, an empirical confidence interval, capturing the range of predictions produced by the s models constructed from the training data. The narrower the interval, the better, indicating the method is more consistent in producing decisions.
4. For classification problems on K classes with n observations, we let p ik the proportion of assignments to class k ∈ [K] of point i ∈ [n] among the s models and define The above score quantifies the stability of predictions made by a method with lower scores leading to higher stability.

The HDL Network
We first introduce the full HDL network for a classification problem over points x ∈ R M whose target y is one of K different classes. For simplicity of notation, we illustrate the HDL network over a fully-connected neural network, but the framework remains the same for convolutional neural networks.
For x ∈ R, define [x] + = max{0, x} (the ReLU function). Given weight matrices W ∈ R r −1 ×r and bias vectors b ∈ R r for ∈ [L], such that r 0 = M, r L = K, the corresponding feed-forward neural network with L layers and ReLU activation functions is defined by the equations: where W denotes the parameters (W , b ) for all ∈ [L]. Consider a data set {(x n , y n )} N n=1 , where y n is the target class of x n . For each point x n , the class predicted by the network is arg max k z L k (W, x n ).
The cross-entropy loss function for a nominal network as described in (3) can be written as: where e k refers to the one-hot vectors with a 1 in the k th coordinate and 0 everywhere else.
On the other hand, our HDL optimization problem for the cross-entropy loss function can be written as: where corresponds to the element-wise product, σ is the standard sigmoid function, z L (·, x) was defined in (3), λ (resp. ρ) is the regularization coefficient corresponding to the 0 (resp. robustness) loss component, and a is the size of the data subsets used for the stability requirement (see Section 2.3).
We observe that robustness adds a term to the output, while stability and sparsity add new parameters (θ and s respectively) to be optimized. Our core contribution is to design a successful holistic deep learning framework that simultaneously trains robust, sparse, and stable feed-forward neural networks at scale. We now describe how to implement each component so that they all remain compatible.

Robustness
This section describes our method to introduce the robust component into neural network training. Since our ultimate goal is to incorporate the sparsity, robustness, and stability of neural networks together in a tractable way, we avoid training algorithms that significantly increase the training time. We follow the approach from Bertsimas et al. (2021a) of using a linear approximation of the neural network to estimate the robust objective. This algorithm is simple to implement, produces good adversarial accuracies, and does not require the extensive training time of other algorithms.
For a given (x, y) pair, the robust problem using the cross-entropy loss and the ∞ -norm uncertainty sets can be upper bounded as is piece-wise linear, we expect the outputs z L k (W, x) and z L k (W, x+δ) to be in the same linear piece when x + δ is close to x. In other words, the linear approximation is exact for small enough δ. Therefore, we approximate the upper bound in (7) as Even though the expression in Eq. (10) is not guaranteed to be an upper bound of Eq. (7) for an arbitrary value of ρ, Bertsimas et al. (2021a) experimentally shows that the average loss obtained using this expression is an upper bound of the average adversarial loss. In fact, for small ρ, their experiments demonstrate that this approach achieves competitive results with state-of-the-art methods while requiring significantly less computational time across a variety of tabular and image data sets. However, we emphasize that the methodology developed in this paper could also be performed with other methods for robust training like adversarial training, which might be more appropriate for large uncertainty sets.

Sparsity
In this work, we use the specific retraining procedure proposed by Savarese et al. (2020) which deterministically approximates the 0 regularization using a sequence of sigmoid functions. Notably, the implementation is easily compatible with our other robustness and stability requirements since this methodology avoids the two-step procedure of the trainthen-sparsify schemes and relies on a penalty term added in the loss function. Therefore, it allows simultaneous optimization of the objective function by gradient descent, comprising the robustness, stability, and sparsity penalties.
The 0 regularization explicitly penalizes the number of non-zero weights in the model to induce sparsity. However, the 0 -norm induces a priori a non-convex and non-differentiable loss function R(W), as follows: where |W| is the dimensionality of the parameters, λ is the regularization factor's weight and L a loss function (e.g., cross-entropy loss).
The goal is to relax the discrete nature of the 0 penalty to preserve an efficient continuous optimization while allowing for exact zeros in the neural network weights. To do this, Savarese et al. (2020) propose to first parameterize the weights is the Heaviside step function, and then approximate the non-differentiable step function with the sigmoid function: σ(βs j ) → H(s j ) when β → ∞. Therefore, β is the hardness parameter that controls how close the approximation is to the 0 regularization, and the final loss function can be written as: To achieve a sparse network, we use this loss function (12) over multiple rounds of retraining to gradually reach a sparse initialization before training an accurate neural network. To reach this initialization, within each round of training, we start with our initialized auxiliary sparsity s 0 and hardness β = 1 parameters. Over the training iterations, we gradually increase β until the training procedure is completed. Then, we take s 0 = min(βs j , s 0 ) to generate the new initialization for the next round of training. This minimization function essentially keeps the information of the suppressed weights, i.e., σ(βs j ) ≈ 0, while reverting those that are not suppressed to their starting position. This process is completed over multiple rounds to find better and sparser initializations for the neural network.
Overall, we implement the methodology as initially proposed by Savarese et al. (2020).
In the results section, we measure sparsity in terms of the number of neuron connections (weights) that are set to 0.

Stability
With the measure of stability defined in Section 2.3, we now proceed to derive a methodology for building stable models. At a high level, this corresponds to constructing a model that is robust to the specific dataset used to train it. One way to think about this is to view the training dataset as a sample from the true data distribution and then require that the resulting model be robust to the specific sample that was received. Viewing the partitioning of the data into training/validation sets as a sampling mechanism from this true data distribution (because for a given choice of a split, we get one training set), we desire to build models that are robust to every subset in the data.
A way to achieve this is to associate each observation (x n , y n ) to a binary variable z n , n ∈ [N ] that indicates whether or not (x n , y n ) participates in the training set. We can then train a given learning algorithm over all possible allocations of these z n 's, resulting in a model that is explicitly built to do well not just over one training set, as is typical, but over all possible training sets.
We begin by considering the model introduced above: where L corresponds to the loss function (in our case the cross entropy loss). Now, we aim to find a model that is robust to the particular data it is trained on. As explained above, one way to do this is to find a model that works well across a set of similar variations of the training data; more specifically, we ask that the model be robust to every subset of fixed size a. Taking a robust optimization approach to encoding this request, we arrive at the following formulation: At an optimal solution of (13), each z n will be equal to either 0 or 1, with the interpretation that if z n = 1, then point (x n , y n ) is assigned to the training set, otherwise, it is assigned to the validation set. The number a indicates the desired proportion between the size of the training and validation sets. For example, by setting a = 0.7n we recover the typical 70/30 training/validation split and by setting a = 0.5n we recover the 50/50 training/validation split.
The above formulation appears as a faithful translation of our earlier intuition: find a model that performs the best against all subsets of size a in the data and in particular the "hardest" subset of size a. In this way, we must now optimize a model that minimizes the worst-case training error across elements of Z.

Computing Stable Solutions
This section describes how to compute stable solutions by solving Problem (13). Our formulation belongs to the class of robust optimization (RO) problems. The two most frequently described methods in the literature for solving such problems are reformulation to a deterministic optimization problem (often called the robust counterpart) or an iterative cutting-plane method. Bertsimas et al. (2015) showed that both approaches are tractable.
In the case of Eq. (13) an exact reformulation is possible (see Bertsimas and Paskov (2020) and Bertsimas et al. (2021b) for more information) and thus we employ this methodology.
Consider again the stable formulation: (14) As the inner maximization problem is linear in z, the problem is equivalent to optimizing over the convex hull of Z: Thus, Problem (14) is equivalent to: where conv(Z) = z : Problem (15) belongs to the class of robust optimization problems (see Bertsimas et al. (2011) for a review) so we leverage techniques from RO to solve Problem (15) efficiently. Namely, to alleviate the multiplication of variables (i.e., the product of z i with f (m, x i , y i )) we take the dual of the inner maximization problem by introducing the dual variable θ for the first constraint and the dual variables u n , n ∈ [N ] for the second set of constraints to obtain: u n subject to θ + u n ≥ L(y n , z L (W, x n )), u n ≥ 0, n ∈ [N ].
Substituting this minimization problem back into the outer minimization we arrive at the following problem: u n subject to θ + u n ≥ L(y n , z L (W, x n )), u n ≥ 0, n ∈ [N ].
Note that variables u n can be solved for exactly, u n = [L(y n , z L (W, x n )) − θ] + . The minimization problem with stability then becomes: which is now an unconstrained problem that can be solved with standard gradient descent optimization algorithms.

Design of the Experiments
This section presents extensive results comparing the different methodologies. We conduct experiments on a variety of standard benchmark datasets, including UCI datasets and MNIST. Our objective is to highlight the merit of HDL and showcase the influence of each studied component -robustness, sparsity, and stability -on the overall performance in terms of natural accuracy, adversarial robustness, stability, and sparsity.

Data
We compute experiments on 44 classification UCI datasets, from the UCI Machine Learning Repository (Dua and Graff, 2017). The data sets give various problem sizes and difficulties to form a representative sample of real-world problems, with the largest data set having 28,056 observations and the highest number of features being 856. We also benchmark our methodologies on a computer vision dataset, MNIST, with 50,000 samples for the training and validation set, and 10,000 for the test set. We flatten each image into a one-dimensional vector to maintain the comparison with 3-layer perceptrons.

Implementation
All our code is written in Python 3.6 (Van Rossum and Drake, 2009). Neural networks are coded using Tensorflow 1.15 (Abadi et al., 2015). For the MNIST and UCI datasets experiments, we train all our models using one Intel(R) Xeon(R) Gold 5120 CPU. Typically, one experiment with 10 seeds on the UCI datasets takes under an hour, and on the MNIST dataset takes under 12 hours.

Training Methodology
On each dataset, we use an 60/20/20 train-validation-test split. Results are validated every 1000 iterations -where an iteration is a backpropagation performed on one data batch.
We select the best model with respect to natural accuracy obtained on the validation set.
We then test this model on the test set and measure its different characteristics (robustness, sparsity, stability). We define an experiment as running each hyperparameter configuration on 10 different seeds that are the same for each distinct methodology (natural, sparse, robust, stable-sparse, holistic) for a fair comparison between techniques. We report the average performance on these 10 seeds. We choose a limit of 50,000 iterations for each experiment, which always represents more than 100 epochs of training. The exact optimization problems solved for each model are presented in Table 1. Table 1: Specific loss functions used in each tested variation of the framework (weight decay not included). The 0 loss function was defined in Eq. 12

Hyperparameters
Neural network architecture For all our experiments, we use a 3-layer feedforward neural network with Glorot uniform initialization (Glorot and Bengio, 2010) and ReLU activations.

Results
This section presents the computational results from the experiments we designed in the previous section. The key question we address is whether HDL achieves a consistent gain compared to standard methods across a range of training conditions.

Comparison of HDL with Natural model
We report in Figures 1 and 2 the natural accuracy improvements over the nominal models that does not employ any robustness, stability or sparsity. We see an overall consistent improvement on the UCI datasets (see Figure 1). Since most performances are in very high-performance ballparks (90%+), we demonstrate how much additional improvement HDL can capture, as calculated by HDL accuracy − Nominal accuracy 100 − Nominal accuracy .
In 17 out of the 44 datasets, HDL captures at least 20% of the potential improvement (see Figure 2), while HDL only performs significantly worse than nominal in 1 out of the 44 datasets. In addition, it does so while also greatly improving adversarial accuracy ( Figure  3), inducing more sparsity (Figure 4), and increasing stability (see Figure 5) When selecting the best model with respect to adversarial accuracy, HDL significantly outperforms the nominal model for all datasets. In Figures 6, 7 and 8, we compare HDL with the nominal model on adversarial accuracy across a range of adversarial strength (10 −3 , 10 −2 , 10 −1 ) on all datasets. We observe that HDL never underperforms the nominal model across any scenario and any dataset. The out-performance is especially strong for the adversarial strength of 10 −2 (Figure 7), where HDL regularly improves over 100% compared to the nominal model. Figure 1: Counts of UCI datasets with specific adversarial accuracy improvement by HDL over the nominal model. Figure 2: Counts of UCI datasets with specific natural accuracy potential improvement captured by HDL from the nominal model. Table 2 references the number of times each methodology achieves the best accuracy on the UCI datasets, classified by their size. We defined a small, medium, or large dataset as one in which the product of the number of data points by the number of features is less than 10,000, between 10,000 and 100,000, or greater than 100,000, respectively. Doing such groupings allows for analysis of the effect of dataset size on a methodology's performance. Namely, on small datasets, the best combination is using robustness and sparsity or robustness and stability. It seems robustness brings the most value to these small datasets, as it improves generalization performance. The tendency changes with larger sizes, where HDL achieves the best performance. Table 3 further records the results of HDL against all methodologies in the MNIST dataset on the four metrics of interest. We observe that HDL concurrently achieves better natural and adversarial accuracy and sparsity than the nominal model, demonstrating that adding the different parts of the framework is valuable. Stability does not appear crucial in the MNIST dataset, likely because the dataset is stable by design and has many samples. Robustness appears valuable and significantly improves the adversarial accuracy and stability   while improving, in addition, the natural accuracy. The sparsity component also benefits from a combination with robustness.

Conclusions
This paper presents a unifying methodology to build holistic deep learning. We designed a successful combination of components to obtain robust, sparse, and stable multi-layer perceptrons that improve the performance of the natural model in all metrics, including Figure 6: Counts of UCI datasets with specific 1e −3 adversarial accuracy improvement by HDL over the nominal model, when the model is chosen with respect to best 1e −3 adversarial accuracy. Figure 7: Counts of UCI datasets with specific 1e −2 adversarial accuracy improvement by HDL over the nominal model, when the model is chosen with respect to best 1e −2 adversarial accuracy.   Table 3: Results on the MNIST dataset. For each method, the parameter combination that resulted in the highest out-of-sample accuracy was chosen as basis of comparison.
the natural accuracy. We presented experiments on various tabular datasets and MNIST to show that the unifying methodology on average is 40% more sparse and 100% more robust to adversarial attacks than the nominal model while simultaneously exceeding it in accuracy. Our framework is architecture-independent and applies to other standard neural networks such as convolutional neural networks.     Table 7: Adversarial accuracy results for all UCI datasets with ρ = 0.001, where n denotes the data size and p denotes the data dimension. Light blue, medium-dark blue and dark blue correspond to the three dataset size categories small, medium, and large, respectively.