{'fraction_non_alphanumeric': 0.025725957822678593, 'fraction_numerical': 0.009164182400353318, 'mean_word_length': 4.9640035118525025, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Assessing Threat of Adversarial Examples on Deep Neural Networks

Deep neural networks are facing a potential security threat from adversarial examples, inputs that look normal but cause an incorrect classification by the deep neural network. For example, the proposed threat could result in hand-written digits on a scanned check being incorrectly classified but looking normal when humans see them. This research assesses the extent to which adversarial examples pose a security threat, when one considers the normal image acquisition process. This process is mimicked by simulating the transformations that normally occur in acquiring the image in a real world application, such as using a scanner to acquire digits for a check amount or using a camera in an autonomous car. These small transformations negate the effect of the carefully crafted perturbations of adversarial examples, resulting in a correct classification by the deep neural network. Thus just acquiring the image decreases the potential impact of the proposed security threat. We also show that the already widely used process of averaging over multiple crops neutralizes most adversarial examples. Normal preprocessing, such as text binarization, almost completely neutralizes adversarial examples. This is the first paper to show that for text driven classification, adversarial examples are an academic curiosity, not a security threat.

Abstract-Deep neural networks are facing a potential security threat from adversarial examples, inputs that look normal but cause an incorrect classification by the deep neural network. For example, the proposed threat could result in hand-written digits on a scanned check being incorrectly classified but looking normal when humans see them. This research assesses the extent to which adversarial examples pose a security threat, when one considers the normal image acquisition process. This process is mimicked by simulating the transformations that normally occur in acquiring the image in a real world application, such as using a scanner to acquire digits for a check amount or using a camera in an autonomous car. These small transformations negate the effect of the carefully crafted perturbations of adversarial examples, resulting in a correct classification by the deep neural network. Thus just acquiring the image decreases the potential impact of the proposed security threat. We also show that the already widely used process of averaging over multiple crops neutralizes most adversarial examples. Normal preprocessing, such as text binarization, almost completely neutralizes adversarial examples. This is the first paper to show that for text driven classification, adversarial examples are an academic curiosity, not a security threat.

I. INTRODUCTION
The solving of classification problems in machine learning has recently made significant progress through the use of deep neural networks, or deep learning [1], [2], [3]. Deep convolutional neural networks (DCNNs) can be used for supervised learning, which is the use of a training set that contains known outputs for the inputs during the training of the DCNN. After training is completed, when presented with unknown inputs, the DCNN is able to classify the inputs with exceptional accuracy.
Although DCNNs have a high accuracy rate, images known as adversarial examples trick DCNNs into classifying an image incorrectly despite humans seeing almost no difference between the original and adversarial image. Recently, researchers have proposed that adversarial examples can "seriously undermine the security of the system supported by the DCNN" [4], because the incorrect classification could potentially lead to an incorrect action with consequences. For example, if a stop sign was crafted as an adversarial example, an autonomous vehicle could complete an incorrect classification of the sign and cause an accident to occur [5]. If the digits signifying an amount on a check were crafted to be adversarial, the amount of money transferred between accounts could be altered. Once the network is trained, the architecture together with its parameter values ✓ F can be considered as a classification function F and the test phase begins: the network is used on unseen inputs X to predict outputs F (X). Weights learned during training hold knowledge that the DNN applies to these new and unseen inputs. Depending on the type of output expected from the network, we either refer to supervised learning when the network must learn some association between inputs and outputs (e.g., classification [1], [4], [11], [26]) or unsupervised learning when the network is trained with unlabeled inputs (e.g., dimensionality reduction, feature engineering, or network pre-training [21], [27], [28]). In this paper, we only consider supervised learning, and more specifically the task of classification. The goal of the training phase is to enable the neural network to extrapolate from the training data it observed during training so as to correctly predict outputs on new and unseen samples at test time.
Adversarial Deep Learning -It has been shown in previous work that when DNNs are deployed in adversarial settings, one must take into account certain vulnerabilities [7], [8], [  In real world applications of deep learning which use inputs from a camera or scanned images, such as an autonomous car or processing of checks as mentioned above, the input images will never be perfectly captured. The input images will always contain slight transformations, such as shifting or blurring, and perturbations, such as noise, due to the imperfect capture of the input, which slightly perturbs the input to the neural network from the intended input. As we will see, these slight transformations render the majority of adversarial examples nonadversarial, thus, no additional defensive technique is necessary. In this paper, the acquisition process is mimicked by performing small transformations that could be expected in normal image acquisition.
The validity of a situation where an adversarial example could be crafted into a real world application input and then survives the image acquisition process needs to be assessed. This research assesses the extent to which adversarial examples are handled and classified correctly, simply through the natural process of acquiring the image, which renders the input nonadversarial.
We also note that most state-of-the-art deep convolutional neural networks (DCNNs) use multiple crops and often multiple networks in reaching their final decision. To date, no paper on adversarial examples has examined whether they could survive this widely used component of DCNNs. We show that even with only 5 crops (on non-transformed), compared to the 10s to hundreds used in state of the art networks, the majority of adversarial images will be correctly classified.
The problem addressed in this paper is whether adversarial  examples are a security threat to DCNNs and the extent to which this is or is not the case.

II. RELATED WORK
Deep neural networks are learning models that produce state-of-the-art results for several types of classification and recognition problems [1], [2]. Szegedy et al. [6] discovered that there exist perturbations which when included in an image cause an incorrect classification by the DCNN but they are "imperceptible" to humans. These examples were named "adversarial examples." Since this discovery, several advancements in the understanding and creation of adversarial examples have taken place. Sabour et al. [7] demonstrated that the existence of adversarial examples could be the result of the architecture of DCNNs themselves. Goodfellow et al. [8] presented the fast gradient sign (FGS) method for generating adversarial examples, which forms perturbations η f gs using the "sign of the elements of the gradient of the cost function with respect to the input," that is defined as where x is the input to the model, y is the target of x, and J(θ, x, y) is the cost used to train the network. Rozsa et al. [9] extended upon the FGS approach for generating adversarial examples and introduced two effective ways to produce more robust adversarial images using the fast gradient value (FGV) and the hot/cold (HC) approaches. The fast gradient value (FGV) approach uses "a scaled version of the raw gradient of loss" to create adversarial examples with distortions even less perceptible to humans. The perturbation formed by the FGV approach can be defined as The hot/cold approach defines a hot class as the target classification class and a cold class as the original classification class. This method then uses these defined classes to create features that cause classification to move towards the hot or target class from the original cold class. In addition to the different approaches to generating adversarial examples, Rozsa et al. also defined a metric for quantifying adversarial examples by measuring both the element-wise difference and probability that the image could be a different perspective of the original input called a Perceptual Adversarial Similarity Score (PASS). This score is a number between 0 and 1, where 1 denotes an adversarial example with no visible difference from the original image.
Rozsa et al. [9] also explored the effectiveness of fine-tuning a DCNN with adversarial examples and showed that such networks were able to correctly classify 86% of previously adversarial examples, demonstrating the ability to increase the robustness of a DCNN to adversarial examples with the right training. In the context of this paper, the results attained by fine-tuning with adversarial examples are complementary to the results shown below.
Rozsa et al. [10] also contributed to the known information about adversarial images by defining adversarial examples that exist in nature as "an image that is misclassified, but that will be correctly classified when an imperceptible modification is applied." Natural adversarial images demonstrate an additional aspect of the security threat of adversarial examples that needs to be considered.
In response to the growing interest and research in adversarial examples, Papernot et al. [4] asserted that by using deep learning algorithms, system designers made security assumptions about DCNNs, specifically in reference to adversarial samples. Papernot et al. [4] attempted to address the problem by demonstrating the use of distillation as an alternative form of training a DCNN to increase the percentage of correctly classified adversarial examples when presented to the DCNN as inputs using the CIFAR-10 [3] and MNIST [11] data sets. However, Carlini et al. [12] demonstrated that the assumptions made in this approach are wrong and the approach itself is not effective.
Papernot et al. [5] also presented a method for attacking a DCNN with adversarial examples without prior knowledge of the architecture of the network itself, and only having access to the targeted network's output and some knowledge of the type of input. To accomplish this, Papernot et al. trained a substitute DCNN on possible inputs for the targeted DCNN. After the network was trained, adversarial examples were crafted with the FGS method [8]. These examples were generated with different values of , as defined in Eq. 1. Examples of the FGS adversarial images generated with various values of can be seen in Fig. 2. The examples generated with higher values of do not fit the portion of the definition of adversarial examples that the perturbations need to be imperceptible to human observers.
A different technique for increasing a DCNN's ability to handle and correctly classify adversarial examples was put forward by Luo et al. [13]. This technique uses a "transformation of the image that selects a region in which the convolutional neural network (CNN) is applied, denoted a foveation, discarding the information from the other regions" as the input to the DCNN. This technique enabled the DCNN to correctly classify over 70% of adversarial examples.
The previous state of the art [14] takes crops of the original input and uses the average prediction over the classified crops which mimics natural perturbations. Crops like the ones used in GoogLeNet predate the discovery of adversarial examples and are used to improve accuracy of DCNNs. The improvement of accuracy in the DCNN also also helped to mitigate classification errors caused by adversarial examples. By taking crops of images, the accuracy of the DCNN when classifying adversarial examples is greatly increased. The networks tested by Papernot et al. [4] did not use this previous state-of-theart. This research aims to assess the level of threat imposed by adversarial examples to DCNNs as suggested by Papernot et al. [4].

III. METHOD
The proposed security threat provided that in a critical situation, the incorrect classification of an adversarial example made by the DCNN could cause actions with immense repercussions.
The image acquisition process as described above, which is necessary in physical all real world applications of classification by a DCNN, is always going to capture an imperfect input.
To assess the extent of the potential security threat that adversarial examples cause for DCNNs after acquisition, it is proposed that a slight transformation -such as a blur or a shift that would occur in normal image acquisition -to be applied to images before they are classified by the DCNN. To evaluate the effectiveness of this method, a trained deep convolutional neural network, a dataset including adversarial examples, and transformations are used.

A. The Deep Neural Network
A LeNet [15] deep convolutional neural network, trained by the authors of [9], is used for completing classification

B. The Dataset
The DCNN is trained on the MNIST dataset of handwritten digits [11], and is tested with the MNIST test set. This dataset provides a basis for a possible security weakness of the DCNN to adversarial examples. If adversarial examples cause an incorrect classification of a handwritten selection, such as an amount on a check, it could cause the amount to be misinterpreted and cause an incorrect amount to be withdrawn.
In addition to the MNIST test set [11], adversarial examples generated using the techniques in [9] and [8] are also tested to initially demonstrate the effectiveness of adversarial examples, and then to assess the effectiveness of the transformations, which mimic the acquisition process, at negating the effect of the perturbations that create an adversarial example.
In initial acquisition, MNIST images were subject to a pipeline of downsampling to 20 × 20 pixels, binarization, and subsequent upsampling to 28 × 28 pixels. The adversarial examples used in the following experiments were generated by the authors of [9] for the network described previously. When generating FGS adversarial examples, the authors of [9] increased , as defined in Eq. 1, until the image was made adversarial, so the entire data set is originally adversarial for the network described above.

C. Transformations
The aim of this research is to assess the extent to which adversarial examples can be handled and classified correctly, simply by the imperfection of the natural image acquisition process -i.e., slight transformations and perturbations added to the input images for classification -and understanding the impact each of these transformations has on classification. In order to mimic the image acquisition process as accurately as possible, an image was printed out, scanned back in, and analyzed for the types of transformation needed to closely replicate the effect of the acquisition. The types of transformations and perturbations that have been used to complete an experiment and their justifications are listed below.
1) Translation: The addition of a translation to the images processed by the DCNN replicates an alignment issue that could occur in the normal image acquisition process. This was implemented by shifting the image to the right by one pixel and filling the pixels in the empty column with values of 0, which in RGB values is black.
2) Noise: In the image acquisition process, it is normal to see additive noise on an image, such as black dots seen when an image is scanned in. To replicate the additive noise on an image, a small amount of generated noise is added to an input image before allowing the image to be processed and classified by the DCNN. The noise mask was generated with a standard deviation of 0.25, and a mean of 0. The noise mask was then added to a copy of the original image.
3) Blurring: When acquiring an image in a real world environment, such as with a camera or scanner, it is virtually impossible to capture an image without any blurring. To replicate this, the amount of blurring was estimated from the printed image. This led to the application of a blur kernel of (2,1) to the input images, as there was approximately one pixel of blur in the x direction and one-half pixel of blur in the y direction on the acquired image, with the asymmetry probably due to the scanner having a moving linear sensor.

4) Cropping & Resizing:
This transformation mimics the event when the acquired image is smaller than the original image. In order to mimic this, input images were cropped, the cropped image was saved, and then the cropped image was resized (using a cubic interpolation function) back to the image size expected by the DCNN of 28 × 28 pixels. 5) Combination: The above transformations each demonstrate pieces of the whole image acquisition process. In order to fully synthetically capture this process, the described transformations must all be applied to the input images. Transformations were applied in the following order: translation, noise, blur, crop and resize. This order was chosen to mimic the order in which the transformations occur in the natural image acquisition process. After transformations were applied, the transformed image was input to the DCNN for classification.

D. Fine-Tuning
The described experiments were run both on the raw LeNet [15] network, and on a fine-tuned network. The network was fine-tuned because when it was trained, the network learned from clean images without transformations. When inputting transformed images for classification, the network was not robust enough to correctly classify transformed inputs -even if the inputs were not adversarial examples -with the same level of accuracy as with clean images.
In order to fine-tune the network, a set of 100k images was taken for training and 20k images were used for validation. The dataset used for fine-tuning contained a total of 60k clean images and 60k transformed images, where the transformed images were the MNIST training set images with the combination of all transformations applied. The fine-tuned network has an accuracy of 99.35% on the chosen validation set, and 99.09% on the MNIST test set.

E. Fusion of Crops
In order to more accurately mimic the previous state-ofthe-art deep neural networks [14], a series of crops was implemented. As was previously mentioned, crops are used to increase the accuracy of DCNNs. With only a 28 × 28 image for MNIST, the number and size of crops is more limited, so we used only 5 crops: a center crop and 4 corner crops of size 27 × 27 pixels rescaled back to 28 × 28 pixels. Each image resizing used a cubic interpolation function. Each crop was classified by the DCNN separately, and their predictions were summed up to determine the predicted label.

F. Binarization
As is common in hand-written text recognition [16], before applying the recognition engine, the image is subject to preprocessing including binarization and noise removal. As mentioned above, the MNIST dataset [11] was subject to such preprocessing before being compiled into the dataset used in training and experimentation. The exact preprocessing of the MNIST images cannot be exactly replicated, due to an unclear description including lack of details on how down-sampling, binarization and subsequent up-sampling were performed. Without details of the rescaling steps, we approximate what we consider the most important step, binarization, using an OpenCV [17] version of OTSU thresholding [18]. Binarization is a critical step and takes into account the fact that machines are trained on basically binary data. When it is forced to deal with data which is not binary, the machine is more easily confused. As we shall see, this single assumption may account for almost all the effectiveness of adversarial examples, and proper preprocessing renders them neutralized. Pure binarization may not be effective on the type of noise in [4], but the despeckeling/noise removal that is commonly used for document processing [16], [19], would likely remove most of that noise as well.

A. Procedure
The experimentation procedure involved three datasets: the MNIST test set [11] of 10k images, a set of 10k randomly chosen FGS [8] adversarial examples, and a set of 10k randomly chosen FGV [9] adversarial examples. All experiments were run on all three datasets in order to generate a basis of comparison for results. After running a baseline with no transformations, experiments consisted of applying a transformation or combination of transformations to input images and then passing the transformed inputs to the DCNN for classification. The results of the classification were measured by the percentage of images correctly classified by the DCNN.    [8] adversarial examples. We show accuracies obtained by testing on a regularly trained "raw" model (in the middle) and a fine-tuned model that was trained on additional transformed images as well. Accuracy is the proportion of correctly classified FGS adversarial images that were transformed before testing as specified on the left.

B. Transformation Results
The specific results of the experiments completed are detailed in Tab. I, II, and III. These results demonstrate that the image acquisition process allows the DCNN to correctly classify a large portion of what used to be adversarial examples.
The most effective transformation to allow the DCNN to compute a correct classification is the cropping and subsequent resizing of input images, which demonstrated 78.28% and 76.16% accuracy on the FGS [8] and FGV [9] datasets respectively for the raw network. This transformation also only slightly altered the accuracy of the raw DCNN on the MNIST test set [11] (from 98.96% to 98.35%). Cropping and resizing also led to the highest accuracy on the fine-tuned network, demonstrating 80.01% and 78.70% accuracy on the FGS [8] and FGV [9] datasets respectively and 99.14% on the MNIST test set. After fine-tuning the network, the classification accuracy of adversarial examples increased, 56.93% and 62.09% on the FGS [8] and FGV [9] datasets respectively without applying transformations.
The results of this portion of the research demonstrates that in the majority of cases, the effect of perturbations added to make FGV adversarial examples are more easily negated than the perturbations formed by the FGS method. Intuitively,   This table reports the results of the conducted experiments on a set of 10k FGV [9] adversarial examples. We show accuracies obtained on a regularly trained "raw" model (in the middle) and a fine-tuned model that was trained on additional transformed images as well. Accuracy is the proportion of correctly classified FGV adversarial images that were transformed before testing as specified on the left.

C. Fusion of Crops Results
Doing experiments with the application and fusion of 5 crops was motivated by using the technique of the previous state of the art [14]

E. Adversarial Examples on ImageNet
After seeing the modest success of the synthetic image acquisition process on handling adversarial examples, an initial experiment was run on a GoogLeNet deep neural network [14] on a subset of the ImageNet dataset [20], with 15k FGS adversarial examples, all of which were provided by the authors of [9]. The experiment consisted of applying the combination of transformations, as is described above. The results of this experiments demonstrated that 63% of adversarial examples were classified correctly for top-1 accuracy and 89.95% of adversarial examples were classified correctly for top-5 accuracy. The fact that the application of transformations are producing similar results for the MNIST dataset and a portion of the ImageNet dataset demonstrates that foveation as described in [13] could be applied to any place in the image to negate the effect of adversarial examples.

V. CONCLUSION & DISCUSSION
This project assessed the extent to which adversarial examples are a security threat and demonstrated the effectiveness of simple solutions, such as slight transformations to the inputs, at mitigating that threat. This research demonstrated that slight transformations do render the majority of FGS and FGV adversarial examples as nonadversarial. The best results of this research, achieved through binarization of the inputs to the DCNN, demonstrated performance near to that of the deep neural network on clean images. For comparison, this method classified 20% more images correctly than the foveation method. These results presented that for the MNIST data set [11], the potential security problem is negligible, as the adversarial examples can be almost completely mitigated through binarization, which is part of the acquisition process of the original images. Therefore, when considering the application of digit classification on checks, adversarial examples are not a problem.
Outside of the classification of handwritten digits, when considering an autonomous car, the camera capturing input for the DCNN has the opportunity to capture a traffic sign hundreds of times, each at a slightly different angle, rotation, alignment and blur. This makes the chances of an adversary producing an adversarial example that would survive the image acquisition process significantly smaller than suggested by [12]. If, independently, each frame that consists of an adversarial example is correctly classified 90% of the time, then to get a majority wrong -say 15 in 30 frames (1 second) -would only have a 30 15 (0.1) 15 ≈ 1.55 × 10 −6 , i.e. about 1 in a million chance of causing an error.
However, further research should focus on the effect of the natural image acquisition process on adversarial examples on a dataset such as ImageNet [20] in order to formalize and assess the extent of the possible security threat to deep neural networks in real world applications.
It should also be noted, that this research does not take into account the case where an input from an adversary does not have to undergo physical image acquisition before classification is performed by a DCNN. For example, this natural defensive mechanism of DCNNs does not apply to an adversary who is attempting to bypass an image filter or change the amount deposited into their own bank account, because the input to the DCNN is coming directly from an adversary without physical acquisition of the image.