{'fraction_non_alphanumeric': 0.024573506604631264, 'fraction_numerical': 0.004625915824375635, 'mean_word_length': 4.869271814187068, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Tutorial on Adversarial Learning Attacks and Countermeasures

Machine learning algorithms are used to construct a mathematical model for a system based on training data. Such a model is capable of making highly accurate predictions without being explicitly programmed to do so. These techniques have a great many applications in all areas of the modern digital economy and artificial intelligence. More importantly, these methods are essential for a rapidly increasing number of safety-critical applications such as autonomous vehicles and intelligent defense systems. However, emerging adversarial learning attacks pose a serious security threat that greatly undermines further such systems. The latter are classified into four types, evasion (manipulating data to avoid detection), poisoning (injection malicious training samples to disrupt retraining), model stealing (extraction), and inference (leveraging over-generalization on training data). Understanding this type of attacks is a crucial first step for the development of effective countermeasures. The paper provides a detailed tutorial on the principles of adversarial machining learning, explains the different attack scenarios, and gives an in-depth insight into the state-of-art defense mechanisms against this rising threat .


Introduction
Our world is becoming increasingly dependent on machine learning for making automated decisions. With rapid rise in malicious cyberattacks, where attackers' incentives are to extract and manipulate the results and models generated by machine learning algorithms, the need for efficient and practical defences for these systems is crucial [1].
Intelligent Security Detection Systems (ISDS) are systems designed to identify and mitigate malicious activity. Anomaly detection is a common example that uses machine learning (ML) algorithms to derive a model of trustworthy behaviour. The current activity is then compared to the normal behaviour learnt by the model, which is then able to classify the activity as normal or anomalous, triggering the appropriate response. These security systems themselves may be vulnerable to adversarial learning attacks which target the ML algorithms they are based upon.
Adversarial learning attacks against machine learning systems exist in an extensive number of variations and categories; however, they can be broadly classified: attacks aiming to poison training data, evasion attacks to make the ML algorithm misclassify an input, and confidentiality violations via the analysis of trained ML models. Many attacks can have their application adapted to both deep learning systems and more traditional machine learning models. Poisoning attacks have a prerequisite of access to the training dataset of the model, before it is trained and have attack implementations for classical machine learning methods including linear regression [1] and Support Vector Machines [2].
The remainder of this paper is structured as follows. Section 2 discusses the threat modelling approaches for adversarial machine learning. Attack mechanisms are discussed in section 3, followed by a review of countermeasures in section 4. Conclusions are drawn in section 5.
2 Threat Modelling for Adversarial Leaning Attacks

Adversarial Threat Model
Papernot et al. [3] provided a summary of the type of adversarial approaches machine learning algorithms are vulnerable to and offered a comprehensive 'unifying threat model' for these algorithms and the encompassing systems that use them. The life cycle of a machine learning system is treated as a generalised data processing pipeline. The entire pipeline as a whole is considered, with a specific focus on the two distinct phases of a machine learning model: the training phase and the inference phase. They divide the threat model into three key components: the attack surface, adversarial capability and adversarial goals.
The attack surface of a ML system describes the point in which the attack takes place. This can be described with respect to the pipeline. For example, such a system may begin by first collecting the data it wishes to feed into the model, from sensors or data repositories. This data usually will need to be processed before being fed into the model, which produces an appropriate output. The model output is then communicated to an encompassing system in order to be acted upon. In such a system, the adversary may want to interfere with collection or processing of input data, corrupt the model itself or tamper with the model's output. Each of these are an appropriate attack surface. Most adversarial learning attacks will occur during the input data collection and processing in order to exploit the vulnerabilities in the model without corrupting it. This can be done by providing specific information for the sensors to pick up, or adding perturbations to images after being collected by the sensors.
The capability of the adversary also plays a large role in defining the threat model. The definition of a system's vulnerabilities is also made with regard to the strength of the attacker, who may have greater access to the target system and its data, and thus a greater ability to inflict damage to the system. The capabilities of an adversary outline the sort of attacks available to them. These relate to what part of the process they have access to, be it the training phase or the interference phase.
If the adversary has the ability to insert themselves into the training phase, they have the capability to learn or influence the training of, and therefore corrupt, the model itself. The simplest form of attacks in this phase involves having access to the training dataset, in full or partially. With this, provided the quality and quantity of data is sufficient, the adversary can train a substitute model on the same dataset in order to test adversarial inputs, and validate their success. Broadly however, there are two general strategies available in the training phase. The first of these is to meddle with the training set. An adversary can insert malicious samples into the training set to sabotage what the model learns and throws the model off. This form of altering the training set it referred to as injection. Similarly, modification, involves directly altering the training set. Alternatively, the adversary may be able to tamper with the actual learning logic of the model. These attacks known as logic corruption, are extremely powerful and would be very tough to defend against.
In the inference phase however, attacks do not attempt to corrupt the model itself, but rather to fool it into producing incorrect outputs, by exploiting inherent vulnerabilities. Alternatively, they may try to perform reconnaissance, merely gathering evidence of the characteristics. Inference time attacks, generally, can be split into white-box or black-box attacks depending on the adversary's access to information on the target model's architecture and therefore their ability to exploit it.
The final component which needs to be considered in Papernot's threat model [3] are the adversarial goals. They model the goals of potential adversaries using the classic cyber security mode, CIA, which defines the three characteristics a system aims to uphold, and an adversary may want to target: confidentiality, integrity and availability. Attacks on the confidentiality of the system correspond to attempting to extract information on the model itself or its training data. For example, in the context of a financial system, the model itself may be regarded as confidential intellectual property, or likewise in the context of a medical system, the training set may contain confidential data used to train it. Attacks on integrity aim to cause the model to behave in an unintelligent and incorrect manner thereby undermining the integrity of the model. For example, causing the misclassification of certain inputs to the model or reducing the confidence the model has in its output. Integrity is crucial to machine learning, and therefore integrity metrics, like model accuracy are often the primary focus of performance metrics. In the same way, attacks on availability look to thwart access to output of such models. One way this can be achieved is by targeting the model's quality and consistency as a model which is erratic and unreliable may consequently become useless.
Attacks in this domain will use similar methods as those used to damage the integrity of the model. Attacks on availability also include attempts to reduce the access to the model, via denial-of-service attacks, or model's performance metrics such as speed so it becomes inconvenient to use.

Intelligent Security Detection System Threat
Model Intelligent Security Detection Systems which employ similar machine learning models fit into this general threat model are therefore vulnerable to adversarial learning attacks. A common approach for such systems is to learn a model of trustworthy behaviour, from which current behaviours can be compared to. Therefore, these systems can generally be attacked in the training stage by poisoning the training set or by logic corruption. Similarly, attacks in the inference stage would aim to evade detection or potentially trigger false positives. These false positives may trigger unnecessary reactions by the system, potentially causing unwanted responses such as the wiping of important data. A stream of false positives may also strain the system, reducing its performance, and as such, its availability.

Attack Attributes
Following on from the general threat model described earlier, the adversarial learning threat model can be broken down into a number of more granular attributes. These attributes are adversarial falsification, adversarial knowledge, adversarial specificity, attack frequency and are selected appropriately depending on the scenario of a specific attack and its accompanying assumptions and quality requirements [4].

Adversarial falsification
Adversarial falsification distinguishes between whether the adversary aims to produce a false positive attack or false negative and what this means for the target system. A false positive attack results in negative samples being incorrectly classified as positive. In the case of an intrusion detection system this may result in the inappropriate triggering of a response to the perceived attack such as deletion of non-threatening or important information. Similarly, the existence of a stream of false positives which the system has to deal with may negatively impact the performance of the model. On the other hand, false negative attacks, also known as evasion attacks, work in the reverse, using the same example of an intrusion detection system, the true intruder or malware is classified as benign and left completely undetected.

Adversarial specificity
Adversarial specificity differentiates between targeted and non-targeted attacks and usually relates to the case of a multiclass classification. Targeted attacks look to guide the output of the model in a certain direction to a specific class. For example, duping the model into predicting all adversarial examples as one class. However, this isn't usually necessary and non-targeted attacks merely aim to classify an adversarial example as any class other than the original. These are therefore easier to implement and are either realised by reducing the probability the model classifies correctly or by selecting one from numerous targeted approaches, the one with the slightest perturbation.

Adversarial frequency
Attack frequency concerns whether the attack is a one-time approached or an iterative one, which updates a number of times to better optimise the attack.
The latter tend to perform better, however come with extra cost, requiring greater computational time. In some cases, the one-time attacks are sufficient or even the only feasible option.

Adversarial Knowledge: White-box vs Black-box
Adversarial knowledge relates to adversarial capabilities and more particularly refers to the case of inference time attacks. Attacks in this case can be broadly split two distinct categories based on the level of detail they require of the target model, to be performed. A white box attack is one which requires an in-depth understanding of the target model. In this way, the mechanics of the model are transparent. The attacker has full knowledge of the system and has access to vital information including details such as the network architecture, parameters, hyper parameters, training data as well as the ability to gather gradients, and prediction results [4]. On the other hand, a black box attack attacks the model with no knowledge of the inner workings of the model. A black box attacker simply requires access to the model in order to query it to produce the prediction result [5].
In the context of a black box attack, an attacker may be able to produce

Types of Attack
Adversarial learning attacks come in different forms. In this section, we review different types of adversarial learning attacks against deep learning ar-chitectures. We focus the attacks against deep learning architectures as these architectures are quickly gaining popularity in industrial application. This is because of the exponentially increasing amount of data that is becoming available which is used to train these models. The performance of deep learning models surpasses that of classical machine learning models when trained on large amounts of data [8].

Fast Gradient Sign Method
The Fast Gradient Sign Method (FGSM) was introduced by I. J. Goodfellow, Shlens, and Szegedy [9] as a part of an explanation of the nature of adversarial examples and what makes neural networks so vulnerable to them. This behaviour had previously been demonstrated by [10] whose L-BFGS method exhibited fundamental weaknesses in neural networks and their training algorithms. [9] argues that neural networks are vulnerable to linear adversarial perturbation attacks due to their linear nature and that the analytical exploiting of this behaviour allows for a cheap and effective method of perturbation generation. This can be demonstrated by taking the dot product of a weight vector w and an adversarial examplex = x + η η η: This perturbation causes the activation to grow by w η η η. By assigning η η η to sign(w) we can maximise the effect of the perturbation, subject to the max norm constraint. With this, we can now see that the maximum bound of the change in the activation is where w has n dimensions and the average magnitude of an element of the weight vector is m.
This shows that the activation given by the perturbation grows linearly with the number of dimensions, n. Therefore, in adequately high-dimensional space, many infinitesimal changes, each capped at , can sum up to produce large enough perturbation to the output, able to fool the model.
FGSM looks to work in the opposite manner to gradient descent in order to maximise the loss. Therefore, we use the cost function used to train the neural network, J, to calculate the gradient of the loss with respect to a particular input x. This allows us to push each pixel in the direction of the gradient, the direction which most increases the loss. This can be expressed as where y are the targets associated with x.
This is then added to the original input to create the adversarial example.
This one-step gradient method was then used a basis for a series of similar approaches. The basic iterative method (BIM) (also known as iterative-FGSM) is one such attack [11]. This extends the FGSM by iterating the process, using a reduced step size and clipping pixel values of the transitional results in order to keep them within a defined range of the original input, described as the ' -neighbourhood'. This can be expressed as: In the same work, [11] also explores the idea of pushing a particular misclassification rather than only pushing in the direction of the target class.
This can result in more extreme and interesting misclassifications. This is especially appropriate in the case that a model classifies a large number of classes and a misclassification may result in a similar class being predicted, for example the classification of one breed of dog over another in ImageNet [12] .
In the paper they try to push the classification of the least likely class (y LL ) according to the trained model. They name this approach 'iterative least-likely class method'. They show that this attempt can very effectively break the truthful classification even at a relatively small value. This is formalised in a similar way but instead takes steps in the direction of sign{∆ x logp(y LL |X}, in order to maximise, logp(y LL |X), the probability of the desired output class y. In a NN which uses a cross-entropy loss function, this looks like: The basic iterative method was further built on to incorporate the idea of momentum, a technique used to accelerate gradient descent-based algorithms.
This approach is known as Momentum-Iterative FGSM [13]. It works by holding the gradients of the previous t iterations which then contribute to the next step by an amount dictated by the decay factor µ. For example, when µ = 0, the algorithm reduces into iterative FGSM and when µ = 1, all previously saved gradients are added to the current update. This gradient update can be formalised as: This can then be used in a similar way as the previous methods to generate the next iterative step in the following way: The use of momentum allows the algorithm to stabilise the updates and navigate through poor local maxima to obtain a method which generally outperforms FGSM and BIM.

Jacobian-based Saliency Map Attack
The The saliency map is then used to traverse through the input space and craft adversarial samples with a target class in mind. Given an input sample X classified as F(X) = Y , the adversary attemps to modify the input X to create an adversarial sample X * such that F(X * ) = Y = Y * . The adversary looks for the smallest perturbation δ X to create an adversarial sample X * , thereby simultaneously avoiding human detection and satisfying the problem below: The first step required in conducting a successful JSMA is taking the forward derivative of the network with respect to the input sample. The Jacobian matrix for a given input sample X is computed and is given by: Computing the Jacobian allows us to see how small variations in the input space lead to large variations in the output of the network. A large forward derivative for the i th feature in a given sample X, indicates that changes to this feature will likely result in large changes to the output of the network.
Conversely, features with smaller forward derivative values are less susceptible to adversarial manipulations, because a small derivative value indicates that the feature has a small effect on the output of the network. Thus, computing forward derivatives provides an efficient method to search the input space and craft adversarial samples thenceforth.
The use of saliency maps was first introduced in [15] as a technique to identify the degree to which pixels in images influence the model to classify the image to a particular class. The aim was to visualize the manner in which classification models use features/pixels to classify a given image. The saliency map was adapted and used to make adversarial saliency maps which allow an attacker to modify the most dominant pixels in an image hence resulting in a misclassification.
The adversary's goal is to highlight features from an input image X that cause the network to classify it to a target class t =ŷ(x) = argmin j F j (X).
The output probabilities for the sample X of target class must be increased while simultaneously decreasing the probabilities of other classes. The adversarial saliency map introduced in [14] elegantly addresses this and is as follows: where Once the saliency map is computed, the input space must be traversed and perturbed to satisfy the equation in (9). Researchers from [14] approach this by introducing an algorithm which iteratively modifies a sample X by considering two pixels at a time. The algorithm conducts the search across the input domain Γ over all input indices [16]. These pixels are selected by the saliency map, they are then perturbed, and removed from the input search domain. This process repeats until the sample is classified by the neural network as the target class, or until a criterion Υ, which specifies the maximum number of iterations, is reached. The criterion Υ tries to put an upper limit on the number of perturbed features so that the adversarial sample remains undetectable by humans.
Results from [14] where researchers conducted the JSMA attack on the LeNet architecture trained on the MNIST dataset, show that any input digit from a source class can be perturbed to fit any other class specified by the adversary with a 97.1% success rate. They also maintain that the adversary only needs to change an average of 4.02% of input features per sample. They also conducted experiments to correlate the percentage of distorted pixels for a given digit, to the humans' ability to recognize the digit. As a result of the experiment, they identified the threshold of 14.29% wherein images which had a greater feature distortion than this, were unrecognizable by humans.

Generative Model Based Method
Developed by I. Goodfellow, Pouget-Abadie, Mirza, et al. [17] in 2014, a Generative Adversarial Network (GAN) is a generative model which solved the data generation problem without the use of Markov Chains. A GAN is comprised of two neural network architectures: a generator, and a discriminator which is sometimes called a critic. These two components act in opposition to one another and converge to a solution wherein the generator is able to produce data from the underlying distribution of input samples x.
Given input samples x, the generator maps noise variables z into data space to try to produce samples G(z) that originate from the input samples underlying distribution P(x). The discriminator on the other hand tries to discern whether the samples come from x or are from the generator. C(x) is the output of the discriminator and is the probability that x came from the input data and not from the generator. The training of the GAN therefore involves simultaneously maximising C(x) and minimising log(1 − C(G(z))).
The loss function can be described as follows: A clever implementation of a GAN to construct adversarial examples was demonstrated by researchers in [18]. introduced by [19]. The loss function is called the Wasserstein loss and it has been proven to help the stabilization of GANs. The loss function is as follows: They firstly train the WGAN until its generator is able to map random noise vectors to samples that come from domain X, then train the inverter I until it is sufficiently able to produce a low dimensional representation of samples from domain X. To train the inverter, they define two error terms: the reconstruction error, and the divergence error. In order to understand these definitions we need to understand the inputs and outputs of parts of the model. The output of the inverter I γ (x) is fed into the trained generator to get G θ (I γ (x)). The difference between this output and the sample x is called the reconstruction error. The generator on the other hand gives the output G θ (z), for a noise vector z. This is fed into the inverter to produce I γ (G θ (z)). The difference between this value and the sampled z is called the divergence error. A representation of their architecture is demonstrated in Figure 1. The loss function used to train the inverter aims to minimise the summation between these two values, shown in Equation 15: Source: Z. Zhao, Dua, and Singh [18]

Universal Attacks
Universal attacks are a special type of attack used to generate a single, fixed, universal perturbation image, which, when added to any natural image, causes it to be misclassified with a high probability. Simply put, a single perturbation vector is generated with the ability to fool the target model on most natural images [20]. These universal perturbations have also been shown to work very well, managing to fool multiple state of the art classification classifiers, with fooling rates of up to 93 %. It has also been shown that some universal perturbations computed for specific networks, generalise well and perform relatively well on completely different architectures, and so are described in [20] as 'to some extent, doubly universal'. For example, a universal perturbation   squeezed input [23]. Singular Value Decomposition can also be used similarly to denoise input data [24].
Reactive approaches include adversarial detection or input reconstruction.
Adversarial detection generally works by utilising a separate mechanism for determining the legitimacy of input data. There have been a number of detection methods proposed with some success. It should be noted that while some approaches naturally lead to a suitable fix [25], some do not and thus would probably deal with the attack in an appropriate manner such as to ignore the adversarial data completely [26].

Adversarial Training
Adversarial training is a method of defending against adversarial examples that is intuitive and commonly accepted as one of the most successful techniques for improving model robustness when put into practice. It reinforces the neural network through the proactive inclusion of adversarial samples in the training data and can be seen through the perspective of a robust optimisation for a min-max problem [27]. The concept behind adversarial training is most simply formulated by Ren, T. Zheng, Qin, et al. [28], representing the adversarial loss as J(θ, x , y), in which the network weights are θ, adversarial inputs are x and the ground truth label is y: The term on the left in the maximisation constraint D(x, x ) < η of Equation 16 denotes a metric for the distance between the original input x and adversarial example x . This maximisation is used to generate an adversarial example that is the most effective and hence has the largest value for the distance metric. The method for generating the example can vary, such as

Defensive Distillation
Defensive distillation is a process that uses knowledge distilled from a neural network architecture in order to increase its robustness to adversarial examples [22]. This idea was extended from work done in [29], where they reasoned that computational complexity of larger architectures could be bypassed by using smaller architectures that contain knowledge of larger, previously trained neural networks. The intuition behind this is that knowledge from complex architectures can be fitted onto smaller resource-constrained devices [22].
The distillation training procedure involves the utilisation of 'soft labels' output by the first neural network, to train a second neural network. The only difference is in the training procedure of soft labels. Papernot, McDaniel, X.
Wu, et al. [22] argue that the use of soft targets in training is justified because the probabilities encode extra knowledge about each class. The vector of soft labels that are output for each datapoint, represents the probability of membership to each of the classes, with the highest probability being the most likely class the datapoint belongs to. The soft labels for a datapoint are calculated in (17), where q i is the class probability for the i-th class, z i the input to the softmax layer and T the temperature, which controls how 'soft' the probability is distributed over the classes [29]. The higher the temperature value used, the more the output probability distribution resembles a uniform one. A breakdown of the distillation procedure is illustrated in Figure 3.
Source: Papernot, McDaniel, X. Wu, et al. [22] Figure 3: The training produce of a distilled network as proposed by [22]. The network on the left is initial network which outputs probability vector prediction. The figure on the left uses these probability targets as labels to train the distilled network.  [30] where a distilled network was used in defending text processing neural networks. Little to no change in the robustness of their network was obtained when a distilled network was used. They speculate that the use of distillation in text processing is misplaced, as gradients are not directly added to the input, but are rather just used to highlight the importance of a given word.
The distillation method seems to succumb to the black box attack proposed in [5]. The researchers recreated the models trained in [22] and successfully implemented an FGSM attack under black box conditions. They accredit the success of the attack to a phenomenon called gradient masking, the method in which distillation defence works. They postulate that the distillation method makes the gradient harder for the adversary to obtain, but does not nullify the attack vector itself.
An extension to the defensive distillation was made by co-authors of the original papers of [22]. In the revised paper [31], Neural networks require an assumption of a continuous input space due to their nature as differentiable models. This is contrary to a digital device's restriction to approximate natural continuous data to a discrete calculation. for each pixel. The human eye is only able to distinguish around ten million colours [32], hence why this representation is often referred to as 'True

Matrices
Colour'. Larger bit colour depths are often preferred by people as the image produced is a more natural representation and closer to the truth, however they are not necessary for understanding and recognising an image, hence why black-and-white images are interpretable.
The implementation of bit depth reduction in W. Xu, Evans, and Qi [23] takes an input and produces an output on an identical numerical scale [0, 1], therefore the NN model requires no alteration. To reduce an image to i-bit depth, such that 1 ≤ i ≤ 7, the product of the input value and 2i-1 is taken and subsequently rounded and re-scaled to [0, 1] before being divided by 2i-1.
The process of rounding to integers during this method causes the reduction in information capacity from an 8-bit to i-bit representation [23].
Spatial smoothing is a common image processing procedure for reducing the amount of noise within the image and is also known as blur. The smoothing applied in the feature squeezing proposed by W. Xu, Evans, and Qi [23] is categorised into to variants, local and non-local smoothing.
Local smoothing is a smoothing mechanism that makes use of the neighbouring pixels within a window (also known as a kernel) around a target pixel, assigning different weights to them depending on the exact smoothing method used, such as Gaussian, mean or median smoothing [33]. In W. Xu, Evans, and Qi [23], median smoothing is used, a window slides over the image in a Due to the random and sparse distribution of noise within an image, it is assumed that the mean of the noise will be zero and hence this method is able to eliminate noise from the image while leaving edges unaffected. The mechanism used to assign weights to similar patches during the averaging closely resembles local smoothing: Gaussian, mean and median operators are commonly used [34]. W. Xu, Evans, and Qi [23] use a popular adaptation of a Gaussian kernel. This enables additional control of the deviation from the mean, as well as the window size across which similar patches are searched for, the patch size and a variable for controlling the strength at which the filter is applied.

SVD
In adversarial attacks against pattern recognition systems, the adversary in- analysing multivariate data [35]. Research has concluded that the SVD of an image can also be used as an adversarial defence by computing an optimal approximation of the image matrix in terms of square loss, as a solution for eliminating the perturbations caused by the adversary [24].
In order to robustly defend against an adversarial attack, it is necessary for the humanly imperceptible perturbations introduced to the input before the input is passed onto the target model. The difference between the unaltered, original input, X, and the adversarial example, X ADV , is calculated as the minute value η: A matrix's distribution characteristics are described by the unique singular values (principal components) δ 1 , δ 2 , . . . , δ r , ∀A ∈ C m * n . A linear transformation of the matrix A maps its points in m-dimensional to n-dimensional space [24]. The SVD of a matrix decomposes the transformation into three separate matrix parts, U , s and V . U and V are complex unitary matrices whereas v is a diagonal matrix. Lossy compression of an image can be achieved through an orthogonal transformation, which retains defining features of the image but with reduced data. This is common for a variety of techniques and processes such as the storage, transmission and analysis of images. A digital image A can be interpreted as having a vertical and horizontal time-frequency in a two-dimensional representation, upon which the singular value decomposition can be computed [35] [24]: In Figure 6,

Adversarial Detection
Adversarial detectors can take many forms. One of the most basic adversarial detection approaches are DNN-based binary classifiers to estimate the probability an input is adversarial. [36]

Input Reconstruction
The key idea behind input reconstruction is to transform the adversarial ex- used as an approximation of the probability of x, p(X), they frame the problem as an optimisation one. The problem is to maximise p(X * ) subject to constraints on the size of the perturbation, def end . I.e max p(X * ) To achieve this, they take a greedy approach not too dissimilar to the generation of images from PixelCNN, with the added constraint that the image should be within the -neighbourhood. As PixelCNN is an autoregressive model, this process is typically a slow one, so they use a faster version proposed by Ramachandran [42].
A similar example is MagNet [43] which is comprised of two components: a detector, which rejects examples too far from the manifold boundary, and the reformer, which given an input x, endeavours to generate an x as a close approximation of x that falls on or near the manifold. It then gives x to the target classifier in the place of x, thereby reconstructing the adversarial example. This process is illustrated in 7 In [43], the detector is realised as a binary classifier, mapping inputs to either adversarial or clean, as discussed above. The implementation opted for is based on probability divergence and is very similar to the kernel den-Source: Meng and H. Chen [43]  sity estimation used in [37]. They make the use of an autoencoder trained to learn the features of the training data, as the detector. An autoencoder is used to encode its input and then decode it in an attempt to recreate the input. A reconstruction error is used to approximate the distance between an input and the learned manifold of clean examples. Now, if an input is drawn from the same data generation process as the training set, the reconstruction error should be low, and thus the reconstruction error can act as distance measure from the manifold of normal, clean data. However as discussed previously, this tends to be less effective when the reconstruction error becomes smaller as adversarial examples are made closer to the original. This can be dealt with by passing the both the inputs and the output of the autoencoder through the model and comparing. If for example, x is clean and drawn from the same process as the training data, the output of the autoencoder ae(x) should be very similar. Therefore, the output of the model should be similar also, i.e f (x) = f (ae(x)). However, if x is an adversarial example, even if the reconstruction error is low, f (x ) and f (ae(x )) can be significantly different.
This works by taking advantage of the target classifier, assuming the use of softmax at the last layer of their network, an assumption which is the case in most neural network classifiers.
The reformer tries to reconstruct adversarial inputs appropriately, so they are close to the original image, while leaving clean inputs unchanged. Similar to the detector, [43] propose the use of autoencoders in the implementation of the reformer, to take advantage of the distribution of normal data. They

Conclusions
This work has provided a comprehensive overview of the principles of adversarial machine learning attacks and existing countermeasures. Our analysis has shown that this threat poses a very real danger to machine learning algorithms, which play a massive and growing role in all aspects of modern society, including safety-critical applications and advance defence systems. There remain several outstanding challenges including efficient deployment mitigating techniques that balance the security risks with the expected overheads. This is specially true for resources-constrained systems. This requires the development of specific implementations of these countermeasures tailored to each application scenario.

Acknowledgments
This research was partly funded by the royal academy of engineering (grant No. IF202136).