{'fraction_non_alphanumeric': 0.02570424784413925, 'fraction_numerical': 0.007831363781539443, 'mean_word_length': 5.140257295262002, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Security and Privacy for Artificial Intelligence: Opportunities and Challenges

The increased adoption of Artificial Intelligence (AI) presents an opportunity to solve many socio-economic and environmental challenges; however, this cannot happen without securing AI-enabled technologies. In recent years, most AI models are vulnerable to advanced and sophisticated hacking techniques. This challenge has motivated concerted research efforts into adversarial AI, with the aim of developing robust machine and deep learning models that are resilient to different types of adversarial scenarios. In this paper, we present a holistic cyber security review that demonstrates adversarial attacks against AI applications, including aspects such as adversarial knowledge and capabilities, as well as existing methods for generating adversarial examples and existing cyber defence models. We explain mathematical AI models, especially new variants of reinforcement and federated learning, to demonstrate how attack vectors would exploit vulnerabilities of AI models. We also propose a systematic framework for demonstrating attack techniques against AI applications and reviewed several cyber defences that would protect AI applications against those attacks. We also highlight the importance of understanding the adversarial goals and their capabilities, especially the recent attacks against industry applications, to develop adaptive defences that assess to secure AI applications. Finally, we describe the main challenges and future research directions in the domain of security and privacy of AI technologies.

machine learning models are being used to drive innovation in the area of health care, gaming and finance, while autonomous car manufacturers rely on the deep learning models to create pipelines for self-driving cars. Machine learning (ML) models and, more recently, deep learning (DL) algorithms used in several AI systems today make it possible to automate tasks and processes, thereby introducing new capabilities and functionalities that were not previously possible. For instance, in 2019, DeepMind's AlphaStar, an AI system based on deep reinforcement learning, reached the Grandmaster level in the video game, StarCraft II, by beating several high-level human players [105].
Despite the noticeable success and benefits of using machine learning, many of the machine learning models in use today are vulnerable to several adversarial examples, where adversaries seek to violate the confidentiality, integrity or availability of machine learning models by using inputs that are specifically crafted to cause the models to make false predictions [10,88]. In many cases, AI systems are designed with no considerations for security, making them highly vulnerable to adversarial examples. Adversarial attacks on AI systems can occur either at the training or testing phase of machine learning [75]. At the training phase, an adversary can inject malicious data into a training dataset to manipulate input features or data labels. These attacks, referred to as poisoning attacks in literature, can easily be carried out in applications that use training data from untrusted sources. Barreno et al. [10] demonstrated how an adversary, with knowledge of a machine learning model, can change the original distribution of a training dataset by modifying the training data. Attacks at the testing phase, known as evasion attacks, are the most prevalent type of attacks on machine learning models as they exploit the vulnerabilities of a model to generate adversarial examples, which are then used to evade the model at test time [75].
The adoption of AI techniques in many applications presents a unique opportunity to solve many socio-economic and environmental challenges. However, this cannot happen without focused research on securing these technologies. The field of adversarial machine learning has recently been receiving significant research interest, and continuing focus in this area will undoubtedly ensure the ubiquitous spread of transformative AI technologies. As AI becomes increasingly integrated into different aspects of human activities and lifestyles, robust algorithms are essential to the progression of a protected and safe innovative future.
Motivation -Adversarial examples make the machine learning models used in many AI systems vulnerable to adversarial attacks. An adversarial attack on the confidentiality of a machine learning model would seek to expose the model structure, the training or test dataset [99], thereby impacting the privacy of data sources. Adversarial examples could also be used to attack the input integrity of machine learning models, thereby exploiting the imperfections made by the learning algorithm during the model's training phase [96]. Attacks on availability fall within the realms of adversarial behaviours which prevents legitimate users from being able to access a model's outputs or features. The security and privacy of AI systems against cyber adversaries have individually attracted attention in the research community over the last few years [30,99,102,104,138]. However, there has been minimal effort to review the literature and experimental results that illustrate a comparative analysis of AI security and privacy, as we present in this paper.
Our Contributions -We summarize the main contributions of our work in five aspects: • We present a detailed analysis of previous related surveys. This constitutes a contribution to literature through the identification of limitations in previous surveys related to the development of secure AI applications, as we addressed in this work. • We present a brief overview of machine learning task categories, including deep learning and federation learning • We present a new adversarial attack framework that illustrates advanced attacks that would exploit AI applications and measures their threats. • We present a new defence framework that demonstrates cyber defence methods for protecting AI systems against adversarial attacks • We discuss challenges in this domain and provide suggestions on future research directions.
The remainder of this paper is structured as follows: First, in Section 2, we explain the most recent surveys and reviews that attempt to explain security or privacy insights of AI applications. Second, we provide an overview of machine and deep learning task categories while also considering federated learning in Section 3. Third, we discuss the most complete set of attack types that would breach AI applications, including real-world attack scenarios and motives, in Section 4. Fourth, we explain in Section 5 defense methods and techniques that would protect AI models against cyber adversaries. Lastly, we explain lessons learned and future research directions based on our systematic review analysis, in section 6.

MOST RECENT REVIEWS
To develop an overview of academic activity relating to security and privacy of AI technologies, we identify and review the most recent surveys and reviews conducted between 2018 and 2020 as presented in Table 1. A total of 16 high-quality systematic reviews were identified; 11 of which were published in peer-reviewed journals such as IEEE Access, Elsevier, ACM, Applied Sciences and ScienceDirect. An analysis of these reviews is presented as follows.
From a data-driven view, Liu et al. [75] presented a systematic survey on the existing security threats and corresponding defensive techniques during the training and testing phases of machine learning. Given the lack of a comprehensive literature review covering the security threats and defensive techniques during the two phases of machine learning, their foundational work provided a detailed summary of existing adversarial attack techniques against machine learning and countermeasures. Particularly, they presented a detailed description of machine learning and introduced the concept of adversarial machine learning. While their survey has informed further research into adversarial machine learning, the authors did not review existing security threats on reinforcement learning.
The first comprehensive survey in the domain of adversarial attacks on deep learning in computer vision was undertaken by Akhtar and Mian [3]. Reviewing existing and proposed defenses for adversarial attack methods, this survey includes a discussion of adversarial attack methods that have been used successfully against deep neural networks in both 'laboratory settings' and real-world scenarios. This work is however limited as it presents the most 'influential' and 'interesting' attacks on deep learning in the restricted context of computer vision.
Based on a growing recognition that machine learning models are increasingly vulnerable to a wide range of adversarial capabilities, Papernot et al. [99] carried out a systematic review of ML security and privacy with specific focus on adversarial attacks on machine learning systems and their countermeasures. In this article, the authors considered ML threat model from the perspective of a data pipeline and reviewed recent attacks on ML and their defenses at the training and testing stages. Their review also covered existing works in the area of differential privacy. A review of the current landscape of research in the area of adversarial machine learning presented by Thomas and Tabrizi [122] analyses the results and trends in adversarial machine learning research. However, this work did not specifically focus on any of the attacks on ML systems and it lacks the depth of many review papers in this domain.
On the security and privacy of deep learning systems in several applications, Bae et al. [7] presents a systematic review of recent research. Covering the foundations of deep learning and An introduction into the foundations of adversarial machine learning presented by Li et al. [73] focuses on recent attack and defensive strategies in deep neural networks. This work describes some of the metrics used in literature for measuring perturbations of input samples. Whilst including adversarial attacks in reinforcement learning, the work only covered a small fraction of the attack and defensive strategies in deep neural networks that have been proposed in literature. A review of some of the well-known adversarial attacks and defenses against deep neural networks provided by Ozdag [95] reports some of the solution models and results presented in the NIPS 2017 Adversarial Learning competition organized by Google Brain. Though showcasing relevant information, the survey did not cover many of the adversarial attacks that have been demonstrated against deep neural networks.
Biggio and Roli [19] in their paper provided an overview of the state of research in the area of adversarial machine learning covering a period of about ten years mainly in the context of computer vision and cybersecurity tasks. The paper covers foundations of deep learning threat models, attack types and their timeline, as well as misconceptions relating to the security evaluation of machine learning algorithms. Furthermore, the paper suggests the need for research into the design of machine learning models that are against adversarial attacks. Similar to other review papers, Qiu et al. [103] presented a comprehensive review of recent research progress on adversarial attacks against deep learning and their defenses. The review focused on attack methods at both the training and testing stages of machine learning in the contexts of computer vision, image classification, natural language processing, cybersecurity and the cyber-physical world. The review also includes a number of proposed defensive strategies in literature. While this work comprehensively reviewed several demonstrated attacks against AI systems and their defenses, it did not cover some of the privacy issues associated with traditional machine learning algorithms.
A review of the security properties of machine learning algorithms in adversarial settings presented by Wang et al. [126] is also similar to other recent reviews in this area, covering foundations of adversarial machine learning. Some of the adversarial attacks that have been demonstrated in literature as well as their countermeasures are also included. A comprehensive review of adversarial attack methods against machine learning models in the visual domain is reported by Wiyatno et al. [128] with specific focus on the application of adversarial examples to machine learning classification tasks. Discussing the strengths and weaknesses of each of the reviewed adversarial attacks and defences, the review covered many of the attacks against machine learning systems in literature. Our work significantly expands their work with an analysis of attacks on unsupervised and reinforcement learning, which is not included in [128].
A taxonomy and review of attacks against machine learning systems undertaken by Pitropakis et al. [102] categorizes attacks based on their key characteristics in order to understand the existing attack landscape towards proposing appropriate defenses. This work adequately covers many of the adversarial attacks on machine learning in the context of intrusion detection, spam filtering, visual recognition and other applications. A limitation of this work however is that it did not include review of defenses against the adversarial attacks. A review of adversarial attacks and potential defenses in non-camera sensor-based cyber-physical systems by Li et al. [74] focuses on adversarial attacks on surveillance sensor data, audio data and textual input, and their defensive strategies. The work introduces a general workflow which could be used to describe adversarial attacks on cyber-physical systems, thus covering recent adversarial attacks against cyber-physical systems. It is however important to note that only classification and regression tasks are covered.
A paper on the review of adversarial attacks and defenses in deep learning by Ren et al. [104] introduces foundational concepts of adversarial attacks and then provides summarized review of adversarial attacks and defensive techniques in deep learning. While some of the adversarial attacks reviewed in this work are based on recent publications, the paper emphasises only attacks applicable to the context of computer vision. Recent research efforts focusing on adversarial attacks on deep learning models in the context of natural language processing (NLP) surveyed by Zhang et al. [138] covers the foundations of adversarial attacks and deep learning techniques in natural language processing. The survey paper includes summaries of black-box and white-box attacks methods that have been demonstrated on deep learning models in NLP. They also reviewed two common defensive techniques that have been proposed in literature for achieving robust textual deep neural networks, namely adversarial training and knowledge distillation. While this work covers most of the attacks that have been demonstrated against textual deep neural networks to date, it did not cover many of the defensive techniques that have been proposed in this area.

MACHINE LEARNING OVERVIEW
The idea of machine learning (ML) is not new; it has been since the 1970s when the first set of algorithms were introduced [77]. Machine learning deals with the problem of extracting features from data in order to solve many predictive tasks; examples of which are forecasting, anomaly detection, spam filtering and credit risk assessment. Its primary goal is to predict results based on some input data. Data is a fundamental component of every machine learning system. For instance, in order to predict if an email is a spam or not, a machine would need to have been trained with samples of spam messages -the more diverse the data used in training a machine, the better the result of the prediction. Input data in machine learning is typically divided into training and test data. Training data is used in developing a machine learning model; and once the accuracy of prediction of the model is satisfactory, test data is then fed into the model.
The main components of Machine Learning are: tasks, models and features [42]. Tasks refer to the problems that can be solved using machine learning. Many machine learning models are designed to solve only a small number of tasks. Models define the output of machine learning. They are simply trained by sample data in order to process additional data for making predictions. Features are an essential component of machine learning as they are characteristics of the input data which simplifies learning of patterns between the input data and output data. Algorithms are used to solve learning problems or tasks. Flach [42] described machine learning as the art of using right features to develop the right models used to solve a given problem. The tasks that are solved using machine learning are commonly divided into: Supervised Learning, Unsupervised Learning, Semi-supervised Learning and Reinforcement Learning [71], as detailed below. Fig. 1 shows the machine learning task categories.
Supervised Learning is a machine learning approach where an algorithm learns patterns from labelled training dataset and uses this for prediction or classification [121]. In supervised learning, the training dataset is either pre-categorised or numerical. Supervised learning tasks can be grouped into either Classification and Regression techniques. On the other hand, Unsupervised learning deals with unlabeled input data and the learning algorithms focuses on analyzing similarities among elements of the input data to infer meaningful features [71,121] which are then extracted to create possible output labels. As the name implies, Semi-supervised learning is an approach to machine learning that combines elements of supervised learning and unsupervised learning. It extend elements of supervised learning type to include additional information typical of the unsupervised learning and vice versa [139]. Reinforcement learning is a machine learning method that enables an agent to learn in an interactive environment through trial and error using feedback that it obtains from its own actions and experiences [117,121]. The reinforcement learning problem involves an agent which interacts with its environment by performing certain actions and then receiving rewards for its actions. The goal of this learning approach is therefore to learn how to take actions in order to maximize the reward.

Deep Learning Methods
Deep learning is a specialized field of machine learning based on deep artificial neural networks. The machine learning methods discussed so far in this section are today referred to as Shallow Learning due to the requirement of a feature engineer to identify relevant characteristics from the input data [4]. A common property of many of the shallow learning techniques is their relatively simple architecture often consisting of a single layer used to convert the input data into a problem-specific feature space [37]. In contrast, the more recent Deep Learning techniques rely on multi-layered representation and abstraction of the input data to perform complex learning tasks and feature selection [4,71]. While many shallow learning algorithms have been effective in solving many well-constrained problems, they have been found to perform poorly in problems that involve extracting well-represented features from data such as in areas of computer vision and natural language processing [36]. However, deep learning solves this problem by building multiple layers of simple features to represent a complex concept [136]. The vast increase in the amount of data available nowadays and the increased chip processing abilities has played an important role in the development of deep learning architectures [127].

Federated Learning
The concept of federated learning was originally introduced by Google in 2016 [81,134]. This learning approach enables the training of a centralized model on unevenly distributed data spread over federated network of nodes [20,65]. The primary motivation for federated learning arises from the need to train models based on data from users' mobile devices which cannot be stored centrally data centers due to privacy concerns [64]. Federated learning provides distinct privacy advantages compared to the other machine learning types due to the fact that only minimal updates necessary to improve a particular model is transmitted for federated learning, and this data is dependent on the training objective. The performance of federated learning improves with more data, that is, the more the nodes available to train the model, the better the model.

ATTACKS ON AI SYSTEMS: A HOLISTIC REVIEW
The widespread use of AI technologies and the fact that most of these technologies are vulnerable to adversarial attacks is concerning. Adversarial attacks are often in the form of manipulations of the input to an ML or DL model with the aim of causing a misclassification of the input data. Adversarial attacks on AI models differ and depends largely on the task category to which the algorithm belongs. In other words, threats to AI systems, in general, are mostly the same, but the approaches to exploitation differ depending on the algorithm in use. In this section we first present a result of literature search covering attacks on AI systems over the last 10 years and then present a detailed analysis of the identified literature. Literature covering attacks on AI systems were identified through Google Scholar search using key words such as Adversarial Attacks, Adversarial Examples, Adversarial Machine Learning, Attacks and Defenses on Deep Learning, Attacks and Defenses on Machine Learning, Membership Inference Attacks, Inversion Attacks, Extraction Attacks, Evasion Attacks and Poisoning Attacks. Many of the papers selected were based on works carried out between 2010 and 2020 and the results of the search are as presented in Table 2 and Table 3. To effectively assess the security and privacy of AI systems, it is important to understand the attack surface of the system, and the attack vectors that an attacker can potentially use to exploit the system. We therefore draw insights from existing frameworks [9,13,16,58,87,99] and present a new framework as a holistic approach to the quantitative analysis of adversarial attacks on AI models (Fig. 2).

AI Attack Surface
The attack surface of an AI system is the total sum of vulnerabilities the AI model is exposed to during both the training and testing phases. It can also be described as a list of inputs that an adversary can use to attempt an attack on the system. As shown in Fig.2, the attack surface of an AI system can be viewed in terms of a generalized data processing pipeline which comprises of the training/test input data or objects, the learning algorithm/model, and the output data. At the testing stage, the input features are processed by the machine learning model to produce the class probabilities, which are further communicated to an external system in the form of an action to be acted upon. An adversary can attempt to attack this system by poisoning the training data, corrupting the learning model or tampering with the class probabilities.

Attacker's Goals
Adversarial goals can be broadly described based on the security properties of an information system, namely: confidentiality, integrity, availability and privacy. An adversary's goal in attacking the confidentiality of an AI system is to gather insights about the internals of the learning model or dataset and to use this information to carry out more advanced attacks. In other words, an attack on confidentiality can be targeted towards a model and its parameters or the training data [99]. The goal of an attack on integrity is to modify the AI logic, through interaction with the system either at the learning or inference stage, while also controlling the model's outputs. In the example of a spam filter, an adversary can poison some data in the training dataset with the goal of changing the classification boundary such that a legitimate email is instead classified as spam and the adversary can evade detection without compromising normal system operations. According to Papernot et al. [98], four goals that impacts the integrity of a deep learning system are: (1) confidence reduction, (2) misclassification (3) targeted misclassification (4) source/target misclassification.
In adversarial settings, the availability of an AI solution can be attacked with the goal of disabling the system's functionality. For instance, an AI system can be flooded with incorrectly classified objects with the aim of causing the model to be unreliable or inconsistent in the target environment  [99]. Attacks on the availability of a machine learning system could lead to many classification errors, making the system effectively unusable. Attacks on confidentiality and privacy are related in goal and method. An adversary seeks to violate the privacy of a machine learning system with the goal of causing it to reveal information about the training data and model. Potential privacy risks may involve an adversary with partial information about a training sample, attempting to manipulate a model into revealing information about unknown portions of the sample [58] or the extraction of the training dataset using the model's predictions.

Attacker's Knowledge and Capabilities
An attacker's knowledge of a machine learning system can be defined based on the knowledge of single components involved in the design of the system [13,16]. An attacker can have different levels of knowledge of the system such as training data D, features X, learning algorithm , objective function L, and parameters w. The adversary's knowledge can therefore be characterised as a vector ∈ Θ where = (D, X, , w). On the other hand, adversarial capabilities refers to the level of information or knowledge about the machine learning system that is available to the adversary [27]. For instance, in the example of a spam detector pipeline described earlier in this section, an adversary with access to or knowledge of the spam detector model used for classification has 'better' adversarial capabilities on the system relative to an adversary with only access to the 'tokenized' text data or incoming email. According to Papernot et al. [99], the adversarial capabilities in machine learning systems can be classified based on how they relate to the training and testing phases, as explained below.  Misclassification Black-box 3 shows a relationship between an adversary's knowledge of a machine learning system and the adversary's capabilities.

Fig. 3. Attacker's Capabilities
Training Phase Capabilities. -Attacks during the training phase seek to learn, influence or alter the performance of the model. The most straightforward attack on the training phase is one in which the adversary attempts to read or access a portion or all of the training data. An adversary that does not have access to the training data nor the learning algorithm can carry out a data injection attack by adding adversarial data to the existing training dataset. An adversary with knowledge of the training dataset but not the learning algorithm can directly poison the data or label (in supervised learning datasets) before it is used to train the target model. This type of attack capability is referred to as data/label modification. Lastly, an adversary with knowledge about the internals of an algorithm can carry out a logic corruption attack by altering the learning logic. Logic corruption attacks are the most advanced attacks on the training phase of a machine learning system and are difficult to defend against.
Testing Phase Capabilities. -Testing phase attacks are referred to as exploratory attacks and according to Barreno et al. [10], these attacks do not alter the training process nor influence the learning, but rather attempt to discover information about the state of the learner. Inference attacks rely on information about the model and its use in the target environment. As illustrated in Fig. 3, adversarial capabilities at the testing phase can be broadly classified into either black-box or white-box attacks. In a White-box attack setting, the adversary has knowledge of everything about the model, including its training data, architecture, parameters, intermediate computations at hidden layers, as well as any hyper-parameters used for predictions [19], so we characterize white-box adversary's knowledge = (D, X, , w).Black-box attacks assumes that the adversary has no knowledge about the model but is able to use information about previous input/output pairs to infer vulnerabilities in the model [99]. The model parameters and architecture are not accessible to the adversary as in the case of Machine Learning as a Service (MLaaS) platforms [90], so = (D,X,ˆ,ŵ).

Attack Strategy
An attack strategy can be described as the method used by an adversary to modify the training and test dataset in other to optimize an attack [126]. Thus, given an adversary's knowledge ∈ Θ and a set of adversarial examples D ′ ∈ Φ(D ), an adversary's goal can be expressed in terms of an objective function A (D ′ , ) ∈ R which is a measure of the effectiveness of an attack with samples D ′ . An adversary's optimal attack strategy can therefore be expressed as:

Adversarial Attack Types
The most common attack types against AI systems are discussed in this section based on the techniques used by an attacker.

Poisoning Attacks.
Poisoning attacks, sometimes referred to as causative attacks, are staged at the training phase where the adversary alters the training dataset by editing, injecting or removing samples with the aim of changing the decision boundary of the target model [99]. They can target the integrity of adaptive or online classifiers, as well as classifiers being retrained on data collected at test time [13]. Some examples of poisoning attacks on AI systems are presented in Table 2.
According to Muñoz-González and Lupu [89], many of the data used in training machine learning systems come from untrusted sources, so the problem of poisoning attacks can be described as related to the reliability of large amount of data collected by these systems. While an adversary may be unable to directly access an existing training dataset, the ability to provide new training data via web-based repositories and honeypots provides an opportunity to poison training data [18]. Some literature [13,19,89] have described two main scenarios of poisoning attacks against multi-class classification systems namely: Error-generic poisoning attacks -These are the most common poisoning attacks in which the attackers aim to cause a denial of service attack in the system, by producing as many misclassifications on targeted data points as possible irrespective of the classes in which they occur. Using formulation of the attack strategy in Eq.1, the error-generic poisoning attack can be expressed as: where the objective function an adversary aims to maximize, A, is defined in terms of loss function, L, computed on set of points, D , targeted by the attacker. The loss function, L, is a function of parameters, w, which depends on injected poisoning points, D .
Error-specific poisoning attacks -The aim of an adversary in error-specific poisoning attacks is to cause specific misclassifications [89]. The attack strategy in this case can be expressed as: where D contains the same samples as D ′ but with labels chosen by the adversary based on desired misclassifications. The loss function L is expressed with negative sign because the adversary aims to minimize loss on the desired labels. Therefore, Eq.3 can equivalently be expressed as: 4.5.2 Oracle Attacks. These are exploratory attacks where an adversary uses samples to collect and infer information about a model or its training data [119]. Many classifiers are considered the intellectual property of the organizations they belong to; for instance in Machine Learning as a Service (MLaaS) platforms such as Microsoft Machine Learning, Amazon Machine Learning and Google Prediction API. In these environments, simple APIs are made available for the customer to upload data and for training and querying models [112]. An adversary can perform an oracle attack against such models by using an API to feed inputs into the learning model in order to observe the model's outputs. An adversary with no knowledge of a classifier can use input-output pairings obtained from oracle attacks to train a surrogate model that operates much like the target model [111]. The steps an adversary uses to carry out an oracle attack are described in Fig. 4. Without prior knowledge of the original classifier and class information of each sample, the adversary can only feed some test data into the classifier and collect the resulting labels which are then used to train a surrogate classifier, that is, a functionally equivalent classifier. The surrogate classifier can then be used to generate adversarial examples for use in evasion attacks against the target model. In oracle attacks, sometimes referred to as exploratory attacks, an adversary uses samples to collect and infer information about a model or its training data. Oracle attack types include membership inference attacks, inversion attacks and extraction attacks [119]. Many machine learning models are considered the intellectual property of the organizations that created them. For instance, Microsoft Machine Learning, Amazon Machine Learning and Google Prediction API are all Machine Learning as a Service (MLaaS) platforms and are owned and operated by Microsoft, Amazon and Google respectively. In these environments, simple APIs are made available for customers to upload data, and for training and querying models [112]. An adversary can perform an oracle attack against such models by using an API to feed inputs into the learning model in order to observe the model's outputs. An adversary with no knowledge of a target model can use input-output pairings obtained from oracle attacks to train a surrogate model which is functionally equivalent to the target model [111]. The surrogate model can then be used to generate adversarial examples for use in evasion attacks against the target model. Oracle attacks include membership inference attacks, inversion attacks and extraction attacks. Membership Inference Attacks -In membership inference attacks, the adversary seeks to determine if a given data point belongs to the training dataset analyzed to learn the model's parameters [99]. For instance, an adversary can violate the privacy of a clinical patient by using membership inference attack to know if the patient's health record was part of the dataset used in training a model that diagnoses a particular disease. Shokri et al. [112] demonstrated how the membership inference attacks can be carried out against black box models by exploiting the difference in the target model's behaviour on data 'seen' during training versus data seen for the first time.
Inversion Attacks -Model inversion attacks, referred to as training data extraction attacks in some literature, enables an adversary to reconstruct training inputs from a model's predictions [43]. The extracted inputs are not specific points but rather an average representation of data that belong to a class. Fredrikson et al. [44] introduced inversion attacks by considering a case where an adversary is able to infer a patient's genetic marker from the predictions of a linear model designed to predict a stable dosage of the drug Warfarin. This approach clearly illustrates privacy issues relating to providing API access to ML models trained on sensitive data [99].
Extraction Attacks -Model extraction attacks involves an adversary who uses observed inputoutput pairs, ( , ), from the predictions of some target model, , to extract a set of parameters and attempts to learn a model,ˆ, that closely approximates . Tramèr et al. [125] demonstrated extraction attacks against two ML-as-a-Service models via their exposed APIs. While extraction attacks could subvert model monetization, it also violates training data privacy and facilitates evasion attacks.

Evasion Attacks.
Evasion attacks are the most common type of attacks on AI systems [27]. An adversary directly manipulates the input samples at test time to avoid detection [16]. While the evasion attack scenarios explored in many literature have mainly focused on machine learning classification tasks, deep neural networks have been found to be highly vulnerable to this kind of threat [12,89,96,118]. However, deep networks also have the advantage of being able to represent functions that can resist adversarial perturbations unlike other shallow linear models [45]. Evasion attacks do not alter the behaviour of a system but instead exploit vulnerabilities in the system using adversarial examples to produce the desired errors. The approach used in evasion attacks is based on the rationale that without prior knowledge of a classifier's decision function, an adversary can learn a surrogate classifier using a surrogate data to reliably evade the targeted classifier. Adversarial samples are often perceptually indistinguishable from the 'clean' samples but yet pose serious security threats for machine learning applications [67]. Some examples of evasion attacks on AI system are presented in Table 3.
Melis et al. [82] described two possible evasion attack settings based on multi-class classifiers namely: Error-generic evasion attacks and Error-specific evasion attacks. Many evasion attack scenarios can be derived from these attack categories.
Error-generic evasion attacks -These refers to attacks where the adversary is simply interested in misleading classification at test time irrespective of the output class predicted by the classifier. In multi-class classifiers, the predicted class * is a class for which the discriminant function, ( ), for a given input sample, , is maximum: The error-generic evasion attack problem can be formulated mathematically as: * ∈ argmin where ( ) is the discriminant function associated with the true class of the source sample and max ≠ ( ) is the discriminant function of the closest competing incorrect class. The optimization problem constraint ( , ) ⩽ limits the perturbation, between the original sample, , and the adversarial sample, , point given in terms of distance in the input space [89].
Error-specific evasion attacks -In this evasion attack setting, the adversary is interested in misleading classification with the aim of producing a specific type of error. The problem of errorspecific evasion attacks is formulated similarly as the error-generic evasion attack with the exception that the objective function is maximized and denotes the discriminant function associated with the class which the adversarial example should be assigned [82]: * ∈ argmin

Adversarial Examples: Crafting Methods
This is an important research direction as it provides guidance on vulnerabilities in AI systems which could potentially be exploited by an adversary. In this section, we review some of the methods for generating adversarial examples in literature.

Box-Constrained L-BFGS.
Szegedy et al. [118] made a groundbreaking discovery on the stability of neural network networks to small imperceptible input perturbations and showed how these perturbations could arbitrarily change the network's prediction. They formalized the problem of finding the optimal perturbation in terms of the L2 norm. To find the optimal perturbation for a given clean input image X ∈ R and target label ∈ {1, ..., }, the aim is to solve a box-constrained optimization problem: An approximate solution to Eq. 9 is obtained by performing a line-search to find the minimum constant > 0 for which the minimizer satisfies (X + ) = . According to Moosavi-Dezfooli et al. [85], this method of generating adversarial perturbations is less efficient and does not scale to large datasets.
where is a hyper-parameter to be selected by the adversary, X represents the input to the model, represents the targets associated with X and ▽ X (X, ) is the gradient of the loss function w.r.t the input X, and can be computed via back-propagation. According to Kurakin et al. [67], this method is simple, fast and computationally efficient as it does not require iterative procedure to compute the adversarial examples. Nevertheless, the FGSM method only provides a coarse approximation of the optimal perturbation vectors [85].

Basic Iterative Method (BIM). The Basic Iterative Method (BIM) is simply an extension of the FGSM and was introduced by
Kurakin et al. [66]. It is an iterative method for crafting adversarial examples by applying the FGSM technique multiple times with small step size: 4.6.4 Jacobian-based Saliency Map Attack (JSMA). The JSMA was introduced by Papernot et al. [98] as a class of algorithms which an adversary can use to reliably generate adversarial examples based on a precise understanding of the input features of a deep neural network that most significantly impacts the output. This approach is distinct from previous approaches for generating adversarial examples because rather than using the gradient of the DNN's cost function to compute the perturbation vector, it instead computes the forward derivative of the neural network directly and then uses the concept of adversarial saliency maps to highlight regions of the input domain that leads to significant changes in the network outputs. The forward derivative F (X) for a given sample X, also referred to as the Jacobian of the multidimensional function F learned by the DNN during training is expressed as: Based on the experimental evaluations carried out in their work on LeNet, a well-studied deep neural network used for handwritten digit recognition, Papernot et al. [98] demonstrated that the JSMA algorithms can reliably produce adversarial examples which can cause misclassifications in specific target DNN models.

Iterative Least-Likely Class Method (ILLC).
The iterative least-likely class method (ILLC), introduced by Kurakin et al. [66], is another technique that has been demonstrated in literature for generating adversarial examples for targeted attacks against deep neural networks in the computer vision context. In contrast to FGSM and BIM, this method enables an adversary to create adversarial examples that will be classified by a DNN as a specific desired target class. The least-likely class according to the prediction of a trained DNN on image X is usually very different from the true class and can be expressed as shown in Eq. 14.
According to Kurakin et al. [67], the procedure for generating an adversarial image which is classified as is:X 4.6.6 Universal Perturbations for Steering to Exact Targets (UPSET). Another method for generating adversarial images that looks similar to the input but fools classifiers is the method known as universal perturbations for steering to exact targets (UPSET) proposed by Sarkar et al. [107]. Given an class setting, UPSET seeks to produce universal perturbations , ∈ {1, 2, ..., } such that when the perturbation is added to any image that does not belong to target class , the perturbed image is classified as being from target class . UPSET uses a residual generating network which uses target class and outputs a perturbation = ( ) which is of similar dimension to input image X. The adversarial imageX is expressed as: where is a scaling factor and the pixel values in X are normalized to be within [-1, 1].
4.6.7 Antagonistic Network for Generating Rogue Images (ANGRI). The Antagonistic Network for Generating Rogue Images (ANGRI) is another method for generating adversarial images proposed by Sarkar et al. [107]. As opposed to UPSET which produces image-agnostic perturbations, ANGRI instead produces image-specific perturbations as its output depends on the input image. Given an input image, X, belonging to class X and a target class, ≠ X , ANGRI generates an adversarial image,X, which the classifier classifies as an image from target class .
4.6.8 DeepFool. DeepFool, introduced by Moosavi-Dezfooli et al. [85] is an iterative method for computing adversarial examples that fools deep neural networks. In comparison to both the box-constrained L-BFGS and FGSM methods discussed earlier, DeepFool is more computationally efficient and generates adversarial perturbations that are more imperceptible. The DeepFool method is based on the assumption that neural networks are linear with each class separated by hyperplanes [25]. Therefore the minimum perturbation of a linearized binary classifier is computed as: where is the perturbation at iteration , is the input image and : R → R is an arbitrary image classification function. Moosavi-Dezfooli et al. also extended DeepFool multi-class classifiers. Experimental results presented in their work from testing the DeepFool algorithm on CNN architectures using the MNIST, CIFAR-10 and ImageNet classification datasets show that method accurately estimates the minimum perturbation required to change classification labels of deep neural networks.

Carlini & Wagner.
Carlini and Wagner [25] introduced one of the most common and effective algorithms for crafting adversarial examples to date. Their approach is also based on the formulation of adversarial examples introduced by Szegedy et al. [118]. Given the problem of finding an adversarial example for input image X: minimize D (X, X + ) such that C(X + ) = , The goal therefore is to find a small perturbation that minimizes some distance metric D (X, X+ ), usually specified in terms of norms, in order to cause a misclassification. Constraint C(X+ ) = ensures that the input image is mis-classified and is highly non-linear. This constraint makes the optimization problem of Eq.18 difficult to solve and while Szegedy et al. [118] solved this problem using the L-BFGS technique, Carlini and Wagner solved the problem by re-formulating it and expressing the constraint as an objective function such that when C(X + ) = is satisfied, (X + ) ≤ 0 is also satisfied. After instantiating distance metric D with an norm, the Carlini and Wagner formulated optimization problem becomes: Constraint X + ∈ [0, 1] , expressed as a box-constraint ensures that the adversarial example is a valid image. To find adversarial examples that will have a low 2 metric distortion, Carlini and Wagner used a change of variable method to optimize a new variable rather than variable , setting: 4.6.10 Houdini. Houdini, introduced by Cisse et al. [33], is an method for reliably generating adversarial examples which are directly tailored for a given combinatory or non-differentiable task loss of interest. Different from many existing methods which are mainly applicable to image classification tasks, Houdini is a more flexible approach which has been successfully applied to a range of applications such as voice recognition, human pose estimation [129] and semantic segmentation. According to Cisse et al., the problem of finding an adversarial example which can fool a neural network model with respect to task loss (.) for a chosen p-norm and noise parameter involves solving:˜= In most cases, the task loss (.) is a combinatory quantity that is difficult to optimize. As such, it is replaced with a differentiable surrogate loss¯( (˜), ). For a given example ( , ), Cisse et al. proposed a surrogate named Houdini and defined as: The first term of Eq. 22 is the stochastic margin and reflects the confidence of the neural network model in its prediction while the second term is the task loss.

Universal Adversarial Perturbations (UAP).
Universal adversarial perturbations, introduced by Moosavi-Dezfooli et al. [84], is a systematic algorithm for generating quasi-imperceptible universal perturbations for deep neural networks. In their work, they demonstrated how the UAP algorithm can be used to generate a single perturbation vector that can be used to fool neural networks on most natural images. Different from the notion of adversarial perturbations presented in [118], Moosavi-Dezfooli et al. examined universal perturbations that are common to most data points belonging to a data distribution. Given a distribution of images ∈ R and classification functionˆthat outputs estimated labelˆ( ) for each image input ∈ R , the focus is to obtain universal perturbation vectors ∈ R that can foolˆon most data points sampled from such that: The UAP algorithm seeks a universal perturbation that satisfies the constraints: where parameter controls the magnitude of and quantifies the target fooling rate for all samples images from distribution .

4.6.12
ATNs. An Adversarial Transformation Networks (ATN) is a feed-forward neural network introduced by Baluja and Fischer [8] for generating adversarial examples. The ATN is formally expressed as a neural network: where is the parameter vector of , is the target network or set of networks, and x ′ ∼ x but argmax (x) ≠ argmax (x ′ ). Obtaining , requires solving the optimization problem: argmin ∑︁ where X is the loss function in the input space, Y is the loss function on the output space of and is the weight that balances both loss functions. Baluja [45], the PGD is a more powerful multi-step attack as it doesn't limit the amount of time and effort an adversary can invest into finding the best attack. The PGD attack algorithm is based on the same intuition as the FGSM which involves solving the saddle-point problem: where D is the data distribution over pairs of examples ∈ R and corresponding labels ∈ [ ], and is the set of model parameters.
4.6.14 One Pixel. Su et al. [115] proposed a method for generating one-pixel adversarial perturbations, based on differential evolution, which can be used for low dimension black-box attacks against deep neural networks. In formalizing the one pixel method, Su et al. represented an input image by an -dimensional vector x = ( 1 , ..., ), with each scalar elements also corresponding to one pixel. Given a target image classifier with original input image x correctly classified as class , the probability of x being classified as belonging to is (x). The vector (x) = ( 1 , ..., ) is an additive adversarial perturbation with respect to x. The goal therefore is to find the optimized solution (x) * : where is a small number and in the case of one-pixel attack, = 1.
While one pixel has only so far been demonstrated in deep neural networks for image classification tasks, the authors argue that it can potentially be extended to other deep learning applications such as natural language processing and speech recognition in future works. , as a robust defense against low-distortion adversarial examples. The OPTMARGIN attack involves creating a surrogate model ( ), which is functionally equivalent to the region-based classifier [23], and uses the surrogate model to classify a small number of perturbed data points, such that ( ) = ( + ); where = 1, ..., , is the point classifier used in the region-based classifier and are perturbations applied to input . If ( ) is the | |-dimensional vector of class weights that internally uses to classify , then the loss function for the surrogate model with data point can be expressed as: where is the confidence margin. In the OPTMARGIN attack, is set to 0, which implies that the model simply misclassifies its input. As an extension to the Carlini & Wagner attacks [25], the OPTMARGIN's objective function uses the sum of the loss function for each perturbed data point, resulting in the minimization problem: where x is defined as the adversarial example of x 0 with target class ≠ 0 , , ≥ 0 are regularization parameters of loss function and and the 1 penalty respectively. The loss function (x, ) for targeted attacks is defined as: where ] ∈ R is the logit layer representation of adversarial example x in the DNN, is the number of classes and ≥ is a confidence parameter.
In evaluating the performance of EAD, Chen et al. performed extensive experimental analysis using MNIST, CIFAR-10 and ImageNet datasets and found that adversarial examples generated by EAD were able to successfully evade deep neural networks trained using defensive distillation [100] in a similar manner as the Carlini & Wagner attacks [25].
4.6.17 Robust Physical Perturbations (RP 2 ). Eykholt et al. [40] introduced the RP 2 method for generating adversarial examples under different physical-world conditions. This method generates adversarial perturbations which can cause targeted misclassification in deep neural networks where the adversary has full knowledge of the model. The RP 2 algorithm was derived in two stages: (1) using the optimization method to generate adversarial perturbation for single-image without considering any physical conditions, (2) update the algorithm by considering different physical conditions. The single-image optimization problem without consideration for environmental conditions is given as: is the target classifier, is a chosen distance function, and * is the target class. Solving Eq. 34 required Eykholt et al. to reformulate it in Lagrangian-relaxed form as: where (·, ·) is the loss function and is a hyper-parameter that controls regularization of the distortion.
After accounting for environmental conditions, the spatially-constrained perturbation optimization problem is defined as: where (·) is the alignment function that maps object transformation to the perturbation transformations.
where is the adversarial perturbation to x, ∥ ∥ 2 2 = ∥ (x + ) − x∥ 2 2 is an ℓ 2 distance metric between the benign image and the adversarial image, loss(·) is an attack loss function and term > 0 is a regularization constant. For a targeted caption denoted by = ( 1 , 2 , ..., , ..., ), where is the index of the -th word in the vocabulary V and is the length of caption , the loss function is given as: where > 0 is the confidence level which accounts for the difference between max ≠ { ( ) } and ( ) , and := [ (1) , (2) , ..., ( | V |) ] ∈ R | V | is a vector of for each possible word in the vocabulary. Based on applying the loss function of Eq. 38 to Eq. 37, the targeted caption method for a given targeted caption is formulated as: Similarly for targeted keyword denoted by K := { 1 , ..., } ⊂ V, the loss function is given as: In evaluating the effectiveness of the Show-and-Fool method, Chen et al. [29] carried out extensive experimental analysis using the Microsoft COCO dataset and the inception-v3 model. They showed that adversarial examples generated using the targeted caption and keyword methods are highly effective against neural image captioning systems and are highly transferable to other models, even those with different architectures.

DEFENSES AGAINST AI SYSTEM ATTACKS
In this section, we categorize defenses against AI system attacks based on whether they provide complete defense against adversarial examples or whether they simply detect and reject the adversarial examples. We then use this categorization, shown in Fig. 5, for our review of existing defense techniques. Complete defenses can be characterized by whether they apply to attacks launched against the training or testing phases of the system operation [119].

Defenses against Training Phase Attacks
Poisoning attacks, like many other attacks on AI systems make learning inherently more difficult. Many training attack defense mechanisms are based on the assumption that poisoned samples are usually out of the expected input distribution [99]. A number of techniques have been proposed as defenses against these training-time attacks, and some of these techniques are discussed as follows: 5.1.1 Data Sanitization. Data Sanitization is a poisoning attack defense strategy that involves filtering out contaminated samples from a training dataset before using it to train a model [28]. Cretu et al. [34] introduced a novel data sanitization technique for defending poisoning attack on out-of-the-box anomaly detection classifiers. Nelson et al. also presented a data sanitization technique [91,92] for defending against poisoning attacks known as Reject On Negative Impact (RONI). This technique was used to successfully filter out dictionary attack messages on the SpamBayes spam filter, and it accurately identified all of the attack emails without flagging any of the non-spam emails. While data sanitization techniques have been effective in defending against some data poisoning attacks developed without explicitly considering defenses, Koh et al. [63] presented three new data poisoning attacks that can simultaneously evade a broad range of data sanitization defenses. The success of their work against anomaly-based data sanitization defenses simply suggests that more research effort is required in finding effective defenses against poisoning attacks.
5.1.2 Robust Statistics. As opposed to data sanitization, this defense approach does not attempt to detect poisoned samples, but instead focuses on design of robust models against poisoning attacks. In their work, Biggio et al. [15] investigated the use of multiple classifier systems (MCSs) for improving the robustness of pattern recognition models in adversarial settings based on the principle that more than one classifier has to be evaded in order to make the system ineffective.
The results they obtained from the experimental investigations suggested that randomisationbased MCS construction techniques can be used to improve the robustness of linear classifiers in adversarial settings. Also, Biggio et al. [11] experimentally demonstrated that 'bagging', an acronym for bootstrap aggregating, is an effective defense technique against poisoning attacks irrespective of the base classification algorithm. Bagging was originally proposed by Breiman [21] as a technique for improving classifier accuracy by generating multiple versions of a classifier and using these to get an aggregate classifier, and it has proved to be effective especially when applied to classifiers whose predictions vary significantly with little variation in training data.

Defenses against Testing (Inference) Phase Attacks
Defenses against testing-phase attacks include various model robustness improvements, differential privacy and homomorphic encryption. Adversarial Training. Adversarial training seeks to improve the robustness of a machine learning model by proactively generating adversarial examples and augmenting them with training data as part of the training process. [123]. These adversarial examples are perturbed examples found by optimizing the input to maximize the model's prediction error [118]. While adversarial training may be time consuming due to the number of iterative computations required to form a robust model, it has proved to be resistant to white-box attacks [109] if the perturbations computed during training closely maximizes the model's loss [80]. On the other hand, deep neural network models that are adversarially trained using fast single-step methods are found to remain vulnerable to simple black-box attacks [123].
Defensive Distillation. Distillation is a technique designed for transferring knowledge from an ensemble of models or large highly regularized models to smaller distilled models while preserving the prediction accuracy [99]. The technique was formally introduced by Hinton et al. [54], after being initially suggested by Ba and Caruana [6] as a method for smoothing a model's prediction accuracy by training shallow neural networks to mimic deeper neural networks for the benefit of deploying deep learning in computationally constrained devices. Papernot et al. [100] also presented defensive distillation as a technique for training models that are more robust to input perturbations. Their experiments demonstrated that distilled DNN-based models are more resilient to adversarial examples. The applicability of this defense against other testing-phase attacks were also demonstrated in [45,98]. Despite the benefits of defensive distillation and it's strong guarantees against adversarial examples, Carlini and Wagner [25] experimentally demonstrated how their attacks using high-confidence adversarial examples can break defensive distillation Gradient Masking. Gradient masking is a technique that seeks to reduce the sensitivity of a DNN model to small input perturbations [119]. In this defense strategy, the first order derivatives of a model is computed with respect to its input and these derivatives are then minimized during the learning phase [99], resulting in a technique that conceals a model's gradient information to an adversary seeking to exploit the gradient-based information. While it is sensible to reduce a model's sensitivity to small changes in the input as a way of defending against adversarial examples, experiments performed by Papernot et al. [97] revealed the flaws in the gradient masking technique by demonstrating how it can be evaded by a black-box attack.
Feature Squeezing. This is a state-of-the-art technique for detecting adversarial examples by reducing the feature input spaces available to an adversary. It achieves this by combining many samples that correspond to different feature vectors in the original input space into a single sample. Xu et al. [131] demonstrated two types of feature squeezing methods in the image space: (1) image color depth reduction and (2) smoothening to reduce variation among pixels (median smoothening), as accurate and robust detection techniques against adversarial inputs. They extended this work in [132] where they showed that the median smoothening is the most effective squeezer in mitigating the Carlini & Wagner attack. While feature squeezing corrupts input features that adversaries rely on for their attacks, experimental evaluations [120] show that it could potentially cause degradation of accuracy on classifying benign inputs.
Ensemble Method. Ensemble methods are learning algorithms used to improve classification decisions of supervised learning models [38]. While many ensemble methods have been introduced in literature over the years, they have only recently been considered as a method for improving defenses of machine learning models against adversarial attacks. As a method of increasing the robustness of convolutional neural networks to adversarial attacks, Abbasi and Gagné [1] proposed the use of an ensemble of diverse specialist classifiers to detect and reject adversarial examples while accepting benign samples based on confidence. The authors showed that this method can be used to reduce the prediction confidence for adversarial examples while preserving the confidence of clean samples to a certain extent. However, He et al. [52] demonstrated that ensembles constructed using this method can be easily evaded by an adaptive adversary who can effectively generate adversarial examples with low distortions. Strauss et al. [114] considered ensemble methods as sole defense against adversarial attacks. Compared to existing ensemble methods, their method improves prediction confidence for clean samples while increasing robustness against adversarial examples at the cost of increased computational complexity. Liu et al. [76] proposed the Random Self-Ensemble (RSE), a defense algorithm that combines the concept of randomness and ensemble to improve the robustness of neural networks. Using experimental results, the authors were able to demonstrate that this method can generalize well and is an effective defense against attacks such as the Carlini & Wagner attack [25].
Reformers/Autoencoders. A reformer is a network that takes an input and reconstructs it to ′ before passing it to a classifier [135]. An ideal reformer is not expected to change the prediction accuracy of benign examples, however it is expected to change adversarial examples such that the reformed examples are close to the benign examples. An autoencoder is an artificial neural network that comprises of an encoder and a decoder, where the encoder learns a set of hidden representation for a given input data and the decoder reconstructs the output using the hidden representation Gu and Rigazio [49]. Meng and Chen [83] introduced MagNet, a robust defense against adversarial attacks on deep neural networks. MagNet, as a framework, incorporates a reformer and a detector. The detector, implemented using an autoencoder, calculates reconstruction error and rejects examples with high reconstruction errors. As an attack-independent defense method, experimental results show that MagNet achieved 99% classification accuracy on adversarial examples generated by the most of the attacks considered.

Differential
Privacy. Lecuyer et al. [69] proposed the use of differential privacy as a means of improving the robustness of deep neural networks against adversarial attacks. In the machine learning context, differential privacy involves the introduction of randomness to training data or model outputs in order to limit the disclosure of private information of individual records included in the training data. Lecuyer et al. [70] introduced PixelDP, a certified defense against adversarial examples based on differential privacy. When applied to deep learning models, differential privacy helps to protect training data from inversion attacks aimed at reconstructing the training data from model parameters [7]. Lecuyer et al. described PixelDP as a scalable, robust defense which applies broadly to different DNN model types. Experimental results show PixelDP to provide more accurate predictions under the euclidean norm attack compared to other certified defenses; however, it adds some computational overhead for training and testing.

Homomorphic Encryption.
Recent advances in homomorphic encryption schemes, specifically the fully homomorphic encryption, enable certain operations, such as addition and multiplication, to be carried out on encrypted data without the need to decrypt the data [5]. Homomorphic encryption has been proposed in recent literature as a means of achieving the privacy-preservation requirements for sensitive data when using predictive models managed by third-parties, such as in the case of machine learning as a service platforms. Xie et al. [130] introduced crypto-nets, a neural network which makes predictions over encrypted data and results in encrypted form. Through theoretical and practical analysis, the authors demonstrated that the computational complexity involved in applying neural networks to encrypted data makes crypto-nets infeasible in specific scenarios. Hesamifard et al. [53] introduced CryptoDL, a solution for implementing deep neural network algorithms over encrypted data. The primary components of CryptoDL are deep convolutional neural networks, trained using low degree polynomials, and homomorphic encryption. The authors designed CryptoDL to be trained using low degree polynomials as this is essential for efficient homomorphic encryption schemes and overcomes the practical limitations of other solutions that use homomorphic encryption, such as crypto-nets. Experimental results show that CryptoDL is scalable and provides efficient and accurate privacy-preserving predictions.

Detection-only defenses
Detection-only defenses have recently attracted research attention due to the limitations of existing complete defenses. Works by Carlini and Wagner [24] and He et al. [52] have shown that many of the complete defenses can easily be broken or evaded. The goal of detection-only defenses is to provide a distinction between clean examples and adversarial examples, and to filter out the adversarial examples for reliable classification. We now discuss some of the proposed detection-only defenses.
5.3.1 Kernel Density Estimation (KDE) based detector. The kernel density estimation method was proposed by Feinman et al. [41] for detecting adversarial examples in deep neural networks. It uses the kernel density estimates calculated in the feature space of the last hidden layer of a neural network to detect points that lie far from the data manifold. According to Feinman et al., the density estimate for a point with predicted class is given as: where ( ) is the last hidden layer activation vector for point , is the training set of class and is the tuned bandwidth. Feinman et al. showed that this defense is effective against the FGSM, BIM, JSMA and C&W attacks based on experiments using the MNIST, CIFAR-10 and SVHN datasets. However, Carlini and Wagner [24] later demonstrated how to break the KDE by using the C&W to generate adversarial examples for MNIST with increased distortion.

Bayesian Neural
Network Uncertainty based detector. The Bayesian Neural Network Uncertainty method for identifying adversarial examples was proposed by Feinman et al. [41] and is used to detect points that lie in the low confidence regions of the input space. According to Feinman et al., this method can be used to obtain additional information about model confidence which are not normally available using methods based on distance metrics such as the KDE. The proposed bayesian uncertainty method is based on adding randomness to the neural network by using dropout, a method introduced in [113] for reducing overfitting when training deep neural networks. The uncertainty estimate of the neural network on a given instance * and stochastic predictions {ˆ * 1 , ...,ˆ * } can be computed as:  [79] proposed the use of Local Intrinsic Dimensionality (LID) to identify adversarial examples from a given input. In contrast to other detection-only defenses which are based on density distribution in dataset, the LID-based detector characterizes the intrinsic dimensionality of adversarial regions of deep neural networks based on local distance distribution from a reference data point to its neighbors [55,56] Formally, given a reference data sample ∈ , where is the data distribution, the maximum likelihood estimator of the LID at is defined as: Ma et al. demonstrated how LID characteristics can facilitate the identification of adversarial examples crafted using a wide range of attack algorithms. They also showed how the characterization of adversarial regions could be used as features in an adversarial example detection process.

CHALLENGES AND FUTURE WORK
In previous sections, we presented a comprehensive review of security and privacy issues in AI systems covering many machine learning models. In this section, we discuss some challenges in the field and provide suggestions on future research directions.

Transferability of adversarial examples
Many machine learning models, including deep neural networks, are subject to the transferability of adversarial examples [128]. The transferability phenomenon means that adversarial examples crafted to fool a particular model can easily be used to fool other models. This property poses a security challenge for many deep neural networks because adversarial examples generated from one model can be used to attack another model without knowledge of the target model's parameters [136], thereby enabling black-box attacks [96]. While many of the works reviewed in this paper show real evidence of transferability of adversarial examples, the fundamental reasons why adversarial examples transfer are not well understood. Based on experimental analysis in [124], the hypothesis on the ubiquity of transferability does not always hold and there are suggestions that transferability property is not inherent in non-robust models despite existence of adversarial examples. Therefore, research efforts focused on understanding the transferability phenomenon is essential for creating robust machine learning models.

Evaluating robustness of defense methods
Many of the existing defenses against adversarial attacks have already been broken or bypassed, raising concerns about the lack of thorough evaluation of the robustness of existing defense methods. In their work, Carlini and Wagner [24] reviewed ten existing methods for detecting adversarial examples and demonstrated how easy it is for an adversary to bypass these methods by constructing new cost functions. Their work shows that existing defenses are not robust against adaptive adversaries while also lacking thorough security evaluations. In Section 5.2.2 we discussed Differential Privacy as a method for improving the robustness of deep learning models against inversion attacks aimed at reconstructing training data from model parameters. Differential Privacy adds randomness to the training data as a means of limiting the disclosure of private information included within the training dataset. While the efficacy of differential privacy has been demonstrated in a number of experiments, there are no practical evaluations or metrics that can be used to measure if the differential privacy bounds are strong enough [7]. Also in Section 5.2.3, we discussed homomorphic encryption as an encryption scheme that can be used to preserve privacy of sensitive data used in many predictive models. While recent implementations of homomorphic encryption such as CryptoDL [53] have demonstrated high prediction rates, the prediction accuracy is not at par with many state-of-the-art deep models and is found to be incompatible with deeper models. A future research direction should, therefore, seek to investigate the properties that defense methods need to have in order to guarantee robustness against adaptive adversaries.

Difficulty in controlling the magnitude of adversarial perturbations
The different methods for generating adversarial examples impose small imperceptible input perturbations in order to change a neural network's prediction. However, determining the exact magnitude of perturbations required to fool a neural network is difficult, because input perturbations that are too small cannot generate adversarial examples, and perturbations that are too large are not imperceptible [137]. Therefore, being able to control the magnitude of input perturbations poses an ongoing challenge.

Lack of research focus on attacks beyond classification tasks
Convolutional neural networks have been hugely successful in computer vision applications. As a result, a majority of the existing methods for generating adversarial examples apply to computer vision applications such as image recognition and object detection which are part of the machine learning classification tasks [45,67,84,85,98,107]. While some attention has been given to adversarial attacks on other machine learning tasks, these are only a handful in comparison to the attention given to classification tasks. It has been established in many literature that all machine learning models, including deep neural networks, are vulnerable to adversarial attacks. Therefore, more research effort on the adversarial threats facing other machine learning task categories, such as reinforcement learning, is required.

Evolving threat of unknown unknowns
Unknown unknowns pose a significant threat to machine learning systems deployed in adversarial environments, similar to how they are a real threat in many cybersecurity problems such as malware and intrusion detection Biggio and Roli [19]. Unlike, known unknowns which are used to model attacks in adversarial machine learning, unknown unknowns are often unpredictable and can cause machine learning models to misclassify with high-confidence due to inputs that are significantly different from the known training data. For machine learning systems to have the capability to detect unknown unknowns using robust techniques for anomaly detection, new research paths would need to be explored in this area.

Randomization of classifier's decision boundary
Introducing some randomization in the placement of the decision boundary for classifiers has been proposed as a method for improving the classifier's security against evasion attacks Barreno et al. [10]. Surprisingly, there has only been a few attempts in literature at investigating this kind of randomization as a viable defense method. While randomization does indeed increase the amount of work that an adversary would need to do in order to move the decision boundary past a targeted point, Barreno et al. acknowledged that a major challenge with using randomization is that it can also increase the classifiers initial error rate, thereby degrading its performance on clean data. The problem of finding an appropriate amount of randomization that can be introduced into a model to achieve robustness against adversarial attacks remains an open problem.

CONCLUSION
The field of adversarial machine learning has received significant research attention in recent years. In particular, many research works have explored adversarial attacks on machine learning models in the context of computer vision and image recognition, natural language processing and cybersecurity. Several defense methods have also been proposed with research efforts focused on evaluating the effectiveness of these defense methods against the continually evolving adversarial attacks.
In this work, we started with a review of recent surveys papers focusing on the issues of security and privacy in AI. We find that many of these existing surveys did not cover attacks and defenses across all machine learning task categories, with most only focusing on deep neural networks in the context of computer vision, natural language processing and cybersecurity. As a basis for describing adversarial attacks on machine learning models, we start by providing a theoretical background of machine learning task categories and make a clear distinction between shallow learning methods and the more recent deep learning methods. We then present a new framework for the holistic review of adversarial attacks on AI systems by first describing an adversary's goals, knowledge and capabilities, followed by a comprehensive analysis of adversarial attacks and defense methods covering many machine learning models.