{'fraction_non_alphanumeric': 0.040543795327427946, 'fraction_numerical': 0.008242566970856638, 'mean_word_length': 4.820560747663551, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
AdvMind: Inferring Adversary Intent of Black-Box Attacks

Deep neural networks (DNNs) are inherently susceptible to adversarial attacks even under black-box settings, in which the adversary only has query access to the target models. In practice, while it may be possible to effectively detect such attacks (e.g., observing massive similar but non-identical queries), it is often challenging to exactly infer the adversary intent (e.g., the target class of the adversarial example the adversary attempts to craft) especially during early stages of the attacks, which is crucial for performing effective deterrence and remediation of the threats in many scenarios. In this paper, we present AdvMind, a new class of estimation models that infer the adversary intent of black-box adversarial attacks in a robust and prompt manner. Specifically, to achieve robust detection, AdvMind accounts for the adversary adaptiveness such that her attempt to conceal the target will significantly increase the attack cost (e.g., in terms of the number of queries); to achieve prompt detection, AdvMind proactively synthesizes plausible query results to solicit subsequent queries from the adversary that maximally expose her intent. Through extensive empirical evaluation on benchmark datasets and state-of-the-art black-box attacks, we demonstrate that on average AdvMind detects the adversary intent with over 75% accuracy after observing less than 3 query batches and meanwhile increases the cost of adaptive attacks by over 60%. We further discuss the possible synergy between AdvMind and other defense methods against black-box adversarial attacks, pointing to several promising research directions.


INTRODUCTION
The recent advances in deep learning [16] have led to breakthroughs in a range of long-standing machine learning tasks (e.g., image classification [9], natural language processing [28], and even playing Go [32]), enabling many scenarios previously considered strictly experimental. However, it is well known that deep neural network (DNN) models are inherently vulnerable to adversarial inputsthose maliciously crafted samples to force the target DNNs to misbehave [34] -which significantly hinder their use in security-sensitive domains. Typically, adversarial inputs are crafted by carefully perturbing legitimate samples under the guidance of the gradient information of target DNNs (i.e., "white-box" attacks) [5,11].
Meanwhile, many cloud-based service providers, including Amazon, Google, Microsoft, BigML, and others all have arisen to provide Machine Learning-as-a-service (MLaaS) platforms. On such platforms, increasingly many commercial and proprietary DNN models are being deployed with publicly accessible interfaces ("predictive APIs"), which allow users to query the backend models with inputs of interests and charge users on a pay-per-query basis. For instance, the Clarifai NSFW 1 ("not safe for work") detection API returns probability scores that a given image contains nudity. The first 2,500 queries are free, it then charges $2.4 per 1,000 queries. Under such settings, the adversary must (i) construct adversarial inputs with only query access to the target DNNs (i.e., without gradient information) and (ii) also minimize the attack cost in terms of the number of queries, which spurs the research on "black-box" adversarial attacks. The existing black-box attacks in the literature can be categorized in two classes. The first one trains a surrogate DNN to emulate the target model and attacks the surrogate model using first-order white-box methods [27]. However, it often requires on the order of millions of queries to construct surrogate models for DNNs used in practice [14]; the second one constructs adversarial inputs by estimating the gradient of target DNNs via coordinatewise finite difference methods [6,14,26], as illustrated in Figure 1. Due to its practical feasibility, in the following, we focus on the second class of black-box adversarial attacks.
Given their reliance on zeroth order methods, such black-box attacks often require issuing a number of queries in the vicinity of a point of interest to accurately estimate its gradient information. Thus, it may be straightforward to detect the undergoing blackbox attacks (e.g., by observing massive similar but non-identical queries [7]). Yet, it is often challenging to accurately infer the adversary's intent -the target class of the adversarial input the adversary attempts to craft -especially during early stages of the attack. Identifying the adversary's target in many scenarios, such as the person the adversary impersonates in face recognition [3], the insurance category the adversary claims in healthcare fraud [29], and the legitimate source the adversary disguises as in fake news detection [17], is critical for deploying proper mitigation strategies or performing prompt remediation against such threats.
Yet, despite the plethora of prior work on black-box adversarial attacks, the research on understanding the adversary's intent is still fairly limited. To the best of our knowledge, this work represents a solid step towards bridging this gap. We present AdvMind, a novel framework for early-stage detection of the adversary's intent in black-box adversarial attacks. At a high level, AdvMind achieves robust and prompt inference of adversary intent with two key strategies: Robust intent estimation -Taking into account that the adversary may purposely issue fake queries to conceal her true intent, AdvMind employs an mean estimation model that is (i) reliable even in the presence of adversarial noise and (ii) agnostic to the underlying distribution from which the adversary samples the queries. Furthermore, by leveraging the fact that camouflage requires significantly increasing the attack cost (e.g., the number of queries issued), AdvMind creates for the adversary the dilemma between intent disclosure and attack cost.
Proactive intent solicitation -To achieve early-stage detection, AdvMind proactively synthesizes plausible query results to solicit subsequent queries from the adversary that maximally exposes her intent. Moreover, generated by slight perturbation on first-order information, the synthetic query results retain sufficient accuracy for legitimate use but can cause significant deviation if the results are used for gradient estimation as in black-box adversarial attacks, which further deters the adversary from successful attacks.
We extensively evaluate the efficacy of AdvMind with respect to benchmark datasets, popular DNNs, and state-of-the-art black-box attacks. We show that across all cases, AdvMind manages to detect the adversary intent with over 75% accuracy after observing less than 3 query batches and meanwhile increases the query cost of adaptive attacks by over 60%. We further discuss the synergy between AdvMind and existing defenses against black-box adversarial attacks, pointing to several promising research directions. 2

FUNDAMENTALS
We begin with introducing a set of concepts and assumptions used throughout the paper. The important symbols and notations are summarized in Table 2 in Appendix.

Threat Models
In the following, we primarily focus on predictive tasks (e.g., image classification [9]). Under this setting, a DNN f represents a function f : X → C, which assigns a given input x ∈ X to one of a set of predefined classes C, f (x) = c ∈ C. DNNs are inherently vulnerable to adversarial inputs, which are maliciously crafted to force target DNNs to misbehave [8,23,25,34]. Here we focus on the setting of targeted attacks. Specifically, an adversarial input x * is often generated by slightly modifying a benign input x • , with the objective of forcing f to misclassify x * into a target class c * , f (x * ) = c * f (x • ). To ensure that x * is perceptually similar to x • , the perturbation is constrained to a set of allowed perturbations (e.g., a norm ball Consider project gradient descent (PGD) [23], a universal firstorder adversarial attack, as an example. At a high level, it is implemented as a sequence of project gradient descent steps on the negative loss function: where Π denotes the projection operator, α (α ≥ 0) represents the learning rate, the loss function ℓ measures the difference of the model prediction f (x) and the class c * desired by the adversary (e.g., cross entropy), and x (0) is initialized as x • .

Algorithm 1: Black-Box Attack
Input: original input x (0) = x • , target class c * , target DNN f , learning rate α , norm constraint ε , and number of iterations n iter Output: adversarial input x * 1 for i = 1, . . . , n iter do Under the black-box setting, the adversary has only query access to the target DNN f (i.e., without the gradient information), as sketched in Algorithm 1. Note that here we focus on an accountoriented setting, where users must create an account before they can query the model [7]. The adversary typically uses coordinatewise finite difference methods to estimate the gradient and then applies gradient descent as in Eqn (1) to generate adversarial inputs x * . Different black-box adversarial attacks mainly differ in their gradient estimation method GradEstimate(x, c * ) in Algorithm 1. In this work, we consider three representative black-box attacks.
To clarify, f (x) returns the classification label c of image x in the above content. However, we'll introduce different attack methods in the following section, within which f (x) is the probability confidence vector after softmax layer. For simplicity, in the remainder of the paper, f (x) stands for the classification label in ℓ(f (x), c * ), and stands for the probability vector when it's alone.
Zeroth Order Natural Evolutionary Strategy. In [14], Ilyas et al. apply natural evolutionary strategies (NES), a derivative-free optimization method [38] to estimate the gradient ofд(x (i) ). Specifically, NES generates n query data points in the neighborhood of x (i) by sampling from a normal distribution, retrieves their predictions from the target DNN f , and estimates the gradientд(x (i) ) as follow: where each sample u (j) is sampled from the standard normal distribution N (0, I ), and σ is the sampling variance.
To accelerate the sampling, it is possible to generate only half of the samples from the distribution and set the other half symmetrically. The gradient estimate is then formulated as follows: Zeroth Order signSGD. In [20], Liu et al. integrate signSGD [2], a compressed optimization method that only uses the sign of gradient, with zeroth order optimization [6], and propose one-sided and twosided estimators ofд(x (i) ). Specifically, the one-sided estimator is given by: where the NES estimator in Eqn (2) can be used to compute the first term above. Also note that since the expectation of each sample u (j) is 0, Eqn (4) and Eqn (2) share the same expectation. Yet, compared with the NES estimator, the second term in Eqn (4) reduces the error caused by the variance of u (j) , leading to faster convergence with the number of samples.
The two-sided estimator in [20] is essentially identical to the improved NES estimator in Eqn (3).
Zeroth Order Hessian-Aware. Note that the NES and signSGD estimators only use first-order information in estimatingд(x (i) ).
In [39], Haishan et al. argue that it is more efficient to estimatê д(x (i) ) if the second-order information (Hessian) is considered. They propose the following gradient estimator: whereH − 1 2 is the Cholesky inverse of the Hessian estimateH . Further,H is estimated as follows: where ⊗ represents the outer product of two vectors. Note that to ensure the feasibility of Cholesky decomposition and invertibility, a small identity matrix (e.g., τ I ) is often added toH .

Overview of AdvMind
Despite their apparent variations, the query-based black-box adversarial attacks share common patterns. Each attack consists of a sequence of iterations. At the i-th iteration, the adversary issues queries regarding the current input x (i) , which we refer to as the query of interest (QOI), and the auxiliary data points in the vicinity of x (i) , which are used to estimateд(x (i) ). Therefore, to infer the adversary's intent (i.e., the target class c * ), AdvMind adopts the following strategies: (i) grouping the incoming queries into a sequence of query batches, each corresponding to an iteration (the feasibility of grouping queries into different iterations is detailed in Appendix B); (ii) estimating the QOI at each iteration -As the auxiliary queries appear in the close vicinity of x (i) , it is feasible to accurately estimatex (i) by clustering the queries; and (iii) estimating the gradient direction followed by the adversary -By linking consecutive QOIs (e.g.,x (i) andx (i+1) ), it is feasible to infer the gradient direction the adversary is following to craft the adversarial input. Intuitively, as the adversary attempts to minimize the loss function with respect to the target class c * , the gradient tends to point to that direction. Further, by combining the inference at multiple iterations, it is possible to reliably infer the adversary's intent c * . This scheme is illustrated in Figure 2. Yet, this naïve inference model suffers the following drawbacks. First, at each iteration, the adversary may attempt to inject fake queries (i.e., irrelevant to the QOI) to conceal her intent. As outliers, the fake queries may significantly deviate AdvMind's estimatê x (i) from the ground-truth QOI x (i) . To account for the possible adversarial noise, AdvMind adopts a robust QOI estimator. With reasonable fake query proportion p fake (e.g., p fake ≤ 40%), the robust estimator is able to estimate the QOI with bounded bias. Further, it creates for the adversary the dilemma between intent disclosure and attack cost (e.g., in terms of the number of queries).
Second, as the adversary performs project descent based on the estimated gradient (Algorithm 1), the descent direction she follows may deviate from the true gradient direction, which in turn causes errors in AdvMind's inference of the adversary's target class c * . To enable the early-stage inference and minimize the inference uncertainty, AdvMind adopts proactive intent solicitation. Specifically, by slightly perturbing the query answers, AdvMind solicits subsequent queries from the adversary that maximally expose the target class. Note that such perturbation retains sufficient accuracy in the query answers for legitimate use but causes significant deviation for gradient estimation.
We elaborate on the strategies of robust intent estimation and proactive intent solicitation in § 3 and § 4 respectively.

QOI ESTIMATION
In this section, we focus on the estimation of query of interest (QOI) at each iteration. We first introduce the naïve QOI estimator, which is however vulnerable to adversarial noise (i.e., fake queries), and then present a robust QOI estimator, which bounds the estimation bias under reasonable noise ratio.

Naïve Estimator
Recall that at i-th iteration, the adversary samples the set of auxiliary queries X (i) in the vicinity of the QOI x (i) following a certain distribution (e.g., Gaussian or uniform). We may therefore estimatê x (i) using the mean or median of the received queries X (i) . 3 Note that compared with the mean estimate, the median estimate is more robust to outliers but has a larger bias as it does not utilize the data points on edges. However, to conceal her intent, the adversary may adaptively inject fake queries into X to deviatex from the true QOI x. As illustrated in Figure 3, the presence of such adversarial noise significantly affects the naïve estimator, even for the median estimate. The adversary may select from a variety of models to generate fake queries, including random sampling, duplicating existing queries, and query blinding [7], which lead to AdvMind's varying estimation error (e.g., measured by ∥x − x ∥).
To be concise, in the following we consider Huber's contamination model [13], which subsumes a number of models above. Specifically, it assumes that the set of queries x 1 , . . . , x n query are randomly sampled from a mixture distribution, with the cumulative distribution defined by: where p fake is the fraction of fake queries (0 ≤ p fake < 0.5) among all the queries, H is the (unknown) cumulative distribution of fake queries, and Φ is the cumulative distribution of true auxiliary queries (e.g., Gaussian or uniform). Next we develop a robust QOI estimator against the influence of fake queries and bounds the estimation bias even under the worst-case distribution H .

Robust Estimator
We extend the robust mean estimator [13,30] to the context of QOI estimation. Specifically, given the set of queries X (|X| = n query ), we estimate the QOI in an iterative manner, in which the k-th round estimatorx (k) is updated as follows: n query (8) where the function ψ is defined as is the median average deviation of the query set X. Concretely, with Φ instantiated as the standard Gaussian distribution, we may estimate the expectation E Φ [ψ ′ ] as follows: The value ofx (0) is initialized as the median of the query set X. As observed in our empirical evaluation (which is also consistent with the results of [30]), one-iteration estimationx (1) is typically accurate enough to approximate the QOI x. Thus we set the number of iterations k = 1 by default.
Further, we may also estimate the bias ofx under the worst-case distribution of adversarial noise, which is needed for inferring the adversary's target in § 4. Specifically, the bias of the k-th iteration estimator, B(x (k ) ), is updated as: with the initial value given by B(x (0) ) = Φ −1 1 2(1−p fake ) , where Φ −1 is the inverse function of the Gaussian distribution.

INTENT INFERENCE
In this section, we introduce AdvMind's intent inference model, which based on the QOIs estimated at consecutive iterations, identifies the adversary's target class. Moreover, to infer the adversary's intent at an early attack stage and to minimize the inference uncertainty, AdvMind employs a proactive solicitation strategy, which solicits subsequent queries from the adversary that maximally expose her target class.

Passive Inference
We begin with the passive inference model, which identifies the target class c * directly from the QOI estimation.
Specifically, assumingx (i) andx (i+1) as the estimated QOIs at two consecutive iterations, we computes the descent direction vector d (i) =x (i+1) −x (i) and the gradient vector д c (x (i) ) atx (i) with respect to each class c ∈ C, and regards the cosine similarity of d (i) and д c (x (i) ) as the score of c: We further normalize the scores of C and transforms them into a probability vector using the softmax function. Let p (i) denote the probability vector estimated at the i-th iteration. We then infer the most likely target classĉ * as the class c leading to the overall largest probability:ĉ * = arg max Algorithm 2 sketches the procedure of adversary intent inference for n iter iterations.

Proactive Solicitation
As illustrated in Figure 4, because the adversary performs descent based on the estimated gradient (Algorithm 1), the descent direction д(x) may deviate from the true gradient direction д(x), which in turn causes possible errors in AdvMind's estimation of c * . The estimate error ofд(x) mainly results from the bias betweenд(x) and д(x). Take signSGD in Eqn (4) as a concrete example; the expectation of д(x) is given by: Where u ∼ N (0, I ) and R(x, u) is the second order remainder of f (x) in its Taylor expansion. We have the following proposition (proof in Appendix). Applying this proposition to Eqn (13), the first term becomes д(x), which is the true gradient. Yet, due to the high-order term R(x, u), Eqn (13) is a biased estimate of д(x). Meanwhile, if we remove the high-order information in the query answers, the adversary would be able to estimate the gradient more accurately, and AdvMind, in turn, would also obtain a more accurate estimation of the adversary's intent. To this end, we define the prediction for a query x i as:f wherex is the estimated QOI,f (x i ) is the modified prediction on x i , G is a gradient matrix (G ∈ R | C |×n ), in which the c-th row is the gradient vector of the class c.
We have the following proposition that ensures the similarity of f (x i ) and its true value f (x i ) (proof in Appendix): where u ∼ N (0, I ) and B(x) is the bias of the QOI estimate in § 3.
Applying Eqn (14), the adversary's gradient descent direction now shares the same expectation with the true gradient direction with respect to the target class c * . Yet, due to the estimation variance, the gradient of other classes may be event might be even closer to the adversary's estimate, as shown in Figure 4. To address this issue, we force the gradient directions of different classes to be orthogonal to each other so that if the adversary follows a descent direction, the target class can be easily distinguished from other classes, as illustrated in Figure 4.
To this end, the new gradient matrix is defined as follows: where the parameter µ ∈ [0, 1] (dispersion coefficient) controls the magnitude of direction dispersion, and M is an orthogonal matrix set by AdvMind (an example given below). Specifically, M separates gradients of different classes since each row of G is the gradient of the corresponding class.
Note that Eqn (17) is only one possible instantiation of M. Let c be the number of classes and d be the data dimensionality. There are essentially (c!) (d /c) different permutation matrices from which the defender is able to arbitrarily select, creating a prohibitively large number of possibilities for the attacker to explore in order to recover the genuine gradient information.
In conclusion, the proactive solicitation modifies the first-order information (i.e., gradient) while preserving the zeroth-order information. By properly controlling the proportion of gradient information released to the adversary, AdvMind is able to expose the adversary's intent at an early stage while also deterring her from achieving successful attacks.

EMPIRICAL EVALUATION
Next we empirically evaluate the performance of AdvMind with respect to benchmark datasets, popular DNNs, and state-of-theart black-box adversarial attacks. The experiments are designed to answer the following questions: • RQ1: Is AdvMind effective to detect the attacker's intent during an early stage of the attack?
• RQ2: Is AdvMind effective against the attacker who purposely attempts to conceal her intent?
• RQ3: Is AdvMind effective against a variety of black-box adversarial attacks? We begin by introducing the experimental setting.

Experimental Setting
Datasets. We primarily use four image classification datasets in our evaluation.
Attack and Inference Models. We consider 3 state-of-the-art blackbox adversarial attack models, NES [14], signSGD [2], and Hes-sAware [39], and their adaptive variants (indicated by the superscript of "A"). More details of the attacks can be found in § 2. We build 3 variants of AdvMind: basic (AdvMind) -which passively observes the attacker's queries; robust (AdvMind R ) -which employs robust intent estimation; and robust + proactive (AdvMind RP )which adopts both robust intent estimation and proactive intent solicitation.
The experiments are conducted on a Linux server with 4 Quodro RTX 6000 GPUs, 2 Intel Xeon processors, and 384G RAM. All the algorithms are implemented in PyTorch. The default parameter setting is summarized in

A CTF Game
To evaluate the performance of AdvMind (the defender) against different black-box attacks and their variants (the attacker). We define the following catch the flag (CTF) game between the attacker and the defender.
Setting. Given the backend DNN f , the attacker randomly selects one input x • from the testing set and a target class c * f (x • ), and aims to generate an adversarial input x * that satisfies (i) x * ∈ B ε (x • ) and (ii) f (x * ) = c * . At the k-th iteration of the attack, the attacker estimates the gradient with respect to the current input x (k−1) and then perturbs x (k −1) to generate x (k ) ; meanwhile, the AdvMind attempts to infer the attacker's target class c * based on the previous queries. We define that the attacker wins the game if she successfully generates the adversarial input x * before AdvMind is able to correctly infer c * with high confidence, and AdvMind wins the game if she correctly infers c * before the attacker generates x * . Note that if none of them succeeds, it is a "draw".
Observations. We measure up to each iteration the percentage of games won by the attacker or the defender, under different configurations for the attacker and the defender. The results on different datasets and DNNs are illustrated in Figure 5 and Figure 6, from which we have the following key observations. Note that we aim at early detection so we only focus on the first 10 iterations, during which most attacks haven't succeeded, especially for VGGface2. O1: Basic AdvMind is able to infer the attacker's intent in nonadaptive attacks to a large extent. AdvMind defeats the attacker in around 50% of the games up to the 10-th iteration on CIFAR10 and CIFAR100, and in around 90% of the games on ISIC and VG-Gface2. Its better performance on higher-dimensional datasets may be attributed to the larger divergence of gradient directions among different classes in higher-dimensional spaces. Yet, the success rate of AdvMind increases gradually with the number of iterations, indicating that the passive inference requires a large number of observations to make reliable estimation. Furthermore, basic AdvMind is highly sensitive to the fake queries injected by the attacker. Under the adaptive attacks, AdvMind fails to correctly infer the attacker's target before the attacker successfully generates the adversarial examples. The detection rate drops around 10% on CIFAR10 and CIFAR100 and about 30% on ISIC and VGGface2. This may be explained by that basic AdvMind relies on naïve mean estimation, which tends to significantly deviate from the real value, under the influence of the injected noise. This deviation is especially evident for higher-dimensional data (e.g., ISIC and VGGface2).
O2: The robust intent estimation of AdvMind R effectively mitigates the influence of fake queries by focusing on the subset of coherent queries to estimate the query of interest. Meanwhile, at each iteration, the attacks attain lower success rates than the case of non-adaptive attacks against basic AdvMind, which is explained by the smaller number of queries useful for gradient estimation.
O3: The proactive intent solicitation of AdvMind RP further significantly improves the inference accuracy, especially during the early stages of the attacks. Across most cases, by the 3rd iteration, AdvMind RP is capable of identifying the attacker's true target with approximately 75% accuracy even against the adaptive attacks. Meanwhile, because of the dominant performance of AdvMind RP , the attacker's winning percentage is kept close to 0 during the first 10 iterations under all the settings.
O4: Overall, AdvMind's performance seems agnostic to different datasets, DNNs, or black-box attacks. In comparison, the attacker's effectiveness varies significantly with the concrete setting. For instance, at the 10-th iteration, the winning percentages of most adaptive attacks on ISIC2018 decrease by around 12.5% compared with their counterparts on CIFAR10. This variations may be explained by that under the black-box setting the adaptive attacks tend to require more iterations to craft adversarial examples in higher-dimensional spaces.

Impact of Key Factors
Next we evaluate the impact of different factors on AdvMind's performance, which shed light on the optimal operation of AdvMind. By    Table 1. To evaluate the inference of AdvMind and the attack effectiveness, we evaluate the inference success rate and the attack success rate up to the 10-th iteration.
Note that to factor out the mutual influence, here we evaluate the attacker's and AdvMind's success rates independently.
Number of Queries per Iteration n query . We first measure the impact of the number of queries issued by the attacker per iteration (n query ). Figure 7 illustrates the attacker's and AdvMind's success rates on different datasets and DNNs as n query varies from 20 to 200. Note that we keep the proportion of fake queries p fake constant (p fake = 0.5) in all the experiments here.
As expected, across all the settings, a large number of queries allow the attacker to generate adversarial inputs more effectively. For instance, on CIFAR10, as n query grows from 20 to 200, the attack success rate increases from about 0.1 to 0.4. At the same time, with more queries, AdvMind is also enabled to estimate the attacker's query of interest more accurately, leading to higher inference accuracy. For instance, on CIFAR10, the inference accuracy increases from around 0.7 to 1.0. Therefore, the attacker faces the dilemma between attack effectiveness and intent exposure.
Proportion of Fake Queries p fake . In this set of experiments, with n query = 100 fixed, we vary the proportion of fake queries p fake to evaluate its impact on the attacker's and AdvMind's performance. Figure 8 plots the results.
It is observed that for both attacker and AdvMind, there exists a critical threshold of p fake : once p fake exceeds this threshold, the attack/inference success rate sharply decreases. For instance, on CI-FAR10, if p fake ≥ 0.6, the success rate of AdvMind drops to around 0.1; once p fake ≥ 0.6, the attack success rate drops to around 0. Interestingly, the accuracy drop is larger for AdvMind than that for the attacker. This may be intuitively explained as follow: p fake mainly affects the attacker's estimation about the gradient information and AdvMind's estimation about the query of interest; yet, as AdvMind needs first to differentiate fake and true queries (which is set by the attacker), p fake tends to have a larger impact on AdvMind. Thus, to hide her intent, the adversary's strategy may be to set p fake sufficiently large (e.g., p fake ≥ 0.6), which however significantly increases the attack cost (measured by the number of queries) and reduces the attack success rate.
Dispersion Coefficient µ. For AdvMind, the parameter µ controls the weight of the dispersion matrix M in Eqn (16). A large µ implies that the gradient directions with respect to different classes tend to be approximately orthogonal to each other, therefore highly indicative of the adversary's target. Figure 9 shows the attack and inference success rates as functions of the dispersion coefficient. It is noted that across all the settings, AdvMind's effectiveness increases with µ, while the attack's success rate drops accordingly. This is due to that (i) the orthogonality of the gradient directions increases with µ, making it increasingly easy for AdvMind to identify the adversary's gradient descent direction, and (ii) meanwhile the error of the gradient information estimated by the adversary grows with µ, making it increasingly difficult to find adversarial inputs. However, it is worth pointing out that as µ increases, the gradient information estimated using the query results becomes less and less informative for the adversary to craft adversarial inputs. Therefore, the adversary may abandon using the query results as µ reaches a critical threshold. To incentivize the adversary to use the gradient information, the defender may keep µ small (e.g., µ = 0.1) in operating AdvMind.

RELATED WORK
In this section, we survey the literature relevant to this work.
White-Box Attacks and Defenses -Due to their use in securitycritical domains, deep neural network (DNN) models are increasingly becoming the targets of malicious attacks. Most existing work focus on the white-box setting, in which the adversary has full access to the model information (e.g., its architecture and parameters). One line of work develops new evasion attacks against DNN models [11,23]. Another line of work attempts to improve DNN resilience against such attacks by inventing new training and inference strategies [22]. Yet, such defenses are often circumvented by even powerful attacks [5] or adaptively engineered adversarial inputs [1], resulting in a constant arms race between adversaries and defenders [18].
Black-Box Attacks and Defenses -The increasing popularity of publicly accessible predictive APIs [36] has spurred research on black-box adversarial attacks in which the adversary has limited or no knowledge about the target DNN. The first class of attacks leverages the property of transferability [34]: certain adversarial inputs crafted against one DNN model are found effective against another model; thus the adversary is able to generate adversarial inputs based on a surrogate model and apply them on the target model [21]. To defend against such attacks, the method of ensemble adversarial training [35] has been proposed recently, which trains a DNN model using data augmented with adversarial inputs crafted on other models.
Recent work notes that adversarial inputs for surrogate do not always transfer to the target model, especially when conducting targeted attacks [6,26]. This line of attacks instead constructs adversarial inputs by estimating the gradient through the target DNNs with coordinate-wise finite difference methods [14,20,39]. The research on defending against query-based black-box attacks is still limited. The recent work [7] proposes a stateful method to detect query-based black-box attacks by measuring the relationships (e.g., similar but non-identical) of a sequence of queries.
This work complements the existing work by focusing on identifying the adversary's intent during an early stage of the attack. Meanwhile, early-stage recognition of the adversary's intent represents a long-standing challenge for the security community in general [19,24,31]. This work studies this problem within the context of adversarial attacks against DNN models. The findings point to a new direction of mitigating black-box adversarial attacks.

DISCUSSION AND CONCLUSION
In this paper, we present AdvMind, a new class of models for inferring adversary intent in black-box adversarial attacks. Combining robust intent estimation and proactive intent solicitation, AdvMind is able to reliably identify the adversary's query of interest and accurately detect the adversary's target class in an early stage of the attack, which facilitates to deploy proper mitigation strategies and to perform prompt remediation against such threats. The empirical evaluation with respect to benchmark datasets, popular DNNs, and state-of-the-art attacks validates the efficacy of AdvMind.
Yet, AdvMind is limited in the following aspects, opening up several avenues for further research. First, we assume that a single adversary launches the attack and therefore can be easily detected [7]. In practice, the attack can be performed by multiple adversaries in a coordinated manner. It is critical to extend AdvMind to more complicated settings. Second, we mainly focus on query-based attacks. There are other types of black-box attacks (e.g., surrogate model [21]). As they do not require query access during drafting adversarial inputs, AdvMind is unable to infer the adversary's target in such attacks. Finally, after identifying the adversary's intent, the next step is to perform effective mitigation or remediation. It is beneficial to integrate AdvMind with model enhancement methods (e.g., adversarial training [23]) to provide end-to-end protection.

APPENDIX A. Symbols and Notations
Notation Definition original input x * adversarial input c * adversary's target class α adversary's learning rate д(x ) gradient of ℓ with respect to x д(x ) adversary's estimate of д(x ) {u (k ) } samples of standard normal distribution n iter number of attack iterations p fake proportion of fake queries ε norm constraint of perturbation

B: Separability of Consecutive Iterations
To infer the adversary's intent, AdvMind separates the queries of different iterations. We now justify the feasibility of separating queries from consecutive iterations. In a query-based black-box attack, at each iteration, the adversary samples queries around a QOI to estimate its gradient; the queries thus form a cluster, as illustrated in Figure 10, The sampling variance σ determines the cluster radius, while the step length α (corresponding to the learning rate in the adversarial attack) determines the distance between consecutive clusters. Next we prove that in order for the adversary to accurately estimate the gradient, α needs to significantly larger than σ (i.e., two consecutive clusters are well separated). First, the upper-bound of the finite sample convergence rate of zero-order stochastic optimization can be written as [37]: where θ is the optimized parameter, R is the Bregman divergence upper-bound in mirror descent, G is the expectation upper-bound of gradient, k is the number of iterations, d is the data dimensionality, s(d) is a d-dependent constraint for the sampling distribution, u is the perturbation magnitude, and α is the step length.