{'fraction_non_alphanumeric': 0.028309671413156167, 'fraction_numerical': 0.010198252332691535, 'mean_word_length': 4.870575083840516, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
On the Privacy Risks of Deploying Recurrent Neural Networks in Machine Learning

We study the privacy implications of deploying recurrent neural networks (RNNs) in machine learning models. We focus on a class of privacy threats, called membership inference attacks (MIAs), which aim to infer whether or not specific data records have been used to train a model. Considering three machine learning applications, namely, machine translation, deep reinforcement learning, and image classification, we provide empirical evidence that RNNs are more vulnerable to MIAs than the alternative feed-forward architectures. We then study differential privacy methods to protect the privacy of the training dataset of RNNs. These methods are known to provide rigorous privacy guarantees irrespective of the adversary's model. We develop an alternative differential privacy mechanism to the so-called DP-FedAvg algorithm, which instead of obfuscating gradients during training, obfuscates the model's output. Unlike the existing work, the mechanism allows for post-training adjustment of the privacy parameters without having to retrain the model. We provide numerical results suggesting that the mechanism provides a strong shield against MIAs while trading off marginal utility.

privacy attacks [1]. Often protecting the privacy of such sensitive information is required by law, e.g., the US Health Insurance Probability and Accountability Act of 1996, which applies to processing healthcare data. Additionally, privacy protection may be associated with strategic values, irrespective of legal considerations, to hide valuable information from competitors [2]. Therefore, it is essential to study the privacy implications of machine learning models prior to their deployment.
We study the privacy implications of deploying recurrent neural networks (RNNs) in machine learning models. These networks are widely used in speech and handwriting recognition [3][4][5], deep reinforcement learning [6,7], semantic segmentation of video sequences [8], etc. State-of-the-art architectures, such as long-short term memory (LSTM) and gated recurrent units, are able to effectively process sequential data and capture dependencies between sequence elements [9]. However, in practice, RNNs are more challenging to be trained than feed-forward neural networks (FFNNs) [10] and are often too slow for real-time inference [11]. Such drawbacks have driven research towards replacing RNNs with feed-forward alternatives that perform at least as well as the recurrent versions [12][13][14]. In this work, we ask the following question.
How do RNNs and FFNNs compare in protecting the privacy of their training datasets?
We focus on membership inference attacks (MIAs) as the privacy threat to compare privacy vulnerabilities. These attacks aim to infer whether or not a given data record belongs to the training dataset of a model, to which we refer as the victim hereinafter. An MIA must be sensitive to the differences between the victim's outputs that correspond to the members of its training dataset and those encountered for the first time. Existing MIAs mainly use the inference confidence or accuracy of the victim as membership indicators [15]. As RNNs generate their outputs in a sequential and dynamic manner, an MIA may be able to associate certain dynamic behaviors to specific data records as an additional indicator. Therefore, we conjecture that RNNs are more vulnerable to MIAs than FFNNs.
As for the main contributions, we provide empirical evidence that RNNs are indeed more vulnerable to MIAs than their task-specific feed-forward alternatives. In particular, we consider three machine learning tasks: image classification, deep reinforcement learning, and machine translation. In all three experiments, we train two agents, an RNN and an FFNN. Upon convergence, we perform an MIA on both agents using the same attack method. Having observed that the MIAs consistently achieve a higher attack accuracy against RNNs than the alternative feed-forward architectures, we conclude that there are higher privacy risks pertaining to the training datasets of RNNs than FFNNs.
In the next contribution, we provide empirical evidence that not only RNNs are more vulnerable to MIAs than their feed-forward counterparts, defending them against MIAs using regularization methods may be more challenging as well. Regularization methods benefit FFNNs in two ways: they prevent overfitting and mitigate vulnerability to MIAs [16,17]. The work in [18] further shows that, under certain assumptions, overfitting is sufficient for vulnerability to MIAs. In other words, agents with lower generalizing power will be more vulnerable to MIAs; however, an agent may generalize well and still leak information to MIAs. We empirically demonstrate that RNNs and FFNNs behave differently upon overtraining and regularization methods provide marginal defense benefits against MIAs for RNNs.
Finally, we study defense measures that leverage the promise of differential privacy [19]. Learning algorithms that satisfy the condition of differential privacy protect the training datasets from any attacker and not just MIAs; however, these methods often cause inaccuracy at the inference level and trade off utility [20][21][22].
Building upon the moments accountant method in [23], the DP-FedAvg algorithm is a prominent differentially private algorithm for training RNNs in natural language processing tasks [24,25]. DP-FedAvg enforces differential privacy by adding noise to gradients while training. As a result, modifying the privacy level of DP-FedAvg requires discarding the model and training from scratch which could be undesirable. For example, for troubleshooting purposes, the operators of such models may require access to the underlying outputs that the model generates without the perturbations of differential privacy. Such an access is not possible when using DP-FedAvg.
We develop an alternative post-training differential privacy mechanism based upon the notion of sensitivitybased differential privacy [26] and using the Dirichlet mechanism [27,28]. The Dirichlet mechanism is espe-cially designed for privatizing probability vectors which happens to be the output of many machine learning models. The mechanism does not alter the training algorithm and obfuscates the outputs, instead; therefore, adjusting the privacy level does not require retraining. We test the mechanism in image classification, deep reinforcement learning, and machine settings and measure the algorithm's utility against the protection it provides against their respective MIAs. The empirical results suggest that the mechanism can weaken the MIA inference accuracy to random guessing while trading off less than 10% utility.
The rest of this paper proceeds as follows: in Section 2, we provide some preliminary background about the experiments. In Section 3, we lay out the methodology that we use to design MIAs and compare the methodology with existing MIAs. In Section 4 we report the empirical results pertaining to vulnerability comparison and regularization methods, respectively. In Section 5, we formally define differential privacy, and then, we introduce the post-training differential privacy mechanism. We conclude the paper in Section 6 and point out the future research directions that we find interesting.

Preliminaries
In this section, we review some background about the differences between RNNs and FFNNs, and the machine learning tasks that we consider in the experiments.

Recurrent vs. Feed-Forward Architecture
A neural network comprises a collection of nodes, each of which accepts an input and produces an output according to a fixed mapping called an activation function. The architecture of a neural network determines how the nodes of the networks are connected to one another. In a feed-forward architecture, the nodes can be stacked into an ordered sequence of layers from the network's input to its output such that the output of each node only affects the nodes in the subsequent layers. Examples of FFNNs include multi-layer perceptrons (MLPs), convolutional neural networks (CNNs), and more sophisticated designs such as transformers [29]. In a recurrent architecture, the connections between the nodes may form a cycle. The backward connection between an RNN's nodes can be unfolded into an in-finite sequence of layers, each of which represents the node activations at different time steps. Therefore, an RNN's output depends on the entire history of its inputs, which results in exhibiting a temporally dynamic behavior, making RNNs suitable for processing sequential data such sentences, videos, audios, etc [9].
Alternatively, FFNNs can also exhibit a temporally dynamic behavior through cascading MLPs or CNNs, or, more intelligently in transformers; however, these methods do not generate their outputs in a sequential manner and their outputs only depend on a finite window in the history of inputs; therefore, RNNs appear to be more expressive than FFNNs. On the other hand, it easier to parallelize the training of FFNNs [12], which models such as generative pre-trained transformers [30] leverage to train natural language processing models on large datasets. Furthermore, making predictions based on the entire history of inputs may be unnecessary as theoretically shown in [31]. As a result, there has been an increasing interest with successful instances in replacing RNNs with FFNNs [10,12,29,32].

RNN Applications Considered
In the experiments, we consider three machine learning tasks: image classification, machine translation, and deep reinforcement learning.
In the image classification task, the model must choose a label from a fixed set of classes to a given image. The model's output is a probability vector that determines its confidence in assigning each of the labels to the given image. CNNs are dominantly used in image classification; however, RNNs may also be used to process images as a sequence of pixels [33].
In the machine translation task, the model must map a series of tokens from a fixed input dictionary to a target dictionary. Both RNN and FFNN solutions use an encoder-decoder framework. The first half of the model computes an encoding of the input sequence through multiple encoder layers. Subsequently, the second half of the model uses the encoding and generates an output sequence through multiple decoder layers. The output of the model is a sequence of probability vectors over the target dictionary words. The dictionaries are appended with start and end tokens to signal the start and completion of sentences, respectively.
Recurrent architectures such as bi-LSTMs [34] use a network of LSTMs to construct both of the encoder and the decoder networks and can process arbitrarylength sequences. Feed-forward architectures such as transformers [29] fix a window of sequence lengths allowed and construct the encoder and the decoder networks using FFNNs. Both of the above network architectures are widely used in machine translation; however, since the debut of transformers in 2017, they have outperformed RNN-based models [35].
In the deep reinforcement learning task, the agent must learn how to reach a target state under partial state observations. The output of the agent casts a probability distribution over the actions that are available to the agent and is called a policy. Under partial state observations, the optimal policy may require memory [36] and RNNs can be integrated with deep reinforcement learning algorithms to capture long-term dependencies [37]. Both MLPs and LSTMs are commonly used in deep reinforcement learning algorithms such as soft actor critic [38], proximal policy optimization (PPO) [39], etc.
In the image classification task, the MIA must infer whether or not an image record has been used to train the model, which can violate the privacy of the individuals in face recognition applications, for example. In the machine translation task, the MIA must infer the membership of a pair of translated sentences in the model's training dataset. These language datasets may be crowdsourced from the users of social media platforms and a successful MIA violates the privacy of the users. Finally, in the deep reinforcement learning task, we consider an MIA that must infer whether or not the agent has been trained in specific regions of the environment, which violates the location privacy of the agent.

Methods
In this section, we describe the methodology that we use to design the MIAs for each of the experiments that we perform. Then, we compare our MIA layouts with the existing MIAs in the literature.

Designing MIAs
We follow the framework of shadow models [16] in designing the MIAs. Intuitively, a shadow model mimics the victim's behavior without directly having access to its training dataset or its learning algorithm. Instead, a data source provides a separate training dataset to the shadow model that is similar to the victim's training dataset. In the image classification and machine trans- lation experiments, we partition the datasets into two parts and use one to train the victim and the other to train the shadow model. In the deep reinforcement learning experiment, we use two disjoint set of environment maps to train the victim and the shadow models.
In the next step, the attacker splits its training dataset into two partitions from which it uses only one to train the shadow model. The attacker later uses the unused partition to train a binary classifier between the shadow model's outputs corresponding to the seen and the unseen data records. The crux of designing an MIA is engineering a proper training dataset for the binary classification. The top box in Figure 1 illustrates a schematic of the training of shadow models.
We use a 3-tuple format to populate the binary classifier's training dataset as shown in the bottom box in Figure 1. The first element of the tuple is the output probability vectors that the shadow model generates from either partitions of its training dataset. The second element consists of some side information about the output in the first element to help the MIA distinguish between the members and non-members. For example, the attacker may leverage the intuition that it is more likely for an image classifier to mislabel unseen images than those used in the training of the model. As a result, it is common to use the correct labels as the side information in supervised learning tasks. Therefore, in the image classification and machine translation tasks, we use the correct labels and correct translations provided in the training datasets, respectively.
Deep reinforcement learning, however, is not an instance of supervised learning task and we must choose a different type of side information. Recall that in the deep reinforcement learning task, the attacker must infer whether or not a given environment map has been used in the training of the victim. At the given environment map, the attacker can train a separate labeling agent to determine the optimal action probability vector at every state observation that the shadow model faces. We then use the labeling agent to provide the side information as follows: for every action probability vector that the shadow model generates, we use the labeling agent's corresponding action probability vector as the side information.
Finally, the last element of the binary classifier's training dataset simply indicates the partition of the shadow model' training dataset to which the first element corresponds. Once the dataset is fully populated, we concatenate the first two components as features, and use the third element as labels to train a binary classifier which conclude the design of the MIA. The attacker can use the trained MIA on the victim to execute its privacy attack as shown in Figure 2.

Existing MIAs
We now compare the MIAs that we designed in the previous section with the existing MIAs in the literature. In the image classification task, our design is identical to the work in [16]. However, in the other two tasks, namely, machine translation and deep reinforcement learning, the existing MIAs are ill-suited to the experiments of this paper.
The audit model [40] is an MIA for text generation and machine translation models that use RNNs. Similar to our design, the audit model also uses shadow models; however, instead of using the prediction probabilities in the first element of the binary classifier's training dataset, the audit model uses the rank of the predicted words over a subset of the dictionary. The audit model uses word ranks in order to design a practical MIA that can be used against the existing natural language processing application programming interfaces (APIs). In another approach adopted in [41], the authors design an MIA for transformers and only use the sequence of the words that the model outputs instead of ranks. The authors find that the resulting MIA's accuracy hovers around 50%, which equals to the attack accuracy of an MIA based on random guessing.
Our goal in using MIAs is to compare the privacy vulnerabilities of RNNs and FFNNs. Therefore, we side with the MIA by providing any information that can improve its accuracy. The authors of [42] show that the Bayes optimal MIA must use the victim's full response for all classes, which translates to using the entire probability values in the output. Having empirically observed that using probability values outperforms the audit model and the MIA in [41], we diverge from these MIAs for machine translation in the way we populate the training dataset of the binary classifier.
We now review the existing MIAs in deep reinforcement learning tasks. The work in [43] is the first to consider a privacy attack against reinforcement learning agents resembling MIAs. However, instead of using a binary classifier, the privacy attack uses a multi-class classifier. As a result, the attacker must train a labeling agent for every single environment map prior to executing the attack. By using a binary classifier, we train labeling agents sequentially for every environment map that the MIA faces on the fly.
The work in [44] develops an MIA that infers the membership of the roll-out trajectories that batchconstrained deep Q-learning agents store in their replay buffers, specifically. We do not restrict the algorithm that is used to train the reinforcement learning agents. Therefore, our design of the MIA against deep reinforcement learning tasks may be of independent interest.

Vulnerability Comparison
In this section, we report and analyze the results of a series of experiments in which we compare the vulnerability of RNNs and FFNNs to MIAs. The implementation details of the experiments may be found in the Appendix.

Deep Reinforcement Learning
We use the MiniGrid toolkit [45] to compare the vulnerability of RNNs and FFNNs to MIAs in deep reinforcement learning. In particular, we choose the MiniGrid-MultiRoom-N4-v0 environment which is shown in Figure 3. In these environments, the agent can be trained in various floor maps, each of which can be uniquely reconstructed by a seed number. The MIA's goal in this experiment is to observe an agent's navigation in a given floor map and infer whether or not the floor map has been used to train it.
We train two agents using the PPO algorithm that is implemented by the RL-Starte-Files library [46]. The first agent uses an MLP neural network and the other uses an MLP with additional LSTM units. Therefore, the former is an FFNN and the latter is an RNN agent. We use the average discounted reward that the agents accumulate as the underlying performance measure. We choose the hyperparameters of the two agents in a way that (a) they have a similar number of trainable parameters (5192 parameters for the FFNN and 5216 parameters for the RNN agent) and (b) their per-  Once the agents are trained, we train an MIA for each of the two agents following the design in Section 3. Revisiting Figure 1, the MiniGrid simulator is the data source and the training datasets contain the floor map seed numbers that are used in the training of the agents. We use two disjoint sets of seed numbers to train the victim and the shadow model. However, we use the same architecture and hyperparameters for the two agents because our intention is to measure vulnerability and we do not seek to design practical MIAs. However, in the experiments, we observed that model-mismatch between the victim and the shadow model does not affect the MIA's performance by much.
In order to measure the attack accuracy of an MIA, we obtain an equal number of output sequences from the victim in floor maps that were used in its training and new floor maps. Then, we report the attack accuracy as the percentage of the instances that the MIA correctly infers membership. In this setting, an MIA that makes arbitrary-at-random guesses achieves a trivial 50% attack accuracy. The results in Figure 4 show that the MIA's attack accuracy against the RNN agent is consistently higher than the accuracy against the FFNN agent.
In the next experiment, we disable a feature of the MiniGrid toolkit that allows for agents concurrently in multiple floor maps. Instead, we train the agents sequentially in the order of the seed numbers in their training dataset. Then, we measure the MIA's attack accuracy across the training history of the agents. Interestingly, the MIA maintains a non-trivial attack accuracy across the entire training history of the RNN agent, whereas the MIA against the FFNN agent quickly loses its accuracy to 50%, as reported in Figure 4.

Machine Translation and Image Classification
We subsequently perform two similar experiments for machine-translation and image-classification tasks. In the former task, we use the En-Fr portion of the Multi-30K dataset [47] to train the models. We use BiLSTM-Attention networks [48] to train the RNN agent and use Transformers [29] to train the FFNN agent. We measure performance using the bilingual evaluation understudy (BLEU) score [49], which captures how a model's translation correlates to that of a human. For the image classification task, we use the CIFAR10 dataset [50]. We use CNNs to train the FFNN agent and use LSTMs to train the RNN agent. We measure performance according to the percentage of the image records that the model labels correctly. To train the corresponding MIAs in both of these tasks, we consider the training datasets themselves as the data source and split them into two partitions to train the victim and the shadow models.
The results in Figure 5 show the same patterns in vulnerability to MIAs as those we observed in the deep reinforcement learning experiment. In the machinetranslation experiment, the FFNN agent slightly outperforms the RNN agent; however, it remains more vulnerable to the MIA across all training times and, in the sequential-training scenario, it retains its vulnerability across the entire history of its training. In the imageclassification experiment, the FFNN agent quickly overfits its training dataset and gradually loses performance as the training proceeds for longer epochs. However, even at low levels of generalizing power, the FFNN agent is less vulnerable to the MIA than the RNN agent.

Relationship between MIA Accuracy and Model Memorization
There appears to be a close relationship between the results pertaining to the sequential training experiments and the notion of unintended memorization [51] and catastrophic interference [52]. Model memorization refers to when the weights of a neural network encode the attributes of the training dataset and is closely related to overfitting [53]. Unintended model memorization can lead to sensitive information leakage. For example, a text-generation model may auto-complete the credit card numbers that may exist in its training dataset [51]. Catastrophic interference or catastrophic forgetting [52] refers to the phenomenon when sequential training of machine learning models leads to a significant loss of performance upon facing new data.
In the experiments, the waning MIA attack accuracy is an indication of catastrophic interference and the higher attack accuracy in the older stages of the training of RNNs signals unintended memorization. For further evidence, we measured the average number of time steps that the two deep reinforcement learning agents take to reach the goal after they are fully trained. When the two agents encounter a new floor map, they both take roughly 35 time steps to reach the goal. However, the RNN agent takes roughly 20 time steps to reach the goal when it encounters a previously seen floor map, whereas the FFNN agent still takes about 35 time steps to reach. Note that the number of time steps does not affect the reward as long as the discount rate is close to 1 and as long as the agents reach the goal eventually. Therefore, the RNN agent appears to be memorizing the floor maps whereas the FFNN agent does not.

Regularization Impacts
We now investigate the effects of overtraining and regularization in these experiments. Increasing the training time of machine learning algorithms may result in overfitting. For example, the validation performance of the FFNN image classification model decreases after training for 10 epochs, whereas its training performance keeps increasing. On the other hand, training models for an extended number of epochs may not always lead to overfitting. Such a phenomenon can be seen especially in RNN models [40] which is consistent with the results in Figures 3 and 5.
Regularization methods such as 2 -regularization are effective in preventing overfitting and they have been shown to be effective in reducing the vulnerability FFNN image-classification models to MIAs [16,17]. However, regularization may add bias to the converging performance levels because they alter the objective function. In particular, these methods compute the 2norm of the node activations as a penalty term, which is subsequently multiplied with a regularization coefficient λ and added to the model's loss function. In Figure 6, we observe that regularization affects the FFNN and RNN models in the image classification and machine translation experiments differently. In particular, the MIA accuracy in the FFNN models is highly sensitive to the regularization coefficient λ, whereas the RNN models are marginally impacted by it.
For the deep reinforcement learning agents, we test a different method of regularization. It is common in deep reinforcement learning algorithms such as the PPO and trust-region policy optimization (TRPO) [54] to regularize the Kullback-Leibler divergence between the policy updates in order to increase model stability [55]. In the PPO algorithm, which we use to train the RNN and FFNN agents in the deep reinforcement learning experiment, a parameter called the clipping epsilon clip controls the policy updates as follows: a small value of clip prevents the agent from taking large gradient steps whereas a large epsilon does not relatively restrict the agent as much. In this case, the validation performance of the both the RNN and FFNN agents are highly sensitive to regularization. However, the RNN agent is still more vulnerable to the MIA than the FFNN agent, and its respective MIA accuracy is relatively less sensitive to regularization.

Differential Privacy
Differential privacy is a characteristic of an algorithm and provides a quantitative definition to data privacy [19]. A differentially private algorithm makes it hard for any observer to link the algorithm's outputs to the respective inputs that generated them. It is best justified to use differential privacy when the purpose of the algorithm is to compute aggregate statistics of a dataset whose individual entries may contain sensitive information. For example, the US Census Bureau uses differential privacy to protect the individuals in its publications [56].
In the sequel, we first formally define differential privacy. Then, we review an existing algorithm that effectively trains neural networks while satisfying differential privacy. Then, we propose an alternative approach for affording differential privacy to the elements of a neural network's training dataset. Finally, we empirically measure the protection that our method provides against the MIAs that we developed in the previous sections and measure the utility that it trades off in order to enforce differential privacy.

Differential Privacy Definition
We denote a machine learning algorithm that trains a neural network by A H : D → Θ, where D is the domain of possible training datasets, H is the hyperparameters, and Θ is the domain of the network weights. For every weight θ ∈ Θ, we denote the resulting neural network by NN θ : X → Y, where X and Y are the network's input and output domains, respectively. We now formally define ( , δ)-differential privacy for machine learning algorithms.

Definition 1 (( , δ)-differential privacy [19]). Let A H be a given machine learning algorithm. Define two training datasets D and D , both in the training dataset domain of A H , adjacent if the number of entries in which the two datasets hold different values is at most
In the definition of ( , δ)-differential privacy, all adjacent training datasets must satisfy the probabilistic condition in (1). Intuitively, the parameter captures the strength of privacy protections and δ captures the probability that the privacy protections fail. Privacy failure could happen due to two reasons: either (1) holds for a larger or no finite satisfies differential privacy. It is customary to choose single-digit values for and choose δ to be O |D| −1 , where |D| is the size of the dataset that we wish to protect [19]. However, larger -values may still provide a strong privacy protection [57].
Differential privacy is immune to post-processing, meaning that post-hoc computations on the output of a differentially private mechanism does not affect the level of differential privacy. However, subsequent queries from the output of a differentially private mechanism may weaken privacy. In general, the overall privacy level of a sequence of k queries from an ( , δ)-differentially private mechanism results in (k , kδ)-differential privacy according to the Composition Theorem [19]. The overall privacy parameters are often referred to as the privacy budget.

Enforcing Differential Privacy
Three well-studied mechanisms are widely used to enforce differential privacy when a function's output is queried. These mechanisms are: the Laplace, Gaussian, and exponential mechanisms [19]. The first two mechanisms add noise to the function's outputs according to the Laplace and Gaussian distributions. The exponential mechanism picks a member of the algorithm's output space according to the exponential distribution parameterized by a score function that computes the utility of the candidate members. These mechanisms calibrate their corresponding probability distributions based on the privacy parameters and the so-called sensitivity of the functions that they seek to protect, which is defined as follows: Let f : D → Y be a mapping from a dataset domain D to a normed space Y. Then, the sensitivity of f , denoted S(f ), is where · is the norm operator and D and D are any two adjacent datasets. The sensitivity determines the variance of the noise that is needed to protect a function. For the case of the exponential mechanism, the sensitivity of the score function affects the variance of the mechanism outputs. In machine learning algorithms, the output of the resulting neural networks also depends on the input at which they are queried, which must be taken into account in the definition of sensitivity. We therefore modify the definition sensitivity to take these inputs into account.

Definition 2 (sensitivity). The sensitivity of a machine learning algorithm A H over input domain X is
, and D and D are any two adjacent training datasets.
In order to design a privacy mechanism for a machine learning algorithm, one must address two main issues. The first issue is related to the structure of the models predictions that belong to the unit simplex. The unit simplex, denoted ∆(n), is the set of all vectors in [0, 1] n that sum to one. By adding Laplace or Gaussian noise to a simplex-valued vector, we destroy its unique structure. Alternatively, using the exponential mechanism requires computing and integrating a score function over the continuous domain of the unit simplex which may be intractable. The second issue is related to the sensitivity of these algorithms. Fixing the 2-norm as the norm operator in the definition of sensitivity, the highest achievable sensitivity within the unit simplex is √ 2. Our experiments indicate that the 2-norm difference between the predictions of a machine learning algorithm can, in rare cases, be close to √ 2 even for two datasets that have the same entries but in different orders. Therefore, these algorithms may have a high sensitivity and the corresponding level of perturbation that is needed to enforce differential privacy may destroy their utility.
The moments accountant method [23] intelligently avoids both of the two issues. Instead of adding noise to the network's output layer, the method alters the stochastic gradient descent algorithm which is dominantly used to train neural networks and injects Gaussian noise to the gradients. In order to address the second issue, the algorithm clips the gradients which artificially bounds the sensitivity of the algorithm.
The DP-FedAvg algorithm [24,25] builds upon the moments accountant method to develop a differentially private algorithm for training natural language processing models using RNNs. The algorithm groups the entries of the training datasets based on the users who populate them. Therefore, their differential privacy guarantees pertain to the individual users, as opposed to the individual data records, which is a positive feature as a dataset may contain multiple data records from the same user.
On the other hand, in a scenario wherein the operator of the models desires to subsequently adjust the level of privacy post training, the DP-FedAvg must train a new model based on the new privacy parameters. In the sequel, we propose a post-training differential privacy mechanism that, instead of altering the training algorithm, perturbs the network outputs. The privacy level of the mechanism can be simply adjusted by calibrating the parameters of the mechanism without having to retrain the model. Such a feature offers more flexibility to the operators of machine learning models in balancing the privacy-utility level. For example, the operator may require a high-precision model output for troubleshooting which the DP-FedAvg does not allow.

Dirichlet Query Handler
We use the notion of sensitivity-induced differential privacy [26] to design the substitute privacy mechanism. Intuitively, the sensitivity-induced differential privacy only guarantees to protect those training datasets that give rise to a certain level of sensitivity. In general,S may be a vector of sensitivity bounds with respect to a collection of norm operators. The next building block to design the privacy mechanism is the Dirichlet mechanism [27,28]. The Dirichlet mechanism Algorithm 1: DQH Input: algorithm A H , training dataset D, Dirichlet mechanism parameters α and r, query point x Output:ỹ θ ← A H (D) ; /* Train NN θ */ y ∼ Dirichlet α,r (NN θ (x)) ; /* Handle query*/ satisfies ( , δ)-differential privacy for the elements of the unit simplex and is described as follows: Definition 4 (dirichlet mechanism [28]). For fixed parameters α ∈ (0, ∞) n and r > 0, the Dirichlet mechanism, denoted Dirichlet α,r , is a probabilistic mapping that maps any simplex-valued vector y to another simplex-valued vectorỹ with probability Recall that the output of the neural networks in image classification, machine translation, and deep reinforcement learning algorithms were all probability vectors. We use the Dirichlet mechanism to privatize the output layer of neural networks and leverage the notion of sensitivity-induced differential privacy to compute its privacy level. In particular, for a given input x and parameters α and r, we invoke the Dirichlet mechanism and draw a sample from Dirichlet α,r (NN θ (x)) as the model's output. We call our method the Dirichlet Query Handler (DQH) because the Dirichlet mechanism acts as a privacy shield in processing the queries made from the neural network's outputs. In the following lemma, we show that the DQH algorithm satisfies sensitivity-induced differential privacy. Lemma 1. Let α ∈ (0, ∞) n and r > 0 be the parameters of a Dirichlet mechanism, ψ(z) = d/dz(log(Γ(z)), and ψ (z) = d/dz(ψ(z)). For all δ ∈ (0, 1), all γ ∈ (0, α min ) and 2-norm and ∞-norm sensitivity boundsS 2 > 0 and S ∞ > 0, respectively, let and (γ, δ) = ρ γ + 2 ρ γ log(1/δ) log(1/δ) ≤ (ω γ − 1) 2 ρ γ , Then, with = inf γ (γ, δ), DQH satisfies (S 2 ,S ∞ )sensitivity-induced ( , δ)-differential privacy.
Proof. By Corollary 3 in [28], for all δ ∈ (0, 1), all γ ∈ (0, α min ), and all simplex-valued p and p such that the Dirichlet mechanism satisfies where R is any subset of the unit simplex and = inf γ (γ, δ) as in (7). Consider two arbitrary adjacent datasets D and D for algorithm A H such that and Then, the corollary implies that, for all R ⊆ Y and all x ∈ X , where X is the input domain and Y is the output domain of the neural networks that A H trains.

Empirical Results
In this section, we deploy the DQH method in the three machine learning tasks that we considered in the previous sections. For each task, we measure the utility of the models across a range of values of the Dirichlet mechanism's parameters r and α. For every value selected, we train a separate MIA assuming that the MIA knows the values of r and α. As a result, we train the MIA's shadow models using the same Dirichlet mechanism that the victims use. We make such an assumption because sharing the parameters of a differential privacy mechanism does not violate differential privacy.
The results in Figure 7 show the following pattern: higher r-values are associated with a higher utility and vulnerability to MIAs, and conversely, a higher α results in a lower utility and vulnerability to MIAs. Note that, in the experiments, we use the same value across the components of α and report a scalar for α. These results are consistent with Lemma 1 because for fixed δ and sensitivity boundsS ∞ andS 2 , r and α have a positive and a negative correlation with , respectively. Next, we compute the level of differential privacy for each of these experiments. In Lemma 1 in which we compute the privacy level, we must first choose δ. With D denoting the training dataset, we choose δ = O(|D| −1 ) which is the conventional choice of δ in the literature [19]. Specifically, for the image classification model, |D| = 10, 000 and we choose δ = 1 × 10 −5 . For training the machine translation model, we split the training dataset into 500 users and choose δ = 1 × 10 −4 . Finally, for the deep reinforcement learning agent we use 32 environment seeds and choose δ = 1 × 10 −3 .  Lemma 1 computes the privacy level with respect to the notion of sensitivity-induced differential privacy and allows for choosing any value for the sensitivity bounds. However, in choosing the bounds, we must make sure that they reflect the model's sensitivity as defined in Definition 2. The sensitivity captures the extend to which the predictions of a model change when an entry is removed from or appended to its training dataset. In a large training dataset and upon the convergence of the learning algorithm, the predictions must not be affected by a single training data record. We therefore expect a low sensitivity for such models.
We use Algorithm 2 to estimate the sensitivity of the machine learning algorithms that we consider. The results for m = 500 are reported in Table 1.

Algorithm 2: SensitivitySampler
Input: machine learning algorithm A, hyperparameters H, sample size m, data source DS, training dataset size N , validation dataset D val Output: We use an over-approximation of the sensitivity values in Table 1 to compute the level of sensitivity-induced differential privacy. In computing the sensitivity, we observed that, by clipping the gradients the same way as in DP-FedAvg, we can significantly reduce the sensitivity. In particular, clipping the gradients whose 2-norms are greater than 10 in the machine translation experiment results in a reduction ofS 2 from 0.0283282 to 0.0015356 without any noticeable effect on the resulting model utility. Therefore, in all of the experiments, we use regularization to maintain a low sensitivity.
The resulting privacy levels for each of the tasks alongside the sensitivity bounds that we considered as an over-approximation are reported in Figure 8. Combining the results in Figures 7 and 8 the operators of these models can decide upon the privacy-utility tradeoff that they deem desirable. For example, in the image classification task, one may use (α = 0.01, r = 1) because it trades off 10% utility while reducing the MIA's accuracy from 70% to 53%. Moreover, from Figure 8, the choice of (α = 0.01, r = 1) corresponds to (1.2, 1 × 10 −5 ) (0.002, 0.002)-sensitivity induced differential privacy.
We note that the -values reported in Figure 8 correspond to a single draw from the Dirichlet mechanism. In the machine translation task, the observers invoke the Dirichlet mechanism as many times as there are words in the output sentence. In these cases, the Composition Theorem applies. The same case applies to the deep reinforcement learning agent who, on average, takes 40 time steps to reach its target. For these two models, we chose δ such that the composition of the queries satisfies δ = O(|D| −1 ).

Conclusion and Future Work
In this work, we provided empirical evidence that MIAs may achieve a higher accuracy when they attack RNNs compared with their feed-forward alternatives. We empirically demonstrated that RNNs memorize their training data in a way that an MIA can maintain a nontrivial attack accuracy over the entire history of their training, whereas the corresponding attack accuracy for the FFNNs quickly drops to 50% as we move back in the training history. Alongside the existing computational drawbacks of training RNNs, our results provide further incentives to replace RNNs with FFNNs.
In the second part of the study, we focused on differential privacy methods. We proposed a post-training differential privacy mechanism to protect the training datasets of machine learning algorithms. In image classification, machine translation, and deep reinforcement learning, we empirically demonstrated that the mechanism trades-off less than 10% utility while enforcing a single-digit -value, which dismisses the impression that output-perturbation mechanisms destroy utility and one must add noise to the gradients, instead.
As for the future work, we are interested to replace Algorithm 2 with verification methods pertaining to neural networks such as POPQORN [58]. In that case, we can strengthen the privacy guarantees by replacing sensitivity-induced differential privacy with ( , δ)differential privacy. Moreover, there seems to be a close relationship between algorithmic stability and sensitivity [59]. Regularization methods can significantly affect the stability of machine learning algorithms. Therefore, it might be interesting to explore regularization methods that reduce the sensitivity of machine learning algorithms without hurting their performance by much.

A.1.0.1 MIA on the reinforcement learning agent:
We use the PPO algorithm to train the agents, for which we use the default parameters set by the RL-Starter-Files toolbox, unless stated below. The feed-forward agent uses an MLP with two hidden layers, each of which consists of 64 neurons. The RNN agent uses the MLP architecture that consists of two 32-neurons layers with 4 additional LSTM units. The first layer is activated by tanh functions and the last layer is activated by a softmax function. We train the agents for a total of 204,800 iterations on seeds 1 to 16 for both agents. We use the default clipping epsilon 0.2 while training.
For the implementation of the MIA, we use an MLP with 5 ReLU-activated hidden layers and 1 LSTM unit. We use 6400 in trajectories and 6400 out trajectories to generate the binary classifier's training dataset. We train the binary classifier using Adam optimizer and the cross-entropy loss function for 15 epochs, each of which consists of 100 gradient updates. We use the Keras li-brary [60] to train the binary classifier with learning rate 0.001 and default parameters, unless stated above.

A.1.0.2 MIA on the machine translation model:
We use an LSTM encoder-decoder network with dot product attention mechanism [61] to construct the sequence-to-sequence model. We use the Multi30K dataset [47] which consists of 30,000 sentence pairs for training and 1,000 pairs for testing. We use 5,000 sentence pairs to train the shadow model and a negative likelihood loss to update gradients. The shadow model is trained for 20 epochs, with a word-embedding dimension 150, a hidden dimension 200, a learning rate of 0.001, and a dropout rate of 0.2. We use PyTorch [62] to implement and train the victim model with default parameters unless specified above. Once the shadow model is fine-tuned, we use 2,000 output sequences to populate the training dataset of the MIA's binary classifier. In the training procedure, we deploy DP-FedAvg with noise n = 0 and q = 1 to reduce the sensitivity. We set the max norm of the gradients to 10 and clip the gradients with norms above the threshold.
The binary classifier consists of 1 LSTM unit, two linear layers, a ReLU-activated layer, and a softmax layer. We implement the MIA classifier using PyTorch and train it using the cross-entropy loss function for 20 epochs with the default parameters.

A.1.0.3 MIA on the image classification model:
We use a convolutional neural network that consists of 3 convolutional layers with ReLU activation, two max pooling layers, and two fully connected layers. We use the image classification dataset Cifar10 that consists of 50,000 training records and 10,000 testing records. We train the target model and shadow model using 10,000 training records and a categorical cross-entropy loss is used to update gradient. We deploy the DP-FedAvg method with noise n = 0 and q = 1 while training. We clip the gradients whose norm is greater than 10.
The recurrent neural network for image classification consists of an LSTM network with hidden dimension 32, an convolutional layer with ReLU activation, a max pooling layer and a dense layer. The training procedure is identical with the convolutional neural network.
For the implementation of the MIA, we use an MLP with 5 ReLU-activated hidden layers. We train the classifier using 20,000 probability pairs with half labeled 'in'. We use the Keras library [60] to train the binary clas-