{'fraction_non_alphanumeric': 0.040600454617063894, 'fraction_numerical': 0.017847095627123985, 'mean_word_length': 4.578656622724419, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Confined Gradient Descent: Privacy-preserving Optimization for Federated Learning

Federated learning enables multiple participants to collaboratively train a model without aggregating the training data. Although the training data are kept within each participant and the local gradients can be securely synthesized, recent studies have shown that such privacy protection is insufficient. The global model parameters that have to be shared for optimization are susceptible to leak information about training data. In this work, we propose Confined Gradient Descent (CGD) that enhances privacy of federated learning by eliminating the sharing of global model parameters. CGD exploits the fact that a gradient descent optimization can start with a set of discrete points and converges to another set at the neighborhood of the global minimum of the objective function. It lets the participants independently train on their local data, and securely share the sum of local gradients to benefit each other. We formally demonstrate CGD's privacy enhancement over traditional FL. We prove that less information is exposed in CGD compared to that of traditional FL. CGD also guarantees desired model accuracy. We theoretically establish a convergence rate for CGD. We prove that the loss of the proprietary models learned for each participant against a model learned by aggregated training data is bounded. Extensive experimental results on two real-world datasets demonstrate the performance of CGD is comparable with the centralized learning, with marginal differences on validation loss (mostly within 0.05) and accuracy (mostly within 1%).


INTRODUCTION
The performance of machine learning largely relies on the availability of large representative datasets. To take advantage of massive data owned by multiple entities, federated learning (FL) is proposed [24,25,46]. It enables participants to jointly train a global model without the necessity of sharing their datasets, demonstrating the potential to address the issues of data privacy and data ownership. It has been incorporated by popular machine learning tools such as TensorFlow [1] and PyTorch [37], and increasingly spread over various industries.
The privacy preservation of FL stems from its parallelization of the gradient descent optimization, which in essence is an application of stochastic gradient descent (SGD) (or the mini-batch * The corresponding author. mode) [25]. During the training process, the participants work on the same intermediate global model via a coordinating server (in the centralized FL) [2,4,34] or a peer-to-peer communication scheme (in the decentralized FL) [22,39]. Each of them obtains the current model parameters, works out a local gradient based on the local data, and disseminates it to update the global model synchronously [2,4] or asynchronously [21]. This paradigm guarantees data locality, but has been found insufficient for data privacy: although the local gradients can be securely synthesized via a variety of techniques such as differential privacy (DP) [2,21,42,48], secure multi-party communication (MPC) [4,11,34], and homomorphic encryption (HE) [31,34,40], the global model parameters that have to be shared are still susceptible to information leakage(cf. Section 5 and [35,36]).
This work further decreases the dependency among participants by eliminating the explicit sharing of the central global model which is the root cause of the information leakage [9,35]. We propose a new optimization algorithm named Confined Gradient Descent (CGD) that enables each participant to learn a proprietary global model. The CGD participants maintain their global models locally, which are strictly confined within themselves from the beginning of and throughout the whole training process. We refer to these localized global models as confined models, to distinguish them from the global model in traditional FL.
CGD is inspired by an observation on the surface of the typical cost function. The steepness of the first derivative decreases slower when approaching the minimum of the function, due to the small values in the Hessian (i.e., the second derivative) near the optimum [5]. This gives the function, when plotted, a flat valley bottom. As such, a gradient descent algorithm A, when applied on an objective function , could start with a set of discrete points (referred to as a colony and their distance is discussed later). Iteratively descending the colony using the joint gradient of the colony would lead A to the neighborhood of 's minimum in the "flat valley bottom". The points in the colony would also end up with similar losses that are close to the loss of the minimum.
In Figure 1, we illustrate a holistic comparison between the workflow of CGD and that of a gradient decent in traditional FL. In traditional FL, every participant updates the same global model using their local gradients , , . In CGD, each participant first independently initializes the starting point of its confined model 1 . Then, in every training iteration, participants independently compute the local gradient from their current confined model and local data, and then jointly work out the sum of all local gradients and use it to update their confined models (the equation in Figure  1b). By doing this, CGD aims to enhance privacy without sacrificing much model accuracy. For the sake of simplicity, we refer to these two properties as privacy and accuracy.
• Privacy. CGD should ensure that, throughout the training process, neither local data of a participant nor intermediate results computed on them can be observed by other participants or an aggregator (if any). • Accuracy. The prediction made by any confined model should approach the centralized model that were to learn centrally on the gathered data.
The desired privacy enhancement of CGD stems from two aspects, i.e., secrecy of confined models and secrecy of local gradients. For the former, besides always hiding the confined models from each other, each participant independently initializes its 1 at random. During the training process, any two confined models keep the same distance and never become closer to each other after descending, preventing any participant from predicting models of others. To further boost the unpredictability, each participant could select its own interval range of initial weights to avoid leaking the average distance between the confined models. CGD withstands interval ranges differing by two orders of magnitude. For the latter, CGD incorporates the secure addition operation on the local gradients to calculate their sum. This has been proved to be viable through the additive secret sharing scheme, in which the sum of a set of secret values is collaboratively calculated without revealing any addends [3,4,29,45]. A previous study [47] demonstrates it is efficient when applied to achieve decentralization in FL. We prove that the adversary's observation in CGD is only the sum of local gradients, and it conceals the extra indicative information that traditional FL would leak (cf. Section 5). We formally prove that CGD ensures convergence. It converges to confined models that are adjacent to the centralized model, and the adjacency is bounded (cf. Section 4). This merit guarantees the accuracy of CGD. We further evaluate the accuracy performance of CGD with two popular benchmark datasets, i.e., MNIST [28] and CIFAR-10 [26]. Our experiments demonstrate that its accuracy closely approaches that of the centralized learning. When the confined models are initialized with the standard initialization scheme (i.e., the Gaussian distribution of mean 0 and the variance 1), it achieves marginal differences on validation loss (mostly within 0.05) and accuracy (mostly within 1%) and outperforms state-ofthe-art federated learning with differential privacy. Its accuracy performance remains stable even when the interval ranges of initial weights among participants differ by two orders of magnitude. Contributions. We summarize the main contributions as follows.
• Confined Gradient Descent For Privacy-enhancing Decentralized Federated Learning. We propose a new optimization algorithm CGD for privacy-preserving decentralized FL. CGD eliminates the explicit sharing of the global model and lets each participant learn a proprietary confined model. CGD retains the merits of traditional FL such as algorithm independence. Therefore, it can easily accommodate any FL schemes regardless of their underlying machine/deep learning algorithms. It also eliminates the necessity of a central coordinating server, such that the optimization can be conducted in a fully decentralized manner. • Convergence Analysis. We theoretically establish a convergence rate for CGD under realistic assumptions on the loss function (such as convexity). We prove CGD converges toward the centralized model as the number of iterations increases. The distance between the trained confined models and the centralized model is bounded, and can be tuned by the hyper-parameter setting. • Enhanced Privacy Preservation Over Traditional FL.
With secrecy of both confined model and local gradients, CGD achieves enhanced privacy preservation over traditional FL. We prove that in CGD, given only the sum of the local gradients, an honest-but-curious white-box adversary, who may control out of participants (where ≤ − 2) including the aggregator for secure addition operation (if any), can learn no information other than their own inputs and the sum of the local gradients from other honest parties, whereas in traditional FL, extra indicative information about local data can be obtained. • Functional Evaluations. We implement CGD and conduct experiments on two popular benchmark datasets MNIST and CIFAR-10. The results demonstrate that CGD can closely approach the performance of centralized model on the validation loss and accuracy, with marginal differences on validation loss (mostly within 0.05) and accuracy (mostly within 1%).

BACKGROUND AND RELATED WORKS
CGD is an optimization method based on the gradient decent. Therefore, in this section, we review the existing techniques for gradient updates in the traditional FL.

Stochastic gradient descent
Stochastic gradient descent (SGD) [6,38] is an efficient variant of the gradient descent algorithm. It is extensively used for optimizing the objective function in machine learning and deep learning. Given a cost function with the parameter , SGD is defined by where are the parameter at the ℎ iteration, ∈ is a randomly selected subset of the training samples at the ℎ iteration, and is the learning rate. Equation 1 can generalize to the mini-batch update when 1 < | | < | |, and to the batch update when = .
In FL, each local participant ∈ L holds a subset of the training samples, denoted by . To run SGD (or the mini-batch update), for each iteration, a random subset , ⊆ from a random participant is selected. The participant then computes the gradient with respect to , , which can be written as ∇ ( , , ), and shares the gradient with other participants (or a parameter server). All the participants (or the server) can thus take a gradient descent step by a) Gradient descent in traditional federated learning. Participants jointly work on the same global model using the descent computed by . Although the local gradients , , can be securely synthesized, by knowing and , the adversary is able to derive information about the local raw data , , . * Each participant strictly confines their own global models from their initialization ( 0 , 0 , 0 ) to optimal values ( * , * , * ). The confined models descend in the same pace, and when CGD converges, reach the bottom of the valley where the centralized model is located. Any two confined models keep the same distance throughout the training process. In other words, , , would not become closer to each other during descending, preventing any participant from predicting models of others. The gradients, if shared in plain text, are subject to information leakage of the local training data. For example, model-inversion attacks [9,32,43] are able to restore training data from the gradients. In the immediately following sections, we summarize the existing privacy-preserving methods for synthesizing the local gradients, which fall into two broad categories, i.e., secure aggregation and learning with differential privacy.
Some existing studies fall into this category. For instance, Bonawitz et al. [4] present a secure aggregation protocol that allows a server to compute the sum of user-held data vectors, which can be used to aggregate user-provided model updates for a deep neural network. Mohassel et al. [34] propose a secure two-party computation (2PC) protocol that supports secure arithmetic operations on shared decimal numbers for calculating the gradient updates using SGD.
Existing FL frameworks employing HE/MPC are mainly based on federated SGD. All the participants in it share and update one and the same global model, and this is subject to membership inference, as revealed by Nasr et al. [35].

Learning with differential privacy
Another line of studies that approaches to privacy-preserving FL is through differential privacy (DP) mechanism [2,8,12,16,21,42,44,48,49]. The common practice of achieving differential privacy is based on additive noise calibrated to ∇ 's sensitivity S 2 ∇ . As such, a differentially private learning framework can be achieved by updating parameters with perturbed gradients at each iteration, for example, to update parameters as where N (0, S 2 ∇ · 2 ) is the Gaussian distribution (a commonly used noise distribution in differentially private learning frameworks [8]) with mean 0 and standard deviation S ∇ · .
The privacy loss is accumulated with repeated access to the data during training epochs [2]. There is also an inherent tradeoff between privacy and utility of the trained model.
In summary, in all of the above approaches, the global model has to be shared with each participant, leading to the leakage of information. This motivates CGD's design to eliminate the explicit sharing of the central global model.

CONFINED GRADIENT DESCENT
CGD optimizes an objective function in FL with multiple local datasets. It starts with a colony of discrete points, and then uses the combination of their gradients to lead the optimization to another colony of points at the neighborhood of the global optimum. In this section, we formalize this problem and present the workflow of CGD optimization.

Problem formulation
3.1.1 Optimization objective. Consider a centralized dataset = {( , )} =1 consisting of training samples. The goal of machine learning is to find a model parameter such that the overall loss, which is measured by the distance between the model prediction ℎ( , ) and the label for each ( , ) ∈ , is minimized. This is reduced to solving the following problem where ( ; ) is the loss function, and ( ) is the regularizer for .
We use * to denote the optimal solution of centralized training (i.e., the centralized model).
In the context of FL, we have a system of local participants, each of which holds a private dataset ⊆ ( ∈ [1, ]) consisting of a part of the training dataset. The part could be a part of training samples, a part of features that have common entries, or both. Assume the training takes iterations, and let denote the confined model of participant at the ℎ iteration, where ∈ [1, ]. Let represent the local gradient with respect to . We use * to denote the final confined model of participant when CGD converges. The objective of CGD is to make * located at a neighborhood of the centralized model * within a bounded gap.
3.1.2 Attacker setting. We assume an honest-but-curious whitebox adversary 1 who may control out of participants (where ≤ − 2), including the aggregator for secure addition operation (if any).  (Figure 2b). In the centralized training, the datasets of all participants are gathered for training a single model. In the traditional FL, every participant owns its local training dataset, and updates the same global model via a parameter server using its local model/gradients. The local gradients ∇ ( , , ) can be protected via either secure aggregation [4,30,31,34,40,47] or differential privacy mechanisms [2,8,21,42,44,48]. This process can be decentralized by replacing the parameter server with a peer-to-peer communication mechanism [22,39]. In CGD, each participant learns its own confined model (represented by different colors), i.e., each 1 A white-box adversary knows the internals of the training algorithms such as the neural network architecture, and can observe the intermediate computations during the training iterations. To better position CGD, we summarize the related studies in the literature in Table 1. We use federated SGD to represent the SGD or the mini-batch update in the FL, including both the plain SGD in which the local gradient/model is shared in plaintext, and privacy-preserving SGD via secure aggregation or differential privacy mechanisms. CGD guarantees that each confined model, when CGD converges, is at the neighborhood of the centralized model, retaining the model accuracy (column 5 in Table 1 and proved in Section 4). It also achieves desired privacy preservation compared to the traditional FL (column 6 in Table 1 and detailed in Section 5).

CGD optimization
Algorithm 1 outlines CGD optimization for training with the confined model . In general, the optimization process consists of the following steps. Theoretically guaranteed convergence [6,38].
The sharing of one and the same global model (even though the local gradients are protected) is still subject to information leakage (refer to [35] and Section 5). Privacy-preserving SGD via differential privacy [2,8,21,42,44,48] Additive noise (in most cases) affects the model accuracy [8].
The privacy cost is accumulated with repeated accesses to the dataset [2].

Confined Gradient Descent
Each participant learns and confines a different global model.
The model update in each iteration embraces the information from all the training samples by summing up the local gradients.
Theoretically guaranteed convergence with bounded gap to the centralized model.
Boosted privacy preservation by eliminating the sharing of the global model. Table 1: A summary of differences between the federated SGD and CGD.
• Initialization. Each participant randomizes its own 1 . A default setting is to sample based on the Gaussian distribution of mean 0 and the variance 1, which is the standard weight initialization scheme used in most machine learning approaches [13,27,33]. To prevent the colluding participants from inferring others' points by the knowledge of the average distance among the confined models, we introduce a hyper-parameter to control the interval range of initial weights 1 , i.e., 1 ∼ [− , ], and allows each participant to independently choose its own . • Step 1. At each iteration , every participant computes the local gradient ( , ) with respect to its current confined model and own dataset using Equation 5.
• Step 2. Securely compute =1 ( , ) which is later used for calculating +1 (double lines in Figure 2c). This step presumes that computational tools exist for securely evaluating the sum of a set of secret values to avoid releasing local gradient ( , ) in plain text. We refer to Section 5 for more detail.

CONVERGENCE ANALYSIS
In this section, we conduct a formal convergence analysis on CGD. Convergence analysis has been extensively used in the literature [19][20][21] to prove the correctness of optimization algorithms. Through the analysis, we demonstrate the bound of the distance between an arbitrary * learned by CGD and the centralized model * . The analysis is centered around a regret function R, which is the difference between the CGD's training loss and the loss of the centralized

Assumptions
We make the following assumptions on the loss function . They all are common assumptions in convergence analyses of most gradientbased methods, and satisfied in a variety of widely used cost functions [6], such as mean squared error (MSE) and cross entropy. Assumption 1. (Lipschitz continuity). The loss function : R → R is continuously differentiable and the gradient function of , namely, Intuitively, this assumption ensures that the gradient of does not change arbitrarily in the course of descending, such that the gradient can be a proper indicator towards the optimum [6]. Assumption 2. (Strong convexity). The loss function : R → R is strongly convex such that A useful fact from Assumption 2 for our analysis is where ⟨·,·⟩ denotes the inner product operation.
In addition, we adopt the same assumption on a bounded solution space used in related studies [6,21].

Main Theorem
We first present our main theorem below, and leave its proof to Section 4.3. It demonstrates our main result on the converge rate of CGD.
Theorem 1. Given a cost function satisfying Assumptions 1 -3, and a learning rate of = ( + ) 2 (0 < < 1), the CGD optimization gives the regret R The theorem implies the following two remarks.
• Convergence rate. Both 1 +1 and ln | +1| approach 0 as increases, implying that CGD will converge toward the optimum. The convergence rate can be adjusted by the parameter (the effect of is investigated in Section 7.2.2).
• Bounded optimality gap. When CGD converges, the gap between the confined models and the centralized model is bounded by . CGD uses the initialization parameter to determine the range of 1 in the way of where Rand is the initialization scheme. A standard Rand used in machine learning is to apply random sampling from the Gaussian distribution of mean 0 and the variance 1, and a standard is 1 √ (where is the sample size) [13].
In CGD, each participant determines its own Rand and independently to avoid leaking the average distance among the confined models. Our experiment finds that CGD keeps robust (in terms of validation accuracy) even when the participants select their s uniformly at random in a range of ( 10 √ , 0.1 √ ) (cf. Section 7.2.1).

Proof of Theorem 1
Our proof aims to identify an upper bound of R. To this end, we consider the trend of the distance from to * , which would shrink as increases, if there is a bound existing. Since is updated using =1 ( , ) (i.e., the descent in CGD), and * is obtained by ∇ (i.e., the descent in the centralized training), the trend of the distance therefore should be related to the deviation between these two. Exploring this leads to the following lemma which describes this relationship. Proof.
Dividing the above equation by , we can prove the lemma. □ In the following, we give the proof of Theorem 1. It calculates a function that is greater than R based on the convexity of the objective function (Inequation 13). The function can be decomposed into three terms based on Lemma 1 (Equation 14). We then explore the boundedness of each term, and taking these bounds together concludes the proof.
Proof. By the definition of the regret function (Equation 6) and Equation 9 in Assumption 2, we have Applying Lemma 1 to Inequation 13 and multiplying it by , we have Next, we explore the boundedness of each term. For the first term, we have Inequation 15 is based on Assumption 3(b), and Inequation 16 is based on the solution to the Basel problem that ∞ =1 1 2 < 2. For the second term, we have Inequation 19 follows reverse triangle inequality, and Inequation 21 reuses the result of the first term (cf. Equation 16).
Determining the bound of the third term is slightly complex. We list it as the following claim, and prove it soon after the proof of Theorem 1.
Combining Inequations 14, 16, 21 and Claim 1, and dividing by T we obtain  Since

PRIVACY PRESERVATION
The participants in CGD have to share the sum of local gradients, i.e.,

=1
( , ). A straightforward way is to let each participant release its local gradient ( , ), but it may leak information about or [47]. To address this, we incorporate the secure addition operation [3,4,29,45] on the local gradients to calculate their sum without releasing each of them. We make use of the additive secret sharing scheme proposed by Bogdanov et al. [3], which uses additive sharing over Z 2 32 for securely evaluating addition operations in a multiparty computation environment. It guarantees the secrecy of the addends even though the majority ( − 2 out of ) of participants are compromised. An brief introduction of the additive secret sharing scheme can be found in Appendix A.
In the rest of this section, we explore the privacy preservation of CGD. We demonstrate CGD's privacy enhancement over traditional FL. We prove that less information is exposed in CGD compared to that of traditional FL.

Information exposed in CGD
Recall that the involved parties are a set L of participants denoted with logical identities ∈ [1, ], and is the adversarial threshold ( ≤ − 2) (Section 3.1.2). Let be any subset of L that includes the compromised and colluding parties.
We demonstrate that, during optimization in CGD, given only the sum of the local gradients which are computed on different confined models, the adversary can learn no information other than their own inputs and the sum of the local gradients from other honest parties, i.e., ( , ) | ∈L\ .
Our analysis is based on the simulation paradigm [14]. It compares what an adversary can do in a real protocol execution to what it can do in an ideal scenario, which is secure by definition. The adversary in the ideal scenario, is called the simulator. An indistinguishability between adversary's view in real and ideal scenarios guarantees that it can learn nothing more than their own inputs and the information required by the simulator for the simulation. A brief introduction of the simulation paradigm is given in Appendix B.
To facilitate the understanding on our analysis, we first present the used notations. Denote L ′ = { ( , )} ∈L ′ as the local gradients of any subset of participants L ′ ⊆ L at ℎ iteration. Let ( L , , P, ) denote their combined views from the execution of a real protocol P. Let ( , , , F , ) denote the views of from an ideal execution that securely computes a function F , where is the information required by the simulator S in the ideal execution for simulation.
The following theorem shows that when executing CGD with the threshold , the joint view of the participants in can be simulated by their own inputs and the sum of the local gradients from the remaining honest nodes, i.e., | ∈L\ . Therefore, | ∈L\ is the only information that the adversary can learn during the execution.
Theorem 2. When executing CGD with the threshold , there exists a simulator S such that for L and , with ⊆ L and | |≤ , the output of S from is perfectly indistinguishable from the output of , namely Proof. We define S through each training iteration as: SIM 1 : 1 is the simulator for the first training iteration. Since the inputs of the parties in do not depend on the inputs of the honest parties in L\ , 1 can produce a perfect simulation by running on their true inputs, and L \ on a set of pseudorandom vectors Since each 1 ( 1 , ) is computed from its respective confined model 1 which is randomized in the initialization, the pseudorandom vectors \ 1 generated by 1 for the inputs of all parties in L \ , and the joint view of in , will be identical to that in , namely

and the information required by
+1 is the simulator for the ( +1) ℎ training iteration.
In execution, (32) In execution, since each +1 ( +1 , ) | ∈ L is also computed from randomized 1 , +1 can produce a perfect simulation by running the parties L \ on a set of pseudorandom vectors As such, the joint view of in , will be identical to that in , and the information required by +1 is = +1 | ∈L\ . By summarizing 1 and +1 , the output of the simulator of each training iteration is perfectly indistinguishable from the output of , and knowledge of is sufficient for the simulation, completing the proof. □

Information exposed in traditional FL
In this section, we demonstrate information exposed in traditional FL, including plain federated SGD, secure aggregated federated SGD, and differentially private federated SGD. Let be the local gradient with respect to training dataset = ( , ).
In traditional FL, is the public global model shared among the participants.
For the sake of simplicity, we assume SGD is not generalized to mini-batch update, i.e., we have 1 | | = 1. Then, Equation 33 can be written as, According to the chain rule in calculus, ∇ ( , , ) is computed as , where is matrix multiplication of training samples and , and ℎ is the hypothesis function which is determined by the learning model. For example, in logistic regression, ℎ is usually a sigmoid function, while in neural network, ℎ is a composite function that is known as forward propagation. Let Δ( , ) = Plain federated SGD. In plain federated SGD, is updated as the following (by combing Equation 1 and 35) in which the local gradient Δ( , ) is shared among the participants. As such, by knowing both Δ( , ) and , the adversary is able to derive indicative information about ( , ). For example, in linear regression, since Δ( , ) = ( − ), the adversary is able to obtain { , }. Secure aggregated federated SGD. In this category of traditional FL [4,30,31,34,40,47], the local gradients is protected by secure aggregation, and the global model is updated as where D ⊆ L. By combing Equation 37 and 35, we have in which the aggregated gradient, Δ( , ), is shared among the participants. As the global model is also shared, by observing the changes of the aggregated gradient during training iterations, i.e., − +1 , the adversary is still able to obtain indicative information about ( , ).
Let L ′ , L ′ respectively denote the concatenated matrix of training samples { } ∈L ′ , and labels { } ∈L ′ of any subset of participants L ′ ⊆ L. The following theorem shows that when executing secure aggregated federated SGD with the threshold , the joint view of the participants in can be simulated by (1)    Proof. SIM ( ≥ 1): is the simulator for the ℎ training iteration.
In execution, ( , ) | ∈ D is computed as (with Equation 37 and 38) As such, the joint view of in , will be identical to that in , since where N denote the noise added at the ℎ iteration. By combing Equation 35, it can be written as The information exposed among participants is Δ( , ) + N , and the additive noise N prevent one from directly deriving Δ( , ) by subtracting and +1 . However, with repeated access to the datasets during training epochs, (the parameter of privacy loss) accumulates, i.e., privacy degrades, as the effect of added noise being canceled out [2,8].

Enhanced privacy over traditional FL
Traditional FL requires all participants to update the same global model during the training process. Every participant thus sees the identical intermediate results, as the same aggregated gradients are shared. This is the root cause of most privacy threats against FL. CGD breaks the mode of single global model, by introducing random variation among the proprietary global models of the participants. The variation hides each global model from other participants, such that the privacy is enhanced in general.
In Table 2, we summarize the privacy enhancement from the perspective of adversary's observation, i.e, the exposed information to the adversary during the optimization. In traditional FL, the sharing of global model , even if the local gradients are protected, is still subject to information leakage about the original dataset. By eliminating the sharing of and letting each participant confine its own , CGD achieves boosted privacy over traditional privacypreserving FL. (1) Compared to secure aggregated FL in which indicative information about original datasets can be observed via the sharing of (Theorem 3), the variation among in CGD prevents such information from disclosure, and guarantees that only the sum of local gradients is exposed during the optimization (Theorem 2). (2) Compared to differential privacy mechanism in which privacy decays with the increasing training epochs, the variation introduced to each hides the local gradients throughout the whole training process, and thus retains privacy regardless of the number of training epochs.

CASE STUDY: CONFINED GRADIENT DESCENT FOR A N-LAYER NEURAL NETWORK
CGD can be applicable to any machine learning algorithms that use gradient descent for optimization. In this section, we apply it to a N-layer neural network to demonstrate its usability. We assume that a centralized dataset is horizontally and vertically partitioned and distributed to participants where = ,..., . Since each participant owns different confined models and proportion of the dataset, the training prediction ( , ) is also different and kept confined in its owner (shown in Figure 3). Algorithm 2 presents the detailed algorithm. The participant first randomly initializes its confined model  which is computed from its own private dataset and confined model (line 12 to 18). Then, they securely evaluate the sum of local gradients from the ℎ layer to the 2 layer (line 19 to 21). For the first layer gradient, since the size of (1) ( , ) can be different, the sum Perturbed local gradients: ( , ) + N , which equals Δ( , ) + N Not applicable with the single access to ( , ). However, with repeated access to ( , ) during epochs, the adversary is able to get a more accurate estimate of Δ( , ), which can be used to obtain { , }.
The decay of privacy with increasing number of training epochs is one of the limitations of most additive-noise based differentially private learning.
Not applicable.
In CGD, each local gradient is computed from a different and private , and only the sum of local gradients is exposed during the training. As such, (1) the indicative information exposed in the secure aggregation cannot be derived in CGD; (2) the randomness introduced in each hides the information about local gradient throughout the training, and thus privacy does not decay. of the local gradients (1) is therefore taken among the vertically partitioned participant groups = { (1, ) , ..., ( ℎ , ) } (line 22 to 24). In the descent, after choosing a learning rate (line 25), the first layer is updated by taking a descent step of (1) (line 26 to 28), and the remaining layers are updated by taking a descent step of ( ) ( ∈ [2, ]) (line 29 to 31).

EXPERIMENTS
We implement CGD and evaluate its performance of model accuracy. It is implemented using C++, and we use Eigen library [15] to handle matrix operations and ZeroMQ library [18] for distributed messaging. Our evaluation focuses on the performance of CGD in terms of validation loss and accuracy, and the influence of the factors (the parameters and ) on its performance.

Performance Evaluation
We evaluate validation loss and accuracy on two popular benchmark datasets: MNIST [28] and CIFAR-10 [26]. We compare CGD with the training on aggregated data (referred to as the centralized training), the training on each participant's local data only (referred to as the local training), and FL with differential privacy (referred to as the DP-FL). CGD is expected to outperform the local training and DP-FL, and approach that of the centralized training. The settings of all trainings are listed below.
• CGD training. The model is trained using Algorithm 1 on the dataset which is partitioned and distributed to participants, where = ( ℎ × ). We observe that the performance of all confined models are proximal, and thus we report the worst performance in this section. • Centralized training (the baseline). We take the performance of centralized training as the baseline. In this setting, the training is on the entire dataset using the batch update. • Local training. In this setting, the data and model are distributed to participants in the same way as in CGD. Each participant separately trains their local models from their local data, without sharing the gradients. • DP-FL. We take the scheme proposed by Abadi et al. [2], one of the state-of-the-art differentially private FL schemes, in our experiments. Due to the different setting of network architecture and hyper-parameters, its centralized baseline is slightly different from ours. Therefore, we report the margin between its performance and the performance of its baseline. (1) ( , ) ← ( , ) (1) ( , ) 7: (1) ( , ) ← (1) ( (1) ( , ) ) 8: ...... for all participants ( , ) do in parallel • the standard weight initialization scheme based on the Gaussian distribution of mean 0 and variance 1. Figure 4 summarizes the performance with varying number of participants. In general, CGD stays close to the centralized baseline in both validation loss and accuracy, and as expected, significantly outperforms the local training. In the worst case of = (1000×112) when CGD contains the greatest number of participants, i.e., each participant owns a small proportion consisting of 60 samples with 7 features, CGD still achieves 0.209 and 93.74% in validation loss and accuracy respectively, close to 0.081 and 97.54% of the centralized baseline. In contrast, the performance of local training declines to 2.348 and 11.64%. Table 3 lists the detailed performance comparison with both centralized and local trainings, and Table 4 lists the comparison with DP-FL. More results are given with varying number of participants in Appendix C ( Figure 9 and Table 8).

CIFAR-10.
Our second set of experiments are conducted on the CIFAR-10 dataset. It consists of 60000 32 × 32 colour images in 10 classes (e.g., airplane, bird, and cat), with 6000 images per class. The images are divided into 50000 training images and 10000 test images. We use the ResNet-56 architecture proposed by He et al. [17]. It takes as input images of size 32 × 32, with the perpixel mean subtracted. Its first layer is 3 × 3 convolutions, and then   [17]. For each training image, we generate a new distorted image by randomly flipping the image horizontally with probability 0.5, and adding 4 amounts of padding to all sides before cropping the image to the size of 32 × 32. Since our focus is on evaluating the proposed CGD, rather than enhancing the state-of-the-art analysis on CIFAR-10, we utilize the transferability 2 of convolutional layers to save the computational cost of computing per-example gradients. We follow the experiment setting of Abadi et al. [2], which treats the CIFAR-10 as the private dataset and CIFAR-100 as a public dataset. CIFAR-100 has the same image types as CIFAR-10, and it has 100 classes containing 600 images each. We use CIFAR-100 to train a network with the aforementioned architecture, and freeze the parameters of the convolutional layers and retrain only the last FC layer on CIFAR-10.
Training on the entire dataset with batch update reaches the validation accuracy of 75.75%, which is taken as our centralized training baseline 3 . For the CGD training, each participant feeds the pre-trained convolutional layers with their private data partitions, generates the input features to the FC layer, and randomly initializes the confined model parameters of the FC layer. They then take the input to the FC layer as the private local training data, and train a model using Algorithm 1 with = 0.01 and = 6000. 2 Transfer learning allows the analyst to take a model trained on one dataset and transfer it to another without retraining [41]. 3 We note that by making the network deeper or using other advanced techniques, better accuracy can be obtained, with the state-of-the-art being about 99.37% [23].    Figure 5 and Table 5 summarize our experimental results against centralized training and local training. The results on the validation loss and accuracy are generally in line with that on the MNIST dataset. In the worst case of = (1000 × 32), CGD achieves 0.724 and 74.29% in validation loss and accuracy respectively, which are relatively near to the centralized baseline (0.675 and 75.72%). Table 6 summarize the results against DP-FL. In line with the results on MNIST, the accuracy difference to the centralized baseline in CGD is smaller than that in DP-FL.

Influencing Factors of CGD
In this section, we study the influence of initialization and the parameter on CGD performance. This study is conducted with the MNIST dataset.

Role of the initialization.
According to Theorem 1, the gap between CGD solution and the centralized model is bounded by = ∥ E ∈ ( 1 − 1 )∥, in which each 1 is decided by in Equation  11. Therefore, we investigate how the initialization setting affects CGD's performance.
To this end, we conduct two experiments in which we mutate parameter while keeping other settings unchanging (i.e., = 0 and = 2000). In the first experiment, we let all participants use the same in {0.1, 0.06, 0.01, 0.001}, as they are around 1 √ 60000 (recall that 60000 is the sample size of MNIST). In the other experiment, we let each participant randomly select its own based on the uniform distribution within the range from 0.001 to 0.1.
The results of our first experiment are shown in Figure 6 and the first five columns in Table 7. In general, as decreases, CGD achieves better performance. This confirms our expectation: decreasing would reduce the value of ∥ E ∈ ℎ ( 1 − 1 )∥, such that the confined models are closer to the centralized optimum. We have not observed significant difference from = 0.1, 0.06 and 0.01. In the case of = (1000 × 112), the validation accuracy and loss achieve 97.08% and 0.096 with = 0.01 which outperform 93.74% and 0.209 in the setting of = 0.1, but both are close to the centralized model whose validation accuracy and loss are 97.61% and 0.078 respectively. However, cannot be set too small, in order to maintain numerical stability in neural network [13]. For example, when we lower to 0.001, the performance starts decreasing.
The results of our second experiment are shown in Figure 7 and the last column in Table 7. When the participants uniformly randomize their s from 0.001 to 0.1, the performance of CGD is still comparable with the centralized mode. For example, with = (1000×112) participants, CGD achieves 95.82% and 0.136 in the validation accuracy and loss. This suggests that CGD keeps robust when the s of its participants differ by two orders of magnitude.   conduct an experiment to investigate this relation. In this experiment, we tune while keeping fixed as 0.1. For the stepsize ( ), we run CGD with a fixed till it (approximately) reaches a preferred point, and then continue our experiment with 1 = . This is to keep CGD practical, as each time the stepsize is diminished, more iterations are required. Therefore, the first 6000 iterations is run with = 0.01, and then diminish in each of the following 8000 iterations. gives a slightly faster speed. CGD reaches the validation accuracy of 95.9% at 11401 ℎ iteration, 2340 iterations (16.7% less) faster than training with < 0.01. For the case of = 1000 × 112), > 0.05 gives a faster speed. CGD reaches the validation accuracy 95.68% at 11821 ℎ iteration, 2140 iterations (15.28% less) faster than training with ≤ 0.05. Even though such slight difference is observed, our experiment suggests that the effect of tuning is relatively limited. The gain of the validation accuracy is only within 0.2% in the cost of around 2000 iterations.

CONCLUSION
We have presented CGD, a novel optimization algorithm for learning confined models to enhance privacy for federated learning. Privacy preservation is achieved against an honest-but-curious adversary even though the majority ( − 2 out of ) of participants are corrupted. We formally proved the convergence of CGD optimization. In our experiments, we achieved validation accuracy of 97.01% with (1000 × 112) participants on the MNIST dataset, and 74.4% with (1000 × 32) participants on the CIFAR-10 dataset. Both are comparable to the performance of centralized training.
A number of future work directions are of interest. In particular, we see new research opportunities in applying our techniques to asynchronous FL, allowing different participants to be at different iterations of model updates up to a bounded delay. We are also considering other types of deep networks, for example, unsupervised neural networks such as autoencoders.