{'fraction_non_alphanumeric': 0.04223980049097923, 'fraction_numerical': 0.008845419475509488, 'mean_word_length': 4.502572898799314, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Privacy-Preserving Federated Learning via System Immersion and Random Matrix Encryption

Federated learning (FL) has emerged as a privacy solution for collaborative distributed learning where clients train AI models directly on their devices instead of sharing their data with a centralized (potentially adversarial) server. Although FL preserves local data privacy to some extent, it has been shown that information about clients' data can still be inferred from model updates. In recent years, various privacy-preserving schemes have been developed to address this privacy leakage. However, they often provide privacy at the expense of model performance or system efficiency, and balancing these tradeoffs is a crucial challenge when implementing FL schemes. In this manuscript, we propose a Privacy-Preserving Federated Learning (PPFL) framework built on the synergy of matrix encryption and system immersion tools from control theory. The idea is to immerse the learning algorithm, a Stochastic Gradient Decent (SGD), into a higher-dimensional system (the so-called target system) and design the dynamics of the target system so that: the trajectories of the original SGD are immersed/embedded in its trajectories, and it learns on encrypted data (here we use random matrix encryption). Matrix encryption is reformulated at the server as a random change of coordinates that maps original parameters to a higher-dimensional parameter space and enforces that the target SGD converges to an encrypted version of the original SGD optimal solution. The server decrypts the aggregated model using the left inverse of the immersion map. We show that our algorithm provides the same level of accuracy and convergence rate as the standard FL with a negligible computation cost while revealing no information about the clients' data.


I. INTRODUCTION
Machine learning (ML) has been successfully used in a wide variety of applications for multiple fields, and industries [1]. In traditional machine learning, training data is centrally held by the server executing the learning algorithm. Distributed learning systems expand this paradigm by branching the learning process to decentralized nodes that only use locally available data. However, when multiple participants are involved, the exchange of local data poses a significant privacy risk.
Federated learning (FL) [2]- [4] has been recently introduced as a decentralized learning framework that can scale to thousands of participants and preserves data privacy. Its core idea is to train machine learning models on separate datasets distributed across several devices or parties. FL schemes train local models on local clients' datasets, and then clients exchange their parameters (e.g., model weights or gradients) with a central server to aggregate a global model. Since clients do not share their training data, FL is suitable for Haleh  sensitive data sharing use cases. This includes health care, the Internet of Things, and other scenarios with high privacy concerns [5]- [7]. Although FL can provide some level of privacy for clients' raw data, private information can still be inferred from model updates throughout the training process. It has been shown that local models can be traced back to their sources [8], [9]. Common attacks to FL are model inversion attacks and gradient inference attacks as identified in [10], [11].
In recent years, various privacy-preserving schemes have been implemented to address the privacy leakage in FL [12], [13]. Most of them rely on cryptography-based techniques such as Secure Multiparty Computation (SMC) [14]- [18] and Homomorphic Encryption (HE) [19], and perturbation-based techniques such as Differential Privacy (DP) [2], [20]- [22]. Bonawitz et al. [14] uses an SMC-based secure aggregation protocol to protect individual model updates by aggregating local clients' updates at a trusted party and sharing the aggregated model with the untrusted server. Although cryptographic algorithms have the advantage of retaining the original accuracy of FL, the resulting solution comes with high additional communication costs. Differential privacy [23] is also commonly used to enforce local and global privacy for machine and federated learning. DP provides strong information-theoretic guarantees, is algorithmically simple, and has a small system overhead. However, there is an inherent tradeoff between DP and the performance of federated learning, both in terms of model accuracy and convergence rate, as introducing noise increases privacy but may compromise accuracy dramatically [22]. Because standard cryptographic techniques have a high computation and communication cost, and differential privacy reduces FL performance, in recent years, hybrid privacy-preserving methods that combine cryptographic tools and DP schemes have been proposed to hold acceptable tradeoffs between data privacy and FL performance [24]- [26].
Although current solutions improve privacy of FL, they often do this at the expense of model performance and system efficiency. Balancing these tradeoffs is a key challenge when implementing private FL systems. It follows that novel Privacy-Preserving FL schemes must be designed to provide strict privacy guarantees, on the one hand, and, on the other hand, have a fair computational cost and use communications efficiently without compromising accuracy excessively.
In this paper, we propose a Privacy-Preserving Federated Learning (PPFL) framework built on the synergy of matrix encryption and systems immersion tools [27] from control theory. The main idea is to treat the learning algorithm used in standard FL -a Stochastic Gradient Decent (SGD) -as a dynamical system that we seek to immerse into a higher-dimensional system (the so-called target system). The dynamics of the target system must be design so that: 1) trajectories of the standard SGD are immersed/embedded in its trajectories; and 2) it learns on encrypted data. We use random matrix encryption, which is reformulated at the server as a random change of coordinates that maps original parameters to a higher-dimensional parameter space and enforces that the target system converges to an encrypted version of the standard SGD optimal solution. The server decrypts the aggregated model using the left inverse of the immersion map.
The proposed framework provides the same accuracy and convergence rate as the standard federated learning (i.e., when no encryption or distortion is induced to protect against data inference), reveals no information about the clients' data, is computationally efficient, and does not degrade the learning performance. To the best of our knowledge, this is the first piece of work that provides a high level of privacy for FL without affecting its performance and excessively increasing communication costs. The main contributions of the paper are summarized as follows: i) using systems immersion tools and random matrix encryption, we develop a privacy-preserving FL scheme that guarantees privacy for local and global models; ii) the proposed scheme is shown to be unconditionally secure [28]; and iii) we validate the performance of the scheme through extensive computer simulations based on a real-world large-scale dataset for a FL network with one server, ten clients, and 199,210 parameters.

A. Standard Federated Learning
To develop the architecture of our scheme, we build upon the standard FL algorithm. In standard FL, multiple distributed devices (the clients) and a centralized server aim to train a global AI model without exchanging local data available at the clients. Clients share local model parameters with the server obtained by training a model on their devices using local data. The server aggregates all local models to create a global model that is shared back with clients. Clients use the new global parameters as initial conditions to retrain local models. This procedure is repeated until convergence is achieved [3].
Consider a standard FL system with one server and N c clients. Let D i denotes the local database held by the i-th client, i ∈ {1, 2, . . . , N c } =: N . At each iteration t ∈ N, the server broadcasts the latest global model, w t ∈ R n (a vector of parameters), to all clients (starting from a random initial value w 0 ). Iteration times t ∈ N are referred to as global iterations. Then, clients determine local AI models, w t i ∈ R n , at their devices by minimizing a given loss function l(w t i , D i ) over local data D i and the latest update on w t . The latter can be formulated as follows: Clients send their local optimal w t i back to the server and the server updates the global model as follows: where |D i | is the size of the i-th dataset, |D| := i |D i |, and w t is the global aggregated model. The process is repeated until convergence to the global optimum: In general, standard FL clients use SGD as the optimization algorithm to minimize their local loss function (1). Each client calculates the stochastic gradient of the local model using a mini-batch X i ⊆ D i randomly sampled from D i and updates its local model following K iterations of the SGD: where w i,k ∈ R n denotes the k-th local iteration of the SGD algorithm at client i, K is the total number of local iterations, and η > 0 is the learning rate of the algorithm. Therefore, at every round, each client initializes the local SGD using the latest received w t and updates w t+1 i via K iterations of the SGD, i.e., w t+1 i = w i,K . Optimal local parameters, w t+1 i , are sent to the server for aggregation and the process repeats until convergence. After a sufficient number of global iterations between clients and the server (in the global counter t) and local updates (in the local counter k), the standard FL scheme converges to the optimal global model (3) (see [2] for details).

B. Privacy Requirements
As discussed in Section 1, information about participants' private data can still be inferred from the model updates throughout the training process [8]- [11]. In addition, privacy leakage can also occur in the broadcasting step by analyzing the global model parameters [8]. In FL, there are two types of actors that can infer private information: internal actors (participating clients, the central server, and third parties) and external actors (model consumers and eavesdroppers) [13]. We assume all the internal actors are untrusted (honestbut-curious), which means that they will faithfully follow the designed FL protocol but attempt to infer private information. External actors are also untrusted; they aim to eavesdrop the communication between internal actors to infer information. We mainly concentrate on privacy of intermediate local and global models. Privacy of the final model, which will be shared with consumers, can be provided by perturbationbased methods.

C. Immersion Map and Target System
The goal of our privacy-preserving FL scheme is to make inference of the clients' datasets, from the local updates w t i and global models w t , as hard as possible without distorting the accuracy and convergence of the learning algorithm (SGD). We aim to design an encryption system through matrix multiplication and system immersion tools from control theory.
System immersion refers to embedding the trajectories of a dynamical system into the trajectories of a different higher-dimensional system (the so-called target system) [27]. That is, there is a bijection between trajectories of both systems (here referred to as the immersion map), and thus having a trajectory of the target system uniquely determines a trajectory of the original system via the immersion map.
In our setting, the idea is to immerse the dynamics of the standard SGD in (4) into a target dynamical system -referred hereafter as the target SGD. The dynamics of the target SGD must be designed so that: 1) trajectories of the standard SGD (4) are immersed in its trajectories via a known immersion map; and 2) the target SGD learns on encrypted data (here, we use random matrix encryption). Once we have designed the target system and the immersion map, we can use them to encrypt model updates and learn on encrypted data.
Consider the original vector of parameters w i,k ∈ R n in (4) at time k, and denote the vector generated by the target SGD asw i,k ∈ R m with m > n. Consider the following general target SGD: with function f : R m → R m to be designed and initial conditionw t (the latest encrypted global update from the server). We say that the standard SGD in (4) is immersed in the target SGD (5), if there exists a left invertible function π : R n → R m satisfying: for all k ∈ K := {1, 2, · · · , K − 1}. We refer to this function π(·) as the immersion map. Because (6) must be satisfied for all k ∈ K, we need to enforce (by designing π(·) and f (·)) that: (a) the initial condition of (5),w i,0 =w t , satisfiesw i,0 = π(w i,0 ) = π(w t ), where w t is the latest unencrypted global update shared by the server (i.e., the update that would be produced by the standard FL scheme in (2)); and (b) the dynamics of both algorithms match under the immersion map, i.e.,w i,k+1 = π (w i,k+1 ). Condition (a) implies that what the server sends to clients is π(w t ).
That is, the immersion map π(·) is the encryption scheme for all clients. It follows that the first constraint on π(·) is that it must comply with the privacy requirements. Next, using the expressions for w i,k+1 andw i,k+1 , in (4) and (5), respectively, and (6), condition (b),w i,k+1 = π (w i,k+1 ), can be written as follows: which is a time-varying nonlinear equation on w i,k . We refer to this equation as the immersion condition.

D. Secure Aggregation and Problem Statement
Once a complete cycle has been finished by the target SGD, so k = K − 1, all clients send their last iteration, w i,K , to a third party for data aggregation. We refer to this party simply as the aggregator. The role of the aggregator is to interface between clients and the server and thus prevent the server from accessing exact local models. The aggregator takes the updated encrypted local models from all clients,w i,K , aggregates them and sends the aggregated model to the server. Consequently, the server cannot access any local model and only has access to the aggregated results. Moreover, since the aggregator has access to the encrypted local updatesw i,K only, it is not required to be trusted. The aggregated model at the t-th iteration is given bỹ where the right-hand side part of (8) follows from the immersion condition (b). The server receivesw t+1 in (8) and aims to retrieve w t+1 = Nc i=1 (|D i |/|D|)w i,K -the aggregated result of the standard SGD in (4). The latter imposes an extra condition on the immersion map, π(·), since to retrieve w t+1 fromw t+1 : (c) there must exists a function π L : R m → R n satisfying the following left-invertibility condition: If such π L (·) and π(·) exist, the server can retrieve the original aggregated parameters Nc i=1 (|D i |/|D|)w i,K by passing the encrypted aggregated results through function π L (·). We have now all the machinery required to state the problem we seek to solve.

III. SOLUTION TO PROBLEM 1 A. Solution
In this section, we introduce the proposed privacypreserving FL algorithm. We construct this algorithm by deriving particular solutions to all functions in Problem 1. Since the problem formulation and solution are based on systems immersion theory, we refer to our algorithm as System Immersion based Federated Learning (SIFL). We start with function f (·) and condition (b), i.e., the immersion condition (7). A natural candidate for f (w i,k ) of the target SGD dynamics (5) is a gradient-dependent function. Let f (w i,k ) have the following form where η > 0 and X i are the same learning rate and data realization as in the standard SGD (4), and ∇l(w i,k , X i ) is a gradient function to be designed. With this f (w i,k ), the immersion condition (7) takes the form Let the immersion map π(·) be an affine function (as privacy mechanism in [29]) for some matrix G ∈ R m×n and b t ∈ R m -with slight abuse of notation, we let b t change with the global counter t independently of the argument s. Then, the immersion condition reduces to Finally, let the modified gradient function be of the form ∇l(w i,k , X i ) = G∇l(Mw i,k , X i ), for some M ∈ R n×m to be designed. Hence, the immersion condition takes the form To satisfy (14) (5) is given as with ∇l(·) and η the gradient and learning rate of the standard SGD in (4), and (G, M ) as defined above.
Note that substitution of the immersion map (12) into (6) leads the solution of the target system (15) to be an affine function of trajectories of the original SGD as follows: By plugging in the designed immersion map (12) in the aggregated encrypted model (8) yields wherew t+1 and w t+1 denote the aggregated encrypted and unencrypted updated models, respectively. We have designed the function f (·) and the immersion map π(·) to satisfy the immersion condition (7). Next, we seek for a function π L (·) satisfying (9) (condition (c) in Problem 1). Given (17), condition (9) can be written as which trivially leads to π L (s) := M s, since M G = I and b t ∈ ker[M ]. Finally, condition (a) in Problem 1 is automatically satisfied for the designed functions as the initial condition of (5),w i,0 =w t , is what the server sends to clients at global iteration t, and the server encrypts the aggregated result with the immersion map, i.e, w i,0 = π(w t ) = π(w i,0 ). At every global iteration t, the server designs a vector b t satisfying M b t = 0 and uses it to construct the immersion map π(s) = Gs + b t (the encryption scheme). To increase security, the server uses this b t to add randomness to the mapping by exploiting the nonempty kernel of M . We let b t be of the form b t = N R t for some matrix N ∈ R m×(m−n) expanding the kernel of M (i.e., M N = 0) and some random vector R t ∈ R (m−n)×1 . Hence, we have M b t = 0 in all iterations and b t = N R t changes randomly with t.
Proof : The proof follows from the analysis provided in the solution section above, Section III.

B. SIFL Algorithm
The flowchart of the SIFL is shown in Figure 1. The summary of the algorithm is as follows: • FL initialization and encryption by the server. The server initializes the global model w 0 and encrypts it asw 0 = Gw 0 + b 0 . Then, it immerses the original SGD into the target SGD as (15) and broadcastsw 0 , target SGD, and other hyperparameters to clients. • Local model training and update by clients. The clients receive the current encrypted global modelw t sent by the server and update their individual local model parameters using their local datasets D i and the target SGD system (15). Then, they send their updated model to the aggregator for aggregation. • Global model aggregation. The aggregator takes the average of local encrypted models and sends the aggregated model (17) to the server. • Global model encryption and broadcasting by the server. The server decrypts the aggregated global model using function π L (·) in (19). Then, it encrypts the new global model using the immersion map π(·) (12) and broadcasts it to all clients for the next round. The pseudo-code of SIFL is shown in Algorithm 1.

IV. SECURITY ANALYSIS
In SIFL, since local and global models are encrypted in all rounds, even if adversaries wish to gain data by attacking the server or a communication channel, they can only get the encrypted models. Even in the case of internal adversaries, they require to break the cryptosystem to access the data. Hence, neither internal adversaries nor external ones (model consumers and eavesdroppers) have access to the original local models. Therefore, they need to break the cryptosystem to infer information about clients' data. Since the encryption keys are random and changed at each iteration, even if adversaries are lucky enough to break some rounds of training results, they cannot access the actual data due to the inclusion of randomness.
In [30], Shannon proves that a necessary condition for an encryption method to be unconditionally secure is that the uncertainty of the secret key is larger than or equal to the uncertainty of the plaintext [28]. He proposes a one-time pad encryption scheme in which the key is randomly selected and never used again. The one-time pad gives unbounded entropy of the key space, i.e., infinite key space, which provides unconditional security [31], [32] (the unconditional secrecy would be lost when the key is not random or if it is reused). Since in SIFL the encryption keys are random and only used once, it provides infinite key space, and thus, it can be considered unconditionally secure.

V. SIMULATION RESULTS
In this section, we implement our proposed scheme for performance evaluation using multi-layer perception (MLP) Algorithm 1: System Immersion based FL algorithm Input : Set of clients and their databases D i , number of FL iterations T , learning rate η, number of local SGD iterations K, privacy matrix M n×m , its right inverse G, and matrix N in its null space.
Handshaking phase: The server sends target SGD (15), the encrypted initialized global modelw 0 , and other hyperparameters to clients for model update.
Server Execution: The aggregator aggregates local models The aggregator sendsw t+1 to the server w t+1 = π L (w t+1 ) = Mw t+1 ; // Server decryption Server creates random vector R t . w t+1 = π(w t+1 ) = Gw t+1 + N R t+1 ; // Server encryption Server sends encryptedw t+1 to clients. end ClientUpdate(w t , D i ): ; // Run on client i X i ← ( split D i into batches ) Initialize:w i,0 ←w t for each local epoch k from 1 to (K − 1) do for batch x i ∈ X i dõ w i,k+1 ←w i,k − ηG∇l(Mw i,k , x i ) end end returnw i,K = Gw i,K + N R t to the aggregator.
[33] and a real-world federated dataset. Since FL is mostly suited for parameterized learning, such as all types of neural networks, MLP is employed for the learning method. We test our algorithm on the standard MNIST dataset for handwritten digit recognition, containing 60000 training and 10000 testing instances of 28× 28 size gray-level images [34]. Our model uses an MLP network with two hidden layers containing 200 hidden units. This feed-forward neural network uses ReLU units and softmax of 10 classes (corresponding to the ten digits) and with ten clients. For the network optimizer, we consider cross-entropy loss and SGD optimizer with the learning rate 0.01 and the local epoch K = 2. To assess model quality, we used the pre-defined MNIST test set. Our implementation uses Keras with a Tensorflow backend.
We design matrix M randomly as the decryption key of SIFL with dimension n × (n + 1), where n = 199, 210 is the total number of model parameters in the MLP network. Next, we calculate its inverse and null space, G and N . Vector R t is randomly chosen in every global iteration. The comparison of training accuracy and loss results of SIFL framework and the standard FL without privacy are shown in Figure 2 and 3, respectively. As can be seen, the accuracy and the evolution of the loss function with the SIFL setting are almost the same as the accuracy and loss with no privacy setting, which shows that SIFL can integrate a cryptographic method in the FL system without sacrificing model accuracy and convergence rate. Therefore, there is no need to hold a trade-off between privacy and the performance of FL.
In the solution section, Section III, we show that in the SIFL framework, the encrypted local model of each client is an affine function of the original local model. This is shown in Figure 4 based on the second norm of local model parameters as ||w t+1 i || = ||Gw t+1 i +N R t || for the first client and in all iterations. Also, in Figure 5, the norm of the relative error betweenw t+1 i and (Gw t+1 i +N R t ) is depicted. The error between updated weights and the expected updated weights is caused by the huge dimension of the parameter vector in this case and the matrix inversion in the calculation of the encryption key, which leads to some calculation errors.
Finally, in Figures 6 and 7, we investigate the effect of encryption and decryption operations in SIFL on the training time of FL. As can be seen, the increased training time compared to the training time of the original FL is negligible.

VI. CONCLUSIONS
In this paper, we proposed a System Immersion Federated Learning, SIFL, as a privacy-preserving FL framework built on the synergy of matrix encryption and system immersion tools from control theory to provide unconditional secrecy for the clients' data in federated learning. As a privacy mechanism, we developed an immersion for the learning algorithm (SGD), and we designed the dynamics of a target system so that trajectories of the original SGD are immersed in its trajectories, and it learns on encrypted data based on the random matrix encryption. Matrix encryption was reformulated at the server as a random change of coordinates that maps original parameters to a higher-dimensional parameter space and enforces that the target SGD converges to an encrypted version of the original SGD optimal solution. SIFL provides the same accuracy and convergence rate as the standard FL, reveals no information about clients' data, and is computationally efficient. It provides a high level of privacy without degrading the performance of FL. The simulation results of SIFL are presented to illustrate the performance of our tool. These results demonstrate that SIFL provides the same accuracy and convergence rate as the standard FL with a negligible computation cost.