{'fraction_non_alphanumeric': 0.03669093055282314, 'fraction_numerical': 0.007820184930159355, 'mean_word_length': 4.638657792567942, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Hierarchical Over-the-Air FedGradNorm

Multi-task learning (MTL) is a learning paradigm to learn multiple related tasks simultaneously with a single shared network where each task has a distinct personalized header network for fine-tuning. MTL can be integrated into a federated learning (FL) setting if tasks are distributed across clients and clients have a single shared network, leading to personalized federated learning (PFL). To cope with statistical heterogeneity in the federated setting across clients which can significantly degrade the learning performance, we use a distributed dynamic weighting approach. To perform the communication between the remote parameter server (PS) and the clients efficiently over the noisy channel in a power and bandwidth-limited regime, we utilize over-the-air (OTA) aggregation and hierarchical federated learning (HFL). Thus, we propose hierarchical over-the-air (HOTA) PFL with a dynamic weighting strategy which we call HOTA-FedGradNorm. Our algorithm considers the channel conditions during the dynamic weight selection process. We conduct experiments on a wireless communication system dataset (RadComDynamic). The experimental results demonstrate that the training speed with HOTA-FedGradNorm is faster compared to the algorithms with a naive static equal weighting strategy. In addition, HOTA-FedGradNorm provides robustness against the negative channel effects by compensating for the channel conditions during the dynamic weight selection process.

Abstract-Multi-task learning (MTL) is a learning paradigm to learn multiple related tasks simultaneously with a single shared network where each task has a distinct personalized header network for fine-tuning. MTL can be integrated into a federated learning (FL) setting if tasks are distributed across clients and clients have a single shared network, leading to personalized federated learning (PFL). To cope with statistical heterogeneity in the federated setting across clients which can significantly degrade the learning performance, we use a distributed dynamic weighting approach. To perform the communication between the remote parameter server (PS) and the clients efficiently over the noisy channel in a power and bandwidth-limited regime, we utilize over-the-air (OTA) aggregation and hierarchical federated learning (HFL). Thus, we propose hierarchical over-theair (HOTA) PFL with a dynamic weighting strategy which we call HOTA-FedGradNorm. Our algorithm considers the channel conditions during the dynamic weight selection process. We conduct experiments on a wireless communication system dataset (RadComDynamic). The experimental results demonstrate that the training speed with HOTA-FedGradNorm is faster compared to the algorithms with a naive static equal weighting strategy. In addition, HOTA-FedGradNorm provides robustness against the negative channel effects by compensating for the channel conditions during the dynamic weight selection process.

I. INTRODUCTION
Federated learning (FL) is a distributed learning framework where many clients train a shared model under the orchestration of a centralized server while keeping the training data decentralized and private. In addition to FL, multi-task learning (MTL) is a learning paradigm that aims to learn multiple related tasks simultaneously by learning a shared representation for all tasks [1], [2]. Integration of MTL in the FL framework leads to personalized federated learning (PFL), in which clients have different tasks, and the clients train the common shared network under the orchestration of the centralized server while each client further trains a small client-specific network for its own specific task, referred to as personalization. The clients in an FL setting can have non-iid training data. Further, task complexities across clients can be different. Both the non-iid nature of the training data across clients and different task complexities cause statistical heterogeneity, leading to overall system performance degradation. Distributed dynamic weighting strategy, called FedGradNorm is previously proposed to handle statistical heterogeneity in PFL by balancing the learning speeds across different tasks [3]. FedGradNorm utilizes the clients' gradients on the serverside before aggregation to be able to do this.
In [3], the characteristics of the communication channel are not considered, although it is known that the clients can be distributed by geographic location in FL [4], [5]. In certain applications, the parameter server (PS) can be far away from the clients, making the communication between the PS and the clients noisy and vulnerable to channel effects. The communication over a shared wireless channel needs to be done over a power-and bandwidth-limited setting, bringing communication cost concerns into FL. To address these issues, over-the-air (OTA) aggregation [6], [7] has become a prominent approach as an efficient strategy for supporting clients on the same bandwidth. In addition, hierarchical federated learning (HFL) framework is introduced by creating clusters of clients around intermediate servers (IS) which communicate with the PS instead of direct communication of clients with the PS. The studies about HFL focus on different aspects such as latency and power analysis [8], [9], resource allocation [10], [11]. These works consider the HFL with error-free channels. [12] investigates HFL with OTA by taking into account the practical wireless channel models. [13] investigates PFL with OTA across clusters in which the clients inside the cluster are assumed to have the same data distribution. In contrast, statistical heterogeneity may exist since clients may be responsible for different tasks, especially when clients are grouped based on their geo-locations. In our work, we introduce hierarchical over-the-air personalized federated learning with FedGradNorm, which we call HOTA-FedGradNorm, by adopting FedGradNorm into HFL by utilizing OTA aggregation for the bandwidth-limited wireless fading multiple access channel (MAC) when the transmission power constraints are imposed on clients. FedGradNorm [3] is modified by taking into account the channel conditions, thereby imposing implicit constraints on the dynamic weighting coefficients. The experiments are conducted to demonstrate how HOTA-FedGradNorm deals with the effect of the channel condition via the dynamic weight selection process. We conduct experiments on a wireless communication system dataset called RadComDynamic. Our experimental results demonstrate that the training speed with HOTA-FedGradNorm is faster compared to algorithms with a naive static equal weighting strategy. In addition, we observe that HOTA-FedGradNorm provides robustness against the negative channel effects by compensating for the channel conditions during the dynamic weight selection process. Finally, extended results for this work and [3] are presented in [14] with theoretical analysis.

II. SYSTEM MODEL AND PROBLEM FORMULATION
The generic form of HFL problem with C clusters each containing an intermediate server (IS) and N clients as depicted in Fig. 1 is, where p (l,i) is the loss weight for client i in cluster l such , and F (l,i) (·) is the local loss function for client i in cluster l.
We consider a PFL setting of N clients within each cluster, in which client i of cluster l has its own local dataset where n l,i is the size of the local dataset. Within cluster l, T l,i denotes the task of client i, i ∈ [N ], and l ∈ [C]. T l,i is assigned from the task set T = {T 1 , T 2 , . . . , T N } such that T l,i = T l,i ′ , for i = i ′ and any l ∈ [C]. Real-life scenarios might involve the same or very similar tasks for clients in a cluster. We assume that tasks are different due to the lack of prior information about them.
Additionally, clients in a cluster are connected to their corresponding intermediate server via local area network (LAN), which are assumed to be error-free and to have high data transmission rate. The ISs are connected to the PS via bandwidth-limited fading MAC for sending the local gradient aggregations in clusters to the PS. The broadcast from the PS to the ISs is considered to be error-free.
The system model in Fig. 1 is composed of a global representation network q ω : R d → R d ′ , which is a function parameterized by ω ∈ W, that maps data points into a lower space of size d ′ . All clients in each cluster share the same global representation network, which is synchronized through global aggregation. The client-specific head q h (l,i) : R d ′ → Y is a function parameterized by h (l,i) ∈ H for all clients i ∈ [N ] of every cluster l ∈ [C], mapping a low dimensional representation space to the label space Y. The local model for client i of cluster l is the composition of the client's global representation model q ω and personalized model q h (l,i) , shown as q l,i (·) = (q h (l,i) • q ω )(·). In addition, the local loss for the ith client of cluster l is shown as Using alternating minimization, the PS and the clients learn the global representation ω together, while only client i learns the the client-specific head h (l,i) in cluster l, i ∈ [N ] and l ∈ [C]. Specifically, client i of cluster l performs τ h local updates to optimize h (l,i) when global network parameters at client i of cluster l, i.e., ω (l,i) are frozen. Then, τ ω local updates are performed to optimize ω (l,i) while the parameters corresponding to the client-specific head are frozen. Thereafter, the lth IS aggregates {ω (l,i) } N i=1 which are sent via LAN, for any l ∈ [C]. The ISs send cluster aggregations to the PS to perform the global aggregation over the wireless fading MAC. The global aggregation is performed over the air by the additive nature of wireless MAC. Considering the overall system model scheme, the optimization problem is III. ALGORITHM DESCRIPTION The algorithm consists of two stages. In the first stage, a distributed dynamic weighting strategy is applied to balance learning speed of tasks across clients within each cluster. Dynamic weighting strategy is combined with power allocation scheme to satisfy the total average transmit power constraint and to be robust against the channel effects of bandwidthlimited fading MAC between the ISs and the PS. In the second stage of the algorithm, the global aggregation is performed over the air by utilizing the additive nature of wireless MAC. Then, the aggregated gradient is estimated on the PS to update the global representation network.

A. Local Aggregation via Dynamic Weighting
During the first stage of the algorithm, each client within a cluster sends its gradient for the global model q ω to its corresponding IS via LAN, where the channels between each client and the corresponding IS are assumed to be errorfree inside a cluster. Then, the corresponding IS performs a modified version of FedGradNorm [3] as a distributed dynamic weighting strategy based on the client's gradients by taking taking the power allocation scheme into account to balance the learning speed across different tasks.
Specifically, the IS of cluster l computes the loss weight p (l,i) k for each client i ∈ [N ] in cluster l via FedGradNorm algorithm to eventually obtain the local weighted aggregation is the local gradient update of client i in cluster l for iteration k. Additionally, the power allocation vector β (l,i) k constructed by the IS of cluster l for each client i in the cluster is designed as: where β k ∈ R |ω| , which represents the effect of the wireless fading channel between the IS of cluster l and the PS. H (l) k (j) is assumed to be independent and identically distributed (iid) according to N (0, σ 2 l ). The threshold H th k is set to satisfy the average transmit power constraint given as follows, • represents the element-wise multiplication. The expectation is taken over the randomness of the channel gains.
From the power allocation scheme in (3), each cluster transmits only the scaled entries of its weighted gradient for which the corresponding channel conditions are sufficiently good. It provides implicit gradient sparsification by saving transmission power. Therefore, we can also modify FedGrad-Norm algorithm by sparsifying the auxiliary loss function F grad in [3] before minimizing it to have {p . Sparsified F grad for the cluster l is the following, where M (l) ∈ {0, 1} |ω| is a mask matrix designed for the sparsification of cluster l as follows: Here,ω (l,i) k is the last layer of the shared network at client i of cluster l at iteration k.Ḡ (l) ω (k) is the average sparsified gradient norm across all clients (tasks) in cluster l at iteration k. r is the relative inverse training rate of task i in cluster l at iteration k, and γ represents the strength of the restoring force which pulls tasks back to a common training rate, which can also be thought of as a metric of task asymmetry across different tasks.
Gradient sparsification used during the calculation of F grad acts as an implicit constraint on F grad minimization problem by considering the channel conditions. Consequently, it ensures that the learning speed of tasks is invariant to the dynamic channel conditions with an appropriate selection process of loss weights. In other words, the implicit constraint of the channel condition preserves the fairness of the learning speed among the clients, as shown in the experimental results.

B. Over-the-Air Aggregation
The second stage of the algorithm involves the process of global aggregation over the wireless fading MAC. The PS obtains a noisy estimate of the aggregated gradient over the wireless fading channel while updating the model parameters.
Due to the additive nature of the wireless MAC, the summation of the signals transmitted by clusters arrives at the PS. The jth entry of the received signal at iteration k, y k ∈ R |ω| is where z k (j) is the jth entry of the Gaussian noise vector z k and is iid according to N (0, 1). M k (j) = {c ∈ [C] : |H (l) k (j)| 2 > H th k } represents the set of clusters contributing to the jth entry of the received signal at the kth iteration. M k (j) is known by the PS, for j ∈ [|ω|] since the PS has the perfect channel state information (CSI).
By considering (3) and the definition of x (l) k in terms of the power allocation vector, we have . The noisy aggregated gradient estimate iŝ Then, the estimated gradient vector is used to update the model parameters as ω k+1 = ω k − βĝ k . The overall algorithm is shown in Algorithm 1. Update(f, h) in Algorithm 1 represents the generic notation for the update of the variable h by using the gradient of f function with respect to the variable h. ω k is the global shared network parameters on IS l at the beginning of the global iteration k, and β is the learning rate for both the client local updates and the PS global updates. F GN Server(·) given in Algorithm 2 performs the auxiliary loss F grad construction and minimization via gradient descent.

IV. EXPERIMENTAL RESULTS
In this section, we evaluate the performance of HOTA-FedGradNorm compared to the naive equal weighting case by analyzing their loss functions.

A. Dataset Specifications
RadComDynamic [15] is used as a dataset for the simulations. This dataset contains 125,000 data points which have the following three attributes: (1) 6 modulation classes which are amdsb, amssb, ask, bpsk, fmcw, pulsed continous wave (PCW). (2) 8 signal types which are AM radio, short-range, radar-altimeter, air-ground-MTI, airborne-detection, airbornerange, ground-mapping. (3) Anomaly behaviour which is defined as having SNR lower than -4 dB since SNR can be a proxy for geo-location information. Low SNR is considered as a signal coming from an outsider.
Algorithm 1 HOTA-FedGradNorm 2: for k=0 to K do 3: The PS broadcasts the current global shared network parameters ω k to the ISs. 4: for Each cluster l ∈ [C] do 5: ω (l) k ← ω k . 6: The IS l broadcasts ω (l) k to clients within cluster. 7: for Each client i ∈ [N ] do 8: Initialize global shared network parameters for local updates by ω for j = 1, . . . , τ ω do 13: to the IS l for dynamic weighting.

16:
The IS l performs the followings: k−1 ) • The IS l constructs the power allocation vector β (l,i) k for each clients in cluster l as given in eq. (3) • aggregates the gradients of clients in cluster l for the global shared network by combining with power allocation scheme as x

17:
The gradients are aggregated over the wireless fading channel as given in eq. (8).

18:
The estimated gradient aggregationĝ k is obtained by the PS as given in eq. (10).
1: Construct the sparsified version of auxiliary loss function F

B. Hyperparameters and Model Specifications
We consider C (number of clusters) = 10 and N (number of clients inside each cluster) = 3. Therefore, data points are divided among all 30 clients where each client owns a specific personalized data. We consider σ 2 l in the channel gain vector N (0, σ 2 l ) as 1 for all clusters. H th defined in (3) is taken as 3.2 × 10 −2 . For F grad construction given in Algorithm 2, γ = 0.6 is used as a measure of task asymmetry. Additionally, α = 0.008 is used as the learning rate for F grad optimization in Algorithm 2 while the learning rate β is taken as 0.0003 for network training in Algorithm 1. We use ADAM optimizer for both network training and F grad optimization. As a shared model, We use a 5-layer FC neural network as explained in Table I Fig. 2 depicts the task losses in the first cluster. The first task (modulation classification) has a lower change of the loss than the second and the third tasks' change of the loss at the beginning of the training. Therefore, we see an increase in the loss weight of the first task in all clusters. After epoch 65, the first task loss decreases significantly, thus, the corresponding loss weight is also decreased. Comparing the result with the result achieved in [3], we observe that considering the wireless MAC channel between the IS servers and the PS leads to slower training. However, as shown in Fig. 2, HOTA-FedGradNorm yields a higher training speed compared to naive equal weighting strategy.

C. Results and Analysis
To demonstrate the effectiveness of F grad to reduce the negative channel effects, we change the first cluster channel gain from σ 2 1 = 1 to σ 2 1 = 0.5 while keeping others as σ 2 l = 1. Decreasing σ l 1 value is equivalent to intensifying the sparsification of the updated gradient based on the definition of H th . Fig. 3 shows how even having a single bad channel in one cluster can deteriorate the entire learning performance if we do not incorporate FedGradNorm into our system model. HOTA-FedGradNorm modifies clients' weights based on the channel conditions, thereby, reducing the channel effects. Fig. 3 illustrates that both the first and second tasks are improved after epoch 85. Additionally, we compare the effects of channels for more diverse σ values in Fig. 4. We observe that HOTA-FedGradNorm is both robust and faster to train under more challenging channel conditions. V. CONCLUSION In this paper, we proposed HOTA-FedGradNorm, a hierarchical over-the-air personalized federated learning framework with a dynamic weighting strategy. Hierarchical federated learning and over-the-air aggregation approaches are integrated with distributed dynamic weighting to cope with the noisy channel in a power-and bandwidth-limited regime while sending updates from clients to the PS. We conducted experiments on the RadComDynamic dataset. We compared the experimental results between the dynamic weighting strategy and the naive equal weighting strategy for both mild and harsh channel conditions. The results demonstrated that HOTA-FedGradNorm improves training speed in most tasks and provides robustness against the negative channel effects by considering the channel conditions during the dynamic weight selection process.  Fig. 4. Comparison between task loss achieved via HOTA-FedGradNorm and naive equal weighting case in RadComDynamic dataset where σ 2 2 = 0.75 and σ 2 l = 1 for ∀l ≥ 3 (a) task 1 (modulation classification) when σ 2 1 = 2, (b) task 1 (modulation classification) when σ 2 1 = 0.25, (c) task 2 (signal classification) when σ 2 1 = 2, (d) task 2 (signal classification) when σ 2 1 = 0.25,