{'fraction_non_alphanumeric': 0.031318126229595365, 'fraction_numerical': 0.01765300154197905, 'mean_word_length': 4.850077760497667, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Federated Self-supervised Speech Representations: Are We There Yet?

The ubiquity of microphone-enabled devices has lead to large amounts of unlabelled audio data being produced at the edge. The integration of self-supervised learning (SSL) and federated learning (FL) into one coherent system can potentially offer data privacy guarantees while also advancing the quality and robustness of speech representations. In this paper, we provide a first-of-its-kind systematic study of the feasibility and complexities for training speech SSL models under FL scenarios from the perspective of algorithms, hardware, and systems limits. Despite the high potential of their combination, we find existing system constraints and algorithmic behaviour make SSL and FL systems nearly impossible to build today. Yet critically, our results indicate specific performance bottlenecks and research opportunities that would allow this situation to be reversed. While our analysis suggests that, given existing trends in hardware, hybrid SSL and FL speech systems will not be viable until 2027. We believe this study can act as a roadmap to accelerate work towards reaching this milestone much earlier.


Introduction
A large amount of audio data, often unlabelled and of private nature, is generated everyday from cellphones, tablets, personal assistants and other IoT devices [1,2,3,4]. Being able to utilise this data to solve various speech related tasks has been of great interest to researchers for over a decade [5,6,7,8,9]. Self-supervised learning (SSL) allows the learning of representations from unlabelled data, which can later be used to solve specific downstream tasks, e.g., automatic speech recognition (ASR), speech translation, keyword spotting (KWS), and others. SSL shares limitations with other forms of unsupervised training favouring large-capacity models trained with vast amounts of data [5,6]. Currently, such workloads can only be realised in data centers, where high-performance hardware is available. Such centralised SSL systems unavoidably raise concerns around privacy specially when dealing with speech data. Additionally, they also cause severe communication overheads due to the transferring of data from devices where it originates.
A natural way to mitigate these issues is to combine SSL with federated learning (FL) [14,15,16,17]. In FL, a distributed population of devices collaboratively trains a single global model while keeping their local data private. A typical FL pipeline is comprised of three stages: first, a global model is initialised in a server to which all FL nodes (or clients) connect; then, a fraction of clients are chosen by the server, receive a copy of the global model and each performs on-device training using their own data; upon completion the clients send the updated model to the server, where a new global model is generated by aggregating the client updates. This process is repeated for a number of rounds. Deploying SSL workloads in a federated setting offers another benefit beyond privacy-preserving training: large-scale distributed feature learning from realworld data without costly and error-prone manual annotations. Despite the combination offering significant benefits, no prior work has studied speech SSL in the context of FL. Figure 1: (left) The pretraining of a SOTA self-supervised model wav2vec 2.0 is several times more time-consuming than SOTA supervised ASR models (transformer [10,11], CRDNN+CTC+Att [12,11]). All of the models are trained with the LibriSpeech dataset using V100 GPUs. (right) FL leads to longer training time compared to centralised settings. The models are trained using V100 GPU in centralised settings and using Jetson Xavier NX in FL [13]. CV is Common Voice dataset.
The SSL models are notoriously costly to train. A SOTA SSL model for speech representation learning, wav2vec 2.0 [5], requires over 7000 GPU hours of pretraining (V100 GPUs). That is an order of magnitude more compute resources than a top-performing model trained with only supervision (Fig. 1  left). However, this leap in resource demands when transitioning from supervised to a SSL solution is not the only significant barrier to performing SSL under FL. It is paired with an empirical phenomena generally seen when migrating from centralised training to a federated version: due to the model update aggregation phase of FL, the overall global convergence is weakened and thus slower ( Fig. 1 right). We again observe resource requirements (in this case raw training time) increasingly alarmingly -although this is now due to the act of federating the training process. While we know these two factors will drive the ultimate performance of systems that combine FL and SSL for speech, curiously, there is large gap in existing empirical knowledge as to how they will interact and translate into resource requirements. How long would a SSL speech model take to converge if federated? Specifically, what would the memory and compute requirements be for the participating FL clients? Would WER decrease in this decentralised setting? All of these questions, ranging from those that are obvious down to questions specific in detail are virtually unknown presently.
In this work, we conduct a systematic investigation as to the precise bottlenecks that cause SSL models to be so resource intensive to train, and further explore how these resource requirements (e.g., memory, compute) translate into the feasibility of SSL models on edge devices in FL environments. We find that the modest edge devices and existing training frameworks are not yet a match for the requirements of federating SSL speech. But more crucially, our empirical results provide for-the-first-time the much needed identification and quantification of the key limiting factors that underpin this situation. As an example of how our findings can accelerate research in this emerging area, we use them to propose a novel modification to FL aggregation that takes into account the training loss of in- dividual FL clients. Our study estimates given existing FL and SSL technique and practices -along with historically guided expectation of hardware improvements [18,19] -that hybrid self-supervised FL for speech will not be feasible until roughly half decade from now, 2027; even though the core principles are available today. Not only is this the first systematic estimate of its type for the speech community to reflect upon, we hope that it acts as a call to arms around which researchers will rally in an effort to reach this milestone much more quickly.
The scientific contributions of this study are: (1) We conduct a systematic analysis that discovers the core reasons why SSL models are resource intensive ( §3.1 and §3.2); (2) We extend this analysis to explore the feasibility of SSL under FL conditions ( §3.3); (3) We perform extrapolative analysis as to future hardware feasibility through faithful GPU simulations ( §4).

Experimental Settings
We conduct all of the measurements using wav2vec 2.0 [5] on the CommonVoice Italian (CV Italian) dataset [20] trained on a variety of hardware platforms. The details are as follows: Model. Our choice of SOTA SSL model for speech representation learning is wav2vec 2.0. This model is comprised of three modules: a convolution layers block that extract latent acoustic features from the raw waveform; a series of transformer layers that generate contextualised embeddings; and, a quantization module that discretises the output of the convolutional module to a finite set of speech representations. The model is jointly trained by a contrastive loss and a diversity loss. We consider both the base and large versions of wav2vec in our evaluation.
Dataset. The Common Voice Italian corpus (version 7.0) contains a total of 211K utterances (317 hours) obtained from volunteers recording sentences on smartphones, computers, tablets, etc. The validated set is used for SSL pretraining and consists of 195K utterances (288 hours) with an average sentence length of 5.5 seconds from more than 6K speakers. The training set is used for downstream ASR fine-tuning which contains 191 hours of speech, while the validation and test sets con-

SSL Speech System Resource Analysis
This section provides detailed analysis on the resource footprint of training wav2vec 2.0 ( §3.1), and the sensitivity of resources to input audio length ( §3.2). These results then enable a feasibility study of federated SSL under existing hardware ( §3.3). Table 1 shows the number of parameters and floating operation points (FLOPs) for each of the three modules in wav2vec 2.0 as well as the total model size and FLOPs required for inference while considering an input audio of duration 5.5 seconds corresponding to the average length of the CV Italian dataset. An immediate observation is the asymmetry between the amount of parameters and FLOPs between the Encoder and Transformer blocks: while the latter has 19× more parameters, it only results in 1.8× more FLOPS compared to the CNN Encoder. When considering the large wav2vec 2.0 model, this difference increases significantly: the Transformer has 66× more parameters but only accounts for 6× more FLOPS than the Encoder. This first-order analysis does not consider the training dynamics and therefore a more in-depth look into the impact of each sub-architectural module is needed to identify bottlenecks, both in terms of memory and compute, occurring at training time. Figure 2 shows the per-layer memory consumption (left) and compute (right) of wav2vec 2.0 base. Immediately, the high memory utilization and compute needed for the CNN Encoder stands out compared to the rest of the model. These two dimensions are a function of the size of the input tensor passed through the layer -which is not captured in high-level analysis as in Table 1. When it comes to extending SSL models to constrained devices for FL, a key factor is the total amount of memory needed at training time. Figure 2 (left) shows a memory peak (red cross) of 2.54GB when training with input sequences of length 5.5s (avg. length) and batch size 4. When considering more desirable batch size of 8 and audio files of 12s, the total memory accumulated prior to backpropagation is 9.89GB. Such a delta between these training settings highlights how seemingly small changes can translate into having to consider an entirely new tier of devices to support such workloads.

On sequence lengths and numerical precision
Memory and FLOPs grow linearly with the length of the input sequence (Fig. 3). From a practical perspective, the sequence length can link to different types of offline speech applications that could be regarded as downstream tasks for the pre-trained SSL model. For instance, KWS may only require short utterances (e.g. 1-3s) [21,22], while tasks such as speech synthesis or recognition may increase this to much longer sentences [23,24]. However, currently released large-scale SSL models including wav2vec 2.0, HuBERT [25] or WavLM [26] are often trained with sentences longer than 15s, making them clearly intractable for current constrained devices. Hence, one could argue that the SSL stage could be tailored to a specific subset of downstream tasks and limit the length of the input sentences for pretraining, reducing the compute and memory footprints.
Parameter precision is another element that affects the memory and compute footprint of an architecture. Therefore, we compare the memory impact of full-precision FP32 with mixed precision (FP32&FP16) [27] (Fig. 4). From the latter results, it appears that the memory savings observed with mixedprecision is not significant and might not be critical while selecting a pool of devices to train SSL models. The impact on the training time, however, provides a different outcome ( §3.3).

Federated SSL feasibility
With this section, we provide an analysis relying on memory consumption and training time answering the feasibility of applying federated learning to large-scale SSL pretraining. Memory cost. Memory is a scarce resource in edge devices and limits the feasibility of on-device training in FL. For instance, some popular mobile processors and edge devices including the Raspberry Pi 4 (RPi), Jetson Xavier NX or the A15 Bionic chip are only accompanied with 8GB of memory while the Jetson Xavier AGX, Google Tensor and Snapdragon 870 chips commonly are given 16GB. According to our findings, the first batch of devices could train base wav2vec 2.0 models while the latest set could use large architectures with sequences of around 7 to 8 seconds at most (Fig. 3). However, if longer sentences are considered as it is typically the case with current SSL models, all the devices may simply be able to train the base model, or even none of them. Nevertheless, we should note that these devices, specially when it comes to smartphones, would likely be running other applications and processes in the background. As a result, the memory ceiling available for FL workloads is lower than the devices' total memory. Therefore, it appears highly probable that even the base wav2vec 2.0 architecture exhibits a training memory cost already higher than what high-end devices have to offer. Training time. Training time is another factor influencing the feasibility of SSL in FL. Typical large-scale SSL pretraining takes weeks or months even with hundreds of GPUs [28]. Training time might simply become intractable with limited compute devices and the slower convergence induced by FL.
Hence, we benchmark the observed training time (i.e. seconds per batch) with different FL devices and a modern GPU. As in prior experiments, sentences are sampled so that the average length is 5.5s. Table 2 reports the obtained measurements. When considering an unlikely yet simple batch size equal to 1, the 8-cores i9 of the MacBook Pro is 30.3× and 39.5× slower than a Nvidia A40 GPU when training on wav2vec 2.0 base and large respectively. In the case of the more modest 4-cores RPi, training 4.4× slower than the MacBook Pro and 138× slower then the A40. The Jetson boards on the other hand are equipped with embedded GPUs and therefore decrease this training time factor to 2.7× and 5.6× slower compared to the A40 for AGX and NX respectively on the base model. Once we increase the batch size to 4, which remains considerably low for SSL training that often rely on large batch size values (above 256 [28]), the per-sequence training times get reduced by 15%, 20%, 29% and 33% for the MacBook Pro, RPi, AGX and NX respectively thanks to a better hardware utilization. Finally, and conversely to the memory cost, mixed precision provides a substantial boost in training speed for devices supporting it: 1.56× and 1.31× faster than full precision for the Xavier NX and AGX with the base model. However, support for mixed precision

Federated SSL Speech Representations
With this last section, we estimate when FL/SSL speech systems will become feasible for edge hardware; before showing results that to-the-best-of-our-knowledge are the first attempt at training SOTA SSL speech under a simulated FL setting.

Feasibility under existing hardware trends
Compute capabilities have been doubling every 18 months for decades [18,19,29]. This implies a 8× acceleration of compute in 5 years. Based on our precise measurement of training time on the NX, which represents a typical FL device with an embedded GPU, we predict edge devices to exhibit the same compute performance than a A40 GPU by 2027. Thus, assuming the existing SSL models (i.e., no substantial growth in model complexity) and FL methodologies, we must wait for around 5 years for hybrid speech systems combining both technologies to become feasible. As discussed in §3.3, a large amount of memory is needed when considering longer input sequences. Currently, the AGX can be purchased with 32GB of memory, therefore we believe that by 2027 mid-tier edge systems (e.g. NX) would come with memory sizes similar to today's A40 48GB. This will support performing SSL suitable to more challenging downstream tasks (e.g. ASR).

Simulated Federated SSL for Speech
We train federated wav2vec 2.0 base using SpeechBrain [11] and Flower [30] on two NVIDIA A40 GPUs. FL partitions. Based on the natural partitioning of CV Italian we have two sets of experiments: (1) 10-client FL and (2) 100client FL. In (1), we simulate a simple FL scenario where clients are generally few but with high availability during all rounds, and similar data distribution [31]. The dataset is randomly split into 10 partitions (i.e. one per client) without overlapping of speakers, each containing roughly the same amount of speech data. Setting (2) rises the number of clients to 100 making it a more challenging scenario as each device participates less frequently and has a smaller portion of data [31]. 20 randomly selected clients participate in each round. Federated SSL pretraining. In each FL round, the SSL model is locally trained on each client with 1 local epoch, batch size of 4 in both FL settings. Then, the updated model weights are sent to the server for aggregation directly with FedAvg [14] or after re-weighting the model updates using the client loss as in [15]. The latter is better suited for heterogeneous data distributions across clients. The SSL model in 10-client setting, trained for 150 rounds, shows a faster converging behaviour than 100client FL which is trained for 500 rounds (Fig. 5 above). This is expected as a 100-client setting is more challenging. ASR downstream evaluation. The learned speech representations from SSL pretraining are evaluated with an ASR downstream task by adding 3 linear layers on the top. The entire network is fine-tuned for 80 epochs with CTC loss following the official SpeechBrain recipe. Figure 5 (bottom) shows the WER on the test set. Federated SSL representations provide competitive performance in both settings, demonstrating its feasibility while benefiting from unconstrained hardware environments (i.e., potentially unrealistic). We show that loss-based aggregation achieves better performance in both settings, which indicates that weakening the effects of low-quality clients can assist the aggregation process in federated SSL.

Conclusion
In this paper, we provided the first empirical study of the feasibility of combining two emerging techniques that will revolutionize the way speech systems are designed and operate: SSL and FL. Our results preview the future in which hybrid FL and SSL systems train universal speech representations. Our study highlights this future will not be easy to reach, with a number of technical challenges ahead of us. But we hope this work will act as a roadmap to the community towards this new milestone.