{'fraction_non_alphanumeric': 0.03089684351166286, 'fraction_numerical': 0.009491733006736069, 'mean_word_length': 5.110034013605442, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Decentralized Distributed Learning with Privacy-Preserving Data Synthesis

—In the medical ﬁeld, multi-center collabora- tions are often sought to yield more generalizable ﬁndings by leveraging the heterogeneity of patient and clinical data. However, recent privacy regulations hinder the possibility to share data, and consequently, to come up with machine learning-based solutions that support diagnosis and prog-nosis. Federated learning (FL) aims at sidestepping this limitation by bringing AI-based solutions to data owners and only sharing local AI models, or parts thereof, that need then to be aggregated. However, most of the existing federated learning solutions are still at their infancy and show several shortcomings, from the lack of a reliable and effective aggregation scheme able to retain the knowledge learned locally to weak privacy preservation as real data may be reconstructed from model updates. Furthermore, the majority of these approaches, especially those dealing with medical data, relies on a centralized distributed learn- ing strategy that poses robustness, scalability and trust issues. In this paper we present a decentralized distributed method that, exploiting concepts from experience replay and generative adversarial research, effectively integrates features from local nodes, providing models able to generalize across multiple datasets while maintaining privacy. The proposed approach is tested on two tasks — tuberculosis and melanoma classiﬁcation — using multiple datasets in order to simulate realistic non-i.i.d. data scenarios. Results show that our approach achieves performance com- parable to both standard (non-federated) learning and federated methods in their centralized (thus, more favourable) formulation.


I. INTRODUCTION
Recent advances of deep learning in the medical imaging domain have shown that, while data-driven approaches represent a powerful and promising tool for supporting physicians' decisions, the availability of large-scale datasets plays a key role in the effectiveness and reliability of the resulting models [1]- [3]. However, the curation of large medical imaging datasets is a complex task: data collection at single M. Pennisi institutions is relatively slow and the integration of historical data may require significant efforts to deal with different data formats, storage modalities and acquisition devices; moreover, medical institutions are often reluctant to share their own data, due to privacy concerns. As a consequence, this affects the quality, reliability and generalizability of models trained on local datasets, which unavoidably suffer from bias and overfitting issues [4]. In order to overcome the lack of largescale datasets, methodological solutions can be adopted: in particular, federated learning [5] encompasses a family of strategies for distributed training over multiple nodes, each with its own private dataset, which typically communicate with a central node by sending local model updates, used to train the main model. In this scenario, no data is explicitly shared between nodes, thus addressing the required privacy issues. However, this family of techniques generally performs well when dataset distributions are approximately i.i.d. and local gradients/models contribute to learning shared features: unfortunately, in practice this hypothesis rarely holds, due to differences in the acquisition and in the clinical nature of data collected by multiple institutions. Moreover, the presence of a central node, besides representing a single point of failure, requires that all nodes trust it to correctly and fairly treat updates from all sources: indeed, privacy issues arise when transferring local updates to the "semi-honest" central node [6], which might attempt to reconstruct original inputs from gradients or parameter variations [7]- [9].
In this work, we propose a decentralized distributed learning approach that combines principles of experience replay [10]- [13] and generative models [14]- [16] to train local independent models that approximately converge to the same solution, while enforcing privacy preservation through the transmission of synthetic data built in a way to obfuscate real data patterns. Specifically, the distributed training procedure envisages multiple nodes that initially train their local models on their own datasets, in order to generate a privacy-preserving synthesized version thereof. Once local training is completed in a node, its model and a "buffer" of generated synthetic data are randomly sent to another node of the network. The receiving node then adapts the incoming model using its own data and the buffer data, in order to limit model's forgetting. Data privacy is ensured through a privacy-preserving generative adversarial network (GAN) that employs a specific loss designed to maximize the distance from real data, while keeping a high level of realism and -as importantly -clinically-consistent features, in order to allow models to be still trained effectively.
Our approach is tested on two tasks, each simulating both i.i.d. and non-i.i.d. scenarios: 1) classification of tuberculosis with X-ray data, using Montgomery County X-ray and Shenzhen Hospital X-ray sets [17]- [19], and 2) melanoma classification using skin images of the ISIC 2019 dataset [20]- [22]. Results show how our approach is able to reach performance similar to using all real data together in a single node. Privacypreserving capabilities are measured quantitatively by evaluating LPIPS distance [23] between real images and samples generated, respectively, through latent space optimization on a standard GAN and by the proposed approach. Qualitatively, we also show several examples of generated images with corresponding closest match in the real dataset, demonstrating significant differences that prevent tracing back to the original real distribution.
In summary, the overall contributions of the proposed work are the following: • We propose, for the first time, a decentralized distributed learning strategy designed for medical imaging data, which performs at least on par with centralized learning approaches and yields performance similar to standard (non-federated) training settings in low-data availability scenarios; • The distributed learning strategy is model and datasetagnostic; • We also propose a privacy-preserving mechanism that supports synthetic data sharing through a GAN-based technique designed to minimize patient information leak; • We demonstrate that distributing the learning at the data level is an effective, secure and scalable solution to create models able to generalize across multiple datasets while keeping local data private.

II. RELATED WORK
Data scarcity in the medical domain often calls for multiinstitutional collaborations, in order to collect enough evidence to draw statistically-significant conclusions, as well as to create automated analysis methods (based on machine learning) to support decision making. However, due to the privacy-sensitive nature of medical information, data cannot be shared, hindering the possibility to create robust learning methods able to effectively support physicians in their clinical routines. Federated Learning (FL) [24] has recently emerged as a family of distributed learning strategies that allow nodes to keep training data private, while supporting the creation of a shared model. In a typical FL setting, a central server sends a model to a set of client nodes; each node fine-tunes the model on its own data, then sends local model updates back to the server; the server aggregates the updates by all nodes into the global model, which is sent back to nodes iteratively until convergence. Given the constraints existing in the medical domain, especially in terms of data sharing, it represents an appropriate test bench for federated learning methods [25]- [29]. The most straightforward way to aggregate information from multiple nodes is through averaging local gradients of each client, as proposed in FedAvg [24]. FedProx [30] demonstrates theoretical convergence guarantee of FedAvg when training over statistically-heterogeneous (non-i.i.d.) data. Statistical data heterogeneity is particularly challenging as it leads to catastrophic forgetting in FL settings [31], [32]. FedCurv [33] addresses this issue by adding a penalty term to the loss function in order to drive the local models to a shared optimum. FedMA [34] builds a shared global model in a layer-wise manner by matching and averaging hidden elements with similar feature extraction signatures. Our method differs from existing feature integration approaches in that, instead of averaging model updates or gradients, which can be subject to input reconstruction attacks [7], [8], [35], each node attempts to learn features that perform well on both its own dataset and other nodes', in a more principled way than crude parameter averaging.
Another novelty of the current work w.r.t. the existing state of the art in the medical domain lies in the network topology. Unlike most solutions in medical federated learning, our strategy is decentralized, i.e., nodes directly exchange information among themselves. Indeed, the presence of a central node that aggregates local updates simplifies the communication protocols when the number of clients is very large (thousands or millions), but introduces several downsides: it represents a single point of failure; it can become a bottleneck when the number of clients increases [36]; in general, it may not always be available or desirable in collaborative learning scenarios [31], [37]. In decentralized learning, the central node is replaced by peer-to-peer communication between clients: there is no longer a global shared model as in standard FL, but the communication protocol is designed so that all local models approximately converge to the same solution. Decentralized distributed learning is particularly suitable to application in the medical domain, where the number of nodes (i.e., institutions) is relatively low; however, research is still ongoing, and no effective solutions have emerged. In [38], a Bayesian approach is proposed to learn a shared model over a graph of nodes, by aggregating information from local observational data with the model of each node's one-hop neighbors. A secure weight averaging algorithm is proposed in [39], where model parameters are not shared between nodes, but all converge to the same numerical values (with the disadvantages associated to parameter averaging with non-i.i.d. data distributions). Other approaches implement different communication strategies based on parameter sharing (e.g., gossip algorithms [37] or decentralized variants on FedAvg [24], [40]). In general, most of the existing solutions do not target, nor are they tested on, the medical domains -most employ toy datasets, such as MNIST and CIFAR10. A work which is similar in spirit to ours is BrainTorrent [26], where a use case of decentralized learning for MRI brain segmentation is presented. However, like other approaches, simple parameter averaging is used to integrate features from multiple nodes.
Finally, a key component in our approach lies in the use of generative adversarial networks (GAN) to synthesize samples that can be shared between nodes in a privacy-preserving way. Among similar approaches proposed in the literaure, GS-WGAN [41] proposes a gradient-sanitized Wasserstein GAN to improve differential privacy, by carefully distorting gradient information in a way that reduces loss of infor- Each node initially trains a privacy-preserving GAN, that is used to sample synthetic data from the local distribution, without retaining features that may be used to identify patients. Then, each node iteratively receives the local model and a buffer of synthetic samples from a random node, and fine-tunes the received model on its own private data, using the buffer to prevent forgetting of previously-learned features. mation and generates more informative samples. Federated CycleGAN [42] is designed to perform unsupervised federated image translation. FedDPGAN [43] designs a distributed DPGAN [35] trained in a FL framework, to generate models for COVID-19 diagnosis from chest X-ray images, without data sharing. While CycleGAN, GS-WGAN and FedGPGAN share either gradient or model updates of the generator with the server, introducing privacy concerns related to the risk of input reconstruction, in our approach nodes share a subset of synthetic images, generated in a way that discourages privacysensitive information leaking, to be used with an experience replay mechanism and enhance feature reuse among nodes.

III. METHOD A. Overview
An overview of the proposed decentralized distributed learning approach is shown in Fig. 1. In this scenario, a federation consists of a set of N peer nodes, each owning a private dataset.
Before the distributed training algorithm is started, each node internally trains a Privacy-Preserving Generative Adversarial Network, which is used to generate synthetic samples from the private data distribution. The training objective of the GAN is designed to enforce the constraint that sampled data do not include privacy-sensitive, while maintaining the clinical features required for successful training.
At each round of distributed training, each node receives the current model and a set of synthetic samples -"buffer" -from a randomly-chosen predecessor node, and sends its own model and buffer to a randomly-chosen successor node. The input model is then fine-tuned on both the private dataset and the buffer, in a way that is reminiscent of experience replay techniques in continual learning (e.g., [13]), in order to learn features that transfer between nodes and that can handle non-i.i.d. distributions. At the end of the current round (i.e., after performing several training iterations), the locally-trained model is sent to a randomly-chosen successor node, and the whole procedure is repeated.
In this work we specifically address the problem of federated learning for medical image classification; thus, the method is presented by considering this task, but the whole strategy can be applied to any other task without losing generalization.

B. Privacy-preserving GAN
In the proposed method, nodes exchange both models and data, implementing a knowledge transfer procedure based on experience replay (see Sect. III-C below). Of course, sharing real samples would go against federated learning policies; hence, exchanged samples are generated so that they are representative of the local data, while taking precautions against privacy violations -which may happen, for instance, if the generative model overfits the source dataset.
Formally, we assume that each node n i owns a private dataset where each x j ∈ X represents a sample in the dataset, and each y j ∈ Y represents the corresponding target 1 . The local dataset can then be used to train a conditional GAN [15], consisting of a generator G, that synthesizes samples for a given label by modeling P (x|y, z)), where z ∈ Z is a random vector sampled from the generation latent space, and a discriminator D, which outputs the probability of an input sample being real, modeling P (real|x, y). The standard GAN formulation introduces a discrimination loss, which trains D to distinguish between real and synthetic samples: and a generation loss, which trains G to synthesize samples that appear realistic to the discriminator: While it has been theoretically proven that, at convergence, the distribution learned by the generator matches and generalizes from the original data distribution [32], unfortunately GAN architectures may be subject to training anomalies, including mode collapse and overfitting: as a consequence, the basic GAN formulation may lead to the generation of samples that are near duplicates of the original samples, which would be unacceptable in a federated learning scenario.
In order to mitigate this risk, we introduce a privacypreserving loss, enforcing the generation of samples that do not retain potentially identifying information, but still include features that are clinically relevant to the target y of the synthetic sample. In other words, if y encodes the diagnosis of a certain disease, we want the generator to learn how to synthesize samples conditioned by y, that exhibit evidence of that disease but cannot be traced back to any of the dataset's samples of the same disease.
To do so, our privacy-preserving loss aims at penalizing the model proportionally to the similarity between pairs of real and synthetic samples. We measure "similarity" by means of the LPIPS metric [23], which has been shown to capture perceptual similarity better than traditional metrics (e.g., SSIM) by calibrating the distance between feature vectors extracted from a pre-trained VGG model [44].
In practice, given a batch of real samples x and a batch of synthetic samples , the privacy-preserving loss term is computed as: where d L is the LPIPS distance. Note that, in this formulation, we ignore the y targets associated to each x: we want to prevent the model from generating near-duplicates of real samples in general, regardless of class correspondence. Also, we intentionally employ a pairwise metric on samples, rather than an aggregated metric such as Frêchet Inception Distance [45], since we want to prevent similarity between samples, not between distributions, which would conflict with the GAN objective. The combined effect of the three loss terms -L D , L G , L PP -pushes the generator to explore the sample space to match the dataset distribution, while "avoiding" latent space mappings that would project to actual real samples.

C. Decentralized learning with experience replay
Current approaches for federated learning are mostly based on parameter averaging (e.g., FedAvg), which is however a very crude way to combine knowledge from multiple sources: feature locations are not aligned over different models and may be disrupted by updates, before slowly converging to consensus: hypothetically, two models could learn the same set of features at different locations of the same layer, to only have them cancel each other when averaging. In a decentralized scenario, this issue is even exacerbated, due to the lack of an entity that enforces global agreement on node features. In our approach, we address the problem by taking inspiration from continual learning principles [46], where it is necessary to learn how to perform a task without forgetting a previouslylearned one: as a consequence, models are encouraged to reuse and adapt features so that they can equally serve the current and previous tasks.
Given these premises, we define a decentralized learning scenario where a node receives another node's model and surrogate data (generated through the privacy-preserving GAN, since original data cannot be shared) -the "previous task" -and fine-tunes that model on its own private date -the "current task" -while using received data as a reference to what is necessary to retain/adapt from the knowledge learned by the previous node.
In the following, we describe our method from the point of view of a single node, n i , which receives information from a predecessor node n i−1 . Note that, for a given n i , the predecessor node n i−1 is not fixed: in a practical asynchronous implementation, a node may receive a model and buffer from any random node in the federation at any time, using queues to handle incoming data.
As above, the local dataset at node n i is , while f θ and are, respectively, the model (with parameters θ) and buffer received from node n i−1 . During a round of the learning algorithm, node n i performs several training iterations on both D i and B i , using the latter to prevent forgetting of previously-learned features, according to the experience replay strategy [13], [47]. Formally, the training loss is defined as: where L is the task-specific loss function (e.g., cross-entropy for classification).
After optimizing the L ni objective through mini-batch gradient descent for a certain number of training iterations, the resulting model f θ , with updated parameters θ , is sent to a random node of the federation, along with a buffer of locallygenerated synthetic samples. The number of training iterations and the size of the buffer is discussed in the next section.
Unlike other distributed learning approaches, each node does not have its own local model: as the distributed learning algorithm proceeds, a node iteratively receives a model from another node and updates it with local information, while preserving previously-learned knowledge, before sending it to the next node. As experimentally shown in the next section, at the end of this procedure all models approximately converge to a solution that performs similarly on all private datasets in the federation.

IV. EXPERIMENTAL RESULTS
We test our federated learning approach on two applications: 1) tuberculosis classification from X-ray images, and 2) skin lesion classification. In this section we present the employed benchmarks, the training procedure and report the obtained results to demonstrate the advantages of the proposed approach w.r.t. the state of the art.

A. Datasets
X-ray image datasets for tuberculosis classification. We employ the Montgomery County X-ray Set and the Shenzhen Hospital X-ray Set [17]- [19]. The Montgomery Set contains 138 frontal chest X-ray images (80 negatives and 58 positives), captured with a Eureka stationary machine (CR) at 4020×4892 or 4892×4020 pixel resolution. The Shenzhen dataset was collected using a Philips DR Digital Diagnostic system. It includes 662 frontal chest X-ray images (326 negatives and 336 positives), with a variable resolution of approximately 3000×3000 pixels. Skin lesion classification. We employ the "ISIC 2019" dataset, which contains 25,331 skin images belonging to nine different diagnostic categories. The data provided belongs to three different sources: 1) the BCN20000 [20] dataset, consisting of 19,424 images of skin lesions captured from 2010 to 2016 in the Hospital Clínic in Barcelona; 2) the HAM10000 dataset [21], which contains 10,015 skin images collected over a period of 20 years from two different sites, the Department of Dermatology at the Medical University of Vienna, Austria, and the skin cancer practice of Cliff Rosendahl in Queensland, Australia; 3) the MSK4 [22] dataset, which is anonymous and includes 819 samples. Among all skin lesion classes, we only consider the melanoma class, posing the problem as a binary classification task.
In our federated learning framework, we consider two settings: • non-i.i.d. setting, the most realistic one, where each dataset is associated to a single node: i.e., two nodes for tuberculosis classification (one with the Montgomery County X-ray Set and the other with the Shenzhen Hospital X-ray set); three nodes for melanoma classification (one with the BCN20000 dataset, one with the HAM10000 dataset and the last one with the MSK4 dataset). This setting is intended to test performance under clear data distribution shifts among nodes. • i.i.d. setting, where a single dataset is randomly split and equally-distributed over multiple nodes. This setting evaluates how the approach scales with the number of nodes. In all tasks and datasets, we employ 80% of the data to train both the privacy-preserving GAN and the classification model, while the remaining 20% of each dataset is used as test set. Test sets are balanced w.r.t. the label to avoid performance biases due to class imbalance.
B. Training procedure and metrics 1) Decentralized distributed learning: In all settings, we employ ResNet-18 [48] as classification model, trained by minimizing the cross-entropy loss with mini-batch gradient descent using the Adam optimizer. Mini-batch size is set to 32 and 8 for the Shenzhen and Montgomery datasets, respectively, and to 64 for skin lesion datasets. The learning rate is set to 10 −4 . Data augmentation is carried out with random horizontal flip; for skin images we additionally apply random 90-degree rotations. All images are resized to 256×256.
The node federation is trained for T rounds. At each round of training, each node receives the model and a buffer of synthetic data from a randomly-chosen predecessor, fine-tunes the received model to its private data for E epochs, and sends the updated model and its own buffer of synthetic samples to a randomly-chosen successor. In our implementation, at each round nodes are randomly ordered to establish each node's predecessor and successor: given our focus on medical applications, we can assume that the number of nodes is low enough that synchronization is not an issue. However, asynchronicity can be achieved by assuming that nodes can store incoming data in queue: if the distribution of successor nodes is uniform and computation times are similar for all nodes, this is on average equivalent to the synchronous case.
2) GAN training: The proposed privacy-preserving GAN uses StyleGAN2-ADA [49] as a backbone, because of its suitability in low-data regimes and its generation capabilities. Training is carried out in two steps: 1) the GAN is initially trained for 800,000 iterations without any privacy-preserving loss, to support learning of high-quality visual features; 2) we enable the privacy-preserving loss and fine-tune the mode for additional 150,000 iterations, in order to limit the embedding of patient-specific patterns in the GAN latent space. In our classification setting, GANs are trained in a label-conditioned fashion with a mini-batch size of 32 and learning rate 0.0025 for both the generator and the discriminator. We evaluate the generator's capability to synthesize realistic images by means of the Frêchet Inception Distance (FID) [45]. In order to quantitatively evaluate privacy preservation, we generate, for each real image, its closest synthetic sample by means of latent space projection (described in Sect. IV-C.2), and compute the average LPIPS distance [23] between such pairs: a sufficiently high value implies the impossibility to reconstruct real images from the generator.

1) Distributed training:
We initially evaluate the performance (in terms of classification accuracy) of our approach, trained for 100 rounds with 100 epochs per round, in the non-i.i.d. setting, and compare its results to those achieved in case of a) non-distributed training of a single model on the union of the datasets, and b) centralized distributed learning where a central node receives a set of 256 privacy-preserving synthetic samples from each node and trains a global model. Classification accuracy is reported for each dataset: in distributed learning scenarios, performance is computed using the final model at the node where that dataset is located. From the results reported in Table I, two insights can be drawn: 1) on large datasets, as in the case of Shenzhen and all the melanoma datasets, our approach performs worse non-distributed training (as can be expected); 2) surprisingly, on smaller datasets such as Montgomery, our approach is able to outperform nondistributed training: a possible explanation lies in the data augmentation effect induced by the use of synthetic examples, which has been shown to reduce overfitting and help improve performance in low-data regimes [50].  We proceed to evaluate the agreement between models trained on each node, by measuring the standard deviation of the classification accuracy obtained by all nodes' models for each specific dataset, for different combinations of numbers of rounds and epochs per round. Table II shows that, in our default configuration with 100 rounds and 100 epochs per round, models tend to converge to similar predictions with limited variability, lower than 0.06 in four cases out of six. The performance on the Shenzhen (tubercolosis) dataset exhibits a larger variability, which can be explained by the lower generalization capabilities of the model trained on the very small Montgomery dataset.
A key component of the proposed architecture is the sharing of synthetic samples among all federated nodes. We evaluate the impact of the buffer size on classification performance, and compare with the case when the buffer is removed (i.e., only the model is shared). Results, evaluated on the tuberculosis task (which is the only one we use also for the subsequent evaluations on the architectural components) with 100 training rounds and different epochs per round, are reported in Table III and indicate that the experience replay strategy is necessary to allow the whole system to effectively learn with non-i.i.d. data. In general, the presence of larger buffer size ensures that common and robust features are learned, leading to significant performance improvements, especially on small datasets such as Montgomery with a performance gain of about 9 percent points between no buffer and a buffer size of 2048 samples. However, in all the experiments, we use buffer size of 512 as it represents the best compromise between accuracy and memory consumption on all the employed datasets.
We then evaluate the capability of the proposed method to scale with the size of the federated network, without losing effectiveness. We quantify this property in the i.i.d. setting on both tuberculosis (Shenzhen dataset) and skin lesion classification (BCN dataset) tasks, by equally splitting the available data on multiple nodes, and training for 100 rounds,  with 100 epochs per round. Fig 2 shows how the proposed approach is able to keep classification accuracy high (on the two datasets there is a performance drop less than 3 percent points from 2 to 8 nodes). A slight decrease in performance can be observed as the number of nodes increase, which can be explained by the lower size of the training set available for each node. Finally, we compare our method with the FedAvg approach [24], which is a suitable baseline as it has been shown to perform generally better than decentralized methods [38], [40] and results are in Table IV in both i.i.d. and noni.i.d. settings for the tubercolosis and melanoma classification tasks 2 . Since centralized methods benefit from the global view of all models by the central node, FedAvg's performance is slightly better on the non-i.i.d. scenario (in this case, with varying number of nodes), while, remarkably, our approach outperforms on average it on the i.i.d. settings, demonstrating better scalability capabilities. Considering all the obtained performance on the various scenarios, we can claim that our approach is at least on par with FedAvg, demonstrating that a principled feature integration approach may lead to better convergence properties over naïve parameter averaging even in a more complex case as the decentralized one.
2) Privacy-preserving Data Synthesis: After demonstrating the impact that the experience replay mechanism has on the proposed decentralized learning strategy, it is important to ensure that no sensitive information is leaving each node. Therefore, we quantify how much information of real samples is stored by our privacy-preserving method, and in particular in the mapping between latent space and synthetic images. To do so, we employ the projection method proposed in [16]: given a real image x, we find an intermediate latent point w such that the generated image G(w) is most similar to x, by optimizing w to minimize the LPIPS distance [23] between x and G(w).
In practice, for each image of the dataset used for GAN training, we perform backprojection to find its most similar synthetic sample, and measure the LPIPS distance between the original and projected images. Fig. 3 shows the histograms of the resulting distances on the Shenzhen dataset, using GAN models trained with and without the proposed privacypreserving loss (both models start from the same w, for fairness). The histograms show that standard GAN training, with no privacy-preserving loss, tends to yield distances close to 0, demonstrating that real images are indeed encoded into the generator latent space; our model significantly mitigates this issue, by synthesizing samples that are significantly more different than the original ones. Fig. 3. Quantitative analysis of privacy-preserving generation. In blue, LPIPS distance histogram between real images and the corresponding images obtained through latent space projection using a GAN trained without the proposed privacy-preserving loss. In red, LPIPS distance histogram between real images and the closest images generated with the proposed approach.
In order to qualitatively substantiate these findings, Fig. 4 compares original samples from the Shenzhen dataset with the corresponding projections, generated with and without our privacy-preserving loss 3 . It is easy to notice that generated samples with a traditional GAN highly resemble real data, making it impossible to share such samples, albeit synthetic, in a privacy-safe manner, as they clearly contain patient information. Instead, comparing real images with the projections obtained from privacy-preserving GAN confirms the inability of the generator to find latent representations that recover real images used during training.
Given the high realism of generated samples, we run additional tests meant to push the level of privacy preservation: a) models are not shared among nodes -only synthetic buffers are sent and received ("buffer-only sharing"); b) models are trained only using synthetic data, even for the local dataset ("synthetic-only training"). Fig. 5 shows the internal architecture of each node in the two variants. Results obtained with these alternative privacy-enhanced strategies are provided in Table V, when training on the Shenzhen and Montgomery datasets in the non-i.i.d. setting for 100 rounds, with 100 epochs per round and a buffer size of 512. It can be noted that the "buffer-only sharing" variant achieves comparable performance to our standard setting, giving encouraging perspectives for further analysis. Instead, the "synthetic-only training" variant performs poorly, showing that using features  from real images to train local models is essential to reach good global performance: nevertheless, further research on the generation process might lead to improvements in this aspect.

V. CONCLUSION
In this paper, we presented a novel decentralized distributed learning framework, that replaces traditional parameters averaging with a more principled feature integration approach based on the combination of experience replay and privacypreserving generative models. In our approach, nodes communicate with each other by sharing local models and buffers of synthetic samples; local model updates are carried out in a way that encourages the reuse and adaptation of features learned by other nodes, thus avoiding potentially disruptive effects due to blind feature averaging. Experimental results show that our method performs at least on par with state-of-theart centralized approaches, and outperforms them in low-data availability scenarios, which is a typical setting in the medical domain. Additionally, quantitative and qualitative analysis shows that our privacy-preserving generation approach is able to synthesize samples that are significantly different from real data, while correctly supporting the learning of discriminative features. In the future, we aim at investigating some unexplored properties of our method: for instance, unlike all other existing methods based on parameter averaging is required, our approach does not strictly require that all nodes share the same model architecture. Model heterogeneity could therefore be employed to create a shared ensemble and combine different feature learning capabilities.